Hiroshi Miura. Scalable Private Cloud Storage with Full OSS Stack. CloudOpen Japan 5, June System Platform Sector, NTT DATA Corporation 年月日

Transcription

2 2 Disclaimer Any product name, service name, software name and other marks are trade mark or registered mark of corresponding companies. This presentation is in a purpose of providing the knowledge gained from our first and biggest swift project. A presenter and NTT Data Corporation provide information in as-is basis and have no responsiveness for results that you got according to information in this presentation material.

3 3 Agenda 1. Back ground and issues 2.Categorize cloud storage 3. Building and utilizing private cloud storage 1. OpenStack Swift (, Ceph) and sheepdog 2. owncloud, seafile

4 4 Top 5 Myth of Cloud Storage

5 5 Myth 1 Cloud Storage is less secure than local storage. It has a risk of information leak.

6 6 Background: Cloud Storage is convenient - free price for some volume - many consumer utilize cloud storage service - many student, professors utilize in college campus - many office worker collaborate with outside of company Several type of Cloud storage etc smart phones and tablet Free wi-fi, mobile connetions Broaden users not only home also campus or office

7 7 Background: but Cloud Storage is less secure - Storing sensitive data out of campus. - Leakage of information Leaking Risk Lost risk Continuous risk - Out of control for external service shift asset to cost Data size limitations Performance limitations

8 8 Myth 2 Impossible to manage growing data such as IoT, smart city and smart factory in private system.

9 9 Growing Data Data Explosion on Enterprise Amount of Unstructured Data has been growing. We need storage with Scalability, Durability, Availability. Growing exponentially Amount of Unstructured Data EB or PB scale Examples of Unstructured Data Media (Images, Videos, Audios) Web Contents Documents Backups/Archives IoT/M2M

10 10 Myth 3 Private storage can archive higher performance, but more expensive than public cloud service.

11 11 Myth 4 Tire-3 storage (SATA, NL-SAS) is cheaper, higher capacity than enterprise grade storage. It is less quality and just a consumer grade.

12 Server for cloud providers There are IA server for storage system High capacity Long term support Fit for cloud storage architecture Main target is Cloud service providers Who can maintain by itself Use a big volume Cloud storage software help us utilize such a hardware Copyright 2015 NTT DATA Corporation 12

13 13 Cloud Storage Architecture Virtualized infrastructure Made up of many distributed resources - federated storage clouds Fault tolerant through redundancy and distribution of data Durable through the creation of versioned copies Typically eventually consistent with regard to data replicas

14 14 You should know about Cloud Storage Cloud Storage Infrastructure OpenStack Swift Ceph Sheepdog Cloud Storage Service software owncloud SeaFile/SeaHub

15 15 OpenStack Swift

16 16 What is Swift? Swift is... A storage system with Scalability, Durability, Availability. The REST-ful Distributed Object Storage likely Amazon S3. One of OpenStack Core Components. Implemented by Python. A Open Source Software. 1 Block Storage (Cinder) 2 Object Storage (Swift)

17 17 3 Key features of Swift 1 Durability 2 Scalability 3 Openness

18 18 Durability Protect data from various defeats Datacenter Rack Node Disk

19 19 Scalability Flexibly adopt to the growth of data 3PB? 100TB 150TB? 10TB 1y 2y 5y

20 20 Openness Free from limited model and maintenance period of hardware Vendor A B C A B C

21 21 3 key features of Swift Durability Scalability Openness Protect data from various defeats Flexibly adopt to the grow of data No vendor lock-in Datacenter 3PB? Vendor A B C Software Hardware 100TB 150TB? Disk 10TB 1y 2y 5y A B C

22 Merit of OpenStack Swift Open Architecture Freedom to change Bundle Low cost server and HDD to single virtual storage Cost effective Long term storage Freedom Stability 6 merit durability Disaster respose automated sync data bet/ Multiple data centers Scalability developed by 370 organization in 139 countries. Small start, extend on demand Copyright 2015 NTT DATA Corporation 2014 年 6 月現在 22

23 23 Swift as a storage for a variety of applications System Backup CMS FTP-like use Digital Distribution Web Apps Cyber Duck REST API Swift

24 24 Architecture: Nodes 2-type of Nodes: Proxy Node and Storage Node. Application Forward Data to node HTTP Load balancer Proxy Node Proxy Node Proxy Node Store data Storage Node Storage Node Storage Node Storage Node

25 25 Architecture: The Ring The Ring (static table for data allocation on storage node) decide the optimal Storage Node by Name. Application HTTP Load balancer Proxy Node Proxy Node Proxy Node Ring Ring Ring Storage Node Storage Node Storage Node Storage Node Ring Ring Ring Ring

26 26 Scalability (1) Expand proxy server Throughput (2)Expand Storage servers or disks volume More Throughput Proxy Proxy (expand) Proxy Proxy Storage Storage Storage Storage Storage Storage Storage (Expand) More Volume

27 27 Replicator Replicator Disk Norma l Node 1 Node 2 Node 3 Node 4 (1) Each nodes checks data in others Node 1 Node 2 Node 3 Node 4 (2) Disk defeat (3) Detect disk trouble Defea t Node 1 Node 2 Node 3 Node 4 (4) Copy data to another node Node 1 Node 2 Node 3 Node 4 (5) Recover disk (6) recover data to original node Recov ery Node 1 Node 2 Node 3 Node 4 (7)Delete temporal data

28 28 Our use case

29 29 Use case: project overview Migration from existing high-end storage to distributed storage Backend Storage for application to store end user s data End Users Application server High-end storage Data migration Swift Distributed Storage

30 30 3 Challenges in the project 1 Durability assurance 2 Geographically distributed cluster 3 Delivery and quality

31 31 Challenge 1: Durability assurance Quality requirement in Japan Our customer is SUPER quality crazy. Everything should be under control System design for normal situation System design for defeat situation Even on distributed system Analyze every behavior before building system

32 Storage Storage Storage Storage Storage Storage Storage Storage Storage Storage Storage Storage Storage Storage Storage Storage Storage Storage 32 Recovery test in variety of defeat pattern Variety of failure pattern (1) The point of failure Disk, NIC, Process, Node, (2) The number of failures 1, 2, 3, 4, (3) The range of failures 1 node, multiple nodes/zones/regions, 100s of test cases!! Case #001 Case #301 Case #501 Case #001 Case #001 Proxy Proxy Storage Storage Storage Proxy Storage Storage Storage Case #101 Proxy Proxy Storage Storage Storage Case #201 Case #201 Zone1 Zone2 Region 1 Proxy Proxy Proxy Storage Storage Storage Storage Storage Storage Zone1 Region 1 Zone2 Zone1 Region 1 Zone2

33 Challenge 2: Geographically distributed cluster Geographically distributed swift cluster to realize disaster recovery Important points to evaluate global distribution 1. Client request 2. Durability Site 1 300km~ Proxy 300km~ private network Site 2 Site 3 Proxy (Secondary) Storage 300km~ Storage Storage 3 Copyright 2015 NTT DATA Corporation 33

34 34 Pseudo-global cluster benchmark Pseudo-global cluster with simulated network latency Proxy and 3 Storage regions placed in different locations 10~200msec latency between locations simulated by tc Proxy TLmsec TLmsec Storage region1 Client Proxy 10~200msec latency 10~200msec latency 10~200msec latency Storage region 1 10~200msec latency Storage region 3 10~200msec latency Storage region 2 10~200msec latency

35 35 Automated building Install and configure all software About 10 software and 50 configurations Kickstart and puppet install & configure Swift OS Swift OS puppet Kickstart swift proxy servers configuration server Swift OS Swift OS swift storage servers

36 36 Result of automated building Speed up of building 200 nodes/day, about 100 times as fast as manual building Elimination of human error in building process 100% automated

37 Automated testing Testing tools for 1. APIs : All swift APIs, including error case 2. Nodes : All swift nodes Extended Tempest and checking tool Tempest checking tool Test all APIs proxy servers Test all nodes storage servers Copyright 2015 NTT DATA Corporation 37

38 38 Result of automated testing Quality assurance 100% API coverage, 100% node coverage Speed up of testing 1 swift cluster(70 swift nodes) / 1 hour

39 Detailed system constitution 3 primary swift clusters over 3 geographical sites 3 backup swift cluster gathered in site 2 12PB total capacity: 1PB*3 for primary clusters, 1PB*1 for backup clusters Site 1 Monitoring (Act/Sby) Visualizing (Act/Sby) Configuration (Act/Sby) Primary Cluster1 Primary Proxy Primary Cluster2 Primary Proxy Primary Cluster3 Primary Proxy Site 2 Site 3 Monitoring (Act/Sby) Visualizing (Act/Sby) Configuration (Act/Sby) Monitoring (Act/Sby) Visualizing (Act/Sby) Configuration (Act/Sby) Backup Cluster1 Proxy Backup Cluster1 Storage(1PB) Primary Cluster1 Secondary Proxy Primary Cluster1 r1 Storage(1PB) Primary Cluster1 r2 Storage(1PB) Primary Cluster1 r3 Storage(1PB) Backup Cluster2 Proxy Backup Cluster2 Storage(1PB) Primary Cluster2 Secondary Proxy Primary Cluster2 r1 Storage(1PB) Primary Cluster2 r2 Storage(1PB) Primary Cluster2 r3 Storage(1PB) Backup Cluster3 Proxy Backup Cluster3 Storage(1PB) Primary Cluster3 Secondary Proxy Primary Cluster3 r1 Storage(1PB) Primary Cluster3 r2 Storage(1PB) Primary Cluster3 r3 Storage(1PB) Copyright 2015 NTT DATA Corporation 39

40 40 Sheepdog

41 Sheepdog distributed object storage Copyright 2015 NTT DATA Corporation 41 Replicated shared storage for VM snapshot, clone, and rollback Massive scale-out : nodes Qemu/libvirt and iscsi Virtual integrated platform Virtual disk image Cut out provide Storage pool IaaS back end storage Consolidate Sheepdog IA servers Internal disks

42 42 Sheepdog developed heavily in east Asia Heavily developed by NTT Lab and Alibaba/Taobao team Version 1.0 will soon be released Yesterday session: Sheepdog is ready!: Distributed Block Storage is turning from experiment to commercial use By Teruaki Ishizaki, NTT

43 43 Feature 1 High reliability Fault tolerant through redundancy and distribution of data Manange data with building block of 4 MB chunk Distributed replica data to multiple servers as ordinal file Realize redundancy Zero data use no actual space Thin provisioning Create chunk of 4MByte data blocks from virtual disk VDI(Virtual Disk Image) - 1 GB 0 MB 4 MB 8 MB 16 MB 1024 MB Distribution of files Sheepdog cluster

44 Feature2: Easy management No manual operation for cluster management Node addition Accept a new node to a cluster automatically, when Sheepdog node connected Automated rebalance of distributed data Node removal Kick out a broken node from a cluster automatically, and continue working by rest of servers in cluster Made replica to another node of the data which has stored in broken node. ex)node addition Sheepdog cluster Automated Rebalance Copyright 2015 NTT DATA Corporation 44

45 45 Feature 3.High performance Archive a good performance in some report FIO benchmark comparison among Sheepdog/Ceph/GlusterFS(LCJ 2014) Test condition 12 node clusters Avr throughput from 4node 12VM configurations Sheepdog:Erasure coding and 3 replica Ceph:3 replica GlusterFS:3 replica +4 stripe Sheepdog(EC) Sheepdog(rep) Ceph GlusterFS MB/s BandWidth (rand-read) MB/s 30.0 BandWidth (rand-write) MB 1MB 512kB 4kB 0.0 4MB 1MB 512kB 4kB

46 Sheepdog architecture Sheepdog clusters Manage raw HDD and connection interfaces Cluster Managers ZooKeeper or CoroSync Client interfaces Qemu NBD Swift iscsi ストレージプール Client or VM Interfaces (qemu/iscsi/swift) VDI Internal HDD IA Servers sheep zookeeper/corosync Linux Softwares Copyright 2015 NTT DATA Corporation 46

47 47 4-type of use case pattern 1Block device for KVM/libvirt 2iSCSI Storage Use case OpenStack Iaas service OpenStack Cinder and Horizon integration iscsi storage for VMwareESX Connection from Windows/Linux OS Backend storage for VirtualBox architecture VM recognize it as block device VM(Linux OS) qemu-dm backing store of iscsi target target tgt Sheepdog driver VDI sheep corosync/zookeeper 仮想ディスク sheep corosync/zookeeper Linux (KVM Hypervisor) Linux

48 48 4-type of use case 構成例 3ObjectStorage 4NFSストレージとして利用 User case Unified Storage with Object storage interface and block interface Shared file server システム概略 OB Container mapped to VDI REST IF ObjectStorage head Sheepdog driver VFS communicat e with Sheep NFS IF NFS head Sheepdog driver VDI One object mapped to sheepdog object sheep corosync/zookeeper Linux VDI sheep corosync/zookeeper Linux

49 49 Distribution algorism Consistent hash NAME: A ID : A 150 Managed with servera NAME: C ID : B C NAME: B ID : 81

50 50 Strong point of Consistent Hashing Less rebalance when changing node count Reduce performance damage when node brokage.l Use Consistent Hashing

51 51 snapshot/clone/rollback Low disk consume when snapshot and clone creation Low overhead when snapshot, clone and rollback odeオブジェクト仮想ディスク VID: fe fe00 fe01 fe02 fe03 オブジェクトのファイルを示す snapshot 取得後仮想ディスク fe00 VID 新規払出し inode 情報はコピー VID: ff fe01 fe02 fe03 snapshot fe00 inodeオブジェクトを引き継ぎ VID: fe fe01 fe02 fe03 fe00 fe01 fe02 fe03

52 52 OwnCloud

53 53 owncloud: Your own Google Drive/Docs owncloud owncloud.org

54 54 owncloud: Your own Google Drive/Docs Building your own Cloud Productivity Application What is owncloud? File hosting application file sync among office, home and mobile devices Web Productivity application edit note, manage calendar, share notes Support Standards: WebDAV, html5 Good for modern browsers Support mobile devices: iphone, ipad and Android Open Source: AGPLv3+GPL3

55 55 owncloud: Many Features Category features Note File Sync Sync files among many type of clients File management Security Extension File share and notification Versioning ODF/PDF Viewer Task Calendar Address book Gallery Music player Video streaming LDAP/Active Directory Encryption Antivirus Application Store External storage User can send and share file securely. Versioning files View ODF/PDF on browser Manage ToDo List and sync with mobile devices Share calendar with users and groups Address book on browser Mange your photos Manage your music files and enjoy everywhere Enjoy your video on browser Authenticate with LDAP ActiveDirectory Encrypt files stored on public cloud Check files Application store for 3 rd party extensions Utilize AmazonS3, DropBox, Swift, GoogleDocs etc.

56 Seafile CC-BY Dan Century 56

57 57 OpenSource Cloud Storage for teams and organizations Easy sharing files in group

58 58 Seahub Web-UI for seafile Django based Web UI for Seafile

60 60 History Start from 2009 Open Sourced in 2012 Young Chinese entrepreneur and hacker start the project and service Internationalized and support many languages. Japanese translation added in 2015.

61 61 Combination of cloud storage software

62 owncloud + Sheepdog Combination of owncloud and Sheepdog Scalable storage Productivity software Run owncloud on VM Add user data disk on demand Scalability issue on RDBMS backend such as MySQL, PostgreSQL ストレージプール Interfaces Qemu VDI Internal HDD IA Servers sheep zookeeper/corosync Linux Copyright 2015 NTT DATA Corporation 62

63 Seafile and OpenStack Swift File Syncing between PCs,Tablets and smartphones with Seafile. File manipulations, versioning management on Webpage. Scalable data architecture using Swift. Scalable head architecture by Seafile processing model. Smartphones Tablets PC Clients Browsers Sync Sync Seahub Scale out Against for users and accesses in every layers. Swift proxy servers Swift storage servers Copyright 2015 NTT DATA Corporation 63

64 64 Conslution Production ready SDS are ready. Cloud service software also is emerged. Combination SDS with private Cloud App provide good usability.