Introduction to OpenStack Swift CloudOpen Japan 2014

2 Agenda 1.What is Swift? 2.Swift s Latest Information 3.Swift s Future

3 Who am I Yuji Hagiwara Platform Engineer, NTT DATA Corp. Since 2011 - Using OpenStack Since 2013 - Developing Searching on Swift Demo App for Searching on Swift

Background Data Explosion on Enterprise Amount of Unstructured Data has been growing. We need storage with Scalability, Durability, Availability. Amount of Unstructured Data EB or PB scale Growing exponentially Examples of Unstructured Data Media (Images, Videos, Audios) Web Contents Documents Backups/Archives 2004 2007 2010 2013 2016 Where should we store these data? One of the Solutions is Swift. Copyright 2014 NTT DATA Corporation 4

What is Swift? Swift is... A storage system with Scalability, Durability, Availability. The REST-ful Distributed Object Storage likely Amazon S3. One of OpenStack Core Components. Implemented by Python. A Open Source Software. 1 Block Storage (Cinder) 2 Object Storage (Swift) Copyright 2014 NTT DATA Corporation 5

6 Usage so simple. $ curl -XPUT --data-binary @mydoc.txt http://swift.example.com:8080/v1/account/container/object $ curl XGET http://swift.example.com:8080/v1/account/container/object $ curl XDELETE http://swift.example.com:8080/v1/account/container/object

7 Use cases of Swift

8 Swift as a storage for a variety of applications System Backup CMS FTP-like use Digital Distribution Web Apps Cyber Duck REST API Swift

9 OpenStack Swift deployments and use cases Name of enterprise Product/ service Description Rackspace(USA) Korean Telecom (Sourth Korea) Cloud Files ucloud storage service Cloud file share service by Rackspace itself. They use same code as OSS except for features such as authentication, Accounting and CDN (<500PB) Object storage service using OpenStack/Swift (16PB+ size) Sina (Republic of China) Sina App Engine(SAE) Public storage service. They moved to OpenStack from another technology MongoDB in 2012. San Diego Supercomputer Center (USA) SDSC Cloud Storage Services Cloud storage service on SDSC. Users can select Amazon/S3 or Rackspace Swift. SME Storage (USA) SMEStorage Open Cloud Platform Cloud storage service based on Rackspace Cloud File SoftLayer (USA) SoftLayer Object Storage Public object storage service. Acquisition by IBM SwiftStack(USA) Swift Stack Provide professional service and Operation and management product HP(USA) HP Cloud Private cloud storage service uses OpenStack. Wikimedia(USA) Wikimedia storage Media files store for Wikipedia. NII(JAPAN) Academic Cloud service Academic cloud service by National Institute of Informatics in Japan (NII)(Integrated and supported by NTT Data)

10 Inside Swift

11 Architecture: Nodes Swift consist of 2-type Nodes: Proxy Node and Storage Node. Application Forward Data to node HTTP Load balancer Proxy Node Proxy Node Proxy Node Store data Storage Node Storage Node Storage Node Storage Node

12 Architecture: The Ring The Ring (static table for data allocation on storage node) decide the optimal Storage Node by Name. Application HTTP Load balancer Proxy Node Proxy Node Proxy Node Ring Ring Ring Storage Node Storage Node Storage Node Storage Node Ring Ring Ring Ring

13 Architecture: The role of Ring If you requested to Store the data A, 3 Replica nodes store the data A. data Application HTTP Load balancer Proxy Node Proxy Node Proxy Node A must be located at 1, 2, 4 Ring Ring Ring Storage Node 1 Storage Node 2 Storage Node 3 Storage Node 4 Data A Data A Data B Data A Data B Data B Ring Ring Ring Ring

14 Architecture: The role of Ring If you requested to Get the data A, One of Nodes reply the data A. data Application data HTTP Load balancer Proxy Node Proxy Node Proxy Node Ring Ring Ring A must be located at 1, 2, 4 Storage Node 1 Storage Node 2 Storage Node 3 Storage Node 4 Data A Data A Data B Data A Data B Data B Ring Ring Ring Ring

15 Scalability (1) Expand proxy server Throughput (2)Expand Storage servers or disks volume More Throughput Proxy Proxy (expand) Proxy Proxy Storage Storage Storage Storage Storage Storage Storage (Expand) More Volume

16 Many processes working together containersync proxyserver Swift objectreplicator accountserver objectauditor objectupdater objectserver objectexpirer accountreplicator accountauditor accountreaper containerserver containerreplicator containerauditor containerupdater

17 Disk Nor mal Node 1 Node 2 Node 3 Node 4 (1) Each nodes checks data in others Node 1 Node 2 Node 3 Node 4 (2) Disk defeat (3) Detect disk trouble Defe at Node 1 Node 2 Node 3 Node 4 (4) Copy data to another node Node 1 Node 2 Node 3 Node 4 (5) Recover disk (6) recover data to original node Reco very Node 1 Node 2 Node 3 Node 4 (7)Delete temporal data

19 Normal state Each Data has replicated. Application HTTP Load balancer Proxy Node Proxy Node Proxy Node Ring Ring Ring Ring Storage Node Storage Node Storage Node Storage Node Server Server Server Server Data A Data A Data B Data A Data B Data B Auditor Auditor Auditor Auditor Ring Ring Ring

21 Defeat state If a disk is broken... Application HTTP Load balancer Proxy Node Proxy Node Proxy Node Ring Ring Ring Ring Storage Node Storage Node Storage Node Storage Node Server Server Server Server Data A Data A Data B Data A Broken Data B Data B Auditor Auditor Auditor Auditor Ring Ring Ring

Defeat state detects the lost data and replicates the data to another node for temporary. Application HTTP Load balancer Proxy Node Proxy Node Proxy Node Ring Ring Ring Ring Storage Node Storage Node Storage Node Storage Node Server Server Server Server Data A Data B Data A Data B Data A Broken Data B Auditor Auditor Auditor Auditor Ring Ring Ring Copyright 2014 NTT DATA Corporation Temporary data When detect a lost data, Replicate the data. 22

24 Recovery state When the broken disk is replaced to a fresh disk... Application HTTP Load balancer Proxy Node Proxy Node Proxy Node Ring Ring Ring Ring Storage Node Storage Node Storage Node Storage Node Server Server Server Server Data A Data B Data A Data B Data A Data B Auditor Auditor Auditor Auditor Ring Ring Ring

Recovery state replicates the data and removes the temporary data. Application HTTP Load balancer Proxy Node Proxy Node Proxy Node Ring Ring Ring Ring Storage Node Storage Node Storage Node Storage Node Server Server Server Server Data A Data B Data A Data B Data A Data B Auditor Auditor Auditor Auditor Ring Ring Ring Copyright 2014 NTT DATA Corporation Removed Replicate the data to the correct node. 25

27 Latest Information

28 History and Trend of Community 2010.6 Start OpenStack Project 2013.4 Grizzly 2013.4 Icehouse 2010.10 1st release "Austin" History of OpenStack 2013.10 Havana 2014.10 Juno Now Hot Topics on Now Erasure Coding Storage Policy Fundamental Global Cluster Development Trend in Swift Timeline in each functions Developing Supported

29 Latest info: Erasure Coding Replication Erasure Coding Data Data Data distribute Data Data Partial Data Partial Data distribute Parity 1 Parity 2 Size 3x original 2x original

30 Latest info: Storage Policy Before Data Data Data After Data Data Data Data Data Data Partial Data Partial Data Parity 1 Parity 2 Data Data Data Data Data Same Policy on cluster Variety Policy on cluster

31 2 concepts: Integrated Searchable Storage Intelligent Resource Management Future Direction of Swift

32 Integrated Searchable Storage Swift should be integrated with Searching. It means to need searching as Scalable, Durable, Available as Swift. Users Operators Store Get Swift Storage System Managers Search

Use cases of Search 1.Content Search 2.Detection for de-duplication 3.Tiered storage Data Hash Already Stored? Data Data Hash Hash Data Hash Copyright 2014 NTT DATA Corporation Hot Content More Modified Cheap Storage Cold Content Less Modified Modifed Date is older than 05/20/2014? 33

34 Future: Integrated Searchable Storage How do we implement? Internal External Search Swift Storage feature Store Index Search feature Search Hook Swift Storage feature Store Index Search preexist Search Engine Where do search Internal Swift with search library (such as Lucene) External Search Engine (such as Solr) Redundancy High Depend on Search engine Availability High Depend on Search engine Scalability High Depend on Search engine Difficulty of implementation Hard Easy

35 Our Implementation Internal Approach Hack Swift to embed the search library. New Search API Application Proxy Node Swift s Ordinary API Indexing Searching Query Distributed by the Ring Data Metadata objectserver accountserver container-server SQLite Search ContainerA DB ContainerB DB Lucene Container A Index Container B Index Indexing

Future: Intelligent Resource Management Swift has more and more different functions. Other arbitrary processes Search...? Compression...? Encryption...? objectexpirer accountreplicator Multi-ring support Swift Storage Policy Erasure Coding objectupdater objectserver objectreplicator accountserver objectauditor proxyserver ecreconstructor ecauditor ec-stripeauditor accountauditor accountreaper containerserver containerreplicator containerauditor containerupdater containersync Copyright 2014 NTT DATA Corporation 36

37 Future: Intelligent Resource Management Resources are drained! IOPS, CPU, Network, Memory Performance Priorities of these functions are different by the Requirement. Ex1) Store performance VS Search performance Ex2) Service Level on Business Hour VS on Outside Hour 0 Outside (High-prio to check durability) 18 Business Hour (High-prio to process requests) 12 6 More Intelligent Resource Management is necessary. with cgroups

38 Summary 1.What is Swift? Swift is a Great OSS, for storing unstructured data. 2.Swift s Latest Information Erasure Coding Storage Policy 3.Swift s Future Integrated Searchable Storage Intelligent Resource management

PR: Demonstration is Now Available! We exhibit the Demo Application(Contents delivery system) built with Swift. On-demand Delivery a lot of contents(pictures or movies) stored at Swift. Implemented Searching on Swift. (Our original implementation) (map for demo booth) Copyright 2014 NTT DATA Corporation 39

Q&A: Do you have any question? Copyright 2011 NTT DATA Corporation Thank you for your attention! Please contact to hagiwarayuj@nttdata.co.jp, if you have any questions or comments. Copyright 2014 NTT DATA Corporation

Challenges and Questions How to integrate Swift with cgroups? How to use cgroups? What is the best toolset for cgroups? VFS? libcgroup? systemd? How to control multiple hosts with cgroups dynamically? How to integrate Swift with search? What is the best implementation way? What is the best search middleware? How to search Multilingual? Copyright 2014 NTT DATA Corporation 41