Consistency and Reliability in Cloud-backed Storage Systems

Transcription

1 Consistency and Reliability in Cloud-backed Storage Systems Alysson Bessani University of Lisbon, Faculty of Sciences (joint work with a lot of people!) 1 Cloud Storage vs Cloud-backed Storage Elasticity Virtualization Scalability Cloud storage gateways File Synchronization Services Personal Storage Services 2 1

2 Cloud-backed Storage Personal Storage Services (e.g., ropbox, Onerive) ropbox System Amazon EC2 and S3 Cloud-backed File Systems (e.g., BlueSky, S3FS) Proxy Cloud Storage 3 State of the Art 4 2

3 Shared Cloud-backed File System (SCFS) Client-based Use existing cloud services Cloud Storage ATA ATA Pay-per ownership Each client pays for its own files SCFS Agent SCFS Agent SCFS Agent Strong Consistency Controlled sharing Access control for security and concurrency Redundant Cloud Services 5 Challenges Internet latency and bandwidth Costs Weak consistency 6 3

4 Outline SCFS esign Consistency Issues Reliability: Using Multiple Clouds Implementation and Evaluation Ongoing work 7 SCFS esign 8 4

5 esign Choices ata layout/access pattern Each file is an object (single-block file) Multiple versions of the files are maintained Always write, avoid reading (exploiting free writes) Cache Persistent file cache Local storage is used to hold copies of all/most client files Opened files are also maintained in main-memory Short lived main-memory metadata cache To deal with bursts of metadata requests 9 esign Choices Consistency Consistency-on-close semantics Control of durability and consistency Locks used to avoid write-write conflicts Modular coordination Separate data from metadata Metadata is stored in a coordination service E.g., Zookeeper [ATC 10], epspace [EuroSys 08] Also used for managing file locks 10 5

6 SCFS Architecture Coordination Service Lock Service Access Control Metadata Computing clouds SCFS Agent Cache SCFS Agent Cloud storage Storage clouds Cache SCFS Agent Cache 11 Security Model Straw man solution: All users share the same account (bill, permissions) SCFS uses the pay-per-ownership model: Several users, each one using its own account Who creates the file, pays for its storage Based on POSIX ACLs (setfacl & getfacl) File ACLs are reflected in the coordination service and the storage clouds Not simple due to access control heterogeneity across different cloud storage providers! 12 6

7 Consistency 13 Strong vs. Weak Consistency x.write(v) x.read() è v x.read() è v Strong Consistency (e.g., Paxos-based Systems, Azure Storage) x.write(v) x.read() è? x.read() è v Weak Consistency (e.g., Amazon S3, Rackspace Files) 14 7

8 Consistency Anchor Problem: How to provide strong consistency on top of weak consistent storage clouds? Composite Storage strong consistency write Algorithm read write read weak consistency Storage Service strong consistency write Consistency Anchor read Key property: the composite storage consistency is the same of the consistency anchor 15 Consistency Anchor WRITE(id, v): REA(id): w1: h Hash(v) r1: h CA.read(id) w2: SS.write(id h, v) r2: do v SS.read(id h) while v = null w3: CA.write(id, h) r3: return (Hash(v) =h)?v : null Figure 3: Algorithm for increasing the consistency of the stor- Garbage collection is needed for limiting storage space r2 needs a timeout for dealing with malicious writers 16 8

9 Consistency Anchor in SCFS If x is cached, it is only validated! REA fd = open( x, );!! 1 Local Storage Coordination read(fd, );! write(fd, );! 2 Service Main Memory! 2 fsync(fd);! Cloud(-of-Clouds)! write(fd, );! Persistent Storage! close(fd);! WRITE 1! 17 Using Multiple Clouds for Reliability 18 9

10 Why use Multiple Cloud Storage Services? Survive datacenter and cloud outages Avoid vendor lock-in Achieve better read performance Tolerance to data corruption Bugs Malicious insiders Attacks and intrusions Amazon S3 Rackspace Windows Azure 19 Cloud-of-clouds Storage Two fundamental characteristics 1. No modification on cloud storage services 2. No collaboration between providers Some works in the area RACS [SCC 10] epsky [EuroSys 11] ICStore [SN 12] Amazon S3 Rackspace Windows Azure 20 10

11 epsky: ependable Cloud-of-Clouds Object Storage Amazon S3 Google Storage Rackspace Windows Azure 21 Key Challenges How to implement an efficient replication protocol using only passive storage nodes? Novel Byzantine quorum systems protocol read How to make it affordable? Erasure codes write 22 11

12 epsky Write WRITE ATA ACK WRITE METAATA ACK Cloud A Cloud B Cloud C Cloud 23 Cloud A Cloud B Cloud C Cloud epsky Read REA METAATA REA ATA ATA METAATA highest version number ata will be fetched from other clouds if needed

13 epsky Confidentiality and Storage Efficiency ata Limitations: 1. ata is accessible by cloud providers 2. Requires n ata storage space Cloud A Cloud B Cloud C Cloud ata ata ata ata 25 epsky Confidentiality and Storage Efficiency ata encrypt K generated key disperse share F 1 F 2 F 3 F 4 S 1 S 2 S 3 S 4 Cloud A Cloud B Cloud C Cloud F 1 S 1 F 2 S 2 F 3 S 3 F 4 S

14 Consistency Proportionality The consistency provided by epsky is the same as the base storage clouds If the weakest consistency cloud provides eventual consistency, epsky provides eventual consistency If the weakest consistency cloud provides read your writes, epsky provides read your writes If the weakest consistency cloud provides regular storage, epsky provides regular storage This notion may be useful for other systems 27 Implementation and Evaluation 28 14

15 SCFS Backends SCFS can use different backends i.e., different cloud storage and a coordination service plugin EC2( S% S% S% S% S% RS( SCFS% Agent% AWS%Backend% S3( BFT$SMaRt* SCFS% Agent% epsky* CoC%Backend% Operation: blocking, non-blocking and non-sharing WA( GS( S3( 29 The Cloud-of-Clouds Backend oes not require trust on any single cloud provider SCFS works correctly as long as less than a third of the providers misbehave epspace/bft-smart [EuroSys 08, SN 14] SCFS Agent Cache epsky CoC Storage [ACM ToS 13] 1. gen. key File ata Client% 2. encrypt 4. secret sharing 3. erasure coding Storage%Services% 1 RS( 1 2 WA( 2 3 GS( 3 4 S3( 4 Figure 6: A write in SCFS using the epsky protocols

16 Everything open source SCFS code available at epsky and epspace/bft-smart also available Latency of CoC and AWS Storage CoC Vs. AWS S3 Latency (ms) 10K 1K 100 Figure 5: epsky read epsky write S3 read S3 write 1K 4K 16K 64K 256K 1M 4M 16M ata size (bytes) S3 is better epsky is better 32 16

17 epspace Latency MS Operation Local EC2 CoC getmetadata 2.21 ± ± ± 1.48 getir 3.26 ± ± ± 0.81 update 5.13 ± ± ± 1.11 put 5.82 ± ± ± 1.36 delete 3.31 ± ± ± 0.68 Table 2: Latency (in milliseconds) of relevant metadata ser- +10 ms AWS EC2 CoC 33 Sharing Latency: SCFS vs ropbox ATA ATA Latency (s) Rackspace Files #! Google Storage Windows Azure Blob % Amazon S3 90% value 50% value CoC-B CoC-NB AWS-B AWS-NB ropbox?$ CoC-B CoC-NB AWS-B AWS-NB ropbox ATA ATA CoC-B CoC-NB AWS-B AWS-NB ropbox 256K 1M 4M 16M ata Size CoC-B CoC-NB AWS-B AWS-NB ropbox Figure 10(b) displays the l marks considering the use of ferent percentages of files sha user. Recall that all previous r (100%), without using PNS, nario. As expected, the result of private files increases, the improves. For instance, when shared more than what was o 34 study we are aware of [33] marks decreases by a factor o and 3.5 (copy files). Non-blocking Blocking 4.5 SCFS Operation C Figure 11 shows the 17 costs as using SCFS. The fixed operati mainly the maintenance of the ning in one or more VMs de Figure 11(a) considers two in

18 Latency (s) Benchmarking (Unmodified) esktop Applications OpenOffice Writer Open Action: 1 open(f,rw), 2 read(f), 3-5 open-write-close(lf1), 6-8 open-read-close(f), 9-11 open-read-close(lf1) Save Action: 1-3 open-read-close(f), 4 close(f), 5-7 open-readclose(lf1), 8 delete(lf1), 9-11 open-write-close(lf2), openread-close(lf2), 15 truncate(f,0), open-write-close(f), open-fsync-close(f), open-read-close(f), 25 open(f,rw) Close Action: 1 close(f), 2-4 open-read-close(lf2), 5 delete(lf2) Figure 7: File system operations invoked in the file synchro- 11 CoC read/writes Open Open Save Save Close Close. S3QL S3QL(L). CoC-NS CoC-NS(L). CoC-NB CoC-NB(L) AWS-NB(L) AWS-NB AWS CoC CoC (NS) S3QL Non-blocking Latency (s) AWS-B(L) AWS-B 55% 40% 80% S3FS. CoC-B(L) CoC-B AWS CoC S3FS Blocking lock files S3FS(L) 35 Financial Evaluation VM Instance EC2 EC2 4 CoC Capacity Large $6.24 $24.96 $ M files Extra Large $12.96 $51.84 $ M files (a) Operation costs/day and expected coordination service capacity. Cost/op (microdollar) read CoC read AWS read CoC write AWS write write File size (MB) (b) Cost per operation (log scale). Cost/day (microdollar) CoC AWS File size (MB) (c) Cost per file per day. +50% 36 18

19 Wrap Up SCFS is a cloud-backed file systems that can be used for backup, disaster recovery and sharing data Key design principles: Always write, avoid reading Strong consistency through consistency anchors Cloud-of-clouds storage (epsky) for super dependability Experience so far The design is possible (CoC not substantially slower) Metadata access much faster than cloud storage access Component reuse is a must: improving BFT-SMaRt improves epspace, while improving epspace and epsky improve SCFS 37 To know more Bessani, Mendes, Oliveira, Neves, Correia, Pasin, Verissimo. SCFS: A Shared Cloud-backed File System. USENIX ATC 14. Bessani, Correia, Quaresma, André, Sousa. epsky: ependable and Secure Storage in the Cloud of Clouds. ACM Transactions on Storage. Nov 2013 (preliminary version on ACM EuroSys 11). Bessani, Alchieri, Correia, Fraga. epspace: A Byzantine Fault-Tolerant Coordination Service. ACM EuroSys 08. Bessani, Santos, Felix, Neves, Correia. On the Efficiency of urable State Machine Replication. USENIX ATC 13. Bessani, Sousa, Alchieri. State Machine Replication for the Masses with BFT-SMaRt. IEEE/IFIP SN