Erasure Coding in Windows Azure Storage

Transcription

1 Erasure Coding in Windows Azure Storage Cheng Huang Microsoft Corporation Joint work with Huseyin Simitci, Yikang Xu, Aaron Ogus, Brad Calder, Parikshit Gopalan, Jin Li, and Sergey Yekhanin

2 Outline Introduction to Windows Azure Storage (WAS) Conventional Erasure Coding in WAS Local Reconstruction Coding in WAS 2

3 North America Region Europe Region Asia Pacific Region Major datacenter CDN PoPs 3

4 Windows Azure Storage Abstractions Blobs File store in the cloud CDN High performance file delivery through proximity caching Drives Durable NTFS volumes for Windows Azure applications Tables Massively scalable NoSQL storage Queues Reliable storage and delivery of messages Easy client access Easy to use REST APIs and Client Libraries Existing NTFS APIs for Windows Azure Drives 4

5 Massive Distributed Storage Systems in Cloud Failures are norm rather than exception As the number of components increase, so does the probability of failure MTTFFirst = MTTFOne / Redundancy is necessary to cope with failures Replication vs. Erasure Coding? n 5

6 Replication vs. Erasure Coding replication a=2 erasure coding a=2 a=2 b=3 b=3 b=3 a=2 a+b=5 a=2 b=3 6

7 Replication vs. Erasure Coding replication a=2 erasure coding a=2 a=2 b=3 b=3 b=3 a=2 a+b=5 b=3 2a+b=7 a=2 b=3 a=2, b=3 7

8 WAS Stream Layer Append-only distributed file system Provides intra-stamp redundancy Streams are very large files Has file system like namespace Ordered list of pointers to extents Extents Unit of replication Sequence of blocks Partition Layer Stream Layer Intra-stamp replication Storage Stamp Storage Location Service Inter-stamp (Geo) replication Partition Layer Stream Layer Intra-stamp replication Storage Stamp 8

9 Replication and Lazy Erasure Coding Extents triple replicated when first created while being appended 9

10 Replica Placement in a Storage Stamp 10

11 Replication and Lazy Erasure Coding Extents triple replicated when first created while being appended Extents lazily erasure coded in the background extents sealed at around 3GB when erasure coding finishes, full replicas are deleted policies to choose between replication, erasure coding, or a mix 11

12 Conventional Erasure Coding RS 6+3 sealed extent ( 3 GB ) p 1 d 0 d 1 d 2 d 3 d 4 d 5 p 2 Reed-Solomon 6+3 p 3 12

13 Erasure Coding Flow Paxos ack with metadata SM SM SM start erasure coding an extent EN 1 EN 2 EN 3 EN 13

14 Space Savings (RS 6+3 over 3-replication) Savings Replicated Data Fragments Code Fragments 14

15 How to Further Reduce Storage Cost? sealed extent ( 3 GB ) overhead (6+3)/6 = 1.5x d 0 d 1 d 2 d 3 d 4 d 5 p 0 p 1 p 2 15

16 How to Further Reduce Storage Cost? sealed extent ( 3 GB ) overhead (6+3)/6 = 1.5x d 0 d 1 d 2 d 3 d 4 d 5 p 0 p 1 p 2 (12+4)/12 = 1.33x d 0 d 1 d 2 d 3 d 4 d 5 d 6 d 7 d 8 d 9 d 10 d 11 p 0 p 1 p 2 p 3 16

17 Challenge p 0 d 0 d 1 d 2 d 3 d 4 d 5 d 6 d 7 d 8 d 9 d 10 d 11 p 1 reconstruction read p 2 p 3 17

18 Reconstruction Read Partition Layer Paxos SM SM SM read data reconstruction read EN 1 EN 2 EN 3 EN 18

19 Challenge p 0 d 0 d 1 d 2 d 3 d 4 d 5 d 6 d 7 d 8 d 9 d 10 d 11 p 1 reconstruction read is expensive reading d 0 during unavailability requiring 12 fragments (12 disk I/Os, 12 net transfers) p 2 p 3 19

20 Reconstruction Read When? Load balancing avoid hot storage node serve reads via reconstruction Rolling upgrade Transient unavailability and permanent failures can we achieve1.33x overhead while requiring only 6 fragments for reconstruction? 20

21 Opportunity Conventional EC all failures are equal same reconstruction cost, regardless of failure # Cloud storage Prob(1 failure) >> Prob(2 or more failures) optimize erasure coding for cloud storage making single failure reconstruction most efficient 21

22 Local Reconstruction Code sealed extent ( 3 GB ) x 0 x 1 x 2 x 3 x 4 x 5 y 0 y 1 y 2 y 3 y 4 y 5 LRC : 12 data fragments, 2 local parities and 2 global parities storage overhead: ( ) / 12 = 1.33x Local parity: reconstruction requires only 6 fragments 22

23 One More Thing Ensuring Reliability in LRC LRC needs to recover arbitrary 3 failures as many 4 failures as possible 23

24 Recover 3 Failures 1 st example q 0 x 0 x 1 x 2 x 3 x 4 x 5 y 0 y 1 y 2 y 3 y 4 y 5 q 1 p x p y recover y 1 from p y (group y) 24

25 Recover 3 Failures 1 st example q 0 x 0 x 1 x 2 x 3 x 4 x 5 y 0 y 1 y 2 y 3 y 4 y 5 q 1 p x p y recover y 1 from p y (group y) recover x 0 and x 2 from q 0 and q 1 25

26 Recover 3 Failures 2 nd example q 0 x 0 x 1 x 2 x 3 x 4 x 5 y 0 y 1 y 2 y 3 y 4 y 5 q 1 p x p y 26

27 Recover 3 Failures 2 nd example q ' 0 x 0 x 1 x 2 x 3 x 4 x 5 y 0 y 1 y 2 y 3 y 4 y 5 q ' 1 p x p y cancel all the y i s from global parities q 0 and q 1 27

28 Recover 4 Failures 1 st example q 0 x 0 x 1 x 2 x 3 x 4 x 5 y 0 y 1 y 2 y 3 y 4 y 5 q 1 p x p y p y is redundant 3 parities cannot recover 4 failures 28

29 Recover 4 Failures 2 nd example q 0 x 0 x 1 x 2 x 3 x 4 x 5 y 0 y 1 y 2 y 3 y 4 y 5 q 1 p x p y Maximal Recoverability swap failure with local parity count remaining failures and global parities 29

30 Properties of LRC Achieving maximal recoverability LRC : arbitrary 3 failures and 86% of 4 failures reliability: RS 12+4 > LRC > RS 6+3 technical paper published at USENIX ATC 2012 Requiring minimum storage overhead, given reconstruction cost fault tolerance technical paper to appear in IEEE Trans. on Information Theory 30

31 Cost & Performance Analysis Vary LRC parameters trade-off points in 3D space storage overhead reconstruction cost reliability (MTTDL) Reliability is a hard requirement set MTTDL 3-replication as target reduce trade-off space to 2D 31

32 LRC vs. Reed-Solomon RS LRC (12+2+2) LRC (12+4+2) Reconstruction Read Cost 10 RS 10+4 same cost 1.5x 1.33x 8 RS 6+3 same overhead 6 half cost (6 3) Storage Overhead Reed-Solomon LRC RS 10+4 : HDFS-RAID at Facebook RS 6+3 : GFS II (Colossus) at Google 32

33 Choice of Windows Azure Storage RS (6 + 3) reconstruction cost = 6 RS (14 + 4) reconstruction cost = 14 LRC ( ) reconstruction cost = 7 14% savings 33

34 Encoding/Decoding Complexity Optimized erasure coding implementation homomorphic transformation convert complex finite field matrix operations to XOR ing large data blocks XOR scheduling further improves throughput Encoding/decoding overhead negligible, compared with network/disk overheads 34

35 Additional Lessons I/O scheduling reconstruction/recovery/on-demand traffic prioritized and throttled carefully erasure encoding rate kept up with data injection rate Data consistency checksum handoff and verification between all levels scrub periodically 35

36 Summary Erasure coding enables significant storage cost savings in Windows Azure Storage with higher reliability than 3-replication LRC achieves additional 14% savings without compromising performance Windows Azure Storage Team Blog 36