Reducing Replication Bandwidth for Distributed Document Databases

Transcription

1 Reducing Replication Bandwidth for Distributed Document Databases Lianghong Xu 1, Andy Pavlo 1, Sudipta Sengupta 2 Jin Li 2, Greg Ganger 1 Carnegie Mellon University 1, Microsoft Research 2

2 Document-oriented Databases { "_id" : "55ca4cf7bad4f75b8eb5c25c", "pageid" : "46780", "revid" : "41173", "timestamp" : " T20:06:22", "sha1" : "6i81h1zt22u1w4sfxoofyzmxd "text" : The Peer and the Peri is a comic [[Gilbert and Sullivan]] [[operetta ]] in two acts just as predicting, The fairy Queen, however, appears to all live happily ever after. " } Update { "_id" : "55ca4cf7bad4f75b8eb5c25d, "pageid" : "46780", "revid" : "128520", "timestamp" : " T20:11:12", "sha1" : "q08x58kbjmyljj4bow3e903uz "text" : "The Peer and the Peri is a comic [[Gilbert and Sullivan]] [[operetta ]] in two acts just as predicted, The fairy Queen, on the other hand, is ''not'' happy, and appears to all live happily ever after. " } Update: Reading a recent doc and writing back a similar one 2

3 Replication Bandwidth Operation logs Primary Database WAN Operation logs { "_id" : "55ca4cf7bad4f75b8eb5c25d, "pageid" : "46780", "revid" : "128520", "timestamp" : " T20:11:12", "sha1" : "q08x58kbjmyljj4bow3e903uz "text" : "The Peer and the Peri is a comic [[Gilbert and Sullivan]] [[operetta ]] in two acts just as predicted, The fairy Queen, on the other hand, is ''not'' happy, and appears to all live happily ever after. " } { "_id" : "55ca4cf7bad4f75b8eb5c25c", "pageid" : "46780", "revid" : "41173", "timestamp" : " T20:06:22", "sha1" : "6i81h1zt22u1w4sfxoofyzmxd "text" : The Peer and the Peri is a comic [[Gilbert and Sullivan]] [[operetta ]] in two acts just as predicting, The fairy Queen, however, appears to all live happily ever after. " } Secondary Secondary 3

4 Operation logs Replication Bandwidth { { "_id" : "55ca4cf7bad4f75b8eb5c25d, "_id" : "55ca4cf7bad4f75b8eb5c25c", "pageid" : "46780", "pageid" : "46780", "revid" : "128520", "revid" : "41173", "timestamp" : " T20:11:12", "timestamp" : " T20:06:22", Primary "sha1" : "q08x58kbjmyljj4bow3e903uz "sha1" : "6i81h1zt22u1w4sfxoofyzmxd "text" : "The Peer and the Peri is a "text" : The Peer and the Peri is a Database comic [[Gilbert and Sullivan]] comic [[Gilbert and Sullivan]] [[operetta ]] in two acts just as [[operetta ]] in two acts just as predicted, The fairy Queen, on the other predicting, The fairy Queen, however, hand, is ''not'' happy, and appears to all appears to all live happily ever after. " live happily ever after. " } Goal: Reduce WAN } bandwidth WAN Operation logs for geo-replication Secondary Secondary 4

5 Why Deduplication? Why not just compress? Oplog batches are small and not enough overlap Why not just use diff? Need application guidance to identify source Dedup finds and removes redundancies In the entire data corpus 5

6 Traditional Dedup: Ideal Chunk Boundary Modified Region Duplicate Region Incoming Data {BYTE STREAM } Deduped Data Send dedup ed data to replicas 6

7 Traditional Dedup: Reality Chunk Boundary Modified Region Duplicate Region Incoming Data Deduped Data 4 7

8 Traditional Dedup: Reality Chunk Boundary Modified Region Duplicate Region Incoming Data Deduped Data 4 Send almost the entire document. 8

9 Similarity Dedup (sdedup) Chunk Boundary Modified Region Duplicate Region Incoming Data Delta! Dedup ed Data Only send delta encoding. 9

10 Compress vs. Dedup 20GB sampled Wikipedia dataset MongoDB v2.7 // 4MB Oplog batches 10

11 sdedup Integration Insertion & Updates Client Database Oplog Source Document Cache Source documents sdedup Encoder Dedup ed oplog entries Unsynchronized oplog entries Oplog Oplog syncer sdedup Decoder Re-constructed oplog entries Source documents Replay Database Primary Node Secondary Node 11

12 sdedup Encoding Steps Identify Similar Documents Select the Best Match Delta Compression 12

13 Identify Similar Documents Consistent Sampling Similarity Sketch Feature Index Table Target Document Rabin Chunking Candidate Documents Doc # Doc #2 Doc #3 Doc #2 Doc #3 Similarity Score 1 Doc #1 2 Doc #2 2 Doc #3 13

14 Select the Best Match Initial Ranking Rank Candidates Score 1 Doc #2 2 1 Doc #3 2 2 Doc #1 1 Final Ranking Rank Candidates Cached? Score 1 Doc #3 Yes 4 1 Doc #1 Yes 3 2 Doc #2 No 2 Is doc cached? If yes, reward +2 Source Document Cache 14

15 Evaluation MongoDB setup (v2.7) 1 primary, 1 secondary node, 1 client Node Config: 4 cores, 8GB RAM, 100GB HDD storage Datasets: Wikipedia dump (20GB out of ~12TB) Additional datasets evaluated in the paper 15

16 Compression Compression Ratio sdedup trad-dedup KB 1KB 256B 64B Chunk Size 20GB sampled Wikipedia dataset 16

17 Memory 800 sdedup trad-dedup Memory (MB) KB 1KB 256B 64B Chunk Size 20GB sampled Wikipedia dataset 17

18 Other Results (See Paper) Negligible client performance overhead Failure recovery is quick and easy Sharding does not hurt compression rate More datasets Microsoft Exchange, Stack Exchange 18

19 Conclusion & Future Work sdedup: Similarity-based deduplication for replicated document databases Much greater data reduction than traditional dedup Up to 38x compression ratio for Wikipedia Resource-efficient design with negligible overhead Future work More diverse datasets Dedup for local database storage Different similarity search schemes (e.g., super-fingerprints) 19