libchop,alibraryfordistributed storage&databackup

Transcription

1 libchop,alibraryfordistributed storage&databackup Ludovic Courtès GNU Hackers Meeting, Den Haag, 24 July 2010

2 The Basics - inception & rationale - storing to remote, untrusted sites - library & tools for data storage The Mechanics The Plan

3 inception& rationale PhD in , LAAS-CNRS, France cooperative backup for the people! flexible & self-managed distributed storage storage to remote (untrusted) sites

4 storing to remote, untrusted sites dissemination of data fragments guarantees for data integrity, confidentiality, & authenticity

5 library& tools for data storage content-addressable storage ("deduplication") encryption via content-hash keys file chopping compression

6 from the command line $ chop-archiver --archive holiday-pic.jpg tree_indexer:hash_block_fetcher:hash_index_handle:64:sha1:onx5ee $ chop-archiver --restore \ tree_indexer:hash_block_fetcher:hash_index_handle:64:sha1:onx5ee \ > restored-pic.jpg

7 fromc chop_indexer_t *indexer; chop_chopper_t *chopper; chop_block_indexer_t *block_indexer; chop_block_store_t *store; chop_index_handle_t *index; /* Instantiate these things */ err = chop_indexer_index_blocks (indexer, chopper, block_indexer, store, store, index);

8 from Guile Scheme (define (archive-file file store) ;; Chop FILE and write the blocks to STORE. ;; Return an ASCII string denoting its index. (let* ((f (file-stream-open file)) (c (anchor-based-chopper-open f)) (bi (hash-block-indexer-open hash-method/sha1)) (i (tree-indexer-open 100))) (let ((index (indexer-index-blocks i c bi store store))) (index-handle-ascii-serialize index))))

9 The Basics The Mechanics - the storage pipeline - block stores - fixed-size chopper - content-based chopper - filters - tradeoffs The Plan

10 the storage pipeline block indexer data stream filter chopper stream indexer filter block store

11 block stores put (key, data); ring, can get (key); a byte string, can be anything

12 block store implementations gdbm, tdb, bdb, etc. file system (à la Git) easily transferred over HTTP, FTP, rsync, etc. remote (chop-block-server) ONC RPC, over TLS discovery using Avahi (chop-store-discover)

13 block indexer: content-addressable storage $ chop-archiver -i hash_block_indexer -I sha1 \ --archive britney.mp3 tree_indexer:hash_block_fetcher:hash_index_handle:64:sha1:7c4 x7e/42 sub-classes of block_fetcher and index_handle hash algo to address blocks & check integrity root block hash

14 block indexer: content-hash keys clear text aes256 cipher text sha256 H $ chop-archiver -i chk_block_indexer \ -I aes256,cbc,sha256,sha1 \ --archive naked.jpg :chk_index_handle:64:aes256,cbc,sha1:7c4 x7e,et4 vek/8a sha256 aes256 cipher algo & integrity check H sha1(cipher-text) key = sha256(clear-text)

15 content-addressable storage& compression single-instance storage single-instance

16 fixed-size chopper The quick brown fox jumps over the quick lazy The quick brown wildebeest jumps over the quick la

17 content-based chopper The quick brown fox jumps over the quick lazy The quick brown wildebeest jumps over the quick

18 filters block indexer data stream zlib bzip2 lzo GPG(?) chopper stream indexer zlib bzip2 lzo GPG(?) block store

19 tradeoffs 0.3 (for content-addressed source code revisions) compression ratio content-defined blocks + zlib d blocks content-defined blocks + no compression 0.0 throughput

20 The Basics The Mechanics The Plan - (built-in) support for directories & versioning - a cooperative/p2p backup daemon - storage in GNU

21 (built-in) support for directories& versioning root root previous date previous date (head) /src/chbouib.c /src/chbouib.c /src/backup.scm /src/backup.scm /new-file.txt R 0 R 1 I 0 I 1 I 2 D 0 D 1 D 2 D 3 D 4

22 a cooperative/p2p backup daemon Backup directories versioning libchop Opportunistic Replication Service Discovery Avahi Storage Contribution put (key, data) Authentication GnuTLS Cooperation & Replication Policies File System Networking put

23 your idea here

24 storage in GNU tar cpio RCS? GNUnet Sharutils

25 Summary libchop is cool!

26 Summary mechanisms, not policy content-addressable storage, content-hash keys, etc. supports disseminated data blocks build distributed backup/storage tools!

27 Thanks!

28 Copyright 2010 Ludovic Courtès You may copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is available at