Trends in Enterprise Backup Deduplication

Transcription

1 Trends in Enterprise Backup Deduplication Shankar Balasubramanian Architect, EMC 1

2 Outline Protection Storage Deduplication Basics CPU-centric Deduplication: SISL (Stream-Informed Segment Layout) Data Invulnerability Architecture 2

3 Data Protection Issues Tapes are archaic Unreliable Inflexible (needs to be streamed) Requires manual intervention for Disaster Recovery Restore performance is abysmal (no random access) Backup is too slow Traditional architecture moves all the data repeatedly Full restore is too slow Primary storage not that good at writes, esp. small files Must restore all data before any is available Introducing disk based protection storage based on deduplication technology 3

4 Protection Storage 4

5 Protection Storage vs. Primary Storage Workload Primary Storage Continuous random accesses Mostly reads Lots of meta-data accesses Protection Storage Large batches of accesses Mostly writes Few meta-data accesses Cost Dollars / IOPS Dollars / TB Performance Latency and throughput, IOPS Sequential throughput Safety What are backups for? Storage of last resort 5

6 Protection Storage Features Protection Storage Cost Performance Global Compression TM for Data Reduction Inline Deduplication Local compression (Lempel-Ziv, GZ, GZ fast style) Excellent sequential throughput Only fair random access and small file performance Safety Data Invulnerability Architecture Network efficient WAN Vaulting 6

7 Deduplication Backup Storage for D2D+DR Backup Clients WAN Backup/ Onsite Offsite media Retention Disaster Recovery servers Storage Storage Backup Retention/ Replication DR Restore Deduplicating storage systems take role of tape libraries Plug/play w/ standard backup software Replication of reduced data for WAN Vaulting Recover locally or remotely Appliance packaging with options: Storage system with controller, firmware and disks Input: NAS-style/Gb Ethernet, VTL/FC, DD Boost 7 Confidential 7

8 Deduplication Basics 8

9 Deduplication techniques Regular Storage Array 1:1 Whitespace Reduction File Level Fixed Blocks, Snapshots LZ Compression ~ 2:1 Single Instance Storage ~ 3:1 Fixed Block ~ 3:1 Backup Target, Variable Segment Variable Segment ~ 20:1 Deduplication Significantly Reduces - Replication WAN Bandwidth - Power - Heat - Cooling - Management 9

10 How Deduplication Works file system /VTL backup /archive SW data stream 1 st full backup 1 st increment 2 nd full backup A B C B D A B E A B C F D E A B C D E F unique variable segments (4KB-12KB) redundant data segments compressed unique segments 10

11 Deduplication work flow Segment Fingerprint Filter Compress Write Data (from a backup stream) sliced into segments Fingerprints for segments are computed Fingerprints compared to fingerprints in summary vector and cache 1. If fingerprint is new, continue 2. If fingerprint is duplicate, reference, then drop duplicate segments Groups of new segments compressed using lz, gz, or gzfast Segments and metadata written to containers, containers written to disk ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ 3 ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~ Data Domain system 5 container disk disk 11

12 Compression Effect Data Stored in TB Disk Savings Weeks in use 1 TB data set 1 full + 6 incr / week Traditional Storage Capacity Optimized First full backup: 2-4x data reduction File-level incrementals: 5-10x Subsequent full backups: 50-60x Aggregate with Weekly Fulls, Daily Incrementals: 20x 12

13 CPU-centric Deduplication: SISL (Stream-informed Segment Layout) 13

14 Performance: CPU-centric vs. Spindle-bound 14

15 Deduplication at backup / recovery speeds Deduplication is done by looking for matching fingerprints 1TB physical storage will have 125 million fingerprints We cannot store all the fingerprints in memory as that will need too much memory Caching the fingerprints will not work because fingerprints are random We cannot read the disks directly as that will not need too many disks to be read in parallel DD answer: SISL Stream-informed Segment Layout; includes: Summary Vector in RAM says if segment is new Segment Localities minimize seeks if answer is on disk 15

16 Summary Vector (Bloom Filter) for new segments Set bits in SV in RAM for each segment stored Check bits when processing a new segment Any 0 means segment is new and should be stored 16

17 Segment Localities Metadata Segment data abcd A B C D efgh E F G H ijkl I J K L... stuv S T U V DDFS log structure Localities Stream-informed storage units Neighboring unique segments stored together Fingerprints and segments stored together with other metadata One seek can retrieve hundreds into RAM Fast caching for fingerprint lookup Fast reads during recovery or copy to tape 17

18 Data Invulnerability Architecture 18

19 Data Invulnerability Architecture Designed from the ground up for data protection File system simplicity & resiliency Five lines of defense against data loss End-to-end verification Fault avoidance and containment Continuous fault detection and healing File system recoverability 19

20 End-to-End Verification at Backup Time DDOS tests recoverability asynchronously after backups File system consistency Data integrity on disk Primary storage can t verify after write It would be too slow Primary storage discovers problems during restore The end-to-end check verifies all file system data and metadata 20

21 Fault Avoidance and Containment Localities Custom log-structured file system architecture New data never overwrites good data Previous backups are not at risk Fewer complex data structures mean fewer bugs No bitmaps and link counts to corrupt NVRAM for fast, safe restart DD-RAID does no partial-stripe writes No RAID vulnerability to power loss or NVRAM failure 21

22 Continuous Fault Detection and Healing DD-RAID 6 Protection against Two disk failures Disk read errors during reconstruction Operator pulling the wrong disk Verifies data integrity and stripe coherency after writes On-the-fly Error Detection and Correction All on-disk structures covered by strong checksums Data correctness verified on every disk read Data errors corrected automatically from data redundancy Continuous Scrub Finds and fixes errors before they become a problem 22

23 File System Recoverability Self-describing data format Metadata structures rebuildable from Locality log File system check (FSCK), if needed, is fast Checks and repairs done on de-duplicated data Run checks on 4 TB of data, not 80 TB No overwrite means it s safe to bring system back on line Metadata Segment data... DDFS log structure 23

24 Summary Data Protection needs to change Eliminate Tape, backup should be fast, restore should be easy Data Domain core technology can eliminate tape Backup to disk at the cost of tape Keep data safe despite reduced copies Versioned Replication changes backup paradigm Integrated Protection Tier completes the revolution 24

25 25