Understanding the Robustness of SSDs under Power Fault Mai Zheng Feng Qin The Ohio State University Joseph Tucek Mark Lillibridge HP Labs
Solid-State Drives (SSDs) - a truly revolutionary and disruptive technology Great performance Low power consumption 2
Solid-State Drives (SSDs) - a truly revolutionary and disruptive technology Great performance Low power consumption *** behavior in adverse conditions? 3
Power Faults - a threat never gone 4
Power Faults - a threat never gone Jan. 2013: A POWER OUTAGE at a key New Jersey data center... Jul. 2012:... human error was responsible for a data center POWER OUTAGE... Jul. 2012: POWER OUTAGE Hits London Data Center... Jun. 2012: Amazon Data Center LOSES POWER During Storm Aug, 2011: Colocation provider Colo4 experienced a POWER OUTAGE Nov. 2010: About 3,000 servers at Montreal web host iweb experienced an OUTAGE May 2010: Car Crash Triggers Amazon POWER OUTAGE 5
Power Faults - a threat never gone Jan. 2013: A POWER OUTAGE at a key New Jersey data center... Jul. 2012:... human error was responsible for a data center POWER OUTAGE... Jul. 2012: POWER OUTAGE Hits London Data Center... Jun. 2012: Amazon Data Center LOSES POWER During Storm Aug, 2011: Colocation provider Colo4 experienced a POWER OUTAGE Nov. 2010: About 3,000 servers at Montreal web host iweb experienced an OUTAGE May 2010: Car Crash Triggers Amazon POWER OUTAGE 6
Potential Failures
Simple Failures before power fault after power fault Bit Corruption record Metadata Corruption all data mess Dead Device 8
Simple Failures before power fault after power fault Shorn Writes new old new old Flying Writes disk block # 1 2 1 2 9
Complex Failure: Unserializable Writes thread A A1 A2 A3 Write Completion Time Serializable state Unserializable state 0 1 2 3 4 block # 0 A2 2 A3 4 0 A1 2 A3 4 10
Testing Framework
Design Scheduler Checker Workers Workers Workers Switcher Control Circuit ❸ read & check ❶ write records ❷ power off/on 12
What to Write? 0000 0000 all 0 s? 0000 ❸ read & check ❶ write records 13
What to Write? 4239 0817 3625 all 0 s? random numbers? ❸ read & check ❶ write records 14
Special Record Format - allows detecting all 6 types of failures checksum fixed-sized header block# timestamp thread_id op_cnt seed 15
Special Record Format - allows detecting all 6 types of failures fixed-sized header checksum block# timestamp thread_id Bit corruption & Shorn writes op_cnt seed 16
Special Record Format - allows detecting all 6 types of failures fixed-sized header checksum block# timestamp thread_id Bit corruption & Shorn writes Flying writes op_cnt seed 17
Special Record Format - allows detecting all 6 types of failures fixed-sized header checksum block# timestamp thread_id Bit corruption & Shorn writes Flying writes Unserializable writes op_cnt seed 18
Special Record Format - allows detecting all 6 types of failures fixed-sized header checksum block# timestamp thread_id Bit corruption & Shorn writes Flying writes Unserializable writes op_cnt seed 19
Special Record Format - allows detecting all 6 types of failures fixed-sized header checksum block# timestamp thread_id op_cnt seed Bit corruption & Shorn writes Flying writes Unserializable writes regenerating records 20
Special Record Format - allows detecting all 6 types of failures fixed-sized header checksum block# timestamp thread_id op_cnt seed Bit corruption & Shorn writes Flying writes Unserializable writes regenerating records Metadata corruption & Dead device 21
Special Record Format - allows detecting all 6 types of failures fixed-sized header extendable padding checksum block# timestamp thread_id op_cnt seed all 0 s? random numbers? duplicates of header 22
Advanced FTL: Compression 23
Randomization of Record Content - avoid interference of compression regular format random mask random format 24
Deriving Completion-time Partial Order - a key step of unserializable writes detection thread A thread B A1 A1 : generating time of B1 1 st record of A A1 B1 A1 : completion time of writing 1 st record of A A2 B2 Time 25
Deriving Completion-time Partial Order - a key step of unserializable writes detection thread A A1 thread B B1 Intra-thread: A1 -> A1 -> A2 A1 B1 A2 Time B2 26
Deriving Completion-time Partial Order - a key step of unserializable writes detection thread A A1 A1 thread B B1 B1 Intra-thread: A1 -> A1 -> A2 B1 -> B1 -> B2 A2 Time B2 27
Deriving Completion-time Partial Order - a key step of unserializable writes detection thread A A1 A1 A2 thread B B1 B1 Intra-thread: A1 -> A1 -> A2 B1 -> B1 -> B2 Inter-thread: A1 -> B1 -> A2 -> B2 Time B2 28
Deriving Completion-time Partial Order - a key step of unserializable writes detection thread A A1 A1 A2 Time thread B B1 B1 B2 Intra-thread: A1 -> A1 -> A2 B1 -> B1 -> B2 Inter-thread: A1 -> B1 -> A2 -> B2 A1 -> B1 or B1 -> A1 Conservatively report no errors 29
Deriving Completion-time Partial Order - a key step of unserializable writes detection thread A A1 A1 A2 Time thread B B1 B1 B2 Intra-thread: A1 -> A1 -> A2 B1 -> B1 -> B2 30
Deriving Completion-time Partial Order - a key step of unserializable writes detection thread A A1 A1 A2 Time thread B B1 B1 B2 Intra-thread: A1 -> A1 -> A2 B1 -> B1 -> B2 Inter-thread: A2 -> B1 31
Deriving Completion-time Partial Order - a key step of unserializable writes detection thread A A1 A1 A2 Time thread B B1 B1 B2 Intra-thread: A1 -> A1 -> A2 B1 -> B1 -> B2 Inter-thread: A2 -> B1 A1 -> B1 More details in our paper & Golab et al. PODC 11 32
Power Fault Injection SSD ❷ power off/on 33
Power Fault Injection Host SSD ❷ power off/on 34
Results
Experimental Environment Block Devices - 15 SSDs and 2 hard drives - SLC & MLC - Manufactured in 2009 2012-4 have power-loss protection - Low-end to high-end ($0.63/GB - $6.50/GB) Host System - Debian 6.0 w/ 2.6.32 kernel - LSI Logic SAS controller - no filesystem on devices - Synchronized & Direct I/O (O_SYNC O_DIRECT) 36
Summary of Observations Failures # of SSDs Bit Corruption 3 Metadata Corruption 1 Dead Device 1 Shorn Writes 3 Flying Writes 0 Unserializable Writes 8 None 2 13 of 15 SSDs exhibit failure(s) 2 perfect SSDs 5 of 6 failures observed 37
Shorn Writes: Subpage Programming pattern #1 new new 4K bytes pattern #2 512 bytes 38
Avg. # of serialization errors per fault Serialization Errors: Avg. Numbers Per Fault 1024 512 256 128 64 32 16 8 4 2 1 0.5 0.25 0.125 0 15 7 6 10 11 12 8 9 4 13 2 5 14 SSD ID increasing price ($/GB) 39
Avg. # of serialization errors per fault Serialization Errors: Avg. Numbers Per Fault 1024 512 256 128 64 32 16 8 4 2 1 0.5 0.25 0.125 0 15 7 6 10 11 12 8 9 4 13 2 5 14 SSD ID increasing price ($/GB) SLC 40
# of serialization errors Serialization Errors: Patterns Over Time 1000 SSD#2 SSD#4 SSD#7 SSD#8 100 10 1 1 10 20 30 40 50 60 70 80 90 100 testing cycle # 41
Metadata Corruption 1 SSD 8 injected power faults lost 31% (72 GB) data Dead Device 1 SSD 136 injected power faults can no long be detected by host 42
Conclusion An effective methodology to expose bugs in block devices under power fault Important implications to storage design e.g. write ahead logging V.S. unserializable writes Thank you! 43
Pristine Version of Our Paper can be Found at: http://www.cse.ohio-state.edu/~zhengm/ 44