Understanding the Robustness of SSDs under Power Fault

Transcription

1 Understanding the Robustness of SSDs under Power Fault Mai Zheng Feng Qin The Ohio State University Joseph Tucek Mark Lillibridge HP Labs

2 Solid-State Drives (SSDs) - a truly revolutionary and disruptive technology Great performance Low power consumption 2

3 Solid-State Drives (SSDs) - a truly revolutionary and disruptive technology Great performance Low power consumption *** behavior in adverse conditions? 3

4 Power Faults - a threat never gone 4

5 Power Faults - a threat never gone Jan. 2013: A POWER OUTAGE at a key New Jersey data center... Jul. 2012:... human error was responsible for a data center POWER OUTAGE... Jul. 2012: POWER OUTAGE Hits London Data Center... Jun. 2012: Amazon Data Center LOSES POWER During Storm Aug, 2011: Colocation provider Colo4 experienced a POWER OUTAGE Nov. 2010: About 3,000 servers at Montreal web host iweb experienced an OUTAGE May 2010: Car Crash Triggers Amazon POWER OUTAGE 5

6 Power Faults - a threat never gone Jan. 2013: A POWER OUTAGE at a key New Jersey data center... Jul. 2012:... human error was responsible for a data center POWER OUTAGE... Jul. 2012: POWER OUTAGE Hits London Data Center... Jun. 2012: Amazon Data Center LOSES POWER During Storm Aug, 2011: Colocation provider Colo4 experienced a POWER OUTAGE Nov. 2010: About 3,000 servers at Montreal web host iweb experienced an OUTAGE May 2010: Car Crash Triggers Amazon POWER OUTAGE 6

7 Potential Failures

8 Simple Failures before power fault after power fault Bit Corruption record Metadata Corruption all data mess Dead Device 8

9 Simple Failures before power fault after power fault Shorn Writes new old new old Flying Writes disk block #

10 Complex Failure: Unserializable Writes thread A A1 A2 A3 Write Completion Time Serializable state Unserializable state block # 0 A2 2 A3 4 0 A1 2 A3 4 10

11 Testing Framework

12 Design Scheduler Checker Workers Workers Workers Switcher Control Circuit ❸ read & check ❶ write records ❷ power off/on 12

13 What to Write? all 0 s? 0000 ❸ read & check ❶ write records 13

14 What to Write? all 0 s? random numbers? ❸ read & check ❶ write records 14

15 Special Record Format - allows detecting all 6 types of failures checksum fixed-sized header block# timestamp thread_id op_cnt seed 15

16 Special Record Format - allows detecting all 6 types of failures fixed-sized header checksum block# timestamp thread_id Bit corruption & Shorn writes op_cnt seed 16

17 Special Record Format - allows detecting all 6 types of failures fixed-sized header checksum block# timestamp thread_id Bit corruption & Shorn writes Flying writes op_cnt seed 17

18 Special Record Format - allows detecting all 6 types of failures fixed-sized header checksum block# timestamp thread_id Bit corruption & Shorn writes Flying writes Unserializable writes op_cnt seed 18

19 Special Record Format - allows detecting all 6 types of failures fixed-sized header checksum block# timestamp thread_id Bit corruption & Shorn writes Flying writes Unserializable writes op_cnt seed 19

20 Special Record Format - allows detecting all 6 types of failures fixed-sized header checksum block# timestamp thread_id op_cnt seed Bit corruption & Shorn writes Flying writes Unserializable writes regenerating records 20

21 Special Record Format - allows detecting all 6 types of failures fixed-sized header checksum block# timestamp thread_id op_cnt seed Bit corruption & Shorn writes Flying writes Unserializable writes regenerating records Metadata corruption & Dead device 21

22 Special Record Format - allows detecting all 6 types of failures fixed-sized header extendable padding checksum block# timestamp thread_id op_cnt seed all 0 s? random numbers? duplicates of header 22

23 Advanced FTL: Compression 23

24 Randomization of Record Content - avoid interference of compression regular format random mask random format 24

25 Deriving Completion-time Partial Order - a key step of unserializable writes detection thread A thread B A1 A1 : generating time of B1 1 st record of A A1 B1 A1 : completion time of writing 1 st record of A A2 B2 Time 25

26 Deriving Completion-time Partial Order - a key step of unserializable writes detection thread A A1 thread B B1 Intra-thread: A1 -> A1 -> A2 A1 B1 A2 Time B2 26

27 Deriving Completion-time Partial Order - a key step of unserializable writes detection thread A A1 A1 thread B B1 B1 Intra-thread: A1 -> A1 -> A2 B1 -> B1 -> B2 A2 Time B2 27

28 Deriving Completion-time Partial Order - a key step of unserializable writes detection thread A A1 A1 A2 thread B B1 B1 Intra-thread: A1 -> A1 -> A2 B1 -> B1 -> B2 Inter-thread: A1 -> B1 -> A2 -> B2 Time B2 28

29 Deriving Completion-time Partial Order - a key step of unserializable writes detection thread A A1 A1 A2 Time thread B B1 B1 B2 Intra-thread: A1 -> A1 -> A2 B1 -> B1 -> B2 Inter-thread: A1 -> B1 -> A2 -> B2 A1 -> B1 or B1 -> A1 Conservatively report no errors 29

30 Deriving Completion-time Partial Order - a key step of unserializable writes detection thread A A1 A1 A2 Time thread B B1 B1 B2 Intra-thread: A1 -> A1 -> A2 B1 -> B1 -> B2 30

31 Deriving Completion-time Partial Order - a key step of unserializable writes detection thread A A1 A1 A2 Time thread B B1 B1 B2 Intra-thread: A1 -> A1 -> A2 B1 -> B1 -> B2 Inter-thread: A2 -> B1 31

32 Deriving Completion-time Partial Order - a key step of unserializable writes detection thread A A1 A1 A2 Time thread B B1 B1 B2 Intra-thread: A1 -> A1 -> A2 B1 -> B1 -> B2 Inter-thread: A2 -> B1 A1 -> B1 More details in our paper & Golab et al. PODC 11 32

33 Power Fault Injection SSD ❷ power off/on 33

34 Power Fault Injection Host SSD ❷ power off/on 34

35 Results

36 Experimental Environment Block Devices - 15 SSDs and 2 hard drives - SLC & MLC - Manufactured in have power-loss protection - Low-end to high-end ($0.63/GB - $6.50/GB) Host System - Debian 6.0 w/ kernel - LSI Logic SAS controller - no filesystem on devices - Synchronized & Direct I/O (O_SYNC O_DIRECT) 36

37 Summary of Observations Failures # of SSDs Bit Corruption 3 Metadata Corruption 1 Dead Device 1 Shorn Writes 3 Flying Writes 0 Unserializable Writes 8 None 2 13 of 15 SSDs exhibit failure(s) 2 perfect SSDs 5 of 6 failures observed 37

38 Shorn Writes: Subpage Programming pattern #1 new new 4K bytes pattern #2 512 bytes 38

39 Avg. # of serialization errors per fault Serialization Errors: Avg. Numbers Per Fault SSD ID increasing price ($/GB) 39

40 Avg. # of serialization errors per fault Serialization Errors: Avg. Numbers Per Fault SSD ID increasing price ($/GB) SLC 40

41 # of serialization errors Serialization Errors: Patterns Over Time 1000 SSD#2 SSD#4 SSD#7 SSD# testing cycle # 41

42 Metadata Corruption 1 SSD 8 injected power faults lost 31% (72 GB) data Dead Device 1 SSD 136 injected power faults can no long be detected by host 42

43 Conclusion An effective methodology to expose bugs in block devices under power fault Important implications to storage design e.g. write ahead logging V.S. unserializable writes Thank you! 43

44 Pristine Version of Our Paper can be Found at: 44