Fault Isolation and Quick Recovery in Isolation File Systems

Transcription

1 Fault Isolation and Quick Recovery in Isolation File Systems Lanyue Lu Andrea C. Arpaci-Dusseau Remzi H. Arpaci-Dusseau University of Wisconsin - Madison 1

2 File-System Availability Is Critical 2

3 File-System Availability Is Critical Main data access interface desktop, laptop, mobile devices, file servers 2

4 File-System Availability Is Critical Main data access interface desktop, laptop, mobile devices, file servers A wide range of failures resource allocation, metadata corruption failed I/O operations, incorrect system states 2

5 File-System Availability Is Critical Main data access interface desktop, laptop, mobile devices, file servers A wide range of failures resource allocation, metadata corruption failed I/O operations, incorrect system states A small fault can cause global failures e.g., a single bit can impact the whole file system 2

6 File-System Availability Is Critical Main data access interface desktop, laptop, mobile devices, file servers A wide range of failures resource allocation, metadata corruption failed I/O operations, incorrect system states A small fault can cause global failures e.g., a single bit can impact the whole file system Global failures considered harmful read-only, crash 2

7 Server Virtualization VM1 VM2 VM3 Guest virtual machines Hypervisor VMDK1 VMDK2 VMDK3 Shared file system 3

8 VM1 VM2 VM3 VMDK1 VMDK2 VMDK3 4

9 VM1 VM2 VM3 VMDK1 VMDK2 VMDK3 e.g., metadata corruption 4

10 VM1 VM2 VM3 VMDK1 VMDK2 VMDK3 5

11 VM1 VM2 VM3 VMDK1 VMDK2 VMDK3 e.g., metadata corruption 5

12 All VMs VM1 VM2 VM3 are affected ReadOnly VMDK1 VMDK2 VMDK3 or Crash 6

13 All VMs VM1 VM2 VM3 are affected ReadOnly VMDK1 VMDK2 VMDK3 or e.g., metadata corruption Crash 6

14 Our Solution 7

15 Our Solution A new abstraction for fault isolation support multiple independent fault domains protect a group of files for a domain 7

16 Our Solution A new abstraction for fault isolation support multiple independent fault domains protect a group of files for a domain Isolation file systems fine-grained fault isolation quick recovery 7

17 Introduction Study of Failure Policies Isolation File Systems Challenges 8

18 Questions to Answer 9

19 Questions to Answer What global failure policies are used? failure types number of each type 9

20 Questions to Answer What global failure policies are used? failure types number of each type What are the root causes of global failures? related data structures number of each cause 9

21 Methodology 10

22 Methodology Three major file systems Ext3 (Linux ), Ext4 (Linux ) Btrfs (Linux 3.8) 10

23 Methodology Three major file systems Ext3 (Linux ), Ext4 (Linux ) Btrfs (Linux 3.8) Analyze source code identify types of global failures count related error handling functions correlate global failures to data structures 10

24 Q1: What global failure policies are used? 11

25 Global Failure Policies 12

26 Global Failure Policies Definition a failure which impacts all users of the file system or even the operating system 12

27 Global Failure Policies Definition a failure which impacts all users of the file system or even the operating system Read-Only e.g., ext3_error(): mark file system as read-only abort the journal 12

28 Read-Only Example ext3/balloc.c, read_block_bitmap(...){ 1 bitmap_blk = desc->bg_block_bitmap; 2 bh = sb_getblk(sb, bitmap_blk); 3 if (!bh){ 4 ext3_error(sb, Cannot read block bitmap ); return NULL; } } 13

29 Read-Only Example ext3/balloc.c, read_block_bitmap(...){ 1 bitmap_blk = desc->bg_block_bitmap; 2 bh = sb_getblk(sb, bitmap_blk); 3 if (!bh){ 4 ext3_error(sb, Cannot read block bitmap ); return NULL; } } 13

30 Read-Only Example ext3/balloc.c, read_block_bitmap(...){ 1 bitmap_blk = desc->bg_block_bitmap; 2 bh = sb_getblk(sb, bitmap_blk); 3 if (!bh){ 4 ext3_error(sb, Cannot read block bitmap ); return NULL; } } 13

31 Read-Only Example ext3/balloc.c, read_block_bitmap(...){ 1 bitmap_blk = desc->bg_block_bitmap; 2 bh = sb_getblk(sb, bitmap_blk); 3 if (!bh){ 4 ext3_error(sb, Cannot read block bitmap ); return NULL; } } 13

32 Read-Only Example ext3/balloc.c, read_block_bitmap(...){ 1 bitmap_blk = desc->bg_block_bitmap; 2 bh = sb_getblk(sb, bitmap_blk); 3 if (!bh){ 4 ext3_error(sb, Cannot read block bitmap ); return NULL; } } 13

33 Global Failure Policies Definition a failure which impacts users of the file system or even the operating system Read-Only e.g., ext3_error(): mark file system as read-only abort the journal Crash e.g., BUG(), ASSERT(), panic() crash the file system or operating system 14

34 Crash Example btrfs/disk-io.c, 3.8 open_ctree(...) { 1 root->node = read_tree_block(...); 2 BUG_ON(!root->node); 15

35 Crash Example btrfs/disk-io.c, 3.8 open_ctree(...) { 1 root->node = read_tree_block(...); 2 BUG_ON(!root->node); 15

36 Crash Example btrfs/disk-io.c, 3.8 open_ctree(...) { 1 root->node = read_tree_block(...); 2 BUG_ON(!root->node); 15

37 Crash Example btrfs/disk-io.c, 3.8 open_ctree(...) { 1 root->node = read_tree_block(...); 2 BUG_ON(!root->node); 15

38 1000 ReadOnly Crash Number of Instances Ext3 Ext4 Btrfs 16

39 Read-only and crash are common in modern file systems Over 67% of global failures will crash the system 17

40 Q2: What are the root causes of global failures? 18

41 Global Failure Causes 19

42 Global Failure Causes Metadata corruption metadata inconsistency is detected e.g., a block/inode bitmap corruption 19

43 Metadata Corruption Example ext3/dir.c, ext3_check_dir_entry(...){ 1 rlen = ext3_rec_len_from_disk(); 2 if (rlen < EXT3_DIR_REC_LEN(1)){ error = rec_len is too small ; 3 ext3_error(sb, error); } 20

44 Metadata Corruption Example ext3/dir.c, ext3_check_dir_entry(...){ 1 rlen = ext3_rec_len_from_disk(); 2 if (rlen < EXT3_DIR_REC_LEN(1)){ error = rec_len is too small ; 3 ext3_error(sb, error); } 20

45 Metadata Corruption Example ext3/dir.c, ext3_check_dir_entry(...){ 1 rlen = ext3_rec_len_from_disk(); 2 if (rlen < EXT3_DIR_REC_LEN(1)){ error = rec_len is too small ; 3 ext3_error(sb, error); } 20

46 Metadata Corruption Example ext3/dir.c, ext3_check_dir_entry(...){ 1 rlen = ext3_rec_len_from_disk(); 2 if (rlen < EXT3_DIR_REC_LEN(1)){ error = rec_len is too small ; 3 ext3_error(sb, error); } 20

47 Global Failure Causes Metadata corruption metadata inconsistency is detected e.g., a block/inode bitmap corruption I/O failure metadata I/O failure and journaling failure e.g., fail to read an inode block 21

48 I/O Failure Example ext4/namei.c, empty_dir(...){ 1 bh = ext4_bread(null, inode, &err); if (bh && err) 2 EXT4_ERROR_INODE(inode, fail to read directory block ); 22

49 I/O Failure Example ext4/namei.c, empty_dir(...){ 1 bh = ext4_bread(null, inode, &err); if (bh && err) 2 EXT4_ERROR_INODE(inode, fail to read directory block ); 22

50 I/O Failure Example ext4/namei.c, empty_dir(...){ 1 bh = ext4_bread(null, inode, &err); if (bh && err) 2 EXT4_ERROR_INODE(inode, fail to read directory block ); 22

51 I/O Failure Example ext4/namei.c, empty_dir(...){ 1 bh = ext4_bread(null, inode, &err); if (bh && err) 2 EXT4_ERROR_INODE(inode, fail to read directory block ); 22

52 Global Failure Causes Metadata corruption metadata inconsistency is detected e.g., a block/inode bitmap corruption I/O failure metadata I/O failure and journaling failure e.g., fail to read an inode block Software bugs unexpected states detected e.g., allocated block is not in a valid range 23

53 Software Bug Example ext3/balloc.c, ext3_rsv_window_add(...){ 1 if (start < this->rsv_start) p = &(*p)->rb->left; 2 else if (start > this->rsv_end) p = &(*p)->rb->right; 3 else { rsv_window_dump(root, 1); 4 BUG(); } 24

54 Software Bug Example ext3/balloc.c, ext3_rsv_window_add(...){ 1 if (start < this->rsv_start) p = &(*p)->rb->left; 2 else if (start > this->rsv_end) p = &(*p)->rb->right; 3 else { rsv_window_dump(root, 1); 4 BUG(); } 24

55 Software Bug Example ext3/balloc.c, ext3_rsv_window_add(...){ 1 if (start < this->rsv_start) p = &(*p)->rb->left; 2 else if (start > this->rsv_end) p = &(*p)->rb->right; 3 else { rsv_window_dump(root, 1); 4 BUG(); } 24

56 Software Bug Example ext3/balloc.c, ext3_rsv_window_add(...){ 1 if (start < this->rsv_start) p = &(*p)->rb->left; 2 else if (start > this->rsv_end) p = &(*p)->rb->right; 3 else { rsv_window_dump(root, 1); 4 BUG(); } 24

57 Software Bug Example ext3/balloc.c, ext3_rsv_window_add(...){ 1 if (start < this->rsv_start) p = &(*p)->rb->left; 2 else if (start > this->rsv_end) p = &(*p)->rb->right; 3 else { rsv_window_dump(root, 1); 4 BUG(); } 24

58 re types stances Ext3 Data Structure MC IOF SB Shared b-bitmap 2 2 Yes i-bitmap 1 1 Yes inode Yes super 1 Yes dir-entry Yes gdt 3 2 Yes indir-blk 1 1 No xattr No block 5 Yes/No journal 3 27 Yes journal head 31 Yes buf head 16 Yes handle 22 9 Yes transaction 28 Yes revoke 2 Yes other 1 11 Yes/No Total =

59 re types stances Ext3 Data Structure MC IOF SB Shared b-bitmap 2 2 Yes i-bitmap 1 1 Yes inode Yes super 1 Yes dir-entry Yes gdt 3 2 Yes indir-blk 1 1 No xattr No block 5 Yes/No journal 3 27 Yes journal head 31 Yes buf head 16 Yes handle 22 9 Yes transaction 28 Yes revoke 2 Yes other 1 11 Yes/No Total =

60 re types stances Ext3 Data Structure MC IOF SB Shared b-bitmap 2 2 Yes i-bitmap 1 1 Yes inode Yes super 1 Yes dir-entry Yes gdt 3 2 Yes indir-blk 1 1 No xattr No block 5 Yes/No journal 3 27 Yes journal head 31 Yes buf head 16 Yes handle 22 9 Yes transaction 28 Yes revoke 2 Yes other 1 11 Yes/No Total =

61 re types stances Ext3 Data Structure MC IOF SB Shared b-bitmap 2 2 Yes i-bitmap 1 1 Yes inode Yes super 1 Yes dir-entry Yes gdt 3 2 Yes indir-blk 1 1 No xattr No block 5 Yes/No journal 3 27 Yes journal head 31 Yes buf head 16 Yes handle 22 9 Yes transaction 28 Yes revoke 2 Yes other 1 11 Yes/No Total =

62 re types stances Ext3 Data Structure MC IOF SB Shared b-bitmap 2 2 Yes i-bitmap 1 1 Yes inode Yes super 1 Yes dir-entry Yes gdt 3 2 Yes indir-blk 1 1 No xattr No block 5 Yes/No journal 3 27 Yes journal head 31 Yes buf head 16 Yes handle 22 9 Yes transaction 28 Yes revoke 2 Yes other 1 11 Yes/No Total =

63 re types stances Ext3 Data Structure MC IOF SB Shared b-bitmap 2 2 Yes i-bitmap 1 1 Yes inode Yes super 1 Yes dir-entry Yes gdt 3 2 Yes indir-blk 1 1 No xattr No block 5 Yes/No journal 3 27 Yes journal head 31 Yes buf head 16 Yes handle 22 9 Yes transaction 28 Yes revoke 2 Yes other 1 11 Yes/No Total =

64 re types stances Ext3 Data Structure MC IOF SB Shared b-bitmap 2 2 Yes i-bitmap 1 1 Yes inode Yes super 1 Yes dir-entry Yes gdt 3 2 Yes indir-blk 1 1 No xattr No block 5 Yes/No journal 3 27 Yes journal head 31 Yes buf head 16 Yes handle 22 9 Yes transaction 28 Yes revoke 2 Yes other 1 11 Yes/No Total =

65 26

66 All global failures are caused by metadata and system states 26

67 All global failures are caused by metadata and system states Both local and shared metadata can cause global failures 26

68 All global failures are caused by metadata and system states Both local and shared metadata can cause global failures 26

69 Not Only Local File Systems 27

70 Not Only Local File Systems Shared-disk file systems OCFS2 inspired by Ext3 design used in virtualization environment host virtual machine images allow multiple Linux guests to share a file system 27

71 Not Only Local File Systems Shared-disk file systems OCFS2 inspired by Ext3 design used in virtualization environment host virtual machine images allow multiple Linux guests to share a file system Global failures are also prevalent a single piece of corrupted metadata can fail the whole file system on multiple nodes! 27

72 Current Abstractions 28

73 Current Abstractions File and directory metadata is shared for different files or directories 28

74 Current Abstractions File and directory metadata is shared for different files or directories Namespace virtual machines, Chroot, BSD jail, Solaris Zones multiple namespaces still share a file system 28

75 Current Abstractions File and directory metadata is shared for different files or directories Namespace virtual machines, Chroot, BSD jail, Solaris Zones multiple namespaces still share a file system Partitions multiple file systems on separated partitions a single panic on a partition can crash the whole operating system static partitions, dynamic partitions management of many partitions 28

76 29

77 All files on a file system implicitly share a single fault domain 29

78 All files on a file system implicitly share a single fault domain 29

79 All files on a file system implicitly share a single fault domain Current file-system abstractions do not provide fine-grained fault isolation 29

80 Introduction Study of Failure Policies Isolation File Systems New Abstraction Fault Isolation Quick Recovery Preliminary Implementation on Ext3 Challenges 30

81 Isolation File Systems 31

82 Isolation File Systems Fine-grained partitioned files are isolated into separated domains 31

83 Isolation File Systems Fine-grained partitioned files are isolated into separated domains Independent faulty units will not affect healthy units 31

84 Isolation File Systems Fine-grained partitioned files are isolated into separated domains Independent faulty units will not affect healthy units Fine-grained recovery repair a faulty unit quickly instead of checking the whole file system 31

85 Isolation File Systems Fine-grained partitioned files are isolated into separated domains Independent faulty units will not affect healthy units Fine-grained recovery repair a faulty unit quickly instead of checking the whole file system Elastic dynamically grow and shrink its size 31

86 New Abstraction 32

87 New Abstraction File Pod an abstract partition contains a group of files and related metadata an independent fault domain 32

88 New Abstraction File Pod an abstract partition contains a group of files and related metadata an independent fault domain Operations create a file pod set / get file pod s attributes failure policy recovery policy bind / unbind a file to pod share a file between pods 32

89 / d1 d2 d3 d4 33

90 / Pod1 Pod2 d1 d2 d3 d4 34

91 Introduction Study of Failure Policies Isolation File Systems New Abstraction Fault Isolation Quick Recovery Preliminary Implementation on Ext3 Challenges 35

92 Metadata Isolation 36

93 Metadata Isolation Observation metadata is organized in a shared manner hard to isolate a failure for metadata 36

94 Metadata Isolation Observation metadata is organized in a shared manner hard to isolate a failure for metadata For example multiple inodes are stored in a single inode block i i i i i i i i i i i i an inode block 36

95 Metadata Isolation Observation metadata is organized in a shared manner hard to isolate a failure for metadata For example multiple inodes are stored in a single inode block an I/O failure can affect multiple files a block read failure i i i i i i i i i i i i an inode block 36

96 37

97 Key Idea 1: 37

98 Key Idea 1: Isolate metadata for file pods 37

99 Localize Failures 38

100 Localize Failures Local Failures convert global failures to local failures same failure semantics only fail the faulty pod 38

101 Localize Failures Local Failures convert global failures to local failures same failure semantics only fail the faulty pod Read-Only mark a file pod as Read-Only 38

102 Localize Failures Local Failures convert global failures to local failures same failure semantics only fail the faulty pod Read-Only mark a file pod as Read-Only Crash crash a file pod instead of the whole system provide the same initial states after crash 38

103 / Pod1 Pod2 d1 d2 d3 d4 39

104 / Pod1 Pod2 d1 d2 d3 d4 e.g., corruption 40

105 / Pod1 Pod2 d1 d2 d3 d4 e.g., corruption 40

106 Introduction Study of Failure Policies Isolation File Systems New Abstraction Fault Isolation Quick Recovery Preliminary Implementation on Ext3 Challenges 41

107 Quick Recovery 42

108 Quick Recovery File system recovery is slow a small error requires a full check many random read requests 7 hours to sequentially read a 2 TB disk 42

109 43

110 a small fault requires a full check (slow!) 43

111 a small fault requires a full check (slow!) 43

112 44

113 Key Idea 2: 44

114 Key Idea 2: Minimize the file system checking range during recovery 44

115 Quick Recovery 45

116 Quick Recovery Metadata Isolation file pod as the unit of recovery check and recover independently both online and offline 45

117 Quick Recovery Metadata Isolation file pod as the unit of recovery check and recover independently both online and offline When recover? leverage internal detection mechanism 45

118 Quick Recovery Metadata Isolation file pod as the unit of recovery check and recover independently both online and offline When recover? leverage internal detection mechanism How to recover more efficiently? only check the faulty pod narrow down to certain data structures 45

119 Introduction Study of Failure Policies Isolation File Systems New Abstraction Fault Isolation Quick Recovery Preliminary Implementation on Ext3 Challenges 46

120 Ext3 Layout 47

121 Ext3 Layout A disk is divided into block groups physical partition for disk locality 47

122 Ext3 Layout A disk is divided into block groups physical partition for disk locality disk layout 47

123 Ext3 Layout A disk is divided into block groups physical partition for disk locality disk layout one block group 47

124 48

125 f1 f2 f3 f4 multiple files can share a single block group 48

126 f1 f2 f3 f4 multiple files can share a single block group 48

127 f1 f2 f3 f4 multiple files can share a single block group 48

128 f1 f2 f3 f4 multiple files can share a single block group 48

129 f1 f2 f3 f4 multiple files can share a single block group 48

130 f5 one file can span multiple block groups f1 f2 f3 f4 multiple files can share a single block group 48

131 f5 one file can span multiple block groups f1 f2 f3 f4 multiple files can share a single block group 48

132 f5 one file can span multiple block groups f1 f2 f3 f4 multiple files can share a single block group 48

133 f5 one file can span multiple block groups f1 f2 f3 f4 multiple files can share a single block group 48

134 f5 one file can span multiple block groups f1 f2 f3 f4 multiple files can share a single block group 48

135 Layout 49

136 Layout A file pod contains multiple block groups one block group only maps to one file pod performance locality and fault isolation 49

137 Layout A file pod contains multiple block groups one block group only maps to one file pod performance locality and fault isolation POD1 POD2 POD3 disk layout 49

138 Data Structures 50

139 Data Structures Pod related structure no extra mapping structures 50

140 Data Structures Pod related structure no extra mapping structures embeds in group descriptors group descriptors are loaded into memory pod a block group 50

141 Algorithms 51

142 Algorithms Pod based inode and block allocation preserve original allocation s locality allocation will not cross pod boundary 51

143 POD1 POD2 POD3 52

144 1. within the same pod 2. an empty block group POD1 POD2 POD3 52

145 Algorithms 53

146 Algorithms Pod based inode and block allocation preserve original allocation s locality allocation will not cross pod boundary De-fragmentation potential internal fragmentation 53

147 Algorithms Pod based inode and block allocation preserve original allocation s locality allocation will not cross pod boundary De-fragmentation potential internal fragmentation de-fragmentation for file pods similar solution in Ext4 53

148 Journaling 54

149 Journaling Virtual transaction contains updates only from one pod Pod 1 Pod 2 Pod 3 independent T1 T2 T3 transactions On-disk journal 54

150 Journaling Virtual transaction contains updates only from one pod better performance isolation Pod 1 Pod 2 Pod 3 independent T1 T2 T3 transactions On-disk journal 54

151 Journaling Virtual transaction contains updates only from one pod better performance isolation commit multiple virtual transactions in parallel Pod 1 Pod 2 Pod 3 independent T1 T2 T3 transactions journal reservation shared journal On-disk journal 54

152 Introduction Study of Failure Policies Isolation File Systems New Abstraction Fault Isolation Quick Recovery Preliminary Implementation on Ext3 Challenges 55

153 Status 56

154 Status What we did a simple prototype for Ext3 provide readonly isolation 56

155 Status What we did a simple prototype for Ext3 provide readonly isolation What we plan to do crash isolation 56

156 Status What we did a simple prototype for Ext3 provide readonly isolation What we plan to do crash isolation quick recovery after failure 56

157 Status What we did a simple prototype for Ext3 provide readonly isolation What we plan to do crash isolation quick recovery after failure other file systems: Ext4 and Btrfs 56

158 Challenges 57

159 Challenges Metadata isolation tree-based directory structure globally shared metadata: super block, journal shared system states: block allocation tree 57

160 Challenges Metadata isolation tree-based directory structure globally shared metadata: super block, journal shared system states: block allocation tree Local failure is it correct to continue to run? light-weight, stateless crash for a pod 57

161 Challenges Metadata isolation tree-based directory structure globally shared metadata: super block, journal shared system states: block allocation tree Local failure is it correct to continue to run? light-weight, stateless crash for a pod Performance potential overhead of managing pods better performance isolation better scalability 57

162 58

163 Failure is not an option. 58

164 Failure is not an option. -- NASA 58

165 59

166 Global failure is not an option; 59

167 Global failure is not an option; local failure with quick recovery 59

168 Global failure is not an option; local failure with quick recovery is an option. 59

169 Global failure is not an option; local failure with quick recovery is an option. -- Isolation File Systems 59

170 60

171 Questions? 60