Theoretical Aspects of Storage Systems Autumn 2009

Transcription

1 Theoretical Aspects of Storage Systems Autumn 2009 Chapter 1: RAID André Brinkmann

2 University of Paderborn Personnel Students: ~ students Professors: ~230 Other staff: ~600 scientific, ~630 non-scientific Five Departments: Cultural studies, economic science, natural science, mechanical engineering, electrical engineering/ computer science/mathematics Master / bachelor degrees, master of education, PhD Research 9 central research institutes 1 DFG Collaborative Research Centers (SFB) 1 European Graduate School on Large Scale Storage Systems 7 graduate colleges / schools 2 Fraunhofer Institutes

3 Computer Science at UPB Computer science department Professors: 21 Other staff: 95 scientific, 23 non-scientific Students: 1600 Research areas Embedded systems and system software Models and algorithms Human-computer interaction Software engineering and information systems Ranking DFG (national science foundation): 7/129 in approval of science funds ACM: best German institution for software engineering CHE ranking: leading position among 77 universities regarding 3rd party funds, teaching, infrastructure, research reputation

4 PC 2 Paderborn Center for Parallel Computing Regional competence center for parallel and distributed data processing in research and application Founded in 1991 Scientific institute of Uni. Paderborn Goal: Foster and explore efficient use of parallel and distributed systems Mandate in research and service Funding from regional and federal government, industrial and EU-projects Research: ~15 PhD students, 2 post docs in 8 projects Services: High performance computing systems, currently ~ 20 user projects and ~40 researchers ~55% resource usage of non-paderborn users

5 Syllabus RAID-Systems Reliability based on efficient multi-error correcting Codes Data De-duplication: Rabin Fingerprints Bloom Filters Delta Encoding Long Term Archiving and Algebraic Signatures Consistent Hashing and Key Value Stores Adaptive, Large Scale Storage Systems

6 Introduction to Disk Arrays Outline Why Disk Arrays? MTTF, MTTR, MTTDL RAID 0, RAID 1, RAID 5 Multiple Disk Failures and RAID 6

7 Use Arrays of Small Disks Katz and Patterson asked in 1987: Can smaller disks be used to close gap in performance between disks and CPUs? Conventional: 4 disk designs Low End High End Disk Array: 1 disk design 3.5 Slide is based on talk from Prof. D. Patterson (Berkeley)

8 Exchange big disks by arrays of small disks (Example 1988) IBM 3390K IBM x70 Improvement Capacity 20 GB 320 MB 23 GB 1x Volume 97 ft ft 2 11 ft 2 9x Power Consumption 3 KW 11 W 1 KW 3x Bandwidth 15 MB/s 1.5 MB/s 120 MB/s 8x IO Rate 600 IOs/s 55 IOs/s 3900 IOs/s 6x MTTF 250 KHrs 50 KHrs?? Hrs Costs $250,000 $2,000 $150,000 1,6x Disk arrays have good performance properties nice MBs/volume and MBs/kW values... but what about reliablity? Slide is based on talk from Prof. D. Patterson (Berkeley)

9 MTTF, MTTR, MTTDL Mean Time to Failure (MTTF): Expected time until a disk fails. MTTF can be defined in terms of the expected value of the failure density function f: Mean Time to Repair (MTTR): Expected time from the failure of a disk until completing its recovery (if possible) Mean Time between Data Loss (MTTDL): Expected time from starting a storage system until the loss of data

10 Bathtub Curve MTTF calculated for normal operating period of a disk Nearly constant over a period of 3 4 years Failure rate before and afterwards significantly higher

11 RAID Parallel Disk Arrays are able to provide high bandwidth good properties concerning MB/Volume und MB/KW, but what about reliability? MTTF of n disks: Inside the example: MTTF 70 = 50,000 h / 70 = 700 h MTTF of the disk array decreases from six years to a single month Arrays (without redundancy) too unreliable to be useful!

12 Reliability / Availability Reliability Reliability is the ability of a system or component to perform its required functions under stated conditions for a specified period of time Based on Mean Time to Failure Availability Availability is the degree to which a system or component is operational and accessible when required for use Assumption: System cannot be accessed when under construction Slide is based on IEEE Standard Computer Dictionary: A Compilation of IEEE Standard Computer Glossaries andtorell / Avelar: Mean Time between Failure- Explanations and Standards

13 Redundant Array of Inexpensive Disks Files are striped across multiple disks Redundancy yields high data availability Availability: Service is still provided to user, even if some components have failed Disks will still fail Contents can be reconstructed from data redundantly stored in the array Capacity penalty to store redundant info Bandwidth penalty to update redundant info See D. A. Patterson, G. A. Gibson, R. H. Katz: A Case for Redundant Arrays of Inexpensive Disks (RAID)

14 RAID I (1989) SUN 4/280 workstation with 128 MByte DRAM Four SCSI-controller disk and Dedicated Disk-Striping Software RAID has become 27 bn $ industry 80% of all non PC-disks are sold inside RAID-systems Berkeley History, RAID-I Slide is based on talk from Prof. D. Patterson (Berkeley)

15 RAID Level RAID = Redundant Array of Independent Disks Standard RAID Levels 0: no Redundancy (JBOD) 1: Mirroring 10: Striped Mirrors 2: Hamming Codes/ECC (not used) 3: Byte-Interleaved Parity 4: Block-Interleaved Parity 5: Rotated Block-Interleaved Parity 6: e.g. Double Parity

16 RAID 0 RAID 0 stripes data over set of disks Size of each data block is several Kbyte Increase of bandwidth for big accesses or for many parallel, but small accesses RAID 0 does not include redundancy information No protection against single disk failures Legend: x-y means 0 1 block 2 x from 3 stripe y Location can be efficiently calculated for n disks Stripe address y = Address / n Disk number x = Address % n Logical address 12 will be mapped to stripe 2 and disk 2

17 RAID 1 Every Disk is fully mirrored 0 1 Very high availability can be achieved Bandwidth sacrifice on write: Logical write = two physical writes Reads can be optimized Legend: x-y means block x on disk y Most expensive RAID solution: 100% capacity overhead

18 Parity RAID Properties of previous RAID levels: Mirroring produces high overhead Striping does not include failure correction Function required with low capacity overhead good failure protection properties low computing costs Idea of RAID 3 and RAID 4 Use striping plus Parity computation Parity computed using XOR Example: Divide data block 1101 in 4 sub blocks plus one Parity block

19 Striping unit Block used to distribute data Stripe Terminology Set of striping units that share parity computation Parity block Block that keeps the parity of a stripe Same size as a striping unit

20 Small Write Problem Read performance of Parity RAID nearly as good as performance of RAID 0 but each small write to a single block x needs to update the parity block! Solution 1: Read all other data blocks except the changed one Calculate new parity Write the new data block and the parity block Overhead proportional to stripe size Solution 2: Use properties of XOR function: and We can read the ONE old data block x and the old parity block p We know the new data block x new Just calculate new parity block to:

21 RAID 5 RAID 4 produces bottleneck at the parity disk Each write access to an arbitrary block produces one write request at the parity disk RAID 4 does not scale concerning the stripe size Idea of RAID 5: Distribute the function of the parity disk over all disks Legend: x-y means block x from stripe y

22 Does redundancy help? Failure probability of 1-error correcting codes for n disks can be calculated to Failure probability during recovery depends on Time to recover the data MTTF of the remaining devices

23 Assumptions Error does not occur in wear-out phase MTTF (of one disk) is constant Exponential failure distribution leads to failure density function: is probability that an element fails; described in failures per unit of measurement It holds that

24 Probability of second Failure Probability of a second failure as integral over density function: Here: and therefore

25 Probability of second Failure Exponential function can be calculated as series with MTTR << MTTF: and MTTDL can be calculated as

26 Does redundancy help? Standard RAID schemes are able to increase MTTDL for n disks from without data protection to Drawbacks writing becomes slower complexity of implementation and administration significantly increases

27 Is this safe enough? Assumption for storage cluster environment: 1 PByte of data stored on 2000 computers Environment is grouped into 200 RAID 5 systems with 10 disks each MTTF of each computer (including disks) is 1000 days Recovery time of a computer is 1 day MTTDL = (1000) d 55d Protection against single disk failures not enough in large scale environments Example taken from Lustre Manual v1.6, August 2007