An Affordable Commodity Network Attached Storage Solution for Biological Research Environments.

Transcription

1 An Affordable Commodity Network Attached Storage Solution for Biological Research Environments. Ari E. Berman, Ph.D. Senior Systems Engineer Buck Institute for Research on Aging

2 Buck Institute Focus: Research on Aging, primarily Small Institute: 200 employees, 19 Faculty Total data footprint: 30TB Low IT budget: $700k/year We are the long tail

3 Scientific Computational Resources Biological science: >50% computational Laboratories produce 100 s of GB s of data High throughput methods Next generation sequencing (SOLiD: 11TB/6days) Mass Spectrometry Proteomics: 500GB/run Imaging: 10 s 100 s GB s Data security as important as research material security (i.e. 80C freezers) Supporting institution should provide solutions

4 Institutional Resource Problems Computational power for larger projects High volume, high speed, high availability, centralized storage Expensive! 80TB Isilon system: ~$140,000 Cloud resources: charge per GB and bottleneck is internet speeds (11TB: 10.8 days at 100Mb and 100% utilization) Home grown RAID systems high failure rate require constant attention by engineers often not enough replication to offset failures

5 Institutional Resource Problems Systems require significant expertise to maintain Institution invests more money in replication to offset failures Larger data > more creative backup solutions Disaster recovery difficult and time intensive No real alternative to hard drives yet SSD getting there, but too expensive for less volatile/high capacity systems The core of the problem: hard drives, they fail!

6 The Problem: hard drives Need Enterpriseclass hard drives Desktop class hard drives: maximum life of 700K hours (10% uptime) Enterprise: faster, higher MTBF, better error checking, seemingly lower failure rate

7 Problems with RAID Generally expensive to achieve useful redundancy (RAID 1: 2x number of disks for useable space) RAID operates through whole disk redundancy, when a disk fails, has to rebuild entire disk (time intensive) RAID will mark disks as failed due to corrupted bits on the drive, not physical problems Bits on disks are corrupted all the time from cosmic radiation Available file systems (ext4, xfs) have size limitations

8 Another Option: ZFS ZFS Zettabyte File System MB = 2 6, GB = 2 9, TB = 2 12, PB = 2 15, ZB = 2 70 ZFS can store up to 256 quadrillion ZB Advantages Dynamically expandable to more disks Copy on write: instant snapshots Continuous integrity checking and repair Redundancy, similar to RAID: bit level vs. drive level Integrity checking removes drive failures due to bit level errors: way fewer failures

9 Another Option: ZFS Disadvantages Not supported well on all OS s: mainly Solaris and FreeBSD (Linux implementation not so great) Requires relatively large learning curve, very different than standard filesystems Basically a mixture of concepts from RAID and lvm

10 What did we do? Commercial system too expensive Small RAID arrays too error prone Wanted to use ZFS for bit level parity Given our data growth rate, needed 250TB system for next 3 5 years Spec d out a system from Supermicro It was both high performance and affordable

11 System Specifications Supermicro 847E16 R1400LPB & 847E16 RJBOD1 126 Seagate TB SAS 6Gb/s 7.2k RPM Dual Intel Xeon CPU 2.40GHz 16 cores 64GB SATADOM for OS OCZ Vertex3 SSD 2x LSI SAS i HBA 24GB RAM 10Gb fiber ethernet

12 FreeNAS Based on FreeBSD 8.2 (NanoBSD) Supports UFS and ZFS out of the box Web based graphical interface NFSv4 ACL s: important for complex sharing

13 Final System Running FreeBSD (FreeNAS, with custom modifications), ZFS 126 2TB disks: 207TB of usable disk space Can handle up to nine simultaneous drive failures

14 ZPool Layout

15 Bigrock Has the capacity to share data natively with Windows, Mac, and Unix machines (86 shares currently) True centralized storage for the institute Current uptime: 82 days. +5TB in that time

16 Performance Sequential Output Sequential Input Random Seeks Per Chr Block Rewrite Per Chr Block Seeks K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU Sequential Output Create Sequential Random Input Create Random Seeks Per Create Chr Block Read Rewrite Delete Per Create Chr Block Read Delete Seeks K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU Sequential Create Random Create Create Read Delete Create Read Delete /sec %CPU /sec %CPU /sec %CPU /sec %CPU /sec %CPU /sec %CPU Performance benchmarked with bonnie++ v1.96 File size: 48GB (2x RAM) Compared against SAS 6Gb fibrechannel RAID 60 (XFS)

17 How Affordable is it? Short answer: very Entire system was built for $40,000 Hard drives were most substantial cost: $22,300 ($177/ea, Aug 2011) Hard drives cost more now: $250/ea Can increase speed by using 2.5 drives Alternate configurations: fibrechannel, infiniband, etc.

18 Why Doesn t Everyone Do This? Not for the faint of heart! High degree of UNIX engineering/architect experience required Basic configuration worked out, works for 99% of users Still working out the kinks Access Control for subdirectories command line only Older platforms needed server modifications But: 100% uptime since release

19 Backups/Disaster Recovery Replication HA NAS Near line Snapshot Smoking hole in theground solution Mostly disks Building offsite system with old arrays and servers Tape Offload LTO 5 Long term Tape Storage Offsite Snapshot

20 The Bottom Line It works! People are happy with it It s fast and reliable Now we need to make it simpler to administer

21 Acknowledgements Darrain Waters Kevin Kennedy SuperMicro Buck Research Community

22 Thank You! Questions?

23 RAID Levels Speed vs. Redundancy More redundancy = safer data Larger = more disk space and speed: parallelism RAID 50 (5+0)

24 A Solution: RAID Redundant Array of Independent Disks (RAID) Many hard drives working together to form a single volume: can be very large

25 FreeNAS Interface

26 RAID Solves Some Problems RAID 5 allows for one disk failure, while maintaining data RAID 6 allows for two failures RAID 1 allows for ½ of all disks to fail while still serving data RAID 50 one failure per stripe RAID 60 two failures per stripe With a potential disk failure rate of 13%, data security requires redundancy

27 Hard Drive Comparison Revision 1.0 April, 2008 Enterprise Platforms and Services Division Marketing, Intel