How vibration affects Hadoop storage performance

Transcription

1 How vibration affects Hadoop storage performance Gus Malek-Madani Founder, CTO/CEO And Amir Youssefi Co-Founder, VP of Software and Electronics Green Platform Corporation 2455 Old Middlefield Way #S Mountain View CA (650)

2 Presenters Gus Malek-Madani 25 years experience in vibration mitigation and carbon fiber design Multiple patents related to carbon fiber Founder and CEO of three companies 1. Green Platform Corp: Vibration management for disk storage 2. Composite Products: Improved high-end audio/video performance 3. Composite Rotor: Centrifuges and Rotors for Biotech Amir Youssefi Distributed Systems Architect with expertise in Big Data, Hadoop/MapReduce, Cloud Computing and Data Warehousing at Fortune/Inc 500 companies such as Yahoo, Apple and Plateau Led software development team for Grid Management System (GMS) operating on Yahoo Hadoop Clusters 2

3 Green Platform Corporation Providing enterprises with superior price/performance Hadoop platforms for Big Data 3

4 Why worry about vibration? Reduces signal/noise ratio Reduces performance Shortens product life Adds inefficiency Hard to resolve Vibration increases cost 4

5 Vibration Primer Mass MX" + CX' + KX = F(t) Acceleration Damping Stiffness Vibration Forces Displacement Velocity C and K of materials are frequency dependent 5

6 Vibration harmonics Disk Drives Cooling Power Distribution Increasing System Density UPS & Floor Fans Unique and Distinctive 6

7 Storage is the Bottleneck Order of magnitude slower Wastes hardware Wastes energy Wastes space 7

8 What s so cool about Carbon Fiber? Very strong, stiff and light Excellent damper of vibration Different properties in different directions Product can be highly customized Hundreds of variations Wide ranges of price/performance Product performance depends on fabrication method and quality A 6 µm diameter carbon filament (running from bottom left to top right) compared to a human hair. 8

9 Novel Technology Patented carbon fiber solution Dissipates vibration passively up to 1000X AVR-1000 Rack US Patent No. 8,240,490 Frictionless implementation Proven Real World results yielding 40%+ improvement in performance 9

10 Hadoop Primer Apache Hadoop is an open-source software for reliable, scalable, distributed computing. Hadoop Distributed File System (HDFS): A distributed file system that provides highthroughput access to application data. 10

11 Hadoop Primer How does it look like in practice? A single Anti Vibration Rack with 5 node Hadoop Cluster Sample Yahoo! Hadoop Cluster 11

12 Vibration on a commodity Node White box commodity node 12 2TB 3.5 7,200 rpm desktop HDD 12 core (24 threads) CPU Typical data center vibration IOzone benchmarking tool Base Vibration.25 grms Hz Write Vibration Re-write Mb/s penalty Mb/s Vibration penalty No % % 12

13 Vibration on a Hadoop cluster A 4-Node Hadoop cluster CDH3u3 (hadoop ) DFSio Rack Base vibration Max vibration on server g rms AVR vibration drop Average Read IO rate MB/s AVR effect on MB/s Metal Random.25 grms Hz AVR Random.25 grms Hz % 67 31% Metal rack on Shake Table 13

14 Terasort test under vibration 1 TB dataset Default 64 MB block size Only one job running on cluster. It gets better with concurrent jobs. Rack Base Max AVR Reduce AVR vibration vibration vibration time - benefit on server g drop min. Metal Random grms Hz AVR Random.25 grms Hz % 30 23% Anti-Vibration Rack On shake table 14

15 In Big Data, is slow OK? Population disease tracking and control Financial fraud detection Portfolio analysis Law enforcement and crime prevention/response Defense Cyber security Health care delivery Pharma research Operational planning and strategic decision-making Insurance claims and outcomes in health care 15

16 Challenges for the Storage Industry Existing approaches to vibration mitigation are no longer good enough Unstructured data relies on disk storage Commodity cluster architectures are very susceptible to vibration Customers are becoming educated about vibration and demanding effective vibration management 16

17 Summary Commodity hardware slowed by vibration MB/s is dropped under Hadoop Reduce times longer under Hadoop Green Platform rack restores hardware performance MB/s: 31% faster Reduce: 23% less time Big Data applications are disk drive intense Shared Infrastructure is very dense Needless hardware sprawl driven by vibration Low risk remedy has been tested 17