Solving Big Data Problems: Storage to the Rescue? PRESENTATION TITLE GOES HERE John Webster Evaluator Group

Transcription

1 Solving Big ata Problems: Storage to the Rescue? PRSTATI TITL GS HR John Webster valuator Group

2 Agenda Big ata Analytics Storage Maxims The Fundamental JB and AS Architecture verview of isk-based Alternatives What are the Advantages and isadvantages? The Solid State and In-memory Alternatives Summary and Q&A ote: References to specific vendors and products are used as real-world examples and do not imply an endorsement 2015 ata Storage Innovation Conference. Insert Your Company ame. All Rights Reserved. 4/1/

3 Big ata Storage Maxim #1 eliver storage performance at large scale and at low cost, and all at the same time (Think early stage Google, Facebook, Twitter) 2015 ata Storage Innovation Conference. Insert Your Company ame. All Rights Reserved. 4/1/

4 Big ata Storage Maxim #2 Minimize the distance between processing and data storage 2015 ata Storage Innovation Conference. Insert Your Company ame. All Rights Reserved. 4/1/

5 Big ata Storage Maxim #3 Big ata analytics is dominated by open source 2015 ata Storage Innovation Conference. Insert Your Company ame. All Rights Reserved. 4/1/

6 Big ata Storage Maxim #4 Big ata analytics software developers manage data at the clustered server level. Storage vendors manage data at the storage system level ata Storage Innovation Conference. Insert Your Company ame. All Rights Reserved. 4/1/

7 Shared othing, Asymmetrical istributed Computing etwork 1 Gb thernet B8GMR3 nly the thernet network is shared 1 Link 2 3 Link 4 5 Link 6 7 Link 8 Pwr Console Compute Commodity Servers Storage 6-12 disks in each server typically JB C T R L AS 1 2 AS AS AS AS 3 Scale to thousands of nodes n In Hadoop, Control = ame ode; ode 1,2 = ata ode 2015 ata Storage Innovation Conference. Insert Your Company ame. All Rights Reserved. 7

8 Apache Hadoop: A Platform for All Applications? ata Integration & Governance ata Workflow ata Lifecycle Falcon Real-time and Batch Ingest Flume Sqoop WebHFS FS ata Acces s Presentation & Application nable both existing and new applications to provide value to the organization Batc Map h Reduc e ata Management nvironmen t Scrip Pig t Appliance nlin Real- SQL Hbase Time e Hive Accumul Storm Metadata Management o HCatalog In- Memory Spark Multitenant Processing: YAR (Hadoop perating System) Storage: HFS (Hadoop istributed File System) Linux n Premise Virtualize Commodity HW Windows ther s Cloud/ Hosted perations mpower existing operations and security tools to manage Hadoop Security Authenticatio n Authorization Accountabilit y ata Protection Across Storage: HFS Resources: YAR Access: Hive, Pipeline: Falcon Cluster: Knox Source: Hortonworks perations Provision, Manage & Monitor Ambari Scheduling ozie 2015 ata Storage Innovation Conference. Insert Your Company ame. All Rights Reserved. 8

9 HFS as a Persistent Storage Advantages Storage performance at large scale and low cost Minimize distance between data and compute ata node failures tolerated pen Source isadvantages Hadoop ameode lacks active/active failover (i.e. it s a SPF) For data integrity and protection, HFS creates three full clone copies of data 3x the storage for each file slow and inefficient If all three copies are corrupted, you re still hosed (reload and start over) o storage tiering (now available in 2.3) Limited ways to respond to corporate security and data governance policies ata in/out processes can take longer than the actual query process Inability to dis-aggregate storage from compute so that the two can be scaled independently 2015 ata Storage Innovation Conference. Insert Your Company ame. All Rights Reserved. 9

10 Shared Storage as Primary Storage etwork B8GMR3 1 Link 2 3 Link 4 5 Link 6 7 Link 8 Pwr Console Compute C T R L n Storage SA or AS, but more commonly Scale-out AS 2015 ata Storage Innovation Conference. Insert Your Company ame. All Rights Reserved. 10 4/1/

11 Shared Storage as Secondary Storage etwork Compute B8GMR3 C T R L 1 Link 2 3 Link 4 5 Link 6 7 Link Pwr Console n Storage SA/AS/bject Storage 2015 ata Storage Innovation Conference. Insert Your Company ame. All Rights Reserved. 11 4/1/

12 Hadoop n Scale-out Storage Scale-out storage replaces node-level AS HFS implemented as over the wire protocol or CMI interface to underlying FS ameode SPF eliminated ecoupled storage and compute layers ata services, data protection, and R by storage-resident services xamples include MC Isilon, IBM lastic Storage, Ceph 2015 ata Storage Innovation Conference. Insert Your Company ame. All Rights Reserved. 4/1/

13 Shared Primary/Secondary Storage Advantages Addresses the enterprise storage management requirements ata protection/disaster recovery/business continuance ata governance and compliance igital records management and archiving isadvantages Additional cost Potential performance impact Using a vendor specific solution introduces proprietary data/storage management software 2015 ata Storage Innovation Conference. Insert Your Company ame. All Rights Reserved. 4/1/

14 What About SS? etwork 10+ Gb thernet B8GMR3 nly the thernet network is shared 1 Link 2 3 Link 4 5 Link 6 7 Link 8 Pwr Console Compute Commodity Servers Storage SS in/attached to each server C T R L AS In Hadoop, Control = ame ode; ode 1,2 = ata ode 2015 ata Storage Innovation Conference. Insert Your Company ame. All Rights Reserved. 1 2 AS AS AS AS 3 Scale to thousands of nodes n 14

15 What About SS? etwork B8GMR3 1 Link 2 3 Link 4 5 Link 6 7 Link 8 Pwr Console Compute C T R L n Storage Scale-out Flash Storage 2015 ata Storage Innovation Conference. Insert Your Company ame. All Rights Reserved. 15 4/1/

16 What About In-Memory Computing? Tachyon UC Berkeley Amp Lab project Reliable, memory-centric storage for Big ata Analytics clusters (i.e. memory as persistent data store across cluster nodes) ne in-memory data copy inside JVM, use operation lineage to re-compute data if failure Initial use in Apache Spark environments 2015 ata Storage Innovation Conference. Insert Your Company ame. All Rights Reserved. 4/1/

17 What About In-memory Computing? Apache Ignite In-memory data fabric istributed in-memory platform for computing and transacting on large-scale data sets in real-time rders of magnitude faster than possible with traditional disk-based or flash technologies. Tier -1 storage? riginated as GridGain ata Fabric 2015 ata Storage Innovation Conference. Insert Your Company ame. All Rights Reserved. 4/1/

18 Summary and Q&A The need for a longer-term, persistent storage layer is now recognized For Hadoop, HFS may or may not be that storage layer nterprise storage architects and administrators will be more directly involved in managing Big ata analytics storage over time ow is the time to research and understand the options 2015 ata Storage Innovation Conference. Insert Your Company ame. All Rights Reserved. 4/1/