Agenda. Big Data & Hadoop ViPR HDFS Pivotal Big Data Suite & ViPR HDFS ViON Customer Feedback #EMCVIPR

Transcription

1 1

2 Agenda Big Data & Hadoop ViPR HDFS Pivotal Big Data Suite & ViPR HDFS ViON Customer Feedback 2

3 A World of Connected Devices Need a new data management architecture for Internet of Things 21% the % of x86 servers deployed at SP PoPs by 2017 (up from 6.6% in 2010) 9 billion Connected devices by 2018 (up from 2 billion today) 50% Of enterprise data will be available for analytics within 5 years 53% YoY growth in scale-out file and object storage 3

4 Companies Are Using Big Data To Transform Big Data is producing game-changing results for companies in all industries University of Southern California Laboratory of Neuro Imaging Monsanto Sustainable agriculture company Using the world s largest repository of brain scans to: Link brain characteristics to neurological diseases Help teams develop treatments for diseases such as Parkinson s Huntington s and Alzheimer's Combine DNA data and brain scans to create individual treatment plans Crop yields have doubled twice in history through mechanization and science To feed 9.7 billion people double crop yields need to double again - Big Data is the key to this Processing data from more than 400M acres of farm land Multiple sources of data; soil, weather, satellite, location, field size/shape 4

5 The Big Data Potential Significant financial value across many verticals US Retail 60+% increase in net margin possible 0.5-1% annual productivity growth Global personal location data $100 billion+ revenue for service providers Up to $700 billion value to end users US Healthcare $300 billion value per year 0.7% annual productivity growth Source: Big Data: The Next Frontier for Innovation, Competition, and Productivity, McKinsey Global Institute Manufacturing Up to 50% decrease in product development, assembly costs Up to 7% reduction in working capital 5

6 Big Data Market Opportunity Create New Business Forbes Survey of board directors. Top business priorities for enterprises with more than $250M in revenue 1. Attracting new customers 2. Retaining and enhancing existing customers 3. Focusing on core competencies 4. Maintaining competitive advantage 5. Fostering innovation 6. Building new competencies 7. Moving into new products or service markets 8. Moving into new geographical markets 9. Increasing speed to market 6

7 Achieving These Outcomes Is Not Easy Getting the results you want requires changes to people, process, and technology access Analytics, Apps, and Data STORE EVERYTHING Structured and unstructured. Internal and external. Historic and real-time streamed data. The nature of data is changing and adaption is needed. ANALYZE ANYTHING Analytics is forward looking and predictive. Data analytics complements the business intelligence, rear-view mirror. BUILD THE RIGHT THING 3 rd platform applications take big data insights and turn them into business value. IT has a role in rapid development, deployment and scaling of new apps. 7

8 The Big Data Problem Velocity, volume and variety make value difficult to obtain Force rich unstructured data into a common schema using ETL operations New datasets generated before processing of previous one can complete Unable to ask new questions of transformed data 8

9 Hadoop for Analytics Descriptive Analytics Predictive Analytics Prescriptive Analytics Pig Hive MR YARN (Distributed Computation) HDFS ViPR HDFS (Hadoop Distributed File System) 9

10 Hadoop Processing Model Direct Attached Storage Model Namenode SPOF Manages HDFS Namespace Data Nodes Compute Storage 10

11 Processing Model Comparison DAS CAPEX investment in new storage infrastructure Dedicated hadoop cluster for a single distro as HDFS is part of the distro Shared Run workloads apart from analytics on the compute Run virtualized compute Scale compute & storage independently 11

12 Challenges with HDFS HDFS not enterprise-grade Requires three full copies of data, no Erasure coding No Geo-distribution, Limited DR, Multi-tenancy Inefficient for handling small files High availability still in progress No active-active failover even with the secondary NameNode DAS architecture not suitable for some customers Lack of enterprise data governance features 12

13 Hadoop Processing Model Shared Storage Model Enables common Data Lake for LOB application storage and analytics Scale compute & storage independently VMAX ECS VNX Isilon 3 rd Party Commodity Multiple distributions, clusters connect to the same data 13

14 EMC Data Lake Solutions Address These Challenges The EVP Data Lake solution lets you bring your analytics tools to your data. Data is shared between projects with centralized control and standardized analytics. INGEST STORE ANALYZE SURFACE ACT Capture data from a wide range of sources, traditional and new Store everything in one environment for cross data analysis Use advanced algorithms to discover new, predictive patterns Share insights with business domain experts Build data driven applications to meet business needs Our EVP Data Lake Solution is a platform that leverages technologies from Pivotal, VMware, and EMC. We also support choice - we believe that Data Lake Solutions can take many forms and can combine both Federation and third party technologies to deliver the required platform and functionality. 14

15 Scale-Out Data Lake Requirements Support multiple workloads and applications Traditional and emerging Multi-protocol support Ability to scale capacity and performance Enterprise data protection Meet security and compliance requirements Efficient and easy to manage to reduce costs Powerful data analytics capabilities 15

16 Geo-scale Big Data Analytics Best in class HDFS Storage on Commodity Hardware Geo-efficient, enterprisegrade storage Optimized for all small and large files Highly optimized for Pivotal 9-28% Cheaper Than Public Cloud Multi-protocol access (Object, Block, HDFS) Works with common distributions 16

17 Data Lake On ECS Appliance vsphere GEM XD DCA - Compute Only ViPR HDFS In-place analytics on top of mixed workload cloud scale storage A single Data Lake that will support multiple protocol access to avoid data movement for analytics Enterprise class storage that provides Erasure coding, Geo replication, DR Bring object storage capabilities to Hadoop to enable new scenarios (Eg: Metadata querying) Object Object Object 17

18 ViPR HDFS Overview Addresses limitations of off-the-shelf HDFS Brings HDFS to existing storage hardware Enables HDFS/Object/File scenarios Flexible software model allows for future co-location of compute and storage 18

19 HDFS Architecture Client RM / AsM NAME NODE SEC NAME NODE Node Manager MapReduce Task Node Manager MapReduce Task Node Manager MapReduce Task Data Store Data Store Data Store Commodity Compute & Storage 19

20 ViPR HDFS ARCHITECTURE Client RM/AsM NAME NODE SEC NAME NODE Node Manager MapReduce Task Node Manager MapReduce Task Node Manager MapReduce Task Client Client Client VNX Isilon 3 rd Party Commodity

21 ViPR HDFS ECS Deployment ViPR Controller ViPR VM Controller ViPR VM Controller VM ViPR Object/HDFS Data Read/ Write ViPR Object/HDFS Customer s Hadoop Compute Cluster 21

22 ViPR HDFS Value Proposition High Availability Built-In, No SPOF Avoids multiple copies of data Erasure Coding Support Geo-Distributed Across Sites Multi-tenancy, Metering, Chargeback Allows Byte-Range Updates Through S3 Interface ViPR Controller aids in Management & Monitoring 22

23 Geo-Distribution & DR For Hadoop SITE 1 Data distributed and replicated across multiple geographic sites Distributed index, meta-data & data Active/Active architecture Write to and read from any location Hadoop Compute Spin up on second site in the event of a site loss SITE 2 23

24 Enable Hadoop-as-a-Service (HaaS) Data Scientists Provision Account Compute Grid Elastic clusters deployed with a commercial Hadoop Distro pointing to ViPR HDFS for Storage Self-Service Portal S3 / Swift / ViPR HDFS VNX Isilon ECS Commodity If ViPR Object (S3 / Swift API) is used for LOB storage, customers can deploy multiple analytics clusters pointing to the same data SP Cloud 24

25 ENTERPRISE DATA CENTER ADVANCED SECURITY A Unique Federation Of Companies Delivering The Software-Defined Enterprise. Solutions & Choice SOFTWARE DEFINED DATA CENTER PLATFORM AS A SERVICE VIRTUAL WORKSPACE BUSINESS DATA LAKE SECURITY ANALYTICS BIG DATA SOLUTIONS PLATFORM AS A SERVICE AGILE APPLICATION DEVELOPMENT ENTERPRISE MOBILITY SOFTWARE-DEFINED DATA CENTER INFORMATION INFRASTRUCTURE CONVERGED INFRASTRUCTURE Partners vcloud Hybrid Service SERVICE PROVIDER 25

26