Hadoop, the Data Lake, and a New World of Analytics

Transcription

1 Hadoop, the Data Lake, and a New World of Analytics Hortonworks. We do Hadoop. Spring 2014 Version 1.0 Page 1 Hortonworks Inc. 2014

2 Traditional Data Architecture Pressured 2.8 ZB in % from New Data Types 15x Machine Data by ZB by 2020 Data source: IDC SOURCES OLTP, ERP, CRM Documents, s Web Logs, Click Streams Social Networks Machine Generated Sensor Data Geolocation Data Page 2 Hortonworks Inc. 2014

3 Modern Data Architecture with Hadoop DATA SYSTEMS APPLICATIONS ROOMS Statistical Analysis RDBMS EDW MPP Repositories BI / Reporting, Ad Hoc Analysis Interactive Web & Mobile Apps Governance & Integra.on Enterprise Applications ENTERPRISE HADOOP Data Access Data Management Security Opera.ons DEV & DATA TOOLS Build & Test OPERATIONS TOOLS Provision, Manage & Monitor SOURCES OLTP, ERP, CRM Documents, s Web Logs, Click Streams Social Networks Machine Generated Sensor Data Geolocation Data Page 3 Hortonworks Inc. 2014

4 YARN Transforms Hadoop s Architecture Mul.- Use Data Pla>orm Store all data in one place, process in many ways Batch Interac.ve Real-.me Streaming YARN : Data Opera.ng System 1 Store any/all raw data sources and processed data over extended periods of time. n Enables deep insight across a large, broad, diverse set of data at efficient scale Page 4 Hortonworks Inc. 2014

5 Unlock New Approach to Insight Current Approach Apply schema on write Dependent on IT Augment with Hadoop Apply schema on read Support range of access paierns to data stored in HDFS: polymorphic access SQL Single Query Engine Repeatable Linear Process Hadoop Mul(ple Query Engines Itera(ve Process: Explore, Transform, Analyze Determine list of ques.ons Design solu.ons Collect structured data Ask ques.ons from list Detect addi.onal ques.ons Batch Interac.ve Real-.me Streaming Page 5 Hortonworks Inc. 2014

6 Schema-on-Write and Schema-on-Read Standard Digital Camera Zoom & focus first Capture limited set of pixels Crop around the focused area Lytro Lightfield Camera Capture entire lightfield Infinite zoom & focus Crop any captured areas Page 6 Hortonworks Inc. 2014

7 Leverage Commodity Compute + Storage Cloud Storage HADOOP NAS Engineered System MPP Fully-loaded Cost Per Raw TB of Data (Min Max Cost) Hadoop Enables Scalable Compute & Storage at a Compelling Cost Structure SAN $0 $20,000 $40,000 $60,000 $80,000 $180,000 Source: Juergen Urbanski, Board Member Big Data & Analytics, BITKOM Page 7 Hortonworks Inc. 2014

8 The Common Journey with Hadoop Scale A Modern Data Architecture What s a Data Lake? RDBMS MPP EDW Governance & Integration Data Access Data Management Security Operations A modern data architecture that provides a shared service for broad insight across a large, diverse set of data at efficient scale New Analytic Apps New types of data LOB-driven Scope Page 8 Hortonworks Inc. 2014

9 Unlock Value in New Types of Data 1. Social Understand how people are feeling and interac(ng right now 2. Clickstream Capture and analyze website visitors data trails and op(mize your website 3. Sensor/Machine Discover paierns in data streaming from remote sensors and machines 4. Geographic Analyze loca(on- based data to manage opera(ons where they occur 5. Server Logs Diagnose process failures and prevent security breaches 6. Unstructured (txt, video, pictures, etc.) Understand paierns in files across millions of web pages, s, and documents Value + Online archive Data that was once purged or moved to tape can be stored in Hadoop to discover long term trends and previously hidden value Page 9 Hortonworks Inc. 2014

10 Business Applications of Hadoop Sensor Server Logs Text Social Geographi c Machine Clickstrea m Structured Unstructur ed Financial Services New Account Risk Screens Trading Risk Insurance Underwriting Telecom Call Detail Records (CDR) Infrastructure Investment Real-time Bandwidth Allocation Retail 360 View of the Customer Localized, Personalized Promotions Website Optimization Page 10 Hortonworks Inc. 2014

11 Business Applications of Hadoop Sensor Server Logs Text Social Geographi c Machine Clickstrea m Structured Unstructur ed Manufacturing Supply Chain and Logistics Assembly Line Quality Assurance Crowd-sourced Quality Assurance Healthcare Use Genomic Data in Medial Trials Monitor Patient Vitals in Real-Time Pharmaceuticals Recruit & Retain Patients for Drug Trials Improve Prescription Adherence Oil & Gas Unify Exploration & Production Data Government Monitor Rig Safety in Real-Time ETL Offload in Response to Budgetary Pressures Sentiment Analysis for Gov t Programs Page 11 Hortonworks Inc. 2014

12 UC Irvine Health does Hadoop UC Irvine Medical Center is ranked among the na(on's best hospitals by U.S. News & World Report for the 12th year More than 400 specialty and primary care physicians Opened in bed medical facility Migrated 22 years of patient data across admissions and clinical Predictive analytics to reduce patient re-admittance Real time monitoring for rapid response to changes in vital signs For healthcare, we have never had the ability to do this. We have always taken the approach that we think we know what data elements are important. Now with all the data, we let the data determine what is important for predictive analysis. Charles Boicey More details at: Page 12 Hortonworks Inc. 2014

13 Some of the Companies Telling Their Stories Page 13 Hortonworks Inc. 2014

14 Enabling Hadoop for the Enterprise Journey Scale A Modern Data Architecture RDBMS 1 Capabili.es Ensure enterprise capabili(es are delivered in 100% open source to benefit all MPP EDW Governance & Integration Data Access Data Management Security Operations 2 Integration Interoperable with existing data center investments New Analytic Apps New types of data LOB-driven 3 Skills Leverage your exis(ng skills: development, analy(cs, opera(ons Scope Page 14 Hortonworks Inc. 2014

15 Our Mission: Enable your Modern Data Architecture by delivering Enterprise Apache Hadoop Open Leadership Drive innovation in the open exclusively via the Apache community-driven open source process Enterprise Rigor Engineer, test and certify Apache Hadoop with the enterprise in mind Ecosystem Endorsement Focus on deep integration with existing data center technologies and skills Headquartered in Palo Alto, CA; 300+ employees and growing Reseller Partners: Page 15 Hortonworks Inc. 2014

16 A Traditional Approach Under Pressure APPLICATIONS Business Analy.cs Custom Applica.ons Packaged Applica.ons OLTP, ERP, CRM Systems Unstructured documents, s 2.8 ZB in 2012 Server logs DATA SYSTEM RDBMS EDW MPP REPOSITORIES 85% from New Data Types 15x Machine Data by 2020 Sen(ment, Web Data 40 ZB by 2020 Source: IDC Sensor. Machine Data SOURCES Exis.ng Sources (CRM, ERP, Clickstream, Logs) Clickstream Geoloca(on Page 16 Hortonworks Inc. 2014

17 An Emerging Modern Data Architecture APPLICATIONS Business Analy.cs Custom Applica.ons Packaged Applica.ons DEV & DATA TOOLS Build & Test DATA SYSTEM RDBMS EDW MPP REPOSITORIES Governance & Integration Data Access Data Management Security Operations OPERATIONS TOOLS Provision, Manage & Monitor SOURCES OLTP, ERP, Documents, Web Logs, CRM Systems s Click Streams Social Networks Machine Generated Sensor Data Geoloca(on Data Page 17 Hortonworks Inc. 2014

18 Drivers of Hadoop Adoption SCALE New Analytic Apps New types of data LOB-driven SCOPE Page 18 Hortonworks Inc. 2014

19 Hadoop Value: New types of data Sentiment Understand how your customers feel about your brand and products right now Clickstream Capture and analyze website visitors data trails and optimize your website Sensors Discover patterns in data streaming automatically from remote sensors and machines Geographic Analyze locationbased data to manage operations where they occur Server Logs Research logs to diagnose process failures and prevent security breaches Unstructured Understand patterns in files across millions of web pages, s, and documents Page 19 Hortonworks Inc. 2014

20 Net New Analytic Applications Are Everywhere $ Financial Services Retail Telecom Manufacturing New Account Risk Screens Fraud Prevention Trading Risk Maximize Deposit Spread Insurance Underwriting Accelerate Loan Processing 360 View of the Customer Analyze Brand Sentiment Localized, Personalized Promotions Website Optimization Optimal Store Layout Call Detail Records (CDRs) Infrastructure Investment Next Product to Buy (NPTB) Real-time Bandwidth Allocation New Product Development Supplier Consolidation Supply Chain and Logistics Assembly Line Quality Assurance Proactive Maintenance Crowdsourced Quality Assurance Healthcare Utilities, Oil & Gas Public Sector Genomic data for medical trials Monitor patient vitals Reduce re-admittance rates Smart meter stream analysis Slow oil well decline curves Optimize lease bidding Analyze public sentiment Protect critical networks Prevent fraud and waste Store medical research data Recruit cohorts for pharmaceutical trials Compliance reporting Proactive equipment repair Seismic image processing Crowdsource reporting for repairs to infrastructure Fulfill open records requests Page 20 Hortonworks Inc. 2014

21 Drivers of Hadoop Adoption SCALE A Modern Data Architecture/Data Lake RDBMS MPP EDW Governance & Integration Data Access Data Management Security Operations New Analytic Apps New types of data LOB-driven SCOPE Page 21 Hortonworks Inc. 2014

22 Unlock A New Approach to Insight Current Reality Apply schema on write Dependent on IT Augment w/ Hadoop Apply schema on read Support range of access patterns to data stored in HDFS: polymorphic access SQL Single Query Engine Repeatable Linear Process Hadoop Mul(ple Query Engines Itera(ve Process: Explore, Transform, Analyze Determine list of ques.ons Design solu.ons Collect structured data Ask ques.ons from list Detect addi.onal ques.ons Batch Interac.ve Real-.me Streaming Page 22 Hortonworks Inc. 2014

23 Architectural Drivers of the MDA EDW Optimization Commodity Compute & Storage OPERATIONS 50% ANALYTICS 20% ETL PROCESS 30% OPERATIONS 50% ANALYTICS 50% Cloud Storage HADOOP NAS Fully-loaded Cost Per Raw TB of Data (Min Max Cost) Current Reality EDW at capacity: some usage from low value workloads Older data archived, unavailable for ongoing exploration Source data often discarded Augment w/ Hadoop Free up EDW resources from low value tasks Keep 100% of source data and historical data for ongoing exploration Mine data for value after loading it because of schema-on-read Engineered System MPP SAN $0 $20,000 $40,000 $60,000 $80,000 $180,000 Hadoop Enables Scalable Compute & Storage at a Compelling Cost Structure Page 23 Hortonworks Inc. 2014

24 Delivering the Core Capabilities of Enterprise Hadoop Page 24 Hortonworks Inc. 2014

25 Core Capabilities of Enterprise Hadoop PRESENTATION & APPLICATION Enable both existing and new application to provide value to the organization ENTERPRISE MGMT & SECURITY Empower existing operations and security tools to manage Hadoop GOVERNANCE & INTEGRATION DATA ACCESS SECURITY OPERATIONS Load data and manage according to policy Access your data simultaneously in multiple ways (batch, interactive, real-time) Store and process all of your Corporate Data Assets Provide layered approach to security through Authentication, Authorization, Accounting, and Data Protection Deploy and effectively manage the platform DATA MANAGEMENT Provide deployment choice across physical, virtual, cloud DEPLOYMENT OPTIONS Page 25 Hortonworks Inc. 2014

26 Core Capabilities of Enterprise Hadoop GOVERNANCE & INTEGRATION DATA ACCESS SECURITY OPERATIONS Data Workflow, Lifecycle & Governance Falcon Sqoop Flume NFS WebHDFS Batch Map Reduce Script Pig SQL Hive/Tez, HCatalog NoSQL HBase Accumulo Stream Storm YARN : Data Opera.ng System Search Solr 1 HDFS (Hadoop Distributed File System) Others In- Memory Analy(cs, ISV engines Authen.ca.on Authoriza.on Accoun.ng Data Protec.on Storage: HDFS Resources: YARN Access: Hive, Pipeline: Falcon Cluster: Knox Provision, Manage & Monitor Ambari Zookeeper Scheduling Oozie N DATA MANAGEMENT Page 26 Hortonworks Inc. 2014

27 HDP Delivers Enterprise Hadoop HDP 2.1 Hortonworks Data Platform GOVERNANCE & INTEGRATION DATA ACCESS SECURITY OPERATIONS Data Workflow, Lifecycle & Governance Falcon Sqoop Flume NFS WebHDFS Batch Map Reduce Script Pig SQL Hive/Tez, HCatalog NoSQL HBase Accumulo Stream Storm YARN : Data Opera.ng System Search Solr 1 HDFS (Hadoop Distributed File System) Others In- Memory Analy(cs, ISV engines Authen.ca.on Authoriza.on Accoun.ng Data Protec.on Storage: HDFS Resources: YARN Access: Hive, Pipeline: Falcon Cluster: Knox Provision, Manage & Monitor Ambari Zookeeper Scheduling Oozie N DATA MANAGEMENT Deployment Choice Linux Windows On-Premise Cloud Page 27 Hortonworks Inc. 2014

28 Driving Innovation Through the Apache Community Apache Project Committers PMC Members Hadoop Tez 10 4 Hive 12 3 HBase 8 3 Pig 6 5 Sqoop 1 0 Ambari Knox 6 2 Falcon 4 2 Oozie 2 2 Zookeeper 2 1 Flume 1 0 Accumulo 2 2 Storm 1 0 TOTAL Total Net Lines Contributed to Apache Hadoop 449, ,933 End Users 614,041 3 LinkedIn 3 IBM 5 Facebook 10 Others 7 Cloudera Total Number of Committers to Apache Hadoop 10 Yahoo 21 Page 28 Hortonworks Inc. 2014

29 The Partners You Rely On, Rely On Hortonworks for Hadoop Page 29 Hortonworks Inc. 2014

30 Broad Ecosystem Integration APPLICATIONS DATA SYSTEM SOURCES BusinessObjects BI RDBMS EDW MPP HANA Exis.ng Sources (CRM, ERP, Clickstream, Logs) Governance & Integration HDP 2.1 Data Access Data Management Security Operations Emerging Sources (Sensor, Sen.ment, Geo, Unstructured) DEV & DATA TOOLS OPERATIONAL TOOLS INFRASTRUCTURE DEPTH Hortonworks engages in deep engineered relationships with the leaders in the data center, such as Microsoft, Teradata, Redhat & SAP BREADTH Hundreds of partners work with us to certify their applications to work with Hadoop so they can extend big data to their users Page 30 Hortonworks Inc. 2014

31 The Partners You Rely on, Rely on Hortonworks HDInsight & HDP for Windows Only Hadoop Distribution for Windows Azure & Windows Server Native integration with SQL Server, Excel, and System Center Extends Hadoop to.net community Teradata Portfolio for Hadoop Seamless data access between Teradata and Hadoop (SQL-H) Simple management & monitoring with Viewpoint integration Flexible deployment options Instant Access + Infinite Scale SAP can assure their customers they are deploying an SAP HANA + Hadoop architecture fully supported by SAP Enables analytics apps (BOBJ) to interact with Hadoop Complete Portfolio for Hadoop UDA Diagram Appliances Page 31 Hortonworks Inc. 2014

32 Hortonworks Data Platform Community Driven Enterprise Apache Hadoop Page 32 Hortonworks Inc. 2014

33 HDP 2.0: Enterprise Hadoop Platform HDP 2.1 Hortonworks Data Platform Hortonworks Data Platform (HDP) GOVERNANCE & INTEGRATION Data Workflow, Lifecycle & Governance Falcon Sqoop Flume NFS WebHDFS Batch Map Reduce Script Pig SQL Hive/Tez, HCatalog DATA ACCESS NoSQL HBase Accumulo Stream Storm YARN : Data Opera.ng System DATA MANAGEMENT Search Solr 1 HDFS (Hadoop Distributed File System) Others In- Memory Analy(cs, ISV engines N SECURITY Authen.ca.on Authoriza.on Accoun.ng Data Protec.on Storage: HDFS Resources: YARN Access: Hive, Pipeline: Falcon Cluster: Knox OPERATIONS Provision, Manage & Monitor Ambari Zookeeper Scheduling Oozie The Only Completely Open Distribution for Apache Hadoop Fundamentally Versatile and Comprehensive enterprise capabilities Wholly Integrated for deep ecosystem interoperability Deployment Choice Linux Windows On-Premise Cloud Page 33 Hortonworks Inc. 2014

34 HDP 2.1: Reliable, Consistent & Current HDP certifies most recent & stable community innovation HDP April HDP 2.0 October 2013 HDP 1.3 May * Hadoop &YARN Tez Pig Hive & HCatalog HBase Storm Mahout Solr Sqoop Flume Falcon Ambari Oozie Zookeeper Knox Data Management Data Access Governance & Integration Operations Security Hortonworks Data Platform Page 34 Hortonworks Inc. 2014

35 Hortonworks Process for Enterprise Hadoop Upstream Community Projects Apache HBase Apache Hive Downstream Enterprise Product Certified at scale using the most advanced Hadoop test bed on the planet 1000 s of production nodes at Yahoo! Over 1500 unit & system tests Apache Pig Apache Falcon Test & Patch Apache Hadoop Design & Develop Release Fixed Issues Design & Develop Integrate & Test HDP 2.1 Package & Certify Apache Knox Apache Storm Stable Project Releases Distribute Virtuous cycle when development & fixed issues done upstream & stable project releases flow downstream Page 35 Hortonworks Inc. 2014

36 Working in the Community for the Enterprise Hortonworks Reseller, System Integrator, Technology, and Training Partners Support Organization Implementation Team Engineering Team Hortonworks De-risks your investment All support and implementation backed by the largest and most experienced Hadoop team on the planet A Two Way Street Gain access to the expertise and knowledge of the Hadoop community All issues or feature requests represented in the community Apache Community Leadership Page 36 Hortonworks Inc. 2014

37 Flexible Support Subscription Programs Our support subscriptions provide unlimited support across development, pilot, staging, & production Resources Available Web /Phone Response Time 5 contacts 24 hours x 7 days Yes Severity One: 1 hour Severity Two: 4 hours All Other: 1 Bus Day Services Provided Customer Support Portal Advanced Knowledgebase Diagnosis of Install, Config & Cluster Mgmt Issues Access to Upgrades, Updates and Patches Diagnosis of Performance Issues Diagnosis of Data Loading, Processing &Query Issues Application Development Support Remote Troubleshooting What s Supported table for specific components Page 37 Hortonworks Inc. 2014

38 Transferring Hadoop Expertise Apache Hadoop Training & Certification World class training programs Designed to help you learn fast Role-based hands on classes with 50% lab time Hadoop Certification demonstrates expertise in Development & Administration Programs designed to transfer knowledge Industry leading Hadoop Sandbox Free download Fastest way to learn Apache Hadoop Personal, portable Hadoop environment Page 38 Hortonworks Inc. 2014

39 Who is Using Hadoop & Hortonworks? Page 39 Hortonworks Inc HORTONWORKS CONFIDENTIAL & PROPRIETARY INFORMATION

40 Hortonworks: A Leader The Forrester Wave : Big Data Hadoop Solutions, Q Hortonworks loves and lives open source innovation Vision & Execution for Enterprise Hadoop. Hortonworks leads with a strong strategy and roadmap for open source innovation with Hadoop and a strong delivery of that innovation in Hortonworks Data Platform. World Class Support and Services. Hortonworks' Customer Support received a maximum score and was significantly higher than both Cloudera and MapR. Key Strategic Partnerships. Hortonworks unique strategic partnerships with Microsoft, SAP, Teradata and others are a key strength as part of its overall strategy of ecosystem partnership to accelerate Hadoop adoption in the enterprise. Page 40 Hortonworks Inc. 2014

41 The Value of Open Connect With the Community Avoid Vendor Lock In Partnerships that Matter Certified for the Enterprise Support from the Experts We employ a large number of Apache project committers & innovators so that you are represented in the open source community Hortonworks Data Platform is close to the open source trunk as possible and is developed 100% in the open so you are never locked in We work with partners to deeply integrate Hadoop with data center technologies so you can leverage existing skills and investments We engineer, test and certify the Hortonworks Data Platform at scale to ensure reliability and stability you require for enterprise use We provide the highest quality of support for deploying at scale. You are supported by hundreds of years of Hadoop experience Page 41 Hortonworks Inc. 2014