Modern Data Architecture for Predictive Analytics

Transcription

1 Modern Data Architecture for Predictive Analytics David Smith VP Marketing and Community - Revolution Analytics John Kreisa VP Strategic Marketing- Hortonworks Hortonworks Inc Page 1

2 Your Presenters David Smith VP Marketing and Community at Revolution Analytics Data Scientist, Blogger and co-author of An Introduction to R John Kreisa (@marked_man) VP Strategic Marketing, Hortonworks Over 20 years in data management as a developer and a marketer Avid camper Hortonworks Inc Page 2

3 Today s Topics Introduction Drivers for the Modern Data Architecture (MDA) Apache Hadoop in the MDA R s role in the MDA Q&A Hortonworks Inc Page 3

4 Poll #1: What stage are you at looking in Hadoop? Research Evaluation Trial Haven t started research Hortonworks Inc Page 4

5 SOURCES DATA SYSTEM APPLICATIONS Existing Data Architecture Business Analytics Custom Applications Packaged Applications DEV & DATA TOOLS BUILD & TEST OPERATIONAL TOOLS RDBMS EDW MPP REPOSITORIES MANAGE & MONITOR Existing Sources (CRM, ERP, Clickstream, Logs) Hortonworks Inc Page 5

6 SOURCES DATA SYSTEM APPLICATIONS Existing Data Architecture Business Analytics Custom Applications Packaged Applications 2.8 ZB in 2012 RDBMS EDW MPP REPOSITORIES 85% from New Data Types 15x Machine Data by ZB by 2020 Source: IDC Existing Sources (CRM, ERP, Clickstream, Logs) Hortonworks Inc Page 6

7 SOURCES DATA SYSTEM APPLICATIONS Modern Data Architecture Enabled Business Analytics Custom Applications Packaged Applications DEV & DATA TOOLS BUILD & TEST OPERATIONAL TOOLS RDBMS EDW MPP REPOSITORIES MANAGE & MONITOR Existing Sources (CRM, ERP, Clickstream, Logs) Emerging Sources (Sensor, Sentiment, Geo, Unstructured) Hortonworks Inc Confidential Page 7

8 Hadoop Powers Modern Data Architecture Hadoop Cluster compute & storage compute & storage Hadoop clusters provide scale-out storage and distributed data processing on commodity hardware Apache Hadoop is an open source project governed by the Apache Software Foundation (ASF) that allows you to gain insight from massive amounts of structured and unstructured data quickly and without significant investment. Hortonworks Inc Confidential Page 8

9 Drivers for Hadoop Adoption Modern Data Architecture Hadoop has a central role in next generation data architectures while integrating with existing data systems Driving Efficiency Business Applications Use Hadoop to extract insights that enable new customer value and competitive edge Driving Opportunity Big Data Sets Existing Traditional Server log Clickstream Emerging Sentiment/Social Machine/Sensor Geo-locations Hortonworks Inc Confidential

10 Opportunity in types of data 1. Sentiment Understand how your customers feel about your brand and products right now 2. Clickstream Capture and analyze website visitors data trails and optimize your website 3. Sensor/Machine Discover patterns in data streaming automatically from remote sensors and machines 4. Geographic Analyze location-based data to manage operations where they occur Value 5. Server Logs Research logs to diagnose process failures and prevent security breaches 6. Unstructured (txt, video, pictures, etc..) Understand patterns in files across millions of web pages, s, and documents Hortonworks Inc Confidential Page 10

11 SOURCES DATA SYSTEM APPLICATIONS Efficiency in the Modern Data Architecture Business Analytics Custom Applications Packaged Applications Drive efficiency via modern data architecture Store data once and access it in many ways RDBMS EDW MPP REPOSITORIES Often referred to a data lake or data repository Infrastructure platform driven Existing Sources (CRM, ERP, Clickstream, Logs) Emerging Sources (Sensor, Sentiment, Geo, Unstructured) IT-oriented, TCO based Hortonworks Inc Confidential Page 11

12 SOURCES DATA SYSTEM APPLICATIONS Engineered for Interoperability BusinessObjects BI DEV & DATA TOOLS OPERATIONAL TOOLS RDBMS EDW MPP HANA INFRASTRUCTURE Existing Sources (CRM, ERP, Clickstream, Logs) Emerging Sources (Sensor, Sentiment, Geo, Unstructured) Hortonworks Inc Confidential Page 12

13 Requirements for Hadoop Adoption Requirements for Hadoop s Role in the Modern Data Architecture Integrated Interoperable with existing data center investments Skills Leverage your existing skills: development, operations, analytics Key Services Platform, operational and data services essential for the enterprise Hortonworks Inc Confidential Page 13

14 SOURCES DATA SYSTEM APPLICATIONS Revolution R Enterprise Architecture Business Analytics Custom Applications Packaged Applications DEV & DATA TOOLS BUILD & TEST OPERATIONAL TOOLS RDBMS EDW MPP REPOSITORIES MANAGE & MONITOR Existing Sources (CRM, ERP, Clickstream, Logs) Emerging Sources (Sensor, Sentiment, Geo, Unstructured) = Revolution R Enterprise Hortonworks Inc Confidential Page 14

15 Today s Topics Introduction Drivers for the Modern Data Architecture (MDA) Apache Hadoop s role in the MDA R s role in the MDA Q&A Hortonworks Inc Page 15

16 Poll #2: Which of the following best describes your use of R and Hadoop? We have R+ Hadoop in Production We have testing R+ Hadoop We have started to investigate but nothing is implemented No current plans Hortonworks Inc Page 16

17 What is the Open Source R Project? Revolution Confidential The R Language: Object-Oriented Language for Stats, Math and Data Science Comprehensive data visualization and statistical modeling capabilities The R Community: 2M+ Users with the Skill to Tackle Big Data Statistical and Numerical Analysis and Machine Learning Projects New graduates with data skills learn R The R Ecosystem: Freely Available Algorithms in CRAN Specialized methods for finance, economics, genomics, linguistics, and every data-driven domain 17

18 R is open source and drives analytic innovation but has Revolution Confidential some limitations for Enterprises Memory Bound Big Data Bigger data sizes Single Threaded Community Support Innovative packages Exponential growth Scale out, parallel processing, high speed Commercial production support Combines with open source R packages where needed Speed of analysis Production support Innovation and scale

19 Revolution R Enterprise Revolution Confidential Revolution R Enterprise is the only commercial big data analytics platform based on open source R statistical computing language High Performance Analytics Big Data Analytics Cross-Platform Easier Build & Deploy Enterprise-Ready 19

20 Modern Data Architecture Extract and Analyze Ad-hoc Data Distillation Exploratory Data Analysis / Data Visualization Model Development SOURCE DATA INTERACTIVE Query/Visualization/ Reporting/Analytical Tools and Apps DBs AMBARI HIVE Server2 Fil Fil es es Files DATA REFINEMENT PIG HIVE CUSTOM ANALYTICAL rhadoop Analytical Tools JMS Queue s - Sensor Logs - Clickstream - Flat Files - Unstructured - Sentiment - Customer - Inventory REST HTTP STREAM LOAD SQOOP FLUME NFS WebHDFS MAPREDUCE YARN HDFS STRUCTURE HCATALOG (metadata services) LOAD SQOOP/Hive Web HDFS Data Sources CSV DATABASES

21 The Data Scientist s Big Data Toolkit Revolution Confidential R Data Step Descriptive Statistics Statistical Tests Sampling Simulation Data Visualization Machine Learning Predictive Models 21

22 Parallel External-Memory Algorithms CPU CPU CPU SMP SERVER 22

23 Parallel External-Memory Algorithms HADOOP NODE HADOOP NODE HADOOP NODE HADOOP CLUSTER 23

24 Modern Data Architecture with RRE7 Revolution Confidential In-Hadoop Predictive Analytics Production Data Distillation (e.g. Semantic Analysis) Production Model Processing / Re-Estimation Production Model Scoring SOURCE DATA INTERACTIVE Query/Visualization/ Reporting/Analytical Tools and Apps DBs AMBARI HIVE Server2 Fil Fil es es Files JMS Queue s - Sensor Logs - Clickstream - Flat Files - Unstructured - Sentiment - Customer - Inventory REST HTTP STREAM LOAD SQOOP FLUME NFS WebHDFS PIG DATA REFINEMENT HIVE MAPREDUCE YARN HDFS DISTILLED DATA FILES CUSTOM ANALYTICAL Revolution R Enterprise STRUCTURE HCATALOG (metadata services) LOAD SQOOP/Hive Web HDFS Analytical Tools Data Sources CSV DATABASES

25 Hadoop As An R Engine Revolution Confidential Hadoop Use Revolution R Enterprise PEMAs in Hadoop No need to change existing R code Simple R programming No need to Think In MapReduce Eliminate data movement to slash latencies Use Hadoop nodes as parallel R computation engines 25

26 Requirements for Hadoop Adoption Requirements for Hadoop s Role in the Modern Data Architecture Integrated Interoperable with existing data center investments Skills Leverage your existing skills: development, operations, analytics Key Services Platform, operational and data services essential for the enterprise Hortonworks Inc Page 26

27 Poll #3: Which of the following would you most like to accomplish with R + Hadoop? Build a model to be put in product in Hadoop Build a model to be put in product elsewhere Create new data from Hadoop to supplement an existing analytics process Something else Hortonworks Inc Page 27

28 Next Steps: More about Revolution Analytics and Hadoop Get started on Hadoop with Hortonworks Sandbox Hortonworks Inc Page 28