Big Data Realities Hadoop in the Enterprise Architecture

Transcription

1 Big Data Realities Hadoop in the Enterprise Architecture Paul Phillips Director, EMEA, Hortonworks +44 (0) Hortonworks Inc Page 1

2 Agenda The Growth of Enterprise Data Hadoop New Data Architecture Hortonworks an Overview The Future of Hadoop and Big Data Page 2

3 Agenda The Growth of Enterprise Data Hadoop New Data Architecture Hortonworks an Overview The Future of Hadoop and Big Data Page 3

4 The Growth of Data in the Enterprise Data Explosion 1 Zettabyte (ZB) = 1 Billion TBs By 2015, organizations that build a modern information management system will outperform their peers financially by 20 percent. Gartner, Mark Beyer, Information Management in the 21st Century 15x growth rate of machine generated data by 2020 Source: IDC Page 4

5 Big Data: Organizational Game Changer Petabytes Terabytes Gigabytes Megabytes BIG DATA WEB CRM ERP Purchase detail Purchase record Payment record Web logs A/B testing Segmentation Mobile Web Sentiment SMS/MMS User Click Stream Speech to Text Behavioral Targeting Customer Touches Support Contacts Offer details Transactions + Interactions + Observations = BIG DATA Social Interactions & Feeds Spatial & GPS Coordinates Click Streams Search Marketing Affiliate Networks Dynamic Funnels Offer history Sensors / RFID / Devices Business Data Feeds External Demographics User Generated Content HD Video, Audio, Images Product/Service Logs Increasing Data Variety and Complexity Page 5

6 Agenda The Growth of Enterprise Data Hadoop New Data Architecture Hortonworks an Overview Model and Strategy The Future of Hadoop and Big Data Page 6

7 Growth Pressures Existing Data Architectures APPLICATIO NS Packaged Analy9c App Custom Analy9c App DEV & DATA TOOLS BUILD & TEST DATA SYSTEMS RDBMS EDW MPP OPERATIONAL TOOLS MANAGE & MONITOR DATA SOURCES Tradi9onal Sources (RDBMS, OLTP, OLAP) Data growth 8% annually Page 7

8 An Emerging Data Architecture APPLICATIO NS Packaged Analy9c App Custom Analy9c App DEV & DATA TOOLS BUILD & TEST DATA SYSTEMS RDBMS EDW MPP ENTERPRISE HADOOP PLATFORM OPERATIONAL TOOLS MANAGE & MONITOR DATA SOURCES Tradi9onal Sources (RDBMS, OLTP, OLAP) New Sources (web logs, , sensors, social media) Data growth 85% annually Page 8

9 Deutsche Telekom s Perspective Hadoop! Coming soon to an enterprise data warehouse near you. I predict that by 2015, fully 80 percent of all new data will enter the typical enterprise on a Hadoop cluster first, making it the de facto enterprise-wide landing zone for large amounts of data. Hadoop is open source and runs on industry-standard hardware, which means it is at least 10 times more economical than conventional data warehouse solutions. Juergen Urbanski, VP Big Data Architectures & Technologies T-Systems, the enterprise IT division of Deutsche Telekom

10 6 Key DATA TYPES as Application Catalysts 1. Sentiment Understand how your customers feel about your brand, products and services right now 2. Clickstream Capture and analyze site visitors data trails and optimize your website 3. Sensor/Machine Track your machine usage, predict failures and schedule maintenance 4. Geographic Analyze location-based data to manage operations where they occur 5. System Logs Research logs to diagnose system failures and prevent future errors 6. Unstructured Text Read and understand web pages, books, PDFs faster than they are written Page 10

11 Agenda The Growth of Enterprise Data Hadoop New Data Architecture Hortonworks an Overview The Future of Hadoop and Big Data Page 11

12 A little history it s 2005

13 A Brief History of Apache Hadoop 2005: Yahoo! creates team under E14 to work on Hadoop Apache Project Established Yahoo! begins to Operate at scale Hortonworks Data Platform : Hortonworks created to focus on Enterprise Hadoop 2013 Enterprise Hadoop Page 13

14 Hortonworks Overview Headquarters: Palo Alto, CA Employees: 230+ and growing Investors: Benchmark, Index, Yahoo MISSION Enable Apache Hadoop to become an enterprise viable data platform Develop Distribute Support We employ the core architects, builders and operators of Apache Hadoop Everything contributed back Endorsed by Strategic Partners We distribute the only 100% Open Source Enterprise Hadoop Distribution: Hortonworks Data Platform Enterprise level support to allow enterprises to deploy at scale We enable the ecosystem to work better with Hadoop Page 14

15 Leadership Starts at the Core Driving next generation Hadoop YARN, MapReduce2, HDFS2, High Availability, Disaster Recovery 420k+ lines authored since 2006 More than twice nearest contributor Deeply integrating w/ecosystem Enabling new deployment platforms (ex. Windows & Azure, Linux & VMware HA) Creating deeply engineered solutions (ex. Teradata big data appliance) All Apache, NO holdbacks 100% of code contributed to Apache Page 15

16 Agenda The Growth of Enterprise Data Hadoop New Data Architecture Hortonworks an Overview The Future of Hadoop and Big Data Page 16

17 HDP: Enterprise Hadoop Distribution OPERATIONAL SERVICES AMBARI FALCON* OOZIE HADOOP CORE PLATFORM SERVICES FLUME SQOOP LOAD & EXTRACT NFS WebHDFS KNOX* DATA SERVICES PIG HIVE & HCATALOG MAP REDUCE* TEZ* YARN* HDFS Enterprise Readiness High Availability, Disaster Recovery, Rolling Upgrades, Security and Snapshots HBASE OTHER HORTONWORKS DATA PLATFORM (HDP) Hortonworks Data Platform (HDP) Enterprise Hadoop The ONLY 100% open source and complete distribution Enterprise grade, proven and tested at scale Ecosystem endorsed to ensure interoperability Page 17

18 The 1 st Generation of Hadoop: Batch HADOOP 1.0 Built for Web-Scale Batch Apps Single App BATCH HDFS Single App INTERACTIVE Single App BATCH HDFS Single App ONLINE Single App BATCH HDFS All other usage patterns must leverage that same infrastructure Forces the creation of silos for managing mixed workloads

19 The Enterprise Requirement: Beyond Batch To become an enterprise viable data platform, customers have told us they want to store ALL DATA in one place and interact with it in MULTIPLE WAYS Simultaneously & with predictable levels of service BATCH INTERACTIVE ONLINE STREAMING GRAPH IN- MEMORY HPC MPI SEARCH HDFS (Redundant, Reliable Storage) Page 19

20 YARN: Taking Hadoop Beyond Batch Created to manage resource needs across all uses Ensures predictable performance & QoS for all apps Enables apps to run IN Hadoop rather than ON Key to leveraging all other common services of the Hadoop platform: security, data lifecycle management, etc. BATCH (MapReduce) Applica9ons Run Na9vely IN Hadoop INTERACTIVE (Tez) ONLINE (HBase) STREAMING (Storm, S4, ) GRAPH (Giraph) IN- MEMORY (Spark) YARN (Cluster Resource Management) HDFS2 (Redundant, Reliable Storage) HPC MPI (OpenMPI) OTHER (Search) (Weave ) Page 20

21 Market Transitioning into Early Majority relative % customers Innovators, technology enthusiasts Early adopters, visionaries The CHASM Early majority, pragmatists Late majority, conservatives Laggards, Skeptics Customers want technology & performance Customers want solutions & convenience time Source: Geoffrey Moore - Crossing the Chasm Page 21

22 The Future of the Hadoop and Big Data The next generation data architecture evolving rapidly Store ALL data in a Hadoop data reservoir Push subsets of data to a final platform for processing Hadoop 2.0 takes Hadoop beyond Batch 2.0 YARN based architecture enabling mixed use workloads with enterprise resource management Enabling a new generation of applications at scale Based on new data types (sensor, sentiment, clickstream, etc.) or keeping existing types for much longer

23 Hortonworks Sandbox Hands on tutorials integrated into Sandbox HDP environment for evaluation Page 23

24 THANK YOU! Paul Phillips Director, EMEA +44(0) Download Sandbox hortonworks.com/sandbox Page 24