Apache Hadoop's Role in Your Big Data Architecture

Transcription

1 Apache Hadoop's Role in Your Big Data Architecture Chris Harris EMEA, Hortonworks Twi<er : cj_harris5 Hortonworks Inc Page 1

2 Agenda The Growth of Enterprise Data Hadoop Market Drivers Hortonworks an Overview The Future of Hadoop and Big Data Page 2

3 The Growth of Data in the Enterprise Data Explosion 1 Zettabyte (ZB) = 1 Billion TBs 15x growth rate of machine generated data by 2020 Source: IDC By 2015, organizations that build a modern information management system will outperform their peers financially by 20 percent. Gartner, Mark Beyer, Information Management in the 21st Century Page 3

4 Next Generation Data Architecture Drivers Business Drivers From reactive analytics to proactive customer interaction Find insights for competitive advantage & optimal returns Technical Drivers Data continues to grow exponentially Data is increasingly everywhere and in many formats Financial Drivers Cost of data systems, as % of IT spend, continues to grow Cost advantages of commodity hardware & open source

5 Market Transitioning into Early Majority relative % customers Innovators, technology enthusiasts Early adopters, visionaries The CHASM Early majority, pragmatists Late majority, conservatives Laggards, Skeptics time Customers want technology & performance Customers want solutions & convenience Source: Geoffrey Moore - Crossing the Chasm Page 5

6 Most Common NEW TYPES OF DATA 1. Sentiment Understand how your customers feel about your brand and products right now 2. Clickstream Capture and analyze website visitors data trails and optimize your website 3. Sensor/Machine Discover patterns in data streaming automatically from remote sensors and machines 4. Geographic Analyze location-based data to manage operations where they occur 5. Server Logs Research logs to diagnose process failures and prevent security breaches 6. Unstructured (txt, video, pictures, etc..) Understand patterns in files across millions of web pages, s, and documents Value + Keep existing data longer!

7 Apache Hadoop Enterprise Use Cases Vertical Use Case Data Type Financial Services Telecom Retail New Account Risk Screens Text, Server Logs Fraud Prevention Server Logs Trading Risk Server Logs Maximize Deposit Spread Text, Server Logs Insurance Underwriting Geographic, Sensor, Text Accelerate Loan Processing Text Call Detail Records (CDRs) Machine, Geographic Infrastructure Investment Machine, Server Logs Next Product to Buy (NPTB) Clickstream Real-time Bandwidth Allocation Server Logs, Text, Sentiment New Product Development Machine, Geographic 360 View of the Customer Clickstream, Text Analyze Brand Sentiment Sentiment Localized, Personalized Promotions Geographic Website Optimization Clickstream Optimal Store Layout Sensor Page 7

8 Call Detail Records (CDRs) Telecom Data: Machine, Geo Business Problem Telcos perform forensics on dropped calls and sound quality Call detail records flow in at a rate of millions per second High volume makes pattern recognition and root cause analysis difficult, which need to happen in real-time Delay causes attrition and harms servicing margins Solution HDP can ingest millions of CDRs per second HDP facilitates data retention and root cause analysis Continuously improve call quality, customer satisfaction and servicing margins Hortonworks Inc Page 8

9 Infrastructure Investment Telecom Data: Machine, Logs Business Problem Telecom marketing and capacity planning are coordinated Consumption of bandwidth and services can be out of sync with plans for new towers and transmission lines Mismatch between infrastructure investments and the actual return on investment puts revenue at risk Solution HDP helps telcos understand service consumption in a particular state, county or neighborhood Analyze Call Detail Records (CDRs) and network loads, more intelligently, over longer periods of time Plan infrastructure with more precision and less variability Hortonworks Inc Page 9

10 Assembly Line Quality Assurance Manufacturing Data: Sensor Business Problem High-tech manufacturing uses sensors to capture data at critical steps in the manufacturing process Sensor data helps diagnose errors with returned products Much data is discarded, because of high storage costs Lean margins mean small budgets for data analysis Solution HDP stores unstructured, streaming, dirty sensor data Manufacturers can proactively analyze more data, over a longer time, to detect subtle issues otherwise undetected Sensor data managed with HDP can help a manufacturer reduce warranty costs and earn a reputation for quality Hortonworks Inc Page 10

11 Supply Chain and Logistics Manufacturing Data: Sensor Business Problem Manufacturers need just-in-time availability of components Stock-outs cause harmful production delays Sensors and RFID tags reduce the cost of capturing more supply chain data, which needs storage and processing Solution HDP stores unstructured, streaming, dirty sensor data Manufacturers get lead time to make alternative arrangements for supply chain disruptions Prevent stock-outs, reduce supply chain costs and improve margins for the finished product Hortonworks Inc Page 11

12 Fraud Prevention Financial Services Data: Server Logs Business Problem Financial institutions are always at risk of fraud Fraudsters test bank systems for vulnerabilities This testing leaves subtle patterns often undetected by bank employees or law enforcement Fraud losses costs banks millions Solution HDP reduces the cost to detect fraudulent activity HDP stores more types of data for longer Analysis of data in the data lake exposes fraudulent patterns that would have gone undetected Hortonworks Inc Page 12

13 360 View of the Customer Retail Data: Clickstream, Text Business Problem Retailers interact with customers across multiple channels Customer interaction and purchase data is often siloed Few retailers can correlate customer purchases with marketing campaigns and online browsing behavior Merging data in relational databases is expensive Solution HDP gives retailers a 360 view of customer behavior Store data longer & track phases of the customer lifecycle Gain competitive advantage: increase sales, reduce supply chain expenses and retain the best customers Hortonworks Inc Page 13

14 Analyze Brand Sentiment Retail Data: Sentiment Business Problem Enterprises lack a reliable way to track their brand health It is difficult to analyze how advertising, competitor moves, product launches or news stories affect the brand Internal brand studies can be slow, expensive and flawed Solution HDP allows quick, unbiased brand sentiment snapshots Analyze sentiment from Twitter, Facebook, LinkedIn or industryspecific social media streams Retailers better understand customer perceptions, to align their communications, products and promotions with those perceptions and expectations Hortonworks Inc Page 14

15 Growth Pressures Existing Data Architectures APPLICATIONS Packaged Analy:c App Custom Analy:c App DEV & DATA TOOLS BUILD & TEST DATA SYSTEMS RDBMS EDW MPP TRADITIONAL REPOS OPERATIONAL TOOLS MANAGE & MONITOR DATA SOURCES Tradi:onal Sources OLTP, POS SYSTEMS (RDBMS, OLTP, OLAP) Data growth 8% annually Page 15

16 An Emerging Data Architecture APPLICATIONS Packaged Analy:c App Custom Analy:c App DEV & DATA TOOLS BUILD & TEST DATA SYSTEMS RDBMS EDW MPP TRADITIONAL REPOS ENTERPRISE HADOOP PLATFORM OPERATIONAL TOOLS MANAGE & MONITOR DATA SOURCES Tradi:onal Sources OLTP, POS SYSTEMS (RDBMS, OLTP, OLAP) New Sources (web logs, , sensors, social media) Data growth 85% annually Page 16

17 Hortonworks & Teradata Unified Data Architecture The right technology on the right analytical problems using best of breed technologies Aster Connector for Hadoop SQL- H Aster- Teradata Connector SQL- H Teradata Connector for Hadoop Viewpoint Integration Common management console for Aster, Teradata and Apache Hadoop TVI: Teradata Vital Infrastructure Proactive reliability, availability, and manageability support service Aster Connector for Hadoop SQL-H integration Teradata Connector for Hadoop Sqoop integration Pre-tuned HDFS and MapReduce parameters for Big Data workloads Page 17

18 Agenda The Growth of Enterprise Data Hadoop Market Drivers An Overview The Future of Hadoop and Big Data Page 18

19 A Brief History of Apache Hadoop Apache Project Established Yahoo! begins to Operate at scale Hortonworks Data Platform : Yahoo! creates team under E14 to work on Hadoop 2011: Hortonworks created to focus on Enterprise Hadoop Enterprise Hadoop Page 19

20 Leadership Starts at the Core Driving next generation Hadoop YARN, MapReduce2, HDFS2, High Availability, Disaster Recovery 420k+ lines authored since 2006 More than twice nearest contributor Deeply integrating w/ecosystem Enabling new deployment platforms (ex. Windows & Azure, Linux & VMware HA) Creating deeply engineered solutions (ex. Teradata big data appliance) All Apache, NO holdbacks 100% of code contributed to Apache Page 20

21 Agenda The Growth of Enterprise Data Hadoop Market Drivers Hortonworks an Overview The Future of Hadoop and Big Data Page 21

22 The 1 st Generation of Hadoop: Batch HADOOP 1.0 Built for Web-Scale Batch Apps Single App INTERACTIVE Single App ONLINE All other usage patterns must leverage that same infrastructure Single App BATCH Single App BATCH Single App BATCH Forces the creation of silos for managing mixed workloads HDFS HDFS HDFS

23 The Enterprise Requirement: Beyond Batch To become an enterprise viable data platform, customers have told us they want to store ALL DATA in one place and interact with it in MULTIPLE WAYS Simultaneously & with predictable levels of service BATCH INTERACTIVE ONLINE STREAMING GRAPH IN- MEMORY HPC MPI SEARCH HDFS (Redundant, Reliable Storage) Page 23

24 YARN: Taking Hadoop Beyond Batch Created to manage resource needs across all uses Ensures predictable performance & QoS for all apps Enables apps to run IN Hadoop rather than ON Key to leveraging all other common services of the Hadoop platform: security, data lifecycle management, etc. Applica:ons Run Na:vely IN Hadoop BATCH (MapReduce) INTERACTIVE (Tez) ONLINE (HBase) STREAMING (Storm, S4, ) GRAPH (Giraph) IN- MEMORY (Spark) HPC MPI (OpenMPI) OTHER (Search) (Weave ) YARN (Cluster Resource Management) HDFS2 (Redundant, Reliable Storage) Page 24

25 The Future of the Hadoop and Big Data The next generation data architecture evolving rapidly Store ALL data in a Hadoop data reservoir Push subsets of data to a final platform for processing Hadoop 2.0 takes Hadoop beyond Batch 2.0 YARN based architecture enabling mixed use workloads with enterprise resource management Enabling a new generation of applications at scale Based on new data types (sensor, sentiment, clickstream, etc.) or keeping existing types for much longer

26 Hortonworks Sandbox Hands on tutorials integrated into Sandbox HDP environment for evaluation Page 26

27 THANK YOU! Chris Harris Download Sandbox hortonworks.com/sandbox Page 27