A Tour of the Zoo the Hadoop Ecosystem Prafulla Wani

Transcription

1 A Tour of the Zoo the Hadoop Ecosystem Prafulla Wani Technical Architect - Big Data Syntel

2 Agenda Welcome to the Zoo! Evolution Timeline Traditional BI/DW Architecture Where Hadoop Fits In 2

3 Welcome to the Zoo! Zookeeper Pig Shark Jaql Hadoop Giraph Hama 3 I am sure you won t find a Shark in any other zoo 3

4 What is Hadoop? Hadoop is an open-source project overseen by the Apache Software Foundation Hadoop is an ecosystem, not a single product Originally based on papers published by Google in 2003 and 2004 Some of the projects in the ecosystem have been inspired based on whitepapers published by Google Google calls it: GFS MapReduce Sawzall BigTable Chubby Pregel Hadoop equivalent HDFS Hadoop MapReduce Hive, Pig HBase ZooKeeper Giraph 4

5 Evolution Timeline Started by Doug Cutting at Yahoo! in early 2006, and named after his kid s toy elephant Hadoop committers work at several different organizations Including Facebook, Yahoo!, LinkedIn, Twitter, Cloudera, Hortonworks Jaql Giraph 5

6 Traditional Data Strategy - BI/DW Architecture ERP Analytics Data Warehouse CRM Database ETL Process Data Marts Ad Hoc Reporting Files OLAP Analysis/BI ETL Tools DW / Marts BI Analytics Commercial Informatica Teradata Microstrategy SAS Oracle Data Integrator Oracle OBIEE TIBCO Spotfire IBM Datastage DB2, Netezza Cognos SPSS Microsoft SSIS SQL server Microsoft SSRS Open source Talend mysql Pentaho, Jaspersoft R, RapidMiner 6

7 How Hadoop fits in? ERP Analytics Data Warehouse CRM Database ETL Process Data Marts Ad Hoc Reporting Files OLAP Analysis/BI Hadoop can complement the existing DW environment as well replace some of the components in a traditional data architecture. 7

8 Data Storage Hadoop Distributed File System (HDFS) It s a file system, not a DBMS Allows storage of both structured and unstructured data Provides distributed, redundant storage for massive amounts of data on cheap, unreliable computers Hadoop 2.0 release (still beta) added some important features HDFS Federation High Availability HBase Distributed, versioned, column-oriented store on top of HDFS Provides an option of low-latency (OLTP) reads/writes along with support for batch-processing model of map-reduce Goal - To store tables with billion rows and million columns 8

9 Data Processing (ETL / Analytics) Extract / Load Source / Target is RDBMS - Sqoop Log collection and aggregation - Flume, Scribe, Chukwa Stream processing - S4, Storm (supports Transformation also) Transformation Map-reduce programming in Java or any other language or high level query languages like Pig, Hive etc. Workflow design and implementation using tools like Oozie, Azkaban etc. Iterative algorithms or in-memory cluster processing using Spark, Shark etc. Analytics Mahout - Scalable machine learning library with most of the algorithms implemented on top Apache Hadoop using map/reduce paradigm RHadoop Provides R packages to access data in HDFS & HBase and also to write map-reduce jobs in R 9

10 Common Industry Use Cases Use cases Solution Comments Cold Data Storage HDFS More cost-effective option compared to most appliances in the market Huge transactional volume Batch processing Log aggregation Real-time message/ stream processing Iterative algorithms / Inmemory processing Machine Learning/ Analytics Graph data storage/processing HBase MapReduce /Hive /Pig Flume, Scribe, Chukwa Storm, S4 Spark / Shark Mahout, RHadoop Giraph StumbleUpon created opentsdb to capture their infrastructure metrics data web-log collection on HDFS in near real-time Used by twitter for real-time tweet processing Predictive analytics, Log Mining Championed at Yahoo! 10

11 Proposed Big Data Roadmap Big Data integration Next steps Throw data open to business users for analysis and they will appreciate the power of new infrastructure 6 Big Data integration Next steps Implement advanced solutions Proof of Concept: Proof of Concept can be performed to demonstrate applicability of Hadoop to enhance DW Big Data integration Initial steps Move cold/warm data to Hive/HBase to reduce expenses on storage infrastructure Bring new data sources like web-logs, which was not possible with traditional storage solutions Big Data integration Next steps Identify the opportunities in ETL & Analytics space Move Hot data to Hadoop Perform real-time data integration using Storm/Spark 1 Kickoff - Assessment Study: Understand the business processes Understand organizational goals & current investments Understand the challenges and painpoints of current setup Hadoop Technology Stack HDFS, Hbase Hive, Pig, MapReduce Mahout, RHadoop 11

12 Thank You