All You Wanted to Know About Big Data Projects Chida Jan 2014

Transcription

1 All You Wanted to Know About Big Data Projects Chida Jan

2 WHAT WE DISCUSS HERE AGENDA > > > > > > Need History Open Source - Hadoop BigData EcoSystem Use Cases Managing BigData Projects 2

3 Data CreaIon According to IBM,we create 2.5 quinillion (Million Trillions!) bytes of data every day. 90% of the data we have has been created in the past two years Structured, Semi- Structured, Unstructured How to Store this Data How To Process this Data How to make Any sense out of this data? At What Cost With What Return On Investment (ROI) h[p://schoollibrarybeyondsurvival.files.wordpress.com/2012/06/ dataneversleeps_4fd61ee2eda5a.jpg 3

4 BigData - Background Search Engine Problem Google Three White Papers - GFS, MapReduce, BigTable MapReduce: Simplified Data Processing on Large Clusters by Jeffrey Dean and Sanjay Ghemawat, 2004 from Google Yahoo! HDFS, MapReduce, Hbase Open Source - Hadoop

5 Existing Data Architecture APPLICATIONS Business Analy;cs Custom Applica;ons Enterprise Applica;ons DEV & DATA TOOLS BUILD & TEST DATA SYSTEMS RDBMS EDW MPP TRADITIONAL REPOS OPERATIONAL TOOLS MANAGE & MONITOR DATA SOURCES Tradi;onal Sources OLTP, POS SYSTEMS (RDBMS, OLTP, OLAP)

6 Next-Generation Data Architecture APPLICATIONS Business Analy;cs Custom Applica;ons Enterprise Applica;ons DEV & DATA TOOLS BUILD & TEST DATA SYSTEMS RDBMS EDW MPP TRADITIONAL REPOS ENTERPRISE HADOOP PLATFORM OPERATIONAL TOOLS MANAGE & MONITOR DATA SOURCES Tradi;onal Sources OLTP, POS SYSTEMS (RDBMS, OLTP, OLAP) New Sources (web logs, , sensors, social media)

7

8 Unified storage provided by distributed file system called HDFS Hadoop The Big Picture Computation Storage Logical Unified computation provided MapReduce distributed computing framework Physical 8 Commodity Hardware Hardware contains bunch of disks and cores 8

9 Hadoop Distributions

10 The 1 st Generation of Hadoop: Batch HADOOP 1.0 Built for Web-Scale Batch Apps Single App Single App INTERACTIVE Single App Single App ONLINE Single App All other usage patterns must leverage that same infrastructure BATCH HDFS BATCH HDFS BATCH HDFS Forces the creation of silos for managing mixed workloads

11 Hadoop as Next-Gen Platform Single Use System Batch Apps HADOOP 1.0 MapReduce (cluster resource management & data processing) HDFS (redundant, reliable storage) Multi Purpose Platform Batch, Interactive, Online, Streaming, HADOOP 2.0 MapReduce (data processing) YARN (cluster resource management) HDFS2 (redundant, reliable storage) Others (data processing) Page 11

12 YARN: Taking Hadoop Beyond Batch Store ALL DATA in one place Interact with that data in MULTIPLE WAYS with Predictable Performance and Quality of Service BATCH (MapReduce) Applica;ons Run Na;vely IN Hadoop INTERACTIVE (Tez) ONLINE (HBase) STREAMING (Storm, S4, ) GRAPH (Giraph) IN- MEMORY (Spark) YARN (Cluster Resource Management) HDFS2 (Redundant, Reliable Storage) HPC MPI (OpenMPI) OTHER (Search) (Weave ) Page12

13 Big Data Use Cases Unstructured data (documents) analysis SenIment analyics Text analyics Social & Enterprise Data Connectors for unified analyics PredicIve AnalyIcs Fraud Detec;on ETL Offloading Real- Time Processing ClickStream Analy;cs Log File processing Geo Spa;al Analyzing Video and Audio Analy;cs Page 13

14 Big Data Team Hadoop Administrators Senior Linux/Unix Admin Infrastructure Engineers Java Developers Hadoop Developers Object Oriented Developers Hadoop Developers Data Analysts FuncIonal Users Hadoop AnalyIcs Users Project Managers! Page 14

15 Big Data Process Agile Ideal Use Case for the company Proof- of- concept Tech Heavy Aware of Available OpIons Many.. Many.. Work with SoluIon Architects Infrastructure Analysis Security OpIons TesIng.. TesIng.. IntegraIng with current Stack Cost.. Cost.. Promises Vs Reality Page 15

16 Big Data Projects Page 16

17 Big Data Vs and Is 3 Vs, 3 - Vs Volume, Velocity, Variability, Veracity, VisualizaIon, Value 3Is Immediate in the sense that you need to do something about it now InImidaIng what if you don t? Ill- defined what is it, anyway? Page 17

18 THANK YOU Looking forward to partner with your thoughts 18

19 Chida LinkedIn Google+ VISIT TO KNOW US