Ge6ng to Know Hadoop. Sriram Mohan Senior Consultant, Avalon Consul,ng, LLC Associate Professor of CSSE, Rose- Hulman Ins,tute of Technology

Transcription

1 Ge6ng to Know Hadoop Sriram Mohan Senior Consultant, Avalon Consul,ng, LLC Associate Professor of CSSE, Rose- Hulman Ins,tute of Technology

2 Presenter Overview Who we are Consultants providing expert technical integra,ons for enterprise- scale Internet, intranet, and extranet sites 50+ staff, mostly senior- level consultants Offices in Dallas and Washington, DC Best known for our work in: Enterprise search Hadoop, big data Enterprise content management Websites & portals E- learning Unstructured, semi- structured content

3 Presenter Overview Who we are Private engineering school in Terre Haute, Indiana 2,000 students, 12:1 faculty- to- student ra,o 10 math, science, and engineering majors Best known for : Best undergraduate engineering school in the country 14 years in a row

4 Overview Hadoop HDFS MapReduce Pig & Hive Sqoop & Flume Oozie

5 What Is Big Data? Volume Large quan,,es (think gigabytes, terabytes of informa,on daily) Velocity Needs to be processed very quickly Variety Data might be structured, unstructured, varying sources

6 Overview Hadoop HDFS MapReduce Pig & Hive Sqoop & Flume Oozie

7 Apache Hadoop Distributed pladorm for data processing Scalable Runs on commodity hardware Data & analysis coloca,on

8 Typical Hadoop Stack

9 Overview Hadoop HDFS MapReduce Pig & Hive Sqoop & Flume Oozie

10 Best Fit Very large files Streaming data access Commodity hardware

11 Bad Fit Low- latency data access Lots of small files Mul,ple writer, arbitrary file modifica,ons

12 HDFS

13 Overview Hadoop HDFS MapReduce Pig & Hive Sqoop & Flume Oozie

14 MapReduce Overview

15 MapReduce Example

16 Overview Hadoop HDFS MapReduce Pig & Hive Sqoop & Flume Oozie

17 Pig What is Pig? Pig La,n Ease of programming Op,miza,on opportuni,es Extensibility Pig execu,on environment Local Distributed

18 Why Pig? ETL pipeline Research on raw data Itera,ve processing

19 Hive What is Hive? HiveQL Data warehouse system More or less compa,ble with SQL standard No support for updates & deletes Project S,nger Metastore Why Hive?

20 Overview Hadoop HDFS MapReduce Pig, Hive Sqoop & Flume Oozie

21 Why Sqoop? Need to move data from tradi,onal data sources to Hadoop for batch processing/ archiving Need to move results of Hadoop jobs to rela,onal data sources that support repor,ng & data visualiza,on

22 What Is Sqoop? Sqoop is a command- line interface applica,on for transferring data between rela,onal databases and Hadoop. It supports incremental loads of a single table or a free- form SQL query as well as saved jobs that can be run mul,ple,mes to import updates made to a database since the last import.

23 What Is Flume? Flume is a distributed, reliable, and available service for efficiently collec,ng, aggrega,ng, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows.

24 Model

25 Overview Hadoop HDFS MapReduce Pig, Hive Sqoop & Flume Oozie

26 Oozie Overview Main features Execute and monitor workflows in Hadoop Periodic scheduling of workflows Trigger execu,on by data availability HTTP and command line interface + web console

27 Oozie Workflow M/R streaming job OK start Java Main OK fork join Pig job OK MORE decision M/R job ENOUGH OK Java Main end OK FS job OK

28 Ques,ons?