Hadoop Job Oriented Training Agenda

Transcription

1 1 Hadoop Job Oriented Training Agenda Kapil CK

2 Module 1 M o d u l e 1 Understanding Hadoop This module covers an overview of big data, Hadoop, and the Hortonworks Data Platform. 1.1 Module Objectives 1.2 Additional Content 1.3 The Three Vs of BigData 1.4 Six Key Hadoop Data Types 1.5 About Use Cases Sentiment Use Case Geolocation Use Case 1.6 About Hadoop Relational Databases vs Hadoop About Hadoop New in Hadoop 2.x The Hadoop Ecosystem 1.7 The Hortonworks Data Platform(HDP) 1.8 The Path to ROI 1.9 Review Questions 1.10 Lab: Start an HDP 2.1 Cluster Objective: Start an HDP cluster in your VM

3 Module 2 M o d u l e 2 The Hadoop Distributed FileSystem(HDFS) This module covers the details of how files are stored and maintained in the Hadoop Distributed File System (HDFS). 2.1 Module Objectives 2.2 Additional Content 2.3 About HDFS Hadoop vs RDBMS An Example of Disk Read Performance HDFS Components 2.4 Understanding Block Storage 2.5 Demonstration: Understanding Block Storage Objective: To understand how data is partitioned into blocks and stored in HDFS 2.6 The NameNode 2.7 The DataNodes DataNode Failure 2.8 HDFS Commands Examples of HDFS Commands HDFS File Permissions 2.9 Review Questions 2.10 Lab: Using HDFS Commands Objective: Become familiar with adding, removing, and viewing files in HDFS

4 Module 3 M o d u l e 3 Inputting Data into HDFS This module covers the various ways to input data into the Hadoop Distributed File System, including the Sqoop and Flume frameworks. 3.1 Module Objectives 3.2 Additional Content The Hadoop Client WebHDFS 3.3 Overview of Flume Flume Example 3.4 Overview of Sqoop The Sqoop Import Tool Importing a Table Importing Specific Columns Importing from a Query The Sqoop Export Tool Exporting to a Table 3.5 Review Questions 3.6 Lab: Importing RDBMS Data into HDFS Objective: Import data from a database into HDFS 3.7 Lab: Exporting HDFS Data to RDBMS Objective: Export data from HDFS into a MySQL table using Sqoop

5 Module 5 Module 4 M o d u l e 4 The MapReduce Framework This module covers the details of the MapReduce programming paradigm. 4.1 Module Objectives 4.2 Additional Content 4.3 Overview of MapReduce Understanding MapReduce The Key/Value Pairs of MapReduce WordCount in MapReduce 4.4 Demonstration: Understanding MapReduce Objective: To understand how MapReduce works 4.5 The Map Phase 4.6 The Reduce Phase 4.7 Review Questions 4.8 Lab: Running MapReduce Job Objective: Run a Java MapReduce job M o d u l e 5 Introduction to Pig This module covers the Pig framework and describes how to load and transform data using the Pig programming language. 5.1 Module Objectives 5.2 Additional Content 5.3 About Pig Pig Latin

6 5.3.2 The Grunt Shell 5.4 Demonstration: Understanding Pig Objective: To understand Pigscripts and relations 5.5 Pig Latin Relation Names Pig Latin Field Names Pig Data Types Pig Complex Types 5.6 Defining a Schema 5.7 Lab: Getting Started with Pig Objective: Use Pig to navigate through HDFS and explore a dataset 5.8 The GROUP Operator GROUP ALL Relations without a Schema 5.9 The FOREACH GENERATE Operator Specifying Ranges in FOREACH Field Names in FOREACH FOREACH with Groups The FILTER Operator 5.10 The LIMIT Operator 5.11 Review Questions 5.12 Lab: Exploring Data with Pig Objective: Use Pig to navigate through HDFS and explore a

7 Module 6 dataset M o d u l e 6 Advanced Pig Programming This module covers some of the more advanced features of Pig, including sorting, parallelization, joins, and user-defined functions. 6.1 Module Objectives 6.2 Additional Content 6.3 The ORDER BY Operator 6.4 The CASE Operator 6.5 Parameter Substitution 6.6 The DISTINCT Operator 6.7 Using PARALLEL 6.8 The FLATTEN Operator 6.9 Lab: Splitting Dataset Objective: Research the WhiteHouse visitor data and look for members of Congress 6.10 Nested FOREACH 6.11 About Joins Performing an InnerJoin Performing an OuterJoin Replicated Joins 6.12 The COGROUP Operator

8 6.13 Pig User-Defined Functions UDF Example Invoking a UDF Tips for Optimizing PigScripts 6.14 Lab: Joining Datasets Objective: Join two datasets in Pig 6.15 Lab: Preparing Data for Hive Objective: Transform and export a dataset for use with Hive 6.16 Overview of the DataFu Library Computing Quantiles 6.17 Demonstration: Computing PageRank Objective: Tounderstand how to use the Page Rank UDF in DataFu 6.18 Review Questions 6.19 Lab: Analyzing Clickstream Data Objective: Become familiar with using the DataFu library to sessionize clickstream data 6.20 Lab: Analyzing StockMarket Data using Quantiles Objective: Use DataFu to compute quantiles

9 Module 7 M o d u l e 7 Hive Programming This module covers the details of the Hive framework and HiveQL programming language. 7.1 Module Objectives 7.2 Additional Content 7.3 About Hive Comparing Hive to SQL Hive Architecture Submitting Hive Queries 7.4 Defining a Hive-Managed Table Defining an External Table Defining a Table LOCATION Loading Data into Hive Table 7.5 Performing Queries 7.6 Lab: Understanding Hive Tables Objective: Understand how Hive table data is stored in HDFS 7.7 Hive Partitions Hive Buckets Skewed Tables 7.8 Demonstration: Understanding Partitions and Skew work Objective: To understand how Hive partitioning and skewed tables 7.9 Sorting Data

10 7.9.1 Using Distribute By Storing Results to File Specifying MapReduce Properties 7.10 Lab: Analyzing Big Data with Hive Objective: Analyze the WhiteHouse visitor data 7.11 Lab: Understanding MapReduce in Hive Objective: To understand how Hivequeries get executed as MapReduce jobs 7.12 Hive Join Strategies Shuffle Joins Map(Broadcast) Joins Sort-Merge-Bucket(SMB) Joins Invoking a Hive UDF Computing ngrams in Hive 7.13 Demonstration: Computing ngrams Objective: To understand how to compute ngrams using Hive 7.14 Review Questions 7.15 Lab: Joining Datasets in Hive Objective: Perform a join of two datasets in Hive 7.16 Lab: Computing ngrams of s in Avro Format Objective: Use Hive to compute ngrams

11 Module 8 M o d u l e 8 Using HCatalog This module covers the details of how HCatalog is used to provide a central repository for defining and sharing schemas for data stored in Hadoop. 8.1 Module Objectives 8.2 Additional Content 8.3 About HCatalog 8.4 HCatalog in the Ecosystem 8.5 Defining a New Schema Using HCat Loader with Pig Using HCat Storer with Pig The Pig SQL Command 8.6 Review Questions 8.7 Lab: Using HCatalog withpig Objective: Use HCatalog to provide the schema for a Pig relation

12 Module 9 M o d u l e 9 Advanced Hive Programming This module covers some of the more advanced features of Hive programming, including views, the windowing functions, and the various optimization capabilities of Hive. 9.1 Module Objectives 9.2 Additional Content 9.3 Performing a Multi-Table/File Insert 9.4 Understanding Views Defining Views Using Views The TRANSFORM Clause The OVERClause 9.5 Using Windows Hive Analytics Functions 9.6 Lab: Advanced Hive Programming Objective: To understand how some of the more advanced features of Hive work 9.7 Hive File Formats Hive SerDes Hive ORCFiles 9.8 Computing Table Statistics Hive Cost Based Optimization Vectorization

13 Module Using Hive Server Understanding Hive on Tez Using Tez for Hive Queries 9.11 Demonstration: Hive Optimizations Objective: To become familiar with someways to optimize Hive 9.12 Hive Optimization Tips Hive Query Tunings 9.13 Review Questions 9.14 Lab: Streaming Data with Hive and Python Objective: Use a custom reducer script to optimize a Hive query M o d u l e 1 0 Hadoop 2 and YARN This module covers the newer features of Hadoop 2, like YARN, HDFS Federation, and NameNode high availability Module Objectives 10.2 Additional Content 10.3 About HDFS Federation Multiple Federated NameNodes Multiple Namespaces Overview of HDFS High Availability Quorum Journal Manager Configuring Automatic Failover

14 Module About YARN Open-source YARN Use Cases The Components of YARN Lifecycle of YARN Application Cluster View Example 10.5 Review Questions 10.6 Lab: Running YARN Application Objective: To run a YARN application M o d u l e 1 1 Defining Workflow with Oozie This module covers how to implement a Hadoop workflow using the Apache Oozie framework Module Objectives 11.2 Additional Content 11.3 Overview of Oozie Defining an Oozie Workflow Pig Actions Hive Actions MapReduce Actions Submitting Workflow Job Fork and Join Nodes 11.4 Defining an Oozie Coordinator Job Schedule Job Based on Time

15 Module Schedule Job Based on Data Availability 11.5 Review Questions 11.6 Lab: Defining an Oozie Workflow Objective: Define and run an Oozie workflow M o d u l e 1 2 Hadoop Streaming This module covers an overview of the streaming capabilities of Hadoop Module Objectives 12.2 Hadoop Streaming 12.3 Running a Hadoop Streaming Job * Note: Contents subject to change