Big Data Course Highlights

Transcription

1 Big Data Course Highlights The Big Data course will start with the basics of Linux which are required to get started with Big Data and then slowly progress from some of the basics of Hadoop/Big Data (like a Word Count program) to some of the advanced concepts around Hadoop/Big Data (like writing triggers/stored procedures to HBase). The main focus of the course will be on how to use the Big Data tools, but we will also focus on how to install and configure some of the Big Data related frameworks on in-premise and cloud based infrastructure. Because of the hype Hadoop is the news all the time. But, there are a lot of frameworks supporting Hadoop (like Pig/Hive) and a lot of frameworks which are alternatives to Hadoop (like Twitter Storm and LinkedIn Samza) to address the limitations of the MapReduce model. Some of these frameworks will also be discussed during the course to give a big picture of what Big Data is about. Also, time will be spent on NoSQL databases. Starting with why NoSQL instead of RDBMS databases to some of advanced concepts like importing data in bulk from a RDBMS to a NoSQL database. Different NoSQL databases will be compared and HBase will be discussed in much more detail. A VM (Virtual Machine) will be provided for all the participants with Big Data frameworks (Hadoop etc.) installed and configured on CentOS with data sets and code to process the same. The VM helps in making the Big Data learning experience less steeper. The training will help the participant get through the Cloudera Certified Developer for Apache Hadoop (CCDH) certification with minimal effort.

2 Pre-Requisites: Knowledge of Java is a definitive plus to get started with Big Data, but not mandatory. Hadoop provides streaming which allows programming MapReduce in non-java languages like Perl, Python and there are also higher level abstracts like Hive/Pig which provides SQL like procedure type interface. Similarly knowledge of Linux would be a definitive plus but the basics of Linux just enough to get started with the different Big data frameworks. A laptop/desktop with minimum of 3GB RAM, 10 GB free HARD Disk and with a decent processor. These specifications would be enough to run the Big Data VM and the framework smoothly. Who should Plan on joining this program Audience: This course is designed for anyone who is Any Developer with skills in other technologies interested in getting into the emerging Big Data field. Any Data Analyst who would like to enhance/transfer their existing knowledge to the Big Data space. Any Architect who would like to design application in conjunction to Big Data or Big Data applications itself.

3 Anyone involved in Software Quality Assurance (Testing). Knowledge of Hadoop will help them to test the application better and will also help them to move into the development cycle. Topics covered in the training Understanding Big Data Understanding Big Data - 3V (Volume-Variety-Velocity) characteristics - Structured and Unstructured Data - Application and use cases of Big Data Limitations of traditional large Scale systems How a distributed way of computing is superior (cost and scale) Opportunities and challenges with Big Data HDFS (The Hadoop Distributed File System) HDFS Overview and Architecture Deployment Architecture Name Node, Data Node and Checkpoint Node (aka Secondary Name Node) Safe mode Configuration files HDFS Data Flows (Read vs Write) How HDFS addresses fault tolerance? CRC Check Sum Data replication Rack awareness and Block placement policy Small files problem

4 HDFS Interfaces Web Interface Command Line Interface File System Administrative Advanced HDFS features Load Balancer DistCP HDFS High Availability Hadoop Cache MapReduce - 1 MapReduce Overview Functional Programming paradigms How to think in a MapReduce way? MapReduce Architecture Legacy MR vs Next Generation MapReduce (YARN) Slots vs Containers Schedulers Shuffling, Sorting Hadoop Data types Input and Output formats Input Splits

5 Partitioning (Hash Partitioner vs Custom Partitioner) Counters Configuration files Distributed Cache MapReduce 2 Developing, debugging and deploying MR programs Standalone mode(eclipse) Pseudo distributed mode (as in the Big Data VM) Fully Distributed Mode MR API Old and new MR API Java Client API Overview of MRUnit Hadoop Data types and custom writables/writable comparables Different input and output formats Saving Binary Data using Sequence Files and Avro Files Optimizing techniques Speculative execution Combiners Compression MR algorithms Sorting (Max Temperature) Different ways of joining data Inverted Index

6 Word co-occurrence Pig Introduction to PIG Why PIG not MapReduce Pig Components Pig Execution Modes Pig Shell - Grunt Pig Latin, Writing PIG Latin scripts Pig Data Types Storage Types Diagnosing Pig commands Macros UDF and External Scripts Hive Introduction and Architecture Different modes of executing Hive queries Metastore implementations HiveQL (DDL & DML operations) External vs Internal Tables Views Partitions & Buckets UDF Comparison of Pig and Hive Flume Overview of Flume

7 Where is Flume used - import/export unstructured data Flume Architecture Using Flume to load data into HDFS Sqoop Overview of Sqoop Where is Sqoop used - import/export structured data Using Sqoop to import data from RDBMS into HDFS Using Sqoop to import data from RDBMS into HBASE Using Sqoop to export data from HDFS into RDMBS Sqoop connectors Impala Overview of Impala Architecture of Impala NoSQL Databases Introduction to NoSQL database Types of NoSQL databases and their features Brewers CAP Theorem Advantage of NoSQL vs. traditional RDBMS ACID vs BASE Different types of NoSQL databases Key value Columnar Document Graph

8 HBase Introduction to HBase Why use HBase HBase Architecture - read and write paths HBase vs. RDBMS Installing and Configuration Schema design in HBase - column families, hot spotting Accessing data with HBase Shell Accessing data with HBase API - Reading, Adding, Updating data from the shell, JAVA API HBase Coprocessors (Endpoints, Observers) POC Click stream analysis Analyzing the Twitter data with Hive