Why all industries are talking about Big Data. What are the technologies support big data. Traditional NO SQL

Transcription

1 Intrductin t Big Data and Hadp Big Data What is Big Data Why all industries are talking abut Big Data What are the issues in Big Data Strage What are the challenges fr string big data Prcessing What are the challenges fr prcessing big data What are the technlgies supprt big data Hadp Data Bases Traditinal NO SQL Hadp What is Hadp Histry f Hadp Why Hadp Why Hadp Advantages and Disadvantages f Hadp Imprtance f Different Ecsystems f Hadp Imprtance f Integratin with ther BigData slutins Big data real time Use Cases. HDFS (Hadp Distributed File System)

2 HDFS Architecture Name Nde Imprtance f Name Nde What are the rles f Name Nde What are the drawbacks in Name Nde Secndary Name Nde Imprtance f Secndary Name Nde What are the rles f Secndary Name Nde What are the drawbacks in Secndary Name Nde Data Nde Imprtance f Data Nde What are the rles f Data Nde What are the drawbacks in Data Nde Data Strage in HDFS Hw blcks are string in DataNdes Hw replicatin wrks in Data Ndes Hw t write the files in HDFS Hw t read the files in HDFS HDFS Blck size Imprtance f HDFS Blck size Why Blck size is s large Hw it is related t MapReduce split size HDFS Replicatin factr Imprtance f HDFS Replicatin factr in prductin envirnment Can we change the replicatin fr a particular file r flder Can we change the replicatin fr all files r flders Accessing HDFS

3 CLI(Cmmand Line Interface) using hdfs cmmands Java Based Apprach HDFS Cmmands Imprtance f each cmmand Hw t execute the cmmand Hdfs admin related cmmands explanatin Cnfiguratins Can we change the existing cnfiguratins f hdfs r nt Imprtance f cnfiguratins Hw t vercme the Drawbacks in HDFS> Name Nde failures Secndary Name Nde failures Data Nde failures Where des it fit and Where desn't fit Explring the Apache HDFS Web UI Hw t cnfigure the Hadp Cluster Hw t add the new ndes ( Cmmissining ) Hw t remve the existing ndes ( De-Cmmissining ) Hw t verify the Dead Ndes Hw t start the Dead Ndes Hadp 2.x.x versin features Intrductin t Namende fedratin Intrductin t Namende High Availabilty Difference between Hadp 1.x.x and Hadp 2.x.x versins MAPREDUCE Map Reduce architecture Jbtracker Imprtance f JbTracker What are the rles f TaskTracker What are the drawbacks in TaskTracker

4 TaskTracker Imprtance f TaskTracker What are the rles f TaskTracker What are the drawbacks in TaskTracker Map Reduce Jb executin flw Data Types in Hadp What are the Data types in Map Reduce Why these are imprtance in Map Reduce Can we write custm Data Types in MapReduce Input Frmat's in Map Reduce Text Input Frmat Key Value Text Input Frmat Sequence File Input Frmat Nline Input Frmat DBInputFrmat Imprtance f Input Frmat in Map Reduce Hw t use Input Frmat in Map Reduce Hw t write custm Input Frmat's and its Recrd Readers Output Frmat's in Map Reduce Text Output Frmat Sequence File Output Frmat DBOutputFrmat Imprtance f Output Frmat in Map Reduce Hw t use Output Frmat in Map Reduce Hw t write custm Output Frmat's and its Recrd Writers Mapper What is mapper in Map Reduce Jb

5 Why we need mapper? What are the Advantages and Disadvantages f mapper Writing mapper prgrams Wrking with identity mapper Reducer What is reducer in Map Reduce Jb Why we need reducer What are the Advantages and Disadvantages f reducer Writing reducer prgrams Wrking with identity reducer Cmbiner What is cmbiner in Map Reduce Jb Why we need cmbiner? What are the Advantages and Disadvantages f Cmbiner Writing Cmbiner prgrams Partitiner What is Partitiner in Map Reduce Jb Why we need Partitiner What are the Advantages and Disadvantages f Partitiner Writing Partitiner prgrams Wrking with default Partitiner Distributed Cache What is Distributed Cache in Map Reduce Jb Imprtance f Distributed Cache in Map Reduce jb What are the Advantages and Disadvantages f Distributed Cache Writing Distributed Cache prgrams Cunters

6 What is Cunter in Map Reduce Jb Why we need Cunters in prductin envirnment Hw t Write Cunters in Map Reduce prgrams Imprtance f Writable and Writable Cmparable Api's Hw t write custm Map Reduce Keys using WritableCmparable Hw t write custm Map Reduce Values using Writable Jins Map Side Jin What is the imprtance f Map Side Jin Where we are using it Reduce Side Jin What is the imprtance f Reduce Side Jin Where we are using it What is the difference between Map Side jin and Reduce Side Jin Cmpressin techniques Imprtance f Cmpressin techniques in prductin envirnment Cmpressin Types NONE, RECORD and BLOCK Cmpressin Cdecs Default, Gzip, Bzip, Snappy and LZO Enabling and Disabling these techniques fr all the Jbs Enabling and Disabling these techniques fr a particular Jb Map Reduce Schedulers FIFO Scheduler Capacity Scheduler Fair Scheduler Imprtance f Schedulers in prductin envirnment

7 Hw t use Schedulers in prductin envirnment Map Reduce Prgramming Mdel Hw t write the Map Reduce jbs in Java Running the Map Reduce jbs in lcal mde Running the Map Reduce jbs in pseud mde Running the Map Reduce jbs in cluster mde Debugging Map Reduce Jbs Hw t debug Map Reduce Jbs in Lcal Mde. Hw t debug Map Reduce Jbs in Remte Mde. YARN (Next Generatin Map Reduce) What is YARN What is the imprtance f YARN Where we can use the cncept f YARN in Real Time What is difference between YARN and Map Reduce Data Lcality What is Data Lcality Will Hadp fllws Data Lcality Speculative Executin What is Speculative Executin Will Hadp fllws Speculative Executin Map Reduce Cmmands Imprtance f each cmmand Hw t execute the cmmand Mapreduce admin related cmmands explanatin Cnfiguratins Can we change the existing cnfiguratins f mapreduce r nt Imprtance f cnfiguratins

8 Writing Unit Tests fr Map Reduce Jbs Cnfiguring hadp develpment envirnment using Eclipse Use f Secndary Srting and hw t slve using MapReduce Hw t Identify Perfrmance Bttlenecks in MR jbs and tuning MR jbs. Map Reduce Streaming and Pipes with examples Use f Ttal Order Partitiner in mapreduce Apache PIG Intrductin t Apache Pig Map Reduce Vs Apache Pig SQL Vs Apache Pig Different data types in Pig Mdes Of Executin in Pig Lcal Mde Map Reduce Mde Executin Mechanism Grunt Shell Script Embedded UDF's Hw t write the UDF's in Pig Hw t use the UDF's in Pig Imprtance f UDF's in Pig Filter's Hw t write the Filter's in Pig Hw t use the Filter's in Pig Imprtance f Filter's in Pig Lad Functins Hw t write the Lad Functins in Pig

9 Hw t use the Lad Functins in Pig Imprtance f Lad Functins in Pig Stre Functins Hw t use the Stre Functins in Pig Imprtance f Stre Functins in Pig Transfrmatins in Pig Hw t write the cmplex pig scripts Hw t integrate the Pig and Hbase Apache Hive Hive Intrductin Hive cmmands Hive architecture Driver Cmpiler Semantic Analyzer Hive Integratin with Hadp Hive Query Language(Hive QL) SQL VS Hive QL Hive Installatin and Cnfiguratin Hive, Map-Reduce and Lcal-Mde Hive DLL and DML Operatins Hive Services CLI Hiveserver Hwi Metastre

10 embedded metastre cnfiguratin external metastre cnfiguratin UDF's Hw t write the UDF's in Hive Hw t use the UDF's in Hive Imprtance f UDF's in Hive UDAF's Hw t use the udaf's in hive Hw t write the UDAF's in Hive Imprtance f udaf's in hive UDTF's Hw t use the UDTF's in Hive Imprtance f UDTF's in Hive Transfrmatins in Pig Hw t write a cmplex Hive queries What is Hive Data Mdel Partitins Imprtance f Hive Partitins in prductin envirnment Limitatins f Hive Partitins Hw t write Partitins Buckets Imprtance f Hive Buckets in prductin envirnment Hw t write Buckets SerDe Imprtance f Hive SerDe's in prductin envirnment Hw t write SerDe prgrams

11 Hw t integrate the Hive and Hbase Apache Zkeeper Intrductin t zkeeper Pseud mde installatins Zkeeper cluster installatins Basic cmmands executin Apache HBase Hbase intrductin Hbase use cases Hbase basics Clumn families Scans Hbase installatin Lcal mde Psued mde Cluster mde Hbase Architecture Strage WriteAhead Lg Lg Structured MergeTrees Blck cache Mapreduce integratin Mapreduce ver Hbase Hbase Usage

12 Key design Blm Filters Versining Cprcessrs Filters Hbase Clients REST Thrift Hive Web Based UI Hbase Admin Schema definitin Basic CRUD peratins Apache Sqp Intrductin t Sqp MySQL client and Server Installatin Sqp Installatin Hw t cnnect t Relatinal Database using Sqp Sqp Cmmands and Examples n Imprt and Exprt cmmands Sqp incremental imprts Sqp with hive integratin Sqp with hbase integratin Apache Flume Intrductin t flume Flume installatin Flume agent usage and Flume examples executin

13 Apache OOZIE Intrductin t zie Ozie installatin Executing zie wrkflw jbs Mnitering Ozie wrkflw jbs Hadp Ecsystems Integratins: Hadp and Hive Integratin Hadp and Pig Integratin Hadp and HBase Integratin Hadp and Sqp Integratin Hadp and Ozie Integratin Hadp and Flume Integratin Hive and Pig Integratin Hive and HBase integratin Pig and HBase integratin Sqp and RDBMS Integratin Hadp Distributins : Wrking with hrtnwrks distributin Wrking with cludera distributin Real Time Prject