APACHE HADOOP JERRIN JOSEPH CSU ID#

Transcription

1 APACHE HADOOP JERRIN JOSEPH CSU ID#

2 CONTENTS Hadoop Hadoop Distributed File System (HDFS) Hadoop MapReduce Introduction Architecture Operations Conclusion References

3 ABSTRACT Hadoop is an efficient Big data handling tool. Reduced the data processing time from days to hours. Hadoop Distributed File System(HDFS) is the data storage unit of Hadoop. Hadoop MapReduce is the data processing unit which works on distributed processing principle.

4 INTRODUCTION What is Big Data?? Bulk Amount Unstructured Lots of Applications which need to handle huge amount of data (in terms of 500+ TB per day) If a regular machine need to transmit 1TB of data through 4 channels : 43 Minutes. What if 500 TB??

5 HADOOP The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models [1] Core Components : HDFS: large data sets across clusters of computers. Hadoop MapReduce: the distributed processing using simple programming models

6 HADOOP : KEY FEATURES High Scalability Highly Tolerant to Software & Hardware Failures High Throughput Best for larger files with less in number Performs fast and parallel execution of Jobs Provides Streaming access to data Can be built out of commodity hardware

7 HADOOP: DRAWBACKS Not good for Low-latency data access Not good for Small files with large in number Not good for Multiple write files Do not encryption at storage level or network level Have a high complexity security model Hadoop is not a Database: Hence cannot alter a file.

8 HADOOP ARCHITECTURE

9 HADOOP DISTRIBUTED FILE SYSTEM (HDFS)

10 HADOOP DISTRIBUTED FILE SYSTEM (HDFS) Storage unit of Hadoop Relies on principles of Distributed File System. HDFS have a Master-Slave architecture Main Components: Name Node : Master Data Node : Slave 3+ replicas for each block Default Block Size : 64MB

11 HDFS: KEY FEATURES Highly fault tolerant. (automatic failure recovery system) High throughput Designed to work with systems with vary large file (files with size in TB) and few in number. Provides streaming access to file system data. It is specifically good for write once read many kind of files (for example Log files). Can be built out of commodity hardware. HDFS doesn't need highly expensive storage devices.

12 HDFS ARCHITECTURE

13 NAME NODE Master of HDFS Maintains and Manages data on Data Nodes High reliability Machine (can be even RAID) Expensive Hardware Stores NO data; Just holds Metadata! Secondary Name Node: Reads from RAM of Name Node and stores it to hard disks periodically. Active & Passive Name Nodes from Gen2 Hadoop

14 DATA NODES Slaves in HDFS Provides Data Storage Deployed on independent machines Responsible for serving Read/Write requests from Client. The data processing is done on Data Nodes.

15 HDFS OPERATION

16 HDFS OPERATION Client makes a Write request to Name Node Name Node responds with the information about on available data nodes and where data to be written. Client write the data to the addressed Data Node. Replicas for all blocks are automatically created by Data Pipeline. If Write fails, Data Node will notify the Client and get new location to write. If Write Completed Successfully, Acknowledgement is given to Client Non-Posted Write by Hadoop

17 HDFS: FILE WRITE

18 HDFS: FILE READ

19 HADOOP MAPREDUCE

20 HADOOP MAPREDUCE Simple programming model Hadoop Processing Unit MapReduce also have Master-Slave architecture Main Components: Job Tracker : Master Task Tracker : Slave From Google s MapReduce Do not fetch data to Master Node; Processed data at Slave Node and returns output to Master

21 HADOOP MAPREDUCE Implemented using Maps and Reduces Split by FileInputFormat Maps Inheriting Mapper Class Produces (key, value) pair as intermediate result from data. Reduces Inheriting Reducer Class Produces required output from intermediate result produced by Maps.

22 JOB TRACKER Master in MapReduce Receives the job request from Client Governs execution of jobs Makes the task scheduling decision TASK TRACKER Slave in MapReduce Governs execution of Tasks Periodically reports the progress of tasks

23 MAPREDUCE ARCHITECTURE

24 MAPREDUCE OPERATIONS

25 MAPREDUCE OPERATIONS

26 MAPREDUCE OPERATIONS

27 MAPREDUCE OPERATIONS

28 APACHE HIVE

29 HIVE Built on top of Hadoop Supports SQL like Query Language : Hive-QL Data in Hive is organized into tables Provides structure for unstructured Big Data Work with data inside HDFS Tables Data : File or Group of Files in HDFS Schema : In the form of metadata stored in Relational Database Have a corresponding HDFS directory Data in a table is Serialized Supports Primitive Column Types and Nestable Collection Types (Array and Map)

30 HIVE QUERY LANGUAGE SQL like language DDL : to create tables with specific serialization formats DML : to load data from external sources and insert query results into Hive tables Do not support updating and deleting rows in existing tables Supports Multi-Table insert Supports custom map-reduce scripts written in any language Can be extended with custom functions (UDFs) User Defined Transformation Function(UDTF) User Defined Aggregation Function (UDAF)

31 HIVE ARCHITECTURE External Interfaces: Web UI : Management Hive CLI : Run Queries, Browse Tables, etc API : JDBC, ODBC Metastore : System catalog which contains metadata about Hive tables Driver : manages the life cycle of a Hive-QL statement during compilation, optimization and execution Compiler : translates Hive-QL statement into a plan which consists of a DAG of map-reduce jobs

32 HIVE ARCHITECTURE

33 HIVE ACHIEVEMENTS & FUTURE PLANS First step to provide warehousing layer for Hadoop(Web-based Map-Reduce data processing system) Accepts only sub-set of SQL: Working to subsume SQL syntax Working on Rule-based optimizer : Plans to build Cost-based optimizer Enhancing JDBC and ODBC drivers for making the interactions with commercial BI tools. Working on making it perform better

34 APACHE HBASE

35 H-BASE Distributed Column-oriented database on top of Hadoop/HDFS Provides low-latency access to single rows from billions of records Column oriented: OLAP Best for aggregation High compression rate: Few distinct values Do not have a Schema or data type Built for Wide tables : Millions of columns Billions of rows Denormalized data Master-Slave architecture

36 H-BASE ARCHITECTURE

37 HMASTER SERVER Like Name Node in HDFS Manages and Monitors HBase Cluster Operations Assign Region to Region Servers Handling Load-balancing and Splitting REGION SERVER Like Data Node in HDFS Highly Scalable Handle Read/ Write Requests Direct communication with Clients

38 INTERNAL ARCHITECTURE Tables Regions Store MemStore FileStore Blocks Column Families

39 APACHE ZOOKEEPER

40 ZOOKEEPER What is ZooKeeper? Distributed coordination service for distributed applications Like a Centralized Repository Challenges for Distributed Applications ZooKeeper Goals

41 ZOOKEEPER ARCHITECTURE

42 ZOOKEEPER ARCHITECTURE Always Odd number of nodes. Leader is elected by voting. Leader and Follower can get connected to Clients and Perform Read Operations Write Operation is done only by the Leader. Observer nodes to address scaling problems

43 ZOOKEEPER DATA MODEL

44 ZOOKEEPER DATA MODEL Z Nodes: Similar to Directory in File system Container for data and other nodes Stores Statistical information and User data up to 1MB Used to store and share configuration information between applications Z Node Types Persistent Nodes Ephemeral Nodes Sequential Nodes Watch : Event system for client notification

45 PROJECTS & TOOLS ON HADOOP HBase Hive Pig Jaql ZooKeeper AVRO UIMA Sqoop

46 CONCLUSION Hadoop is a successful solution for Big Data Handling Hadoop expanded from a simple project to the level of a platform The projects and tools on Hadoop are proof for the successfulness of Hadoop.

47 REFERENCES [1] "Apache Hadoop", [2] Apache Hive, [3] Apache HBase, [4] Apache ZooKeeper, [5] Jason Venner, "Pro Hadoop", Apress Books, 2009 [6] "Hadoop Wiki", [7] Jiong Xie, Shu Yin, Xiaojun Ruan, Zhiyang Ding, Yun Tian, James Majors, Adam Manzanares, Xiao Qin, " Improving MapReduce Performance through Data Placement in Heterogeneous Hadoop Clusters", 19th International Heterogeneity in Computing Workshop, Atlanta, Georgia, April 2010

48 REFERENCES [8] Dhruba Borthakur, The Hadoop Distributed File System: Architecture and Design, The Apache Software Foundation [9] "Apache Hadoop", [10] "Hadoop Overview", [11] Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler, The Hadoop Distributed File System, Yahoo!, Sunnyvale, California USA, Published in: Mass Storage Systems and Technologies (MSST), 2010 IEEE 26th Symposium.

49 REFERENCES [12] Vinod Kumar Vavilapalli, Arun C Murthy, Chris Douglas, Sharad Agarwal, Mahadev Konar, Robert Evans, Thomas Graves, Jason Lowe, Hitesh Shah, Siddharth Seth, Bikas Saha, Carlo Curino, Owen O Malley, Sanjay Radia, Benjamin Reed, Eric Baldeschwieler, Apache Hadoop YARN: Yet Another Resource Negotiator, ACM Symposium on Cloud Computing 2013, Santa Clara, California. [13] Raja Appuswamy, Christos Gkantsidis, Dushyanth Narayanan, Orion Hodson, and Antony Rowstron, Scale-up vs Scale-out for Hadoop: Time to rethink?, Microsoft Research, ACM Symposium on Cloud Computing 2013, Santa Clara, California.