WA2341 Hadoop Programming EVALUATION ONLY

Transcription

1 WA2341 Hadoop Programming Web Age Solutions Inc. USA: Canada: Web:

2 The following terms are trademarks of other companies: Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. UNIX is a registered trademark of The Open Group in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. IBM, WebSphere, DB2 and Tivoli are trademarks of the International Business Machines Corporation in the United States, other countries, or both. Other company, product, and service names may be trademarks or service marks of others. For customizations of this book or other sales inquiries, please contact us at: USA: , getinfousa@webagesolutions.com Canada: toll free, getinfo@webagesolutions.com Copyright 2014 Web Age Solutions Inc. This publication is protected by the copyright laws of Canada, United States and any other country where this book is sold. Unauthorized use of this material, including but not limited to, reproduction of the whole or part of the content, re-sale or transmission through fax, photocopy or is prohibited. To obtain authorization for any such activities, please write to: Web Age Solutions Inc. 439 University Ave Suite 820 Toronto Ontario, M5G 1Y8

3 Table of Contents Chapter 1 - MapReduce Overview MapReduce Defined Google's MapReduce MapReduce Explained MapReduce Explained MapReduce Word Count Job MapReduce Shared-Nothing Architecture Similarity with SQL Aggregation Operations Example of Map & Reduce Operations using JavaScript Example of Map & Reduce Operations using JavaScript Problems Suitable for Solving with MapReduce Typical MapReduce Jobs Fault-tolerance of MapReduce Distributed Computing Economics MapReduce Systems Summary...18 Chapter 2 - Hadoop Overview Apache Hadoop Apache Hadoop Logo Typical Hadoop Applications Hadoop Clusters Hadoop Design Principles Hadoop's Core Components Hadoop Simple Definition High-Level Hadoop Architecture Hadoop-based Systems for Data Analysis Hadoop Caveats Summary...23 Chapter 3 - Hadoop Distributed File System Overview Hadoop Distributed File System Hadoop Distributed File System Data Blocks Data Block Replication Example HDFS NameNode Directory Diagram Accessing HDFS Examples of HDFS Commands Client Interactions with HDFS for the Read Operation Read Operation Sequence Diagram Client Interactions with HDFS for the Write Operation Communication inside HDFS Summary...30 Chapter 4 - MapReduce with Hadoop Hadoop's MapReduce JobTracker and TaskTracker...33

4 4.3 MapReduce Programming Options Java MapReduce API The Structure of a Java MapReduce Program The Mapper Class The Reducer Class The Driver Class Compiling Classes Running the MapReduce Job The Structure of a Single MapReduce Program Combiner Pass (Optional) Hadoop's Streaming MapReduce Python Word Count Mapper Program Example Python Word Count Reducer Program Example Setting up Java Classpath for Streaming Support Streaming Use Cases The Streaming API vs Java MapReduce API Amazon Elastic MapReduce Amazon Elastic MapReduce Amazon Elastic MapReduce Summary...43 Chapter 5 - Apache Pig Scripting Platform What is Pig? Pig Latin Apache Pig Logo Pig Execution Modes Local Execution Mode MapReduce Execution Mode Running Pig Running Pig in Batch Mode What is Grunt? Pig Latin Statements Pig Programs Pig Latin Script Example SQL Equivalent Differences between Pig and SQL Statement Processing in Pig Comments in Pig Supported Simple Data Types Supported Complex Data Types Arrays Defining Relation's Schema The bytearray Generic Type Using Field Delimiters Referencing Fields in Relations Summary...54 Chapter 6 - Apache Pig HDFS Interface...55

5 6.1 The HDFS Interface FSShell Commands (Short List) Grunt's Old File System Commands Summary...57 Chapter 7 - Apache Pig Relational and Eval Operators Pig Relational Operators Example of Using the JOIN Operator Example of Using the JOIN Operator Example of Using the Order By Operator Caveats of Using Relational Operators Pig Eval Functions Caveats of Using Eval Functions (Operators) Example of Using Single-column Eval Operations Example of Using Eval Operators For Global Operations Summary...63 Chapter 8 - Apache Pig Miscellaneous Topics Utility Commands Handling Compression User-Defined Functions Filter UDF Skeleton Code Summary...67 Chapter 9 - Apache Pig Performance Apache Pig Performance Performance Enhancer - Use the Right Schema Type Performance Enhancer - Apply Data Filters Use the PARALLEL Clause Examples of the PARALLEL Clause Performance Enhancer - Limiting the Data Sets Displaying Execution Plan Summary...72 Chapter 10 - Hive What is Hive? Apache Hive Logo Hive's Value Proposition Who uses Hive? Hive's Main Systems Hive Features Hive Architecture HiveQL Where are the Hive Tables Located? Hive Command-line Interface (CLI) Summary...78 Chapter 11 - Hive Command-line Interface Hive Command-line Interface (CLI) The Hive Interactive Shell Running Host OS Commands from the Hive Shell...82

6 11.4 Interfacing with HDFS from the Hive Shell The Hive in Unattended Mode The Hive CLI Integration with the OS Shell Executing HiveQL Scripts Comments in Hive Scripts Variables and Properties in Hive CLI Setting Properties in CLI Example of Setting Properties in CLI Hive Namespaces Using the SET Command Setting Properties in the Shell Setting Properties for the New Shell Session Summary...86 Chapter 12 - Hive Data Definition Language Hive Data Definition Language Creating Databases in Hive Using Databases Creating Tables in Hive Supported Data Type Categories Common Primitive Types Example of the CREATE TABLE Statement The STRUCT Type Table Partitioning Table Partitioning Table Partitioning on Multiple Columns Viewing Table Partitions Row Format Data Serializers / Deserializers File Format Storage More on File Formats The EXTERNAL DDL Parameter Example of Using EXTERNAL Creating an Empty Table Dropping a Table Table / Partition(s) Truncation Alter Table/Partition/Column Views Create View Statement Why Use Views? Restricting Amount of Viewable Data Examples of Restricting Amount of Viewable Data Creating and Dropping Indexes Describing Data Summary Chapter 13 - Hive Data Manipulation Language Hive Data Manipulation Language (DML)...103

7 13.2 Using the LOAD DATA statement Example of Loading Data into a Hive Table Loading Data with the INSERT Statement Appending and Replacing Data with the INSERT Statement Examples of Using the INSERT Statement Multi Table Inserts Multi Table Inserts Syntax Multi Table Inserts Example Summary Chapter 14 - Hive Select Statement HiveQL The SELECT Statement Syntax The WHERE Clause Examples of the WHERE Statement Partition-based Queries Example of an Efficient SELECT Statement The DISTINCT Clause Supported Numeric Operators Built-in Mathematical Functions Built-in Aggregate Functions Built-in Statistical Functions Other Useful Built-in Functions The GROUP BY Clause The HAVING Clause The LIMIT Clause The ORDER BY Clause The JOIN Clause The CASE Clause Example of CASE Clause Summary Chapter 15 - Apache Sqoop What is Sqoop? Apache Sqoop Logo Sqoop Import / Export Sqoop Help Examples of Using Sqoop Commands Data Import Example Fine-tuning Data Import Controlling the Number of Import Processes Data Splitting Helping Sqoop Out Example of Executing Sqoop Load in Parallel A Word of Caution: Avoid Complex Free-Form Queries Using Direct Export from Databases Example of Using Direct Export from MySQL More on Direct Mode Import...120

8 15.16 Changing Data Types Example of Default Types Overriding File Formats The Apache Avro Serialization System Binary vs Text More on the SequenceFile Binary Format Generating the Java Table Record Source Code Data Export from HDFS Export Tool Common Arguments Data Export Control Arguments Data Export Example Using a Staging Table INSERT and UPDATE Statements INSERT Operations UPDATE Operations Example of the Update Operation Failed Exports Summary Chapter 16 - Cloudera Impala What is Cloudera Impala? Impala's Logo Benefits of Using Impala Key Features How Impala Handles SQL Queries Impala Programming Interfaces Impala SQL Language Reference Differences Between Impala and HiveQL Impala Shell Impala Shell Main Options Impala Shell Commands Impala Common Shell Commands Cloudera Web Admin UI Impala Browse-based Query Editor Summary Chapter 17 - Apache HBase What is HBase? HBase Design HBase Features The Write-Ahead Log (WAL) and MemStore HBase vs RDBS HBase vs Apache Cassandra Interfacing with HBase HBase Thrift And REST Gateway HBase Table Design Column Families A Cell's Value Versioning...143

9 17.12 Timestamps Accessing Cells HBase Table Design Digest Table Horizontal Partitioning with Regions HBase Compaction Loading Data in HBase HBase Shell HBase Shell Command Groups Creating and Populating a Table in HBase Shell Getting a Cell's Value Counting Rows in an HBase Table Summary Chapter 18 - Apache HBase Java API HBase Java Client HBase Scanners Using ResultScanner Efficiently The Scan Class The KeyValue Class The Result Class Getting Versions of Cell Values Example The Cell Interface HBase Java Client Example Scanning the Table Rows Dropping a Table The Bytes Utility Class Summary...156

10

11 Chapter 1 - MapReduce Overview Objectives In this chapter, participants will learn about: MapReduce Programming Model Main MapReduce design principles 1.1 MapReduce Defined There are different definitions of what MapReduce (single word) is: a programming model parallel processing framework a computational paradigm batch query processor This technique (model, etc.) was influenced by functional programming languages that have the map and reduce functions 1.2 Google's MapReduce MapReduce was first introduced by Google back in 2004 Google applied for and was granted US Patent 7,650,331 on MapReduce called "System and method for efficient large-scale data processing" The patent lists Jeffrey Dean and Sanjay Ghemawat as its inventors The value proposition of the MapReduce framework is its scalability and fault-tolerance achieved by its architecture Notes: ABSTRACT A large-scale data processing system and method includes one or more application-independent map modules configured to read input data and to apply at least one application-specific map operation to the input data to produce intermediate data values, wherein the map operation is automatically parallelized across multiple processors in the parallel processing environment. A plurality of intermediate data structures are used to store the intermediate data values. One or more application-

12 Chapter 1 - MapReduce Overview independent reduce modules are configured to retrieve the intermediate data values and to apply at least one application-specific reduce operation to the intermediate data values to provide output data. Source: MapReduce Explained MapReduce works by breaking the data processing task into two phases: The map phase and the reduce phase executed sequentially with the output of the map operation piped (emitted) as input into the reduce operation The map phase is backed up by a Map() procedure breaks up the chunks of original data into a list of key/value pairs There is a data split operation feeding data into the Map() procedure The reduce phase is backed up by a Reduce() procedure that performs some soft of an aggregation operation by key (e.g. counting, finding the maximum of elements in the data set, etc.) on data received from the Map() procedure There is also an additional step between the map and reduce phases, called "Shuffle", that prepares (sorts, etc.) and directs output of the map phase to the reduce phase 12

13 Chapter 1 - MapReduce Overview 1.4 MapReduce Explained (Source: Wikipedia) 13

14 Chapter 1 - MapReduce Overview 1.5 MapReduce Word Count Job Notes: The key in the Word Count MapReduce operation is the word itself. Each word's associated value is the number of occurrences of this word in the input data set. The map phase emits the key/value pairs as word,1 (each word gets counted as occurring only once) with potentially multiple duplications which will be combined in the shuffle step and then aggregated in the reduce phase. 1.6 MapReduce Shared-Nothing Architecture MapReduce is designed around the shared-nothing architecture which leads to computational efficiencies in distributed computing environments Shared-nothing means "independent of others" A mapper process is independent from other mapper processes, and so are reducer processes This architecture allows the map and reduce operations to be executed in parallel (in their respective phases) 14

15 Chapter 1 - MapReduce Overview MapReduce programming model is linearly scalable 1.7 Similarity with SQL Aggregation Operations It may help compare MapReduce with aggregation operations used in SQL In SQL, aggregation is achieved by using COUNT(), AVG(), MIN(), and other such functions (which act as some sort of reducers) with the GROUP BY clause, e.g. SELECT MONTH, SUM(SALES) FROM YEAR_END_REPORT GROUP BY MONTH MapReduce offers a more fine-grained programmatic control over data processing in multi-node computing environments using parallel programming algorithms 1.8 Example of Map & Reduce Operations using JavaScript JavaScript supports some elements of functional programming in the form of the map() and reduce() functions Problem: Find out the sum of elements of the [1,2,3] array after all its elements have been increased by 10% Note: An alternative solution is to find the sum of the array elements before the increase and then apply the 10% increase to the total 1.9 Example of Map & Reduce Operations using JavaScript Solution: The Map phase (apply the function of a 10% value increase to each element of the input array of [1,2,3]): [1,2,3].map(function(x){return x + x/10}); Result: [1.1, 2.2, 3.3] The Reduce phase (use the result array of the Map operation as input and sum up all its elements): 15

16 Chapter 1 - MapReduce Overview [1.1, 2.2, 3.3].reduce(function(x,y){return x + y}); Result: 6.6 Note: While the map() and reduce() functions here can also be used independently from each other; MapReduce is always a single operation 1.10 Problems Suitable for Solving with MapReduce The following is the criteria that you can use to see if the problem at hand can be efficiently solved by MapReduce: The problem can be split into smaller problems with no shared state that can be solved in parallel Those smaller problems are independent from one another and do not require interactions The problem can be decomposed into the map and reduce operations The map operation: execute the same operation on all data The reduce operation: execute the same operation on each group of data produced by the map operation Basically, you should see the generic "divide and conquer" pattern 1.11 Typical MapReduce Jobs Counting tokens (words, URLs, etc.) Finding aggregate values in the target data set, e.g. the average value Processing geographical data etc... Google Maps uses MapReduce to find the nearest feature, like coffee shop, museum, etc., to a given address 1.12 Fault-tolerance of MapReduce MapReduce operations have a degree of fault-tolerance and built-in recoverability from certain types of run-time failures which is leveraged in 16

17 Chapter 1 - MapReduce Overview production-ready systems (e.g. Hadoop) MapReduce enjoys these quality of service due to its architecture based on process parallelism Failed units of work are rescheduled and resubmitted for execution should some mappers or reducers fail (provided the source data is still available) Note: High Performance (e.g. Grid) Computing systems that use the Message Passing Interface communication for check-points are more difficult to program for failure recovery 1.13 Distributed Computing Economics MapReduce splits the workload into units of work (independent computation tasks) and intelligently distributes them across available worker nodes for parallel processing To improve system performance, MapReduce tries to start a computation task on the node where the data to be worked on is stored Notes: This helps avoid unnecessary network operations "Data locality" (collocation of data with the compute node) underlies the principles of Distributed Computing Economics Distributed Computing Economics was a topic covered by Jim Gray of Microsoft Research in his paper published back in 2003 ( His main conclusion was: "Put the computation near the data" or "One puts computing as close to the data as possible in order to avoid expensive network traffic" MapReduce Systems Many systems leverage MapReduce programming model: Hadoop has a built-in MapReduce engine for executing both regular and streaming MapReduce jobs Amazon WS offers their clients Elastic MapReduce (EMR) Service (running on a Hadoop cluster) 17

18 Chapter 1 - MapReduce Overview Notes: Amazon Elastic MapReduce works as a Job flow where the client needs to follow pre-defined steps: 1. Provide the map and reduce applications/scripts to the Hadoop Framework via Amazon S3 bucket upload or use pre-installed ones 2. Specify the S3 bucket containing data input file(s) and the output S3 bucket to receive the result of the MapReduce workflow 3. Allocate the needed compute (EC2) instances 4. Execute the Job and pickup the output from the output S3 bucket 1.15 Summary The MapReduce programming model was influenced by functional programming languages MapReduce (one word) has the map and reduce components working together in a distributed computational environment Production MapReduce system offer a number of quality of services, such as fault-tolerance 18

19 Chapter 2 - Hadoop Overview Objectives In this chapter, participants will learn about: Apache Hadoop and its core components Hadoop's main design considerations Notes: 2.1 Apache Hadoop Apache Hadoop is a distributed fault-tolerant computing platform written in Java Designed as a massively parallel processing (MPP) system based on a distributed master-slave architecture Hadoop allows for the distributed processing of large data sets across clusters of computers using simple programming models, e.g. MapReduce Hadoop is an open source project Hadoop is the name the son of Doug Cutting (the project's creator) gave to his yellow elephant toy. Doug Cutting is presently Chairman of the Apache Software Foundation and Cloudera's Chief Architect. Massively parallel processing (MPP) is the system configuration that enlists hundreds and thousands of CPUs simultaneously for solving a particular computational task. Each computer in the MPP system controls its own resources along with task execution coordination with other nodes in the system. The coordination aspect sets Hadoop aside from MPP systems as Hadoop is designed on the shared-nothing architecture delegating the coordination activities to the dispatcher layer (made up of the JobTracker and TaskTrackers). According to the Wikipedia article ( "A shared nothing architecture is a distributed computing architecture in which each node is independent and selfsufficient, and there is no single point of contention across the system. More specifically, none of the nodes share memory or disk storage." Yahoo provided funds to make Hadoop a Web-scale technology; the initial use case for Hadoop at Yahoo was to create and analyze a WebMap graph that consists of about one trillion (10 12 ) Web links and 100 billion distinct URLs.

20 Chapter 2 - Hadoop Overview 2.2 Apache Hadoop Logo 2.3 Typical Hadoop Applications Log and/or clickstream analysis Web crawling results processing Marketing analytics Machine learning and data mining Data archiving (e.g. for regulatory compliance, etc.) See for the list of educational and production uses of Hadoop 2.4 Hadoop Clusters First versions of Hadoop were only able to handle 20 machines; newer versions are capable to run Hadoop clusters comprising thousands of nodes Hadoop clusters run on moderately high-end commodity hardware ($2-5K per machine) Hadoop clusters can be used as a data hub, data warehouse or a business analytics platform 2.5 Hadoop Design Principles Hadoop's design was influenced by ideas published in Google File System (GFS) and MapReduce white papers Hadoop's core component, Hadoop Distributed File System (HDFS) is the counterpart of GFS 20

21 Chapter 2 - Hadoop Overview Hadoop uses functionally equivalent to Google's MapReduce data processing system also called MapReduce (term coined by Google's engineers) One of the main principle of Hadoop's architecture is "design for failure" To deliver high-availability quality of service, Hadoop detects and handles failures at the application layer (rather than relying on hardware) 2.6 Hadoop's Core Components The Hadoop project is made up of the following main components: Common Contains Hadoop infrastructure elements (interfaces with HDFS, system libraries, RPC connectors, Hadoop admin scripts, etc.) Hadoop Distributed File System (HDFS) Hadoop's persistence component designed to run on clusters of commodity hardware built around the "load once and read many times" concept MapReduce A distributed data processing framework used as data analysis system 2.7 Hadoop Simple Definition In a nutshell, Hadoop is a distributed computing framework that consists of: Reliable data storage (provided via HDFS) Analysis system (provided by MapReduce) 21

22 Chapter 2 - Hadoop Overview 2.8 High-Level Hadoop Architecture Notes: The HDFS NameNode acts as a meta-data server for all information stored in HDFS on data nodes. The MapReducer master is responsible for allocating the map and reduce jobs in a distributed environment in a most efficient way. 2.9 Hadoop-based Systems for Data Analysis Hadoop (via HDFS) can host the following systems for data analysis: The MapReduce engine (the major data analytics component of the Hadoop project) Apache Pig HBase database Apache Hive data warehouse system Apache Mahout machine learning system etc. 22

23 Chapter 2 - Hadoop Overview 2.10 Hadoop Caveats Hadoop is a batch-oriented processing system Many Hadoop-centric business analytics systems have high processing latency which comes from their dependencies on the MapReduce subsystem Querying even small data sets (under a gigabyte in size) may take up to several minutes This is in sharp contrast with querying speed of relational databases such as MySQL, Oracle, DB2, etc. where functionally similar work can be done several orders of magnitude faster by applying indexes and other techniques Systems that bypass MapReduce when building queries have near realtime querying times (e.g. HBase and Cloudera Impala) 2.11 Summary Apache Hadoop is a distributed fault-tolerant computing platform written in Java used for processing large data sets across clusters of computers Hadoop clusters may comprise thousands of nodes One of the main principle of Hadoop's architecture is "design for failure" 23

24

25 Chapter 3 - Hadoop Distributed File System Overview Objectives In this chapter, participants will learn about: Hadoop Distributed File System (HDFS) Ways to access HDFS 3.1 Hadoop Distributed File System The Hadoop Distributed File System (HDFS) is a distributed, scalable, fault-tolerant and portable file system written in Java HDFS's architecture is based on the master/slave design pattern An HDFS cluster consists of: Notes: A single NameNode (metadata server) holding directory information about files on DataNodes A number of DataNodes, usually one per machine in the cluster HDFS design is "rack-aware" to minimize network latency HDFS includes a server called a secondary NameNode, which is not a fail-over NameNode, rather, the secondary NameNode connects to the primary NameNode at specified intervals and pulls out the primary NameNode's directory information for building file system current snapshots. These snapshots can be used to restart a failed primary NameNode from the most recent checkpoint without having to replay the entire journal of file-system actions. The secondary NameNode has been deprecated in favor of the newly introduced Checkpoint node which takes over the former's tasks and adds more functionality. Work is under way to provide automatic fail-over for the NameNode to prevent a single point of failure of a cluster. Rack-awareness means taking into account a machine's physical location (rack) while scheduling tasks and allocating storage. Basically, HDFS is aware of the fact that network bandwidth between machines sitting in the same server rack is greater than that between machines in different racks. The creators of HDFS made a design decision in favor of using machines with internal hard drives as it helps ensure data locality (data is stored on the hard drive of the machine which CPU is used for processing it, e.g. with MapReduce). For that reason, Storage Area Network (SAN) or similar storage technologies are not recommended for performance considerations.

26 Chapter 3 - Hadoop Distributed File System Overview 3.2 Hadoop Distributed File System HDFS is designed for efficient implementation of the following data processing pattern: Write once (normally, just load the data set on the file system) Read many times (for data analysis, etc.) HDFS functionality (and that of Hadoop) is geared towards batch-oriented rather than real-time scenarios HDFS does not have a built-in cache Processing of data-intensive jobs can be done in parallel 3.3 Data Blocks A data file in HDFS is split into blocks (a typical block size used by HDFS is 64 MB) For large files, bigger block sizes will help reduce the amount of metadata stored in the NameNode (metadata server) Data reliability is achieved by replicating data blocks across multiple machines By default, data blocks get replicated to three nodes: two on the same server rack, and one on a different rack for redundancy 26

27 Chapter 3 - Hadoop Distributed File System Overview 3.4 Data Block Replication Example Example of a three-way (default) replication of a single data block for redundancy and achieving high data availability (the block size is usually a multiple of 64M) 3.5 HDFS NameNode Directory Diagram Notes: The HDFS NameNode maintains the directory metadata about all data nodes and data blocks stored for each file registered on HDFS. 27

28 Chapter 3 - Hadoop Distributed File System Overview 3.6 Accessing HDFS HDFS is not a Portable Operating System Interface (POSIX) compliant file system It is modeled after traditional hierarchical file systems containing directories and files File-system commands (copy, move, delete, etc.) against HDFS can be performed in a number of ways: Through the HDFS Command-Line Interface (CLI) which supports Unixlike commands: cat, chown, ls, mkdir, mv, etc. Using Java API for HDFS Via the C-language wrapper around the Java API Using regular HTTP browser for file-system and file content viewing This is made possible through Web server (Jetty) embedded in the NameNode and DataNodes 3.7 Examples of HDFS Commands The common way to invoke the HDFS CLI: hadoop fs {HDFS commands} Copying a file from the local file system over to HDFS (with the same name): hadoop fs -put <filename_on_local_sysetm> Copying the file from HDFS to the local file system (with the same name): hadoop fs -get <filename_on_hdfs> Recursively listing files in a directory: hadoop fs -ls -R <directory_on_hdfs> Creating a directory under the current user's home directory (e.g. /user/userid/) on HDFS hadoop fs -mkdir REPORT 28

29 Chapter 3 - Hadoop Distributed File System Overview Notes: Instead of the put command, you can use the functionally equivalent but more descriptive copyfromlocal command, and the get command also has its more descriptive equivalent: copytolocal. 3.8 Client Interactions with HDFS for the Read Operation Whenever a Hadoop client needs to read a file on HDFS, it first contacts the NameNode The NameNode locates block ids associated with the file as well as IP address of the DataNodes storing those blocks The NameNode returns the related information to the client The client contacts the related DataNodes and supplies the block ids for DataNodes to locate the blocks on their local HDFS storage The DataNodes serve the blocks of data back to the client 3.9 Read Operation Sequence Diagram 29

30 Chapter 3 - Hadoop Distributed File System Overview 3.10 Client Interactions with HDFS for the Write Operation Notes: Whenever a Hadoop client needs to write a file to HDFS, it first contacts the NameNode The NameNode generates and registers meta-data about the file and allocates suitable DataNodes to keep the file's replicas Information about the allocated DataNodes is sent back to the client Client uses HDFS I/O facilities to stream content to the first DataNode in the list (the "primary" DataNode) The "primary" DataNode makes a local copy of related data and engages other DataNodes in a peer-to-peer data sharing communication for data replication Each DataNode sends acknowledgments of receiving their data back to the "primary" DataNode The client gets an acknowledgment from the "primary" DataNode as a confirmation of the HDFS write operation The client notifies the NameNode on completion of the operation Once all DataNodes receive their replicas of the file, they cache the content in their memory (on Java Heap) and send acknowledgments to the "primary" DataNode. The actual data persistence to the local file system is done by DataNodes asynchronously at a later time Communication inside HDFS HDFS is designed for batch processing rather than for interactive use This affected the design of communication protocols inside HDFS Communication inside HDFS is modeled after the Remote Procedure Call (RPC) application protocol layered on top of the TCP/IP protocol 3.12 Summary File-system commands against HDFS can be performed in a number of 30

31 Chapter 3 - Hadoop Distributed File System Overview ways: Through the fs command-line interface (which supports Unix-like commands: cat, chown, ls, mkdir, mv, etc.) Using Java API for HDFS Via the C-language wrapper around the Java API 31

32