Istanbul Şehir University Big Data Camp 14 HDFS: Hadoop Distributed File System Aslan Bakirov Kevser Nur Çoğalmış
Agenda Distributed File System HDFS Concepts HDFS Interfaces HDFS Full Picture Read Operation WorkFlow Network Topology and Hadoop Write File WorkFlow Future Concepts Demo Q&A
Distributed File System What is distributed systems? A network of interconnected computers is distributed systems A single computer can also be viewed as a distributed system in which the central control unit, memory units and IO channels are separate processes A system is distributed if the message transmission delay is not negligible compared to the time between events in a single process
Distributed File System File systems that manage the storage across a network of machines are called distributed file systems Must be; Consistent (all nodes see the same data at the same time) Partition tolerant (the system continues to operate despite arbitrary message loss or failure of part of the system)
HDFS Concepts Hadoop Distributed File System: part of Apache Hadoop Project Two Types of Nodes NameNode(Master) - Holds metadata and keeps track of block locations in DataNodes DataNode - Slave nodes that store and retrieve data blocks. DNs periodically report to namenode about list of blocks that they are storing Files split into 128mb(default) blocks Replicated to 3 datanodes(default)
HDFS Concepts HDFS is good for: Very Large Files: Hdfs is designed and optimized for very large files, in size of GBs, TBs, PBs etc. Streaming Data Access: Hdfs is good for write once and read many times pattern. Reading whole dataset is more important than reading particular block.
HDFS Concepts HDFS is bad for: Low-latency data Access: HDFS is designed for high throughput of data, HBase is better for low latency data Access. Lots of small files: Since the namenode holds metadata in memory, the limit to the number of files in file system is governed by amount of memory on NN. Each file costs about 150 Bytes of memory. Multiple writers: Writes are always done at the end of file. There are no support for multiple writers at arbitrary offsets in the file.
Block HDFS Concepts A disk has a block size, minimum amount of data that it can read or write. (512 bytes by default). HDFS has block size of 128MB by default but it is configurable. Files in HDFS are broken into block-sized chunks and stored as independent units. Unlike local file system, a file in HDFS smaller than a single block does not occupy full block. hadoop fsck / -files blocks: list blocks make up each file in hdfs
HDFS Interfaces (Most Used Ones) C: C interface uses libhdfs library which uses JNI to call Java file system client FUSE: File system in Userspace (FUSE) allows file system that are implemented in user space to be mounted as a UNIX file system. FUSE-DFS allows any HDFS to be mounted as a UNIX file system WebDAV: WebDAV is a set of extensions to http to support editing and retrieving files in HDFS Java API: FileSystem, FSDataInputStream and FSDataOutputStream classes are used to read and write data to HDFS
HDFS Full Picture Image Source: http://yoyoclouds.files.wordpress.com/2011/12/hadoop_arch.png
Read Operation WorkFlow Image Source: Hadoop: The Definitive Guide Book, P: 63
Read Operation Workflow 1. HDFS client opens the file it wishes to read by calling open() on DistributedFileSystem for HDFS 2. DistributedFileSystem calls the namenode using RPC, to determine the locations of the blocks 3. Namenode returns the addresses of the datanodes that have copy of that block. (Datanodes are sorted according to the proximity to the client. Nearest one first). If the client is datanode itself, then it will read from local datanode, if it has the copy of block
Read Operation Workflow 4. The DistributedFileSystem returns an FSDataInputStream to the client to read blocks 5. The client then calls read() method on the stream. FSDataInputStream connects to the first nearest datanode for the first block. When end of stream reaches, DFSInputStream closes connection to that datanode and connects to the next datanode for the next block. This process are done for all blocks of file 6. DFSInputStream also does checksum over data it receives from datanode. And if a corrupted block found, it will reported to the namenode before start reading same block from another datanode
Network Topology And Hadoop What we mean close about the distance between two nodes in datacenter? Hadoop takes network as a tree and levels of tree are datacenter, rack and node Bandwidth available for each of the following scenarios becomes progressively less Process on the same node Different nodes on the same rack Nodes of different racks in the same datacenter Nodes in different datacenter
Network Topology And Hadoop Image Source: Hadoop: The Definitive Guide Book, P: 65
Write Operation Workflow Image Source: Hadoop: The Definitive Guide Book, P: 66
Write Operation Workflow 1. Client creates the file by calling create() on DistributedFileSystem. 2. DistributedFilesystem makes an RPC call to the namenode to create a new file in the file system s namespace with no blocks associated with it. 3. Namenodes makes some validation checks like, file already exist or permission problems. 4. If checks are passed, namenode makes a record of new file. 5. DistributedFileSystem returns FSDataOutputStream to the client to start writing.
Write Operation Workflow 6. As client writes data, FSDataOutputStream splits it into packets and writes them to an internal queue called data queue. 7. The data queue is consumed by the DataStreamer, whose responsibility is to ask the namenode to allocate new blocks by picking list of suitable datanodes to store the replicas. 8. The list of datanodes forms a pipeline we will assume that replication parameter is three. 9. The DataStreamer streams the packets to the first datanode in the pipeline, which stores the packet and forwards it to the second datanode in the pipeline. Similarly second node stores and forwards to the third node in the pipeline.
Write Operation Workflow 10. DFSOutputStream (FSDataOutputStream) also maintains the internal queue of packets that are waiting to be acknowledged by datanodes, called ack queue. 11. The packet is removed from ack queue only when it has been acknowledged by all the datanodes in the pipeline 12. When client finishes writing data, it calls close() on the stream
Write Operation Workflow Image Source: Hadoop: The Definitive Guide Book, P: 68
Future Concepts Resource Management Security
Demo