Distributed File Systems
|
|
|
- Whitney Boone
- 8 years ago
- Views:
Transcription
1 Distributed File Systems Alemnew Sheferaw Asrese University of Trento - Italy December 12, 2012 Acknowledgement: Mauro Fruet Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 1 / 55
2 Outline 1 Distributed File Systems Introduction Most Important DFS 2 The Google File System (GFS) Introduction Design Overview System Interactions Master Operation Fault Tolerance and Diagnosis 3 The Hadoop Distributed File System (HDFS) Introduction Architecture File I/O Operations and Replica Management Writing and Reading Data Replica Placement 4 Conclusions 5 Bibliography Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 2 / 55
3 Distributed File Systems Introduction Introduction Definition of DFS File system that allows access to files from multiple hosts via a computer network Features Share files and storage resources on multiple machines No direct access to data storage for clients Clients interact using a protocol Access restrictions to file system Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 3 / 55
4 Distributed File Systems Introduction DFS Goals Access transparency Location transparency Concurrent file updates File replication Hardware and software hetereogenity Fault tolerance Consistency Security Efficiency Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 4 / 55
5 Distributed File Systems Most Important DFS Most Important DFS NFS (Network File System), developed by Sun Microsystems in 1984 Small number of clients, very small number of servers Any machine can be a client and/or a server Stateless file server with few exceptions (file locking) High performance AFS (Andrew File System), developed by the Carnegie Mellon University Support sharing on a large scale (up to 10,000 users) Client machines have disks (cache files for long periods) Most files are small Reads more common than writes Most files read/written by one user Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 5 / 55
6 Distributed File Systems Most Important DFS Need for Large Streams of Data Standard DFS not adequate to manage large streams of data: Google : 24 PB/day Facebook: 130 TB/day for logs, 500+ TB/day new date entry, 300 M photos uploaded per day, TB data scanned in Hive per day LHC-CERN: 25 PB/year Such workloads need different assumptions and goals. Proposed solutions: The Google File System (GFS) Hadoop Distributed File System (HDFS) by Apache Software Foundation Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 6 / 55
7 The Google File System (GFS) Introduction The Google File System (GFS) [1] Proprietary DFS, only key ideas available Developed and implemented by Google in 2003 Shares many goals with standard DFS (performance, scalability, reliability, availability,...) Needed to manage data-intensive applications Provides fault tolerance Provides high performance Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 7 / 55
8 The Google File System (GFS) Introduction GFS - Motivations Component failures are the norm: A storage cluster is built from hundreds or thousands of inexpensive commodity servers Constant monitoring, error detection, fault tolerance, and automatic recovery is needed Huge files (Multi- GB) with many application objects Appending new data rather than overwriting existing Co-designing the applications and the file system API to increase flexibility Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 8 / 55
9 The Google File System (GFS) Design Overview Assumptions Composed of inexpensive commodity components that often fail Modest number of huge files (Multi GBs) Large streaming reads and small random reads Sequential writes that append to files rather than overwrite Files seldom modified again Semantics for concurrently append to the same file by multiple clients High bandwidth more important than low latency Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 9 / 55
10 The Google File System (GFS) Design Overview GFS Interface Familiar file system interface Don t implement Standard API such as POSIX Supports Common Operations (create, delete,open, close, read, and write files) Snapshot Record append Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 10 / 55
11 Architecture The Google File System (GFS) Design Overview Figure: GFS Architecture Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 11 / 55
12 The Google File System (GFS) Design Overview Architecture Master has file system metadata Master controls: Chunk lease management Garbage collection of Orphaned chunks Chunk migration Periodically HeartBeat messages exchanged between master and chunkservers to check the state of chunkserver Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 12 / 55
13 The Google File System (GFS) Design Overview Single Master Enables to make sophisticated chunk placement and replication decisions Must minimize its involvement in reads and writes Clients never read and write file data through the master Client asks the master which chunkservers it should contact Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 13 / 55
14 The Google File System (GFS) Design Overview Single Master Cont... Problems Bandwidth bottleneck A single point of failure Opportunities Central place to control replication and garbage collection and many other activities Solution Around 2009 a Distributed Master is proposed and implemented for the current schema. BigTable and file count are being used. Details are not clearly available :) Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 14 / 55
15 The Google File System (GFS) Design Overview ChunkServer Large chunk size 64 MB - larger than typical File system Large chunk size: Pros Cons Reduces client master interaction Reduce network overhead by keeping a persistent TCP connection Reduces the size of the metadata stored on the master internal fragmentation (solution: lazy space allocation) large chunks can become hot spots (solutions: higher replication factor, staggering application start time, client-to-client communication) Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 15 / 55
16 Metadata The Google File System (GFS) Design Overview All metadata is kept in main memory of the master Namespace File-to-chunk mapping Chunks replicas location Access control information Namespace and file-to-chunk mapping are persistently stored Mutations logged to an operation log No persistent record of replicas locations Operation Log: Contains critical metadata changes Defines order of concurrent operations Replicated on multiple remote machines Used to recover the master Checkpoint of master if log beyond certain size: Master switches to a new log Checkpoint created with all mutations before switch Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 16 / 55
17 The Google File System (GFS) Design Overview Consistency Model File namespace mutations are atomic Namespace locking guarantees atomicity and correctness Mutations applied in same order on all replicas Chunk version numbers used to detect any stale replica (missed mutations while chunkserver was down) Stale replicas are garbage collected Checksum for detection of data corruption A file region is consistent if all the clients will always see the same data regardless of which replicas they read from. Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 17 / 55
18 The Google File System (GFS) System Interactions Leases and Mutation Order Master uses leases to maintain consistent mutation order Master grants a chunk lease to a replica, called primary Primary chooses serial order between Mutations Lease grant order defined by the master Order within a lease defined by the primary Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 18 / 55
19 The Google File System (GFS) Control Flow of a Write System Interactions Figure: Write Control and Data Flow Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 19 / 55
20 Data Flow The Google File System (GFS) System Interactions Separated flow of data and control Goals: 1 Fully use outgoing bandwidth of each machine: Data pushed in a chain of chunkservers 2 Avoid network bottlenecks and high-latency links: Each machine forwards data to closest machine 3 Minimize latency: Once a chunkserver receives data, it starts forwarding immediately Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 20 / 55
21 The Google File System (GFS) System Interactions Atomic Record Appends Traditional Write Client specifies the offset Concurrent writes to the same region are not serializable Record Append Client specifies only the data GFS Appends it to the file at least once atomically at an offset of GFSs choosing and returns that offset to the client Used by Distributed application in which many clients on different machines append to the same file concurrently Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 21 / 55
22 Snapshot The Google File System (GFS) System Interactions Used to: 1 Quickly create copies of huge data sets 2 Easily checkpoint current state The master receives snapshot request: 1 Revokes leases on chunks of the files it is about to snapshot 2 Logs the operation to disk 3 Duplicates the corresponding metadata New snapshot file points to same chunk C of source file New chunk C created for first write after snapshot Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 22 / 55
23 The Google File System (GFS) Master Operation Namespace Management and Locking No per-directory data structure that lists all files in the directory Doesn t support aliases for the same file (No hard or symbolic links) Namespace maps full pathnames to metadata Each master operation acquires a set of locks before it runs Read lock on internal nodes and read/write lock on the leaf Locking allows concurrent mutations in the same directory Read lock on directory prevents its deletion, renaming or snapshot Write locks on file names serialize attempts to create a file with the same name twice Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 23 / 55
24 The Google File System (GFS) Master Operation Replica Placement Hundreds of chunkservers across many racks Chunkservers accessed from hundreds of clients Goals: Maximize data reliability and availability Maximize network bandwidth utilization Replicas spread across machines and racks Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 24 / 55
25 The Google File System (GFS) Master Operation Creation, Re-replication and Rebalancing Creation Place new replicas on chunkservers with below-average disk usage Limit number of recent creations on each chunkservers Replication When number of available replicas falls below a user-specified goal Prioritization: based on how far it is from its replication goal, live files than recently deleted files... Rebalancing Periodically, for better disk utilization and load balancing Distribution of replicas is analyzed Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 25 / 55
26 The Google File System (GFS) Master Operation Garbage Collection File deletion logged by master File renamed to a hidden name with deletion timestamp instead of reclaiming the resources immediately Master regularly scan and removes hidden files older than 3 days (configurable) Until then, hidden file can be read and undeleted When a hidden file is removed, its in-memory metadata is erased Orphaned chunks identified removed during a regular scan of the chunk namespace Safety against accidental irreversible deletion Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 26 / 55
27 The Google File System (GFS) Master Operation Stale Replica Detection Need to distinguish between up-to-date and stale replicas Chunk version number: Increased when master grants new lease on the chunk Not increased if replica is unavailable Stale replicas deleted by master in regular garbage collection Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 27 / 55
28 The Google File System (GFS) Fault Tolerance and Diagnosis High Availability Fast Recovery Master and chunkservers have to restore their state and start in seconds no matter how they terminated Chunk Replication Each chunk is replicated on multiple chunkservers on different racks Master Replication Master state replicated for reliability on multiple machines When master fails: It can restart almost instantly A new master process is started elsewhere Shadow (not mirror) master provides only read-only access to file system when primary master is down Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 28 / 55
29 The Google File System (GFS) Fault Tolerance and Diagnosis Data Integrity Checksumming to detect corruption of stored data Integrity verified by each chunkserver independently Corruptions not propagated to other chunkservers Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 29 / 55
30 The Google File System (GFS) Fault Tolerance and Diagnosis Diagnostic Tools Extensive and detailed diagnostic logging Diagnostic logs record many significant events(chunkservers going up and down, RPC requests and replies)... can be easily deleted without affecting the system Bibliography Case Study, GFS: Evolution on Fast-forward, A discussion between Kirk McKusick and Sean Quinlan; ACM Queue, 2009 Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 30 / 55
31 The Hadoop Distributed File System (HDFS) Introduction What is Hadoop? Summary An Apache project Yahoo! has developed and contributed to 80% of the core of Hadoop (HDFS and MapReduce). Components: HBase - originally by Powerset, then Microsoft Hive - Facebook Pig, ZooKeeper, Chukwa - Yahoo! Avro - Originally Yahoo, and co-developed with cloudera Uses MapReduce Distributed Computation framework Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 31 / 55
32 The Hadoop Distributed File System (HDFS) Introduction The Hadoop Distributed File System (HDFS) [2] Sub-project of Apache Hadoop project Designed to store very large data sets separately: Metadata on dedicated server called NameNode Application data on DataNodes Creates multiple replicas of data blocks Distributes replicas on nodes Highly fault-tolerant Designed to run on commodity hardware Suitable for applications for large data sets Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 32 / 55
33 Alemnew S. Asrese (UniTN) Figure: Distributed HDFSFile Architecture Systems 2012/12/12 33 / 55 Architecture The Hadoop Distributed File System (HDFS) Architecture
34 The Hadoop Distributed File System (HDFS) Architecture NameNode Namespace is a hierarchy of files and directories Files and directories are represented on the NameNode by inodes Maintains the namespace tree and the mapping of file blocks to DataNodes Authorization and authentication A multithreaded system and processes requests simultaneously from multiple clients Keeps the entire namespace in RAM, allowing fast access to the metadata. Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 34 / 55
35 The Hadoop Distributed File System (HDFS) Architecture DataNodes Responsible for serving read and write requests from the file systems clients Perform block creation, deletion, and replication upon instruction from the NameNode Performs handshake by connecting with NameNode during startup To verify the Namespace ID and software version of the DataNode Nodes with a different namespace ID will not be able to join the cluster Incompatible version may cause data corruption or loss Send heartbeats to the NameNode to confirm that the DataNode is operating and the block replicas it hosts are available. Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 35 / 55
36 HDFS Client The Hadoop Distributed File System (HDFS) Architecture Figure: HDFS Client, NameNode and DataNode interaction for creating a file Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 36 / 55
37 The Hadoop Distributed File System (HDFS) Architecture Image and Journal Image File system metadata that describes the organization of application data as directories and files A persistent record of the image written to disk is called a checkpoint Journal A write-ahead commit log for changes to the file system that must be persistent During startup the NameNode initializes the namespace image from the checkpoint, and then replays changes from the journal A new checkpoint and empty journal are written back to the storage directories before the NameNode starts serving clients Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 37 / 55
38 The Hadoop Distributed File System (HDFS) Architecture CheckpointNode Periodically combines the existing checkpoint and journal to create a new checkpoint and an empty journal Usually runs on a different host wrt NameNode Downloads the current checkpoint and journal files from the NameNode, Merges them locally,and returns the new checkpoint back to the NameNode Multiple CheckPoint Nodes may be used simultaneously Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 38 / 55
39 The Hadoop Distributed File System (HDFS) Architecture BackupNode Create periodic checkpoints As Checkpoint Node, but it applies edits to its own namespace copy Without downloading checkpoint and journal files from the active NameNode But in addition it maintains an in-memory, up-to-date image of the file system namespace Always synchronized with the state of the NameNode Can be viewed as a read-only NameNode Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 39 / 55
40 The Hadoop Distributed File System (HDFS) File Read and Write, Block Placement File I/O Operations and Replica Management File Read and Write Implements a single-writer, multiple-reader model Block Placement Spread the nodes across multiple racks Replicas placed: The first on the node where the writer located, The second and third on different nodes at different racks The others palced on random nodes: 1 No Datanode contains more than one replica of any block 2 No rack contains more than two replicas are in the same rack Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 40 / 55
41 The Hadoop Distributed File System (HDFS) File I/O Operations and Replica Management Replication management Detects that a block has become under- or over-replicated When a block becomes over replicated, the NameNode chooses a replica to remove When a block becomes under-replicated, it is put in the replication priority queue A block with only one replica has the highest priority A block with a number of replicas that is greater than two thirds of its replication factor has the lowest priority Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 41 / 55
42 The Hadoop Distributed File System (HDFS) File I/O Operations and Replica Management Inter-Cluster Data Copy A tool called DistCp used for large inter/intra-cluster parallel copying A MapReduce job; each of the map tasks copies a portion of the source data into the destination file system Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 42 / 55
43 The Hadoop Distributed File System (HDFS) Writing Data in HDFS Cluster Writing and Reading Data Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 43 / 55
44 The Hadoop Distributed File System (HDFS) Writing Data in HDFS Cluster Writing and Reading Data Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 44 / 55
45 The Hadoop Distributed File System (HDFS) Writing Data in HDFS Cluster Writing and Reading Data Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 45 / 55
46 The Hadoop Distributed File System (HDFS) Reading Data in HDFS Cluster Writing and Reading Data Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 46 / 55
47 The Hadoop Distributed File System (HDFS) Types of Faults and Their Detection Writing and Reading Data Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 47 / 55
48 The Hadoop Distributed File System (HDFS) Types of Faults and Their Detection Writing and Reading Data Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 48 / 55
49 The Hadoop Distributed File System (HDFS) Writing and Reading Data Handling Reading and Writing Failures Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 49 / 55
50 The Hadoop Distributed File System (HDFS) Handling DataNode Failures Writing and Reading Data Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 50 / 55
51 The Hadoop Distributed File System (HDFS) Replica Placement Strategy Replica Placement Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 51 / 55
52 The Hadoop Distributed File System (HDFS) Replica Placement Cluster Rebalancing Balanced cluster: no under-utilized or over-utilized DataNodes Common reason: addition of new DataNodes Requirements No block loss No change of the number of replicas No reduction of the number of racks a block resides in No saturation of the network Rebalancing issued by an administrator Rebalancing decisions made by a Rebalancing Server Goal of each iteration: reduce imbalance in over/under-utilized DataNodes Source and destination selection based on some priorities Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 52 / 55
53 The Hadoop Distributed File System (HDFS) Replica Placement Balancer A tool that balances disk space usage on an HDFS cluster. Iteratively moves replicas from DataNodes with higher utilization to DataNodes with lower utilization Bibliography http: // hadoop. apache. org/ docs/ hdfs Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 53 / 55
54 Conclusions Conclusions GFS Proprietary file system All the details in the paper of 2003 HDFS Open-source project Details are available in the paper of 2010 and in Hadoop website Used by many big companies Inspired by GFS Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 54 / 55
55 Bibliography Bibliography Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. The Google File System. In Proceedings of the Nineteenth ACM Symposium on Operating Systems Principles, SOSP 03, pages 29 43, New York, NY, USA, ACM. Shvachko Konstantin, Kuang Hairong, Radia Sanjay, and Robert Chansler. The Hadoop Distributed File System. In Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), MSST 10, pages 1 10, Washington, DC, USA, IEEE Computer Society. Alemnew S. Asrese (UniTN) Distributed File Systems 2012/12/12 55 / 55
Distributed File Systems
Distributed File Systems Mauro Fruet University of Trento - Italy 2011/12/19 Mauro Fruet (UniTN) Distributed File Systems 2011/12/19 1 / 39 Outline 1 Distributed File Systems 2 The Google File System (GFS)
THE HADOOP DISTRIBUTED FILE SYSTEM
THE HADOOP DISTRIBUTED FILE SYSTEM Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler Presented by Alexander Pokluda October 7, 2013 Outline Motivation and Overview of Hadoop Architecture,
Lecture 5: GFS & HDFS! Claudia Hauff (Web Information Systems)! ti2736b-ewi@tudelft.nl
Big Data Processing, 2014/15 Lecture 5: GFS & HDFS!! Claudia Hauff (Web Information Systems)! ti2736b-ewi@tudelft.nl 1 Course content Introduction Data streams 1 & 2 The MapReduce paradigm Looking behind
The Hadoop Distributed File System
The Hadoop Distributed File System The Hadoop Distributed File System, Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler, Yahoo, 2010 Agenda Topic 1: Introduction Topic 2: Architecture
The Hadoop Distributed File System
The Hadoop Distributed File System Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler Yahoo! Sunnyvale, California USA {Shv, Hairong, SRadia, Chansler}@Yahoo-Inc.com Presenter: Alex Hu HDFS
Distributed File System. MCSN N. Tonellotto Complements of Distributed Enabling Platforms
Distributed File System 1 How do we get data to the workers? NAS Compute Nodes SAN 2 Distributed File System Don t move data to workers move workers to the data! Store data on the local disks of nodes
Distributed File Systems
Distributed File Systems Paul Krzyzanowski Rutgers University October 28, 2012 1 Introduction The classic network file systems we examined, NFS, CIFS, AFS, Coda, were designed as client-server applications.
Journal of science STUDY ON REPLICA MANAGEMENT AND HIGH AVAILABILITY IN HADOOP DISTRIBUTED FILE SYSTEM (HDFS)
Journal of science e ISSN 2277-3290 Print ISSN 2277-3282 Information Technology www.journalofscience.net STUDY ON REPLICA MANAGEMENT AND HIGH AVAILABILITY IN HADOOP DISTRIBUTED FILE SYSTEM (HDFS) S. Chandra
Architecture for Hadoop Distributed File Systems
Architecture for Hadoop Distributed File Systems S. Devi 1, Dr. K. Kamaraj 2 1 Asst. Prof. / Department MCA, SSM College of Engineering, Komarapalayam, Tamil Nadu, India 2 Director / MCA, SSM College of
The Google File System
The Google File System By Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung (Presented at SOSP 2003) Introduction Google search engine. Applications process lots of data. Need good file system. Solution:
Comparative analysis of mapreduce job by keeping data constant and varying cluster size technique
Comparative analysis of mapreduce job by keeping data constant and varying cluster size technique Mahesh Maurya a, Sunita Mahajan b * a Research Scholar, JJT University, MPSTME, Mumbai, India,maheshkmaurya@yahoo.co.in
The Google File System
The Google File System Motivations of NFS NFS (Network File System) Allow to access files in other systems as local files Actually a network protocol (initially only one server) Simple and fast server
Sunita Suralkar, Ashwini Mujumdar, Gayatri Masiwal, Manasi Kulkarni Department of Computer Technology, Veermata Jijabai Technological Institute
Review of Distributed File Systems: Case Studies Sunita Suralkar, Ashwini Mujumdar, Gayatri Masiwal, Manasi Kulkarni Department of Computer Technology, Veermata Jijabai Technological Institute Abstract
Hadoop Distributed File System. T-111.5550 Seminar On Multimedia 2009-11-11 Eero Kurkela
Hadoop Distributed File System T-111.5550 Seminar On Multimedia 2009-11-11 Eero Kurkela Agenda Introduction Flesh and bones of HDFS Architecture Accessing data Data replication strategy Fault tolerance
Google File System. Web and scalability
Google File System Web and scalability The web: - How big is the Web right now? No one knows. - Number of pages that are crawled: o 100,000 pages in 1994 o 8 million pages in 2005 - Crawlable pages might
HDFS Under the Hood. Sanjay Radia. Sradia@yahoo-inc.com Grid Computing, Hadoop Yahoo Inc.
HDFS Under the Hood Sanjay Radia Sradia@yahoo-inc.com Grid Computing, Hadoop Yahoo Inc. 1 Outline Overview of Hadoop, an open source project Design of HDFS On going work 2 Hadoop Hadoop provides a framework
CS2510 Computer Operating Systems
CS2510 Computer Operating Systems HADOOP Distributed File System Dr. Taieb Znati Computer Science Department University of Pittsburgh Outline HDF Design Issues HDFS Application Profile Block Abstraction
CS2510 Computer Operating Systems
CS2510 Computer Operating Systems HADOOP Distributed File System Dr. Taieb Znati Computer Science Department University of Pittsburgh Outline HDF Design Issues HDFS Application Profile Block Abstraction
HDFS Architecture Guide
by Dhruba Borthakur Table of contents 1 Introduction... 3 2 Assumptions and Goals... 3 2.1 Hardware Failure... 3 2.2 Streaming Data Access...3 2.3 Large Data Sets... 3 2.4 Simple Coherency Model...3 2.5
Take An Internal Look at Hadoop. Hairong Kuang Grid Team, Yahoo! Inc hairong@yahoo-inc.com
Take An Internal Look at Hadoop Hairong Kuang Grid Team, Yahoo! Inc hairong@yahoo-inc.com What s Hadoop Framework for running applications on large clusters of commodity hardware Scale: petabytes of data
HDFS Users Guide. Table of contents
Table of contents 1 Purpose...2 2 Overview...2 3 Prerequisites...3 4 Web Interface...3 5 Shell Commands... 3 5.1 DFSAdmin Command...4 6 Secondary NameNode...4 7 Checkpoint Node...5 8 Backup Node...6 9
Massive Data Storage
Massive Data Storage Storage on the "Cloud" and the Google File System paper by: Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung presentation by: Joshua Michalczak COP 4810 - Topics in Computer Science
Overview. Big Data in Apache Hadoop. - HDFS - MapReduce in Hadoop - YARN. https://hadoop.apache.org. Big Data Management and Analytics
Overview Big Data in Apache Hadoop - HDFS - MapReduce in Hadoop - YARN https://hadoop.apache.org 138 Apache Hadoop - Historical Background - 2003: Google publishes its cluster architecture & DFS (GFS)
Snapshots in Hadoop Distributed File System
Snapshots in Hadoop Distributed File System Sameer Agarwal UC Berkeley Dhruba Borthakur Facebook Inc. Ion Stoica UC Berkeley Abstract The ability to take snapshots is an essential functionality of any
The Hadoop Distributed File System
The Hadoop Distributed File System Konstantin Shvachko, Hairong Kuang, Sanjay Radia, Robert Chansler Yahoo! Sunnyvale, California USA {Shv, Hairong, SRadia, Chansler}@Yahoo-Inc.com Abstract The Hadoop
Prepared By : Manoj Kumar Joshi & Vikas Sawhney
Prepared By : Manoj Kumar Joshi & Vikas Sawhney General Agenda Introduction to Hadoop Architecture Acknowledgement Thanks to all the authors who left their selfexplanatory images on the internet. Thanks
Hadoop Architecture. Part 1
Hadoop Architecture Part 1 Node, Rack and Cluster: A node is simply a computer, typically non-enterprise, commodity hardware for nodes that contain data. Consider we have Node 1.Then we can add more nodes,
Hadoop Distributed File System. Dhruba Borthakur Apache Hadoop Project Management Committee dhruba@apache.org June 3 rd, 2008
Hadoop Distributed File System Dhruba Borthakur Apache Hadoop Project Management Committee dhruba@apache.org June 3 rd, 2008 Who Am I? Hadoop Developer Core contributor since Hadoop s infancy Focussed
Processing of massive data: MapReduce. 2. Hadoop. New Trends In Distributed Systems MSc Software and Systems
Processing of massive data: MapReduce 2. Hadoop 1 MapReduce Implementations Google were the first that applied MapReduce for big data analysis Their idea was introduced in their seminal paper MapReduce:
Design and Evolution of the Apache Hadoop File System(HDFS)
Design and Evolution of the Apache Hadoop File System(HDFS) Dhruba Borthakur Engineer@Facebook Committer@Apache HDFS SDC, Sept 19 2011 Outline Introduction Yet another file-system, why? Goals of Hadoop
IJFEAT INTERNATIONAL JOURNAL FOR ENGINEERING APPLICATIONS AND TECHNOLOGY
IJFEAT INTERNATIONAL JOURNAL FOR ENGINEERING APPLICATIONS AND TECHNOLOGY Hadoop Distributed File System: What and Why? Ashwini Dhruva Nikam, Computer Science & Engineering, J.D.I.E.T., Yavatmal. Maharashtra,
HDFS Space Consolidation
HDFS Space Consolidation Aastha Mehta*,1,2, Deepti Banka*,1,2, Kartheek Muthyala*,1,2, Priya Sehgal 1, Ajay Bakre 1 *Student Authors 1 Advanced Technology Group, NetApp Inc., Bangalore, India 2 Birla Institute
Hadoop Distributed File System. Jordan Prosch, Matt Kipps
Hadoop Distributed File System Jordan Prosch, Matt Kipps Outline - Background - Architecture - Comments & Suggestions Background What is HDFS? Part of Apache Hadoop - distributed storage What is Hadoop?
Big Data Processing in the Cloud. Shadi Ibrahim Inria, Rennes - Bretagne Atlantique Research Center
Big Data Processing in the Cloud Shadi Ibrahim Inria, Rennes - Bretagne Atlantique Research Center Data is ONLY as useful as the decisions it enables 2 Data is ONLY as useful as the decisions it enables
Comparative analysis of Google File System and Hadoop Distributed File System
Comparative analysis of Google File System and Hadoop Distributed File System R.Vijayakumari, R.Kirankumar, K.Gangadhara Rao Dept. of Computer Science, Krishna University, Machilipatnam, India, vijayakumari28@gmail.com
CLOUD- SCALE FILE SYSTEMS THANKS TO M. GROSSNIKLAUS
Google File System (GFS) Data Management in the Cloud CLOUD- SCALE FILE SYSTEMS THANKS TO M. GROSSNIKLAUS While produc@on systems are well disciplined and controlled, users some@mes are not Ghemawat, Gobioff
Jeffrey D. Ullman slides. MapReduce for data intensive computing
Jeffrey D. Ullman slides MapReduce for data intensive computing Single-node architecture CPU Machine Learning, Statistics Memory Classical Data Mining Disk Commodity Clusters Web data sets can be very
Seminar Presentation for ECE 658 Instructed by: Prof.Anura Jayasumana Distributed File Systems
Seminar Presentation for ECE 658 Instructed by: Prof.Anura Jayasumana Distributed File Systems Prabhakaran Murugesan Outline File Transfer Protocol (FTP) Network File System (NFS) Andrew File System (AFS)
CSE-E5430 Scalable Cloud Computing Lecture 2
CSE-E5430 Scalable Cloud Computing Lecture 2 Keijo Heljanko Department of Computer Science School of Science Aalto University keijo.heljanko@aalto.fi 14.9-2015 1/36 Google MapReduce A scalable batch processing
Erlang Distributed File System (edfs)
Indian Institute of Technology Bombay CS 492 : BTP Stage I Erlang Distributed File System (edfs) By: Aman Mangal (100050015) Coordinator: Prof. G. Sivakumar November 30, 2013 Contents Abstract 3 1 Introduction
MASSIVE DATA PROCESSING (THE GOOGLE WAY ) 27/04/2015. Fundamentals of Distributed Systems. Inside Google circa 2015
7/04/05 Fundamentals of Distributed Systems CC5- PROCESAMIENTO MASIVO DE DATOS OTOÑO 05 Lecture 4: DFS & MapReduce I Aidan Hogan aidhog@gmail.com Inside Google circa 997/98 MASSIVE DATA PROCESSING (THE
HDFS Design Principles
HDFS Design Principles The Scale-out-Ability of Distributed Storage SVForum Software Architecture & Platform SIG Konstantin V. Shvachko May 23, 2012 Big Data Computations that need the power of many computers
Cloud Computing at Google. Architecture
Cloud Computing at Google Google File System Web Systems and Algorithms Google Chris Brooks Department of Computer Science University of San Francisco Google has developed a layered system to handle webscale
Distributed Metadata Management Scheme in HDFS
International Journal of Scientific and Research Publications, Volume 3, Issue 5, May 2013 1 Distributed Metadata Management Scheme in HDFS Mrudula Varade *, Vimla Jethani ** * Department of Computer Engineering,
Apache Hadoop FileSystem and its Usage in Facebook
Apache Hadoop FileSystem and its Usage in Facebook Dhruba Borthakur Project Lead, Apache Hadoop Distributed File System dhruba@apache.org Presented at Indian Institute of Technology November, 2010 http://www.facebook.com/hadoopfs
Processing of Hadoop using Highly Available NameNode
Processing of Hadoop using Highly Available NameNode 1 Akash Deshpande, 2 Shrikant Badwaik, 3 Sailee Nalawade, 4 Anjali Bote, 5 Prof. S. P. Kosbatwar Department of computer Engineering Smt. Kashibai Navale
RAID. Tiffany Yu-Han Chen. # The performance of different RAID levels # read/write/reliability (fault-tolerant)/overhead
RAID # The performance of different RAID levels # read/write/reliability (fault-tolerant)/overhead Tiffany Yu-Han Chen (These slides modified from Hao-Hua Chu National Taiwan University) RAID 0 - Striping
Big Data With Hadoop
With Saurabh Singh singh.903@osu.edu The Ohio State University February 11, 2016 Overview 1 2 3 Requirements Ecosystem Resilient Distributed Datasets (RDDs) Example Code vs Mapreduce 4 5 Source: [Tutorials
Index Terms : Load rebalance, distributed file systems, clouds, movement cost, load imbalance, chunk.
Load Rebalancing for Distributed File Systems in Clouds. Smita Salunkhe, S. S. Sannakki Department of Computer Science and Engineering KLS Gogte Institute of Technology, Belgaum, Karnataka, India Affiliated
Welcome to the unit of Hadoop Fundamentals on Hadoop architecture. I will begin with a terminology review and then cover the major components
Welcome to the unit of Hadoop Fundamentals on Hadoop architecture. I will begin with a terminology review and then cover the major components of Hadoop. We will see what types of nodes can exist in a Hadoop
Accelerating and Simplifying Apache
Accelerating and Simplifying Apache Hadoop with Panasas ActiveStor White paper NOvember 2012 1.888.PANASAS www.panasas.com Executive Overview The technology requirements for big data vary significantly
INTERNATIONAL JOURNAL OF PURE AND APPLIED RESEARCH IN ENGINEERING AND TECHNOLOGY
INTERNATIONAL JOURNAL OF PURE AND APPLIED RESEARCH IN ENGINEERING AND TECHNOLOGY A PATH FOR HORIZING YOUR INNOVATIVE WORK A COMPREHENSIVE VIEW OF HADOOP ER. AMRINDER KAUR Assistant Professor, Department
Data-Intensive Computing with Map-Reduce and Hadoop
Data-Intensive Computing with Map-Reduce and Hadoop Shamil Humbetov Department of Computer Engineering Qafqaz University Baku, Azerbaijan humbetov@gmail.com Abstract Every day, we create 2.5 quintillion
CSE 590: Special Topics Course ( Supercomputing ) Lecture 10 ( MapReduce& Hadoop)
CSE 590: Special Topics Course ( Supercomputing ) Lecture 10 ( MapReduce& Hadoop) Rezaul A. Chowdhury Department of Computer Science SUNY Stony Brook Spring 2016 MapReduce MapReduce is a programming model
Department of Computer Science University of Cyprus EPL646 Advanced Topics in Databases. Lecture 14
Department of Computer Science University of Cyprus EPL646 Advanced Topics in Databases Lecture 14 Big Data Management IV: Big-data Infrastructures (Background, IO, From NFS to HFDS) Chapter 14-15: Abideboul
Hadoop & its Usage at Facebook
Hadoop & its Usage at Facebook Dhruba Borthakur Project Lead, Hadoop Distributed File System dhruba@apache.org Presented at the The Israeli Association of Grid Technologies July 15, 2009 Outline Architecture
Hadoop & its Usage at Facebook
Hadoop & its Usage at Facebook Dhruba Borthakur Project Lead, Hadoop Distributed File System dhruba@apache.org Presented at the Storage Developer Conference, Santa Clara September 15, 2009 Outline Introduction
Role of Cloud Computing in Big Data Analytics Using MapReduce Component of Hadoop
Role of Cloud Computing in Big Data Analytics Using MapReduce Component of Hadoop Kanchan A. Khedikar Department of Computer Science & Engineering Walchand Institute of Technoloy, Solapur, Maharashtra,
Hadoop Distributed File System. Dhruba Borthakur Apache Hadoop Project Management Committee dhruba@apache.org dhruba@facebook.com
Hadoop Distributed File System Dhruba Borthakur Apache Hadoop Project Management Committee dhruba@apache.org dhruba@facebook.com Hadoop, Why? Need to process huge datasets on large clusters of computers
International Journal of Advance Research in Computer Science and Management Studies
Volume 2, Issue 8, August 2014 ISSN: 2321 7782 (Online) International Journal of Advance Research in Computer Science and Management Studies Research Article / Survey Paper / Case Study Available online
Parallel Processing of cluster by Map Reduce
Parallel Processing of cluster by Map Reduce Abstract Madhavi Vaidya, Department of Computer Science Vivekanand College, Chembur, Mumbai vamadhavi04@yahoo.co.in MapReduce is a parallel programming model
AN EFFICIENT APPROACH FOR REMOVING SINGLE POINT OF FAILURE IN HDFS
AN EFFICIENT APPROACH FOR REMOVING SINGLE POINT OF FAILURE IN HDFS Pranav Nair, Jignesh Vania LJ Institute of Engineering and Technology Ahmedabad, Gujarat, India Abstract: -HDFS is a scalable, highly
DATA MINING WITH HADOOP AND HIVE Introduction to Architecture
DATA MINING WITH HADOOP AND HIVE Introduction to Architecture Dr. Wlodek Zadrozny (Most slides come from Prof. Akella s class in 2014) 2015-2025. Reproduction or usage prohibited without permission of
Hadoop Distributed File System (HDFS) Overview
2012 coreservlets.com and Dima May Hadoop Distributed File System (HDFS) Overview Originals of slides and source code for examples: http://www.coreservlets.com/hadoop-tutorial/ Also see the customized
Data-Intensive Programming. Timo Aaltonen Department of Pervasive Computing
Data-Intensive Programming Timo Aaltonen Department of Pervasive Computing Data-Intensive Programming Lecturer: Timo Aaltonen University Lecturer timo.aaltonen@tut.fi Assistants: Henri Terho and Antti
GraySort and MinuteSort at Yahoo on Hadoop 0.23
GraySort and at Yahoo on Hadoop.23 Thomas Graves Yahoo! May, 213 The Apache Hadoop[1] software library is an open source framework that allows for the distributed processing of large data sets across clusters
HADOOP MOCK TEST HADOOP MOCK TEST I
http://www.tutorialspoint.com HADOOP MOCK TEST Copyright tutorialspoint.com This section presents you various set of Mock Tests related to Hadoop Framework. You can download these sample mock tests at
PERFORMANCE ANALYSIS OF A DISTRIBUTED FILE SYSTEM
PERFORMANCE ANALYSIS OF A DISTRIBUTED FILE SYSTEM SUBMITTED BY DIBYENDU KARMAKAR EXAMINATION ROLL NUMBER: M4SWE13-07 REGISTRATION NUMBER: 117079 of 2011-2012 A THESIS SUBMITTED TO THE FACULTY OF ENGINEERING
Hadoop IST 734 SS CHUNG
Hadoop IST 734 SS CHUNG Introduction What is Big Data?? Bulk Amount Unstructured Lots of Applications which need to handle huge amount of data (in terms of 500+ TB per day) If a regular machine need to
Hadoop Distributed File System. Dhruba Borthakur June, 2007
Hadoop Distributed File System Dhruba Borthakur June, 2007 Goals of HDFS Very Large Distributed File System 10K nodes, 100 million files, 10 PB Assumes Commodity Hardware Files are replicated to handle
Hadoop: Embracing future hardware
Hadoop: Embracing future hardware Suresh Srinivas @suresh_m_s Page 1 About Me Architect & Founder at Hortonworks Long time Apache Hadoop committer and PMC member Designed and developed many key Hadoop
Hadoop Scalability at Facebook. Dmytro Molkov (dms@fb.com) YaC, Moscow, September 19, 2011
Hadoop Scalability at Facebook Dmytro Molkov (dms@fb.com) YaC, Moscow, September 19, 2011 How Facebook uses Hadoop Hadoop Scalability Hadoop High Availability HDFS Raid How Facebook uses Hadoop Usages
!"#$%&' ( )%#*'+,'-#.//"0( !"#$"%&'()*$+()',!-+.'/', 4(5,67,!-+!"89,:*$;'0+$.<.,&0$'09,&)"/=+,!()<>'0, 3, Processing LARGE data sets
!"#$%&' ( Processing LARGE data sets )%#*'+,'-#.//"0( Framework for o! reliable o! scalable o! distributed computation of large data sets 4(5,67,!-+!"89,:*$;'0+$.
Distributed Filesystems
Distributed Filesystems Amir H. Payberah Swedish Institute of Computer Science amir@sics.se April 8, 2014 Amir H. Payberah (SICS) Distributed Filesystems April 8, 2014 1 / 32 What is Filesystem? Controls
NoSQL and Hadoop Technologies On Oracle Cloud
NoSQL and Hadoop Technologies On Oracle Cloud Vatika Sharma 1, Meenu Dave 2 1 M.Tech. Scholar, Department of CSE, Jagan Nath University, Jaipur, India 2 Assistant Professor, Department of CSE, Jagan Nath
Distributed File Systems
Distributed File Systems File Characteristics From Andrew File System work: most files are small transfer files rather than disk blocks? reading more common than writing most access is sequential most
EXPERIMENTATION. HARRISON CARRANZA School of Computer Science and Mathematics
BIG DATA WITH HADOOP EXPERIMENTATION HARRISON CARRANZA Marist College APARICIO CARRANZA NYC College of Technology CUNY ECC Conference 2016 Poughkeepsie, NY, June 12-14, 2016 Marist College AGENDA Contents
Fault Tolerance in Hadoop for Work Migration
1 Fault Tolerance in Hadoop for Work Migration Shivaraman Janakiraman Indiana University Bloomington ABSTRACT Hadoop is a framework that runs applications on large clusters which are built on numerous
Storage Architectures for Big Data in the Cloud
Storage Architectures for Big Data in the Cloud Sam Fineberg HP Storage CT Office/ May 2013 Overview Introduction What is big data? Big Data I/O Hadoop/HDFS SAN Distributed FS Cloud Summary Research Areas
Hadoop: A Framework for Data- Intensive Distributed Computing. CS561-Spring 2012 WPI, Mohamed Y. Eltabakh
1 Hadoop: A Framework for Data- Intensive Distributed Computing CS561-Spring 2012 WPI, Mohamed Y. Eltabakh 2 What is Hadoop? Hadoop is a software framework for distributed processing of large datasets
Hadoop and Map-Reduce. Swati Gore
Hadoop and Map-Reduce Swati Gore Contents Why Hadoop? Hadoop Overview Hadoop Architecture Working Description Fault Tolerance Limitations Why Map-Reduce not MPI Distributed sort Why Hadoop? Existing Data
<Insert Picture Here> Big Data
Big Data Kevin Kalmbach Principal Sales Consultant, Public Sector Engineered Systems Program Agenda What is Big Data and why it is important? What is your Big
Big Data Technology Core Hadoop: HDFS-YARN Internals
Big Data Technology Core Hadoop: HDFS-YARN Internals Eshcar Hillel Yahoo! Ronny Lempel Outbrain *Based on slides by Edward Bortnikov & Ronny Lempel Roadmap Previous class Map-Reduce Motivation This class
Big Data and Hadoop with components like Flume, Pig, Hive and Jaql
Abstract- Today data is increasing in volume, variety and velocity. To manage this data, we have to use databases with massively parallel software running on tens, hundreds, or more than thousands of servers.
There's Plenty of Room in the Cloud
There's Plenty of Room in the Cloud [Shameless reference to Feynman s talk from 1959] Lecturer: Zoran Dimitrijevic Altiscale, Inc. Spring 2015 CS290B -- Cloud Computing 50 Years of Moore
Apache Hadoop. Alexandru Costan
1 Apache Hadoop Alexandru Costan Big Data Landscape No one-size-fits-all solution: SQL, NoSQL, MapReduce, No standard, except Hadoop 2 Outline What is Hadoop? Who uses it? Architecture HDFS MapReduce Open
HFAA: A Generic Socket API for Hadoop File Systems
HFAA: A Generic Socket API for Hadoop File Systems Adam Yee University of the Pacific Stockton, CA adamjyee@gmail.com Jeffrey Shafer University of the Pacific Stockton, CA jshafer@pacific.edu ABSTRACT
A REVIEW: Distributed File System
Journal of Computer Networks and Communications Security VOL. 3, NO. 5, MAY 2015, 229 234 Available online at: www.ijcncs.org EISSN 23089830 (Online) / ISSN 24100595 (Print) A REVIEW: System Shiva Asadianfam
ZooKeeper. Table of contents
by Table of contents 1 ZooKeeper: A Distributed Coordination Service for Distributed Applications... 2 1.1 Design Goals...2 1.2 Data model and the hierarchical namespace...3 1.3 Nodes and ephemeral nodes...
Load Rebalancing for File System in Public Cloud Roopa R.L 1, Jyothi Patil 2
Load Rebalancing for File System in Public Cloud Roopa R.L 1, Jyothi Patil 2 1 PDA College of Engineering, Gulbarga, Karnataka, India rlrooparl@gmail.com 2 PDA College of Engineering, Gulbarga, Karnataka,
Distributed Lucene : A distributed free text index for Hadoop
Distributed Lucene : A distributed free text index for Hadoop Mark H. Butler and James Rutherford HP Laboratories HPL-2008-64 Keyword(s): distributed, high availability, free text, parallel, search Abstract:
What Is Datacenter (Warehouse) Computing. Distributed and Parallel Technology. Datacenter Computing Application Programming
Distributed and Parallel Technology Datacenter and Warehouse Computing Hans-Wolfgang Loidl School of Mathematical and Computer Sciences Heriot-Watt University, Edinburgh 0 Based on earlier versions by
BookKeeper. Flavio Junqueira Yahoo! Research, Barcelona. Hadoop in China 2011
BookKeeper Flavio Junqueira Yahoo! Research, Barcelona Hadoop in China 2011 What s BookKeeper? Shared storage for writing fast sequences of byte arrays Data is replicated Writes are striped Many processes
Hadoop and ecosystem * 本 文 中 的 言 论 仅 代 表 作 者 个 人 观 点 * 本 文 中 的 一 些 图 例 来 自 于 互 联 网. Information Management. Information Management IBM CDL Lab
IBM CDL Lab Hadoop and ecosystem * 本 文 中 的 言 论 仅 代 表 作 者 个 人 观 点 * 本 文 中 的 一 些 图 例 来 自 于 互 联 网 Information Management 2012 IBM Corporation Agenda Hadoop 技 术 Hadoop 概 述 Hadoop 1.x Hadoop 2.x Hadoop 生 态
Hadoop Big Data for Processing Data and Performing Workload
Hadoop Big Data for Processing Data and Performing Workload Girish T B 1, Shadik Mohammed Ghouse 2, Dr. B. R. Prasad Babu 3 1 M Tech Student, 2 Assosiate professor, 3 Professor & Head (PG), of Computer
Lecture 2 (08/31, 09/02, 09/09): Hadoop. Decisions, Operations & Information Technologies Robert H. Smith School of Business Fall, 2015
Lecture 2 (08/31, 09/02, 09/09): Hadoop Decisions, Operations & Information Technologies Robert H. Smith School of Business Fall, 2015 K. Zhang BUDT 758 What we ll cover Overview Architecture o Hadoop
Introduction to Hadoop HDFS and Ecosystems. Slides credits: Cloudera Academic Partners Program & Prof. De Liu, MSBA 6330 Harvesting Big Data
Introduction to Hadoop HDFS and Ecosystems ANSHUL MITTAL Slides credits: Cloudera Academic Partners Program & Prof. De Liu, MSBA 6330 Harvesting Big Data Topics The goal of this presentation is to give
CDH AND BUSINESS CONTINUITY:
WHITE PAPER CDH AND BUSINESS CONTINUITY: An overview of the availability, data protection and disaster recovery features in Hadoop Abstract Using the sophisticated built-in capabilities of CDH for tunable
Highly Available Hadoop Name Node Architecture-Using Replicas of Name Node with Time Synchronization among Replicas
IOSR Journal of Computer Engineering (IOSR-JCE) e-issn: 2278-0661, p- ISSN: 2278-8727Volume 16, Issue 3, Ver. II (May-Jun. 2014), PP 58-62 Highly Available Hadoop Name Node Architecture-Using Replicas
HDFS scalability: the limits to growth
Konstantin V. Shvachko HDFS scalability: the limits to growth Konstantin V. Shvachko is a principal software engineer at Yahoo!, where he develops HDFS. He specializes in efficient data structures and