Index Terms : Load rebalance, distributed file systems, clouds, movement cost, load imbalance, chunk.

Transcription

1 Load Rebalancing for Distributed File Systems in Clouds. Smita Salunkhe, S. S. Sannakki Department of Computer Science and Engineering KLS Gogte Institute of Technology, Belgaum, Karnataka, India Affiliated to Visvesvaraya technological University A B S T R A C T Distributed file systems are key building blocks for cloud computing applications based on the MapReduce programming paradigm. In such file systems, nodes simultaneously serve computing and storage functions; a file is partitioned into a number of chunks allocated in distinct nodes so that MapReduce tasks can be performed in parallel over the nodes. However, in a cloud computing environment, failure is the norm, and nodes may be upgraded, replaced, and added in the system. Files can also be dynamically created, deleted, and appended. This results in load imbalance in a distributed file system; that is, the file chunks are not distributed as uniformly as possible among the nodes. Emerging distributed file systems in production systems strongly depend on a central node for chunk reallocation. This dependence is clearly inadequate in a largescale, failure-prone environment because the central load balancer is put under considerable workload that is linearly scaled with the system size, and may thus become the performance bottleneck and the single point of failure. In this paper, a fully distributed load rebalancing algorithm is presented to cope with the load imbalance problem. Our algorithm is compared against a centralized approach in a production system and a competing distributed solution presented in the literature. The simulation results indicate that our proposal is comparable with the existing centralized approach and considerably outperforms the prior distributed algorithm in terms of load imbalance factor, movement cost, and algorithmic overhead. The performance of our proposal implemented in the Hadoop distributed file system is further investigated in a cluster environment. Index Terms : Load rebalance, distributed file systems, clouds, movement cost, load imbalance, chunk. I. INTRODUCTION CLOUD Computing is a compelling technology. In clouds, clients can dynamically allocate their resources on-demand without sophisticated deployment and management of resources. Key enabling technologies for clouds include the MapReduce programming paradigm, distributed file systems, virtualization, and so forth. These techniques emphasize scalability, so clouds can be large in scale, and comprising entities can arbitrarily fail and join while maintaining system reliability. Distributed file systems are key building blocks for cloud computing applications based on the MapReduce programming paradigm. In such file systems, nodes simultaneously serve computing and storage functions; a file is partitioned into a number of chunks allocated in distinct nodes so that MapReduce tasks can be performed in parallel over the nodes. For example, consider a word count application that counts the number of distinct words and the frequency of each unique word in a large file. In such an application, a cloud partitions the file into a large number of disjointed and fixed-size pieces (or file chunks) and assigns them to different cloud storage , IJAFRC and NCRTIT 2015 All Rights Reserved

2 nodes (i.e., chunkservers). Each storage node (or node for short) then calculates the frequency of each unique word by scanning and parsing its local file chunks. In such a distributed file system, the load of a node is typically proportional to the number of file chunks the node possesses. Because the files in a cloud can be arbitrarily created, deleted, and appended, and nodes can be upgraded, replaced and added in the file system, the file chunks are not distributed as uniformly as possible among the nodes. Load balance among storage nodes is a critical function in clouds. In a load-balanced cloud, the resources can be well utilized and provisioned, maximizing the performance of MapReduce-based applications. In this paper, we are interested in studying the load rebalancing problem in distributed file systems specialized for large-scale, dynamic and data-intensive clouds. (The terms rebalance and balance are interchangeable used.) Such a large-scale cloud has hundreds or thousands of nodes (and may reach tens of thousands in the future). Our objective is to allocate the chunks of files as uniformly as possible among the nodes such that no node manages an excessive number of chunks. Additionally, we aim to reduce network traffic (or movement cost) caused by rebalancing the loads of nodes as much as possible to maximize the network bandwidth available to normal applications. Moreover, as failure is the norm, nodes are newly added to sustain the overall system performance resulting in the heterogeneity of nodes. Exploiting capable nodes to improve the system performance is, thus, demanded. II. LOAD REBALANCING ALGORITHM The load rebalancing problem in distributed file systems specialized for large-scale, dynamic and dataintensive clouds to allocate the chunks of files as uniformly as possible among the nodes of chunks and aims to reduce network traffic (or movement cost ). We consider a large-scale distributed file system consisting of a set of chunkservers V in a cloud, where the cardinality of V is V =n. Typically, n can be 1,000, 10,000, or more. In the system, a number of files are stored in the n chunkservers. First, let us denote the set of files as F. Each file f F is partitioned into a number of disjointed, fixed- size chunks denoted by C f. For example, each chunk has the same size, 64 bytes, in Hadoop HDFS [4]. Second, the load of a chunkserver is proportional to the number of chunks hosted by the server [3]. Third, node failure is the norm in such a distributed system, and the chunkservers may be upgraded, replaced and added in the system. Finally, the files in F may be arbitrarily created, deleted, and appended. The net effect results in file chunks not being uniformly distributed to the chunkservers. Our objective in the current study is to design a load rebalancing algorithm to reallocate file chunks such that the chunks can be distributed to the system as uniform uniformly as possible while reducing the movement cost as much as possible. Here, the movement cost is defined as the number of chunks migrated to balance the loads of the chunkservers. Let A be the ideal number of chunks that any chunkserver i V is required to manage in a system-wide load-balanced state, A. Necessity of Load Rebalancing Eliminates the dependence on central nodes. Operates in a distributed manner in which nodes perform their load-balancing tasks independently without synchronization or global knowledge regarding the system. Reduce algorithmic overhead introduced to the DHTs as much as possible , IJAFRC and NCRTIT 2015 All Rights Reserved

3 III. SYSTEM DESIGN System Architecture shows a small-scale cluster environment consisting of a single, dedicated name node and 25 data nodes. Specifically, the name node is equipped with Intel Core 2 processor and 1 Gbytes RAM. As the number of file chunks in our experimental environment is small, the RAM size of the name node is sufficient to cache the entire name node process and the metadata information, including the directories and the locations of file chunks. In proposed work, experimental environment, a number of clients are established to issue requests to the name node. The requests include commands to create directories with randomly designated names, to remove directories arbitrarily chosen, etc Name Node Data node Data node Load Balancer Data node Network Clients Figure 1 System Architecture A. Load Balancer Fig. 2 Shows load balancer, A file is partitioned into a number of chunks allocated in distinct nodes so that MapReduce tasks can be performed in parallel over the nodes. For example, consider a word count application that counts the number of distinct words and the frequency of each unique word in a large file. In such an application, a cloud partitions the file into a large number of disjointed and fixed-size pieces (or file chunks) and assigns them to different cloud storage nodes (i.e., chunkservers) , IJAFRC and NCRTIT 2015 All Rights Reserved

4 Flowchart: Annexure 1. IV. CONCLUSION A novel load-balancing algorithm to deal with the load rebalancing problem in large-scale, dynamic, and distributed file systems. Proposed System will balance the loads of nodes and reduce the demanded movement cost as much as possible, while taking advantage of physical network locality and node heterogeneity. Proposal is comparable to the centralized algorithm in the Hadoop HDFS production system. Proposed Load-balancing algorithm exhibits a fast convergence rate. Our proposal is comparable to the centralized algorithm in the Hadoop HDFS production system and dramatically out- performs the competing distributed algorithm in terms of load imbalance factor, movement cost, and algorithmic overhead. Particularly, our load-balancing algorithm exhibits a fast convergence rate. V. REFERENCES [1] Hung-Chang Hsiao, Member, IEEE Computer Society, Hsueh-Yi Chung, Haiying Shen, Member, IEEE, and Yu-Chang Chao, Load Rebalancing for Distributed File Systems in Clouds, IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 24, NO. 5, MAY , IJAFRC and NCRTIT 2015 All Rights Reserved

5 [2] S. Ghemawat, H. Gobioff, and S.-T. Leung, The Google File System, Proc. 19th ACM Symp. Operating Systems Principles (SOSP 03),pp , Oct [3] Hadoop Distributed File System, hdfs/, [4] VMware, [5] Xen, [6] Apache Hadoop, [7] Hadoop Distributed File System Rebalancing Blocks, [8 ] K. McKusick and S. Quinlan, GFS: Evolution on Fast-Forward, Comm. ACM,vol. 53, no. 3, pp , Jan [9] HDFS Federation, [10] I. Stoica, R. Morris, D. Liben-Nowell, D.R. Karger, M.F. Kaashoek,F. Dabek, and H. Balakrishnan, Chord: A Scalable Peer-to-Peer Lookup Protocol for Internet Applications, , IJAFRC and NCRTIT 2015 All Rights Reserved

6 Annexure , IJAFRC and NCRTIT 2015 All Rights Reserved