Big Application Execution on Cloud using Hadoop Distributed File System

Big Application Execution on Cloud using Hadoop Distributed File System Ashkan Vates*, Upendra, Muwafaq Rahi Ali RPIIT Campus, Bastara Karnal, Haryana, India ---------------------------------------------------------------------***--------------------------------------------------------------------- Abstract Big data is defined as large amount of data which requires new technologies and architectures so that it becomes possible to extract value from it by capturing and analysis process. Due to such large size of data it becomes very difficult to perform effective analysis using the existing traditional techniques. Big data is the term for collection of data sets so large and complex that it becomes difficult to process using traditional database management tools and that is main problem.90% of the data in the world today has been created in the last two years alone. 80% of data captured today is unstructured from sensors used together climate information. Post to social media sites, digital pictures and videos, purchase transaction records and cell phone GPS signals, to name a few all of this unstructured data is big data. For solving the big data application problem Hadoop is used. With Hadoop no data is too big. Hadoop has gained momentum mainly due to its ability to analyze unstructured Big Data to draw important predictions for businesses. In this paper presented the main issues and challenges along with the complete descriptions of the methods being employed for tackling the storage and processing problems associated with big data. Keywords Big data ; Hadoop; Hadoop Distributed File System; MapReduce. I. INTRODUCTION Data is growing at a huge speed making it difficult to handle such large amount of data (exabytes).the main difficulty in handling such large amount of data is because that the volume is increasing rapidly in comparison to the computing resources. The Big data term which is being used now a days is kind of misnomer as it points out only the size of the data not putting too much of attention to its other existing properties. Big data challenges at three levels: Data being produced is not of single category as it not only includes the traditional data but also the semi structured data from various resources like web Pages, Web Log Files, social media sites, e-mail, documents, sensor devices data both from active passive devices. All this data is totally different consisting of raw, structured, semi structured and even unstructured data which is difficult to be handled by the existing traditional analytic systems. a. Volume The Big word in Big data itself defines the volume. At Present the data existing is in petabytes and is supposed to increase to zettabytes in nearby future. The social networking sites existing are themselves producing data in order of terabytes every day and this amount of data is definitely difficult to be handled using the existing traditional systems. b. Velocity Velocity in Big data is a concept which deals with the speed of the data coming from various sources. This characteristic is not being limited to the speed of incoming data but also speed at which the data flows. For example the data from the sensor devices would be constantly moving to the database store and this amount won t be small enough. Thus our traditional systems are not capable enough on performing the analytics on the data which is constantly in motion. Big data is different from the data being stored in traditional warehouses. The data stored there first needs to be cleansed, documented and even trusted. Moreover it should fit the basic structure of that warehouse to be stored but this is not the case with Big data it not only handles the data being stored in traditional warehouses but also the data not suitable to be stored in those warehouses. Thus there comes the point of access to mountains of data and better business strategies and decisions as analysis of more data is always better. Big data is difficult to work with using most relational database management systems and desktop statistics and visualization packages, requiring instead massively parallel software running on tens, hundreds, or even thousands of servers. What is considered "big data" varies depending on the capabilities of the organization 2014, IRJET.NET- All Rights Reserved Page 23

managing the set, and on the capabilities of the applications that are traditionally used to process and analyze the data set in its domain. For some organizations, facing hundreds of gigabytes of data for the first time may trigger a need to reconsider data management options. For others, it may take tens or hundreds of terabytes before data size becomes a significant consideration. The solution of the big data processing is Hadoop because Hadoop provides the function of parallel execution of task with the help of MapReduce function. In industry, a number of public cloud service providers have rolled out hosted versions of MapReduce-based clusters as their data processing frameworks. For example, the world s largest Hadoop cluster is run by Facebook to operate thousands of online social media applications per second. When a workflow submitted to Hadoop, MapReduce runtime packages each service request as a MapReduce job that contains a large number of map and reduce tasks. Scheduler inside the runtime is responsible for scheduling tasks to compute nodes and monitoring their completion, which exerts great impact on the QoS and the performance of cluster. LITERATURE SURVEY 1) Traditional Approach: In traditional approach an enterprise will get a very powerful computers and it will speed in whatever the data is available to this computer to crunch the number and this computer will go good job but until a certain point will come in this computer will not able to do the processing anymore because this is not scalable and the big data is growing, so traditional enterprise approach does have its limitations when it comes in big data. Fig 1: Traditional Approach Hadoop takes a very different approach than the enterprise approach breaks the data into smaller pieces and that s why it deals with a big data. Breaking the data into smaller pieces is a good idea but that what how are going to perform the computation breaks the computation as well down into smaller pieces and it sends each piece of computer to each piece of data. 2) Comparison with HPC and Grid Computing Tools: The approach in HPC and Grid computing includes the distribution of work across a cluster and they are having a common shared File system hosted by SAN. The jobs here are mainly compute intensive and thus it suits well to them unlike as in case of Big data where access to larger volume of data as network bandwidth is the main bottleneck and the compute nodes start becoming idle. Map Reduce component of Hadoop here plays an important role by making use of the Data Locality property where it collocates the data with the compute node itself so that the data access is fast. HPC and Grid Computing basically make use of the API s such as message passing Interface (MPI). Though it provides great control to the user, the user needs to control the mechanism for handling the data flow. On the other hand Map Reduce operates only at the higher level where the data flow is implicit and the programmer just thinks in terms of key and value pairs. Coordination of the jobs on large distributed systems is always challenging. Map Reduce handles this problem easily as it is based on shared-nothing architecture that is the tasks are independent of each other. The implementation of Map Reduce itself detects the failed tasks and reschedules them on healthy machines. Thus the order in which the tasks run hardly matters from programmer s point of view. But in case of MPI, an explicit management of check pointing and recovery system needs to be done by the program. This gives more control to the programmer but makes them more difficult to write. 2014, IRJET.NET- All Rights Reserved Page 24

3) Comparison with RDBMS: The traditional database deals with data size in range of Gigabytes as compared to MapReduce dealing in petabytes. The Scaling in case of MapReduce is linear as compared to that of traditional database. In fact the RDBMS differs structurally, in updating, and access techniques from MapReduce. I. PROPOSED MODEL The following tools and techniques are available A)Hadoop: Hadoop is an open source project hosted by Apache Software Foundation. It consists of many small sub projects which belong to the category of infrastructure for distributed computing. Hadoop mainly consists of: 1) File System (The Hadoop File System) 2) Programming Paradigm (Map Reduce) The other subprojects provide complementary services or they are building on the core to add higherlevel abstractions. There exist many problems in dealing with storage of large amount of data. Though the storage capacities of the drives have increased massively but the rate of reading data from them hasn t shown that considerable improvement. The reading process takes large amount of time and the process of writing is also slower. This time can be reduced by reading from multiple disks at once. Only using one hundredth of a disk may seem wasteful. But if there are one hundred datasets, each of which is one terabyte and providing shared access to them is also a solution. There occur many problems also with using many pieces of hardware as it increases the chances of failure. This can be avoided by Replication i.e. creating redundant copies of the same data at different devices so that in case of failure the copy of the data is available. The main problem is of combining the data being read from different devices. Many a methods are available in distributed computing to handle this problem but still it is quite challenging. All the problems discussed are easily handled by Hadoop. The problem of failure is handled by the Hadoop Distributed File System and problem of combining data is handled by Map reduce programming Paradigm. Map Reduce basically reduces the problem of disk reads and writes by providing a programming model dealing in computation with keys and values. Hadoop thus provides: a reliable shared storage and analysis system. The storage is provided by HDFS and analysis by MapReduce. B) Hadoop Components in detail Hadoop Distributed File System: Hadoop comes with a distributed File System called HDFS, which stands for Hadoop Distributed File System. HDFS is a File System designed for storing very large files with streaming data access patterns, running on clusters on commodity hardware. HDFS block size is much larger than that of normal file system i.e. 64 MB by default. The reason for this large size of blocks is to reduce the number of disk seeks. A HDFS cluster has two types of nodes i.e. namenode (the master) and number of datanodes (workers). The name node manages the file system namespace, maintains the file system tree and the metadata for all the files and directories in the tree. The datanode stores and retrieve blocks as per the instructions of clients or the namenode. The data retrieved is reported back to the namenode with lists of blocks that they are storing. Without the namenode it is not possible to access the file. So it becomes very important to make namenode resilient to failure. These are areas where HDFS is not a good fit: Low-latency data access, Lots of small file, multiple writers and arbitrary file modifications. C) MapReduce: MapReduce is the programming paradigm allowing massive scalability. The MapReduce basically performs two different tasks i.e. Map Task and Reduce Task. A map-reduce computation executes as follows: Map tasks are given input from distributed file system. The map tasks produce a sequence of key-value pairs from the input and this is done according to the code written for map function. These value generated are collected by master controller and are sorted by key and divided among reduce tasks. The sorting basically assures that the same key values ends with the same reduce tasks. The Reduce tasks combine all the values associated with a key working with one key at a time. Again the combination process depends on the code written for reduce job. The Master controller process and some number of worker processes at different compute nodes are forked by the user. Worker handles map tasks (MAP WORKER) and reduce tasks (REDUCE WORKER) but not both. The Master controller creates some number of map and reduce tasks which is usually decided by the user program. The tasks 2014, IRJET.NET- All Rights Reserved Page 25

are assigned to the worker nodes by the master controller. Track of the status of each Map and Reduce task (idle, executing at a particular Worker or completed) is kept by the Master Process. On the completion of the work assigned the worker process reports to the master and master reassigns it with some task. The failure of a compute node is detected by the master as it periodically pings the worker nodes. All the Map tasks assigned to that node are restarted even if it had completed and this is due to the fact that the results of that computation would be available on that node only for the reduce tasks. The status of each of these Map tasks is set to idle by Master. These get scheduled by Master on a Worker only when one becomes available. The Master must also inform each Reduce task that the location of its input from that Map task has changed. D) MapReduce cloud Architectutre Fig 2:- MapReduce E) Execution overview of MapReduce Model The Map invocations are distributed across multiple machines by automatically partitioning the input data into a set of M splits. Fig 3: MapReduce Execution The input splits can be processed in parallel by different machines. Reduce invocations are distributed by partitioning the intermediate key space into R pieces using a partitioning function (e.g., hash(key) mod R). The number of partitions (R) and the partitioning function are specified by the user. Figure 3 shows the overall flow of a MapReduce operation in our implementation, when the user program calls the MapReduce function. The following sequence of actions occurs (the numbered labels in Figure 1 correspond to the numbers in the list below): The MapReduce library in the user program first splits the input files into M pieces of typically 16 megabytes to 64 megabytes (MB) per piece (controllable by the user via an optional parameter). It then starts up many copies of the program on a cluster of machines. One of the copies of the program is special-the master. The rest are workers that are assigned work by the master. There are M map tasks and R reduce tasks to assign. The master picks idle workers and assigns each one a map task or a reduce task. A worker who is assigned a map task reads the contents of the corresponding input split. It parses key/value pairs out of the input data and passes each pair to the userdefined Map function. The intermediate key/value pairs produced by the Map function are buffered in memory. 2014, IRJET.NET- All Rights Reserved Page 26

Periodically, the buffered pairs are written to local disk, partitioned into R regions by the partitioning function. The locations of these buffered pairs on the local disk are passed back to the master, who is responsible for forwarding these locations to the reduce workers. When a reduce worker is notified by the master about these locations, it uses remote procedure calls to read the buffered data from the local disks of the map workers. When a reduce worker has read all intermediate data, it sorts it by the intermediate keys so that all occurrences of the same key are grouped together. The sorting is needed because typically many different keys map to the same reduce task. If the amount of intermediate data is too large to fit in memory, an external sort is used. The reduce worker iterates over the sorted intermediate data and for each unique intermediate key encountered, it passes the key and the corresponding set of intermediate values to the user's Reduce function. The output of the Reduce function is appended to a final output file for this reduces partition. When all map tasks and reduce tasks have been completed, the master wakes up the user program. At this point, the MapReduce call in the user program returns back to the user code. After successful completion, the output of the MapReduce execution is available in the R output files (one per reduce task, with file names as specified by the user). Typically, users do not need to combine these R output files into one file. they often pass these files as input to another MapReduce call, or use them from another distributed application that is able to deal with input that is partitioned into multiple files. F) Execution of Big Application on Hadoop Platform Hadoop is Linux based set of tools we have Linux on all this low cost numerous computers. All this computers will have two components of Hadoop that is Task Tracker and Data node. Fig 4: Execution of Application on Hadoop platform The job of task tracker component is to process the small piece of task that has been given to this particular node and all this computers will be called slaves. Why would they be called slaves? Because we have a master and difference between master and slaves? The main difference is the master has two additional components that are running on computer. It would have task tracker and data node as well just like any other but additional to those two components it will have job tracker and the name node. So we have learned that Hadoop has two main components that are very high level MapReduce and file system. How does that relate to the job tracker and name node, task tracker and data node pieces. So the job tracker and task tracker off course are part of the high level component MapReduce, so they all fall under umbrella of MapReduce and similarity the name node and all this data node fall under the umbrella of Hadoop file system called HDFS. The applications that are running on Hadoop will contact with the master node. One of the attribute of Hadoop is that it is batch processing set of tools, so application would assign or provide a task for Hadoop to perform and its going into queue. Once the task is completed application will be inform and results will give back to the application. Keep in mind there are certain projects that provides real time capabilities to an application to get results from Hadoop but mainly it is batch processing set of tool, so the role of job tracker component is running on the master node is to break the hire bigger task into smaller pieces and to send each small 2014, IRJET.NET- All Rights Reserved Page 27

piece of computation to the task tracker so they will perform smaller piece and send the result back to the job tracker then it will back to the job tracker then it will combine the hole result together and send the final result back to the application and the name node running on master computer is responsible to keep an index of which data node it tells the application go to this particular to get the data so it has all index, so the name node tells the application where the data is residing then it goes directly node and will get the directly from that node, so application is not dependent on name node to give data back it goes and get the data directly but it has to no way to get residing the network failures are bound to happen there will happen and good thing about Hadoop is that it is built keeping hardware failure in mind, it has built in tolerance by default Hadoop maintains three copies of each file. These copies are scattered along different computers so this way when a computers fails the system keeps running. Data is available from different node and once you fix the failed nodes then Hadoop will take care of that and it will copy from other file to that node so that is one of the very important feature of Hadoop file system fault tolerance is not limiting to the disk failing at the one of the slave node or even on master node. It is also applicable to the task tracker services that are running on slave computers. If any of the computer fails or even just service fails job tracker would detect the failure and it will be some other task tracker to perform the same job. The fault tolerance is fair but one can argue the fault tolerance their only as far as the computers and concern. If master computer dies then that would be single point of failure. Hadoop has taken care of that has table that are maintain by name node has the entire index where data has residing. All those tables are backup and backup copies are copied over different computer and enterprise version of Hadoop also keeps two masters, one as main masters in case one master dies that is no single point of failure so we have talk about how Hadoop has a solve challenges provided by big data and talk about architecture of Hadoop. This brings back to the bottom line. The programming is now easy for program for programmer. Programmer do not have to worry about taking care of file is located the master node is there to manage that and also they don t have to worried about to how to manage the failures Hadoop is taking care of that and also they don t have to break the bigger computation down into smaller pieces that is the task for Hadoop is going to break into the computation down and will send the smaller task to the individual nodes and also they don t have to do the programming for scaling. II. CONCLUSIONS MapReduce is a pervasive programming model for big data analysis. This paper described the concept of Big data and its importance and this existing projects. Reduce the time complexity rather than traditional execution of application. Hadoop tool for Big data is described in detail focusing on the areas where it needs to be improved. Running a big application parallel on distributed machine using Hadoop technology so that it improve the performance of the execution of application. REFERENCES [1] Avita katal and Mohammad Wazid, Big Data:Issues,challenges,Tools and Good Practices Department of CSE Graphic Era University Dehradun,India, pp. 404-409, 2013 [2] Fei Teng and Hao Yang Scheduling Real-time Workflow on MapReduce-based Cloud, School of Information Science and Technology Southwest Jiaotong University Chengdu,China,: Academic, 2013, pp. 117-122. [3] Jeffrey Dean and Sanjay Ghemawat, MapReduce:Simplified Data Processing on Large Cluters, 2004, pp.1-14. [4] Novia Nurain, Hasan Sarwar and Md.Parvez Sajjad, An In-depth study of Map Reduce in cloud Environment, Bangladesh University of Engineering and technology,2013, pp. 263-268. [5] Huan Liu, Dan Orban Cloud MapReduce: a MapReduce Implementation on top of a Cloud Operating System, pp. 1-14. [6] Lui Yi-di, Research on Iterative Method in Solving Linear Equations on the Hadoop platform city institute,dalian university of technology, 2013 [7] Thilina Gunarathne, Tak-Lon Wu, Judy Qiu, MapReduce in the Clouds for Science Geoffrey Fox School of Informatics and Computing Pervasive Technology Institute Indiana University, Bloomington. [8] Matthew Smith, Christian Szongott, Benjamin Henne, Gabriele von Voigt, Big Data Privacy Issues in Public Social Media, IEEE, 6 th International Conference on Digital Ecosystems Technologies (DEST), 18-20 June 2012. [9] Martin Courtney, The Larging-up of Big Data, IEEE, Engineering & Technology, September 2012. 2014, IRJET.NET- All Rights Reserved Page 28