A Survey on Big Data Concepts and Tools

Transcription

1 A Survey on Big Data Concepts and Tools D. Rajasekar 1, C. Dhanamani 2, S. K. Sandhya 3 1,3 PG Scholar, 2 Assistant Professor, Department of Computer Science and Engineering, Sri Krishna College of Engineering and Technology, Coimbatore, Tamilnadu, India, Abstract The term Big Data refers to the massive amount of structured, unstructured and semi-structured data that are so large and unwieldy the regular database management tools or statistics tools have difficulty among capturing, analysis, storing, sharing, visualize and managing the information. Big Data includes messages, snaps, business deals, surveillance video recordings, posts of social medias, mobile phone GPS signals and RFID readers, microphones, cameras, sensors and activity logs. That s why, the data that exceeds the processing capacity of conventional database systems. That data is very big, moves very fast or doesn t fit the structure of your traditional database architecture. This paper presents an overview of Big Data concept, challenges and its Tools. Keywords Big Data, variety, volume, value, veracity, velocity I. INTRODUCTION In earlier days, internet is used for only the purpose of sending s messages, browsing queries. But recently internets are used for more purpose. So the internet generates very huge amount of data at every seconds and it will increased day by day. So such type of data is difficult to process because it contains the billons records of million people. Information that includes the web sale, social networks, online transactions, s, audios, videos, images, sensor data, science research, health records, search queries, click streams, posts, cloud computing, mobile phones and their applications. It cannot be deals with traditional database like RDBMS, DBMS, and ORDBMS. The advantages of MapReduce over traditional systems are shown in Table 1. II. HOW BIG IS BIG DATA In the last twenty years, the data is increasing day by day across the world. There are more than 2 billion internet users in the world today and 4.6 billion mobile phones in Interestingly, approximately 80% of these data are unstructured, 20% of data are only a structured with this massive quantity of data, business need fast, authentic, deeper data insight. TABLE I RDBMS VS MAPREDUCE Traditional RDBMS MapReduce Data Size Gigabytes Petabytes Access Interactive and Batch Batch Updates Read and Write many times Write once, Read many times Structure Static Schema Dynamic Schema Integrity High Low Scaling Nonlinear Linear Some facts about the data are, there are more than 250,000 tweets every minute, more than 2 million searchings on Google every minute, more than 70 hours of videos are uploaded to YouTube, There are more than 100 million s are sent, more than 400 GB of data is processing in facebook and more than 570 websites are created every minute on internet. Facebook has more than 1 billion people active accounts from which 751 million using facebook from a mobile. During 2012, 2.5 quintillion bytes of data were created every day. Big data and its analysis are the center of recent science and business areas. Large amount of information is generated from the varied sources either in structured or unstructured kind. Such type of data stored in databases and then it became difficult to extract, transform and load. IBM indicates that 2.5 exabytes data is formed every day that is extremely tough to investigate. The estimation about the generated data is that till 2003 it was represented about 5 EB of data, then until 2012 is 2.7 ZB (Unit name and Sizes of data are show in Table 2) of data and till 2015 it is expected to increase 3 times. TABLE 2 UNIT NAME AND SIZES Unit Name Sizes in Bytes Kilobyte (KB) 1,000 Megabyte (MB) 1,000,000 Gigabytes (GB) 1,000,000,000 Terabyte (TB) 1,000,000,000,000 Petabyte (PB) 1,000,000,000,000,000 Exabyte (EB) 1,000,000,000,000,000,000 Zettabyte (ZB) 1,000,000,000,000,000,000,000 Yottabyte (YB) 1,000,000,000,000,000,000,000,000 80

2 III. CHALLENGES IN BIG DATA The need of big data generated from the large companies like YouTube, Facebook, Google, yahoo, etc for the purpose of analysis of enormous amount of data which is in structured form or even in unstructured form. Google contains the large amount of information. So; there is the need of Big Data Analytics that is the processing of the complex and massive datasets This data is different from structured data (which is stored in relational database systems) in terms of five parameters variety, volume, value, veracity and velocity (5V s) are the challenges of big data management are: 1. Volume Data is ever-growing day by day of all types ever Kilo Byte, Mega Byte, Peta Byte, Yotta Byte, Zetta Byte, Tera Byte of information. The data results into massive files. Excessive volume of information is main issues of storage. This main issue is resolved by reducing storage value. Data volumes are expected to grow more than 50 times by Variety Data sources (even in the same field or in distinct) are extremely heterogeneous. The files comes in various formats and of any type, it may be unstructured or structured such as text, audio, log files, videos and more. The varieties are endless, and the data enters the network without having been quantified or qualified in any way. 3. Velocity The data comes at high speed. Sometimes one minute is too late so big data is time sensitive. Most organisations data velocity is main challenge. The credit card transactions and social media messages done in millisecond and data generated by this putting in to databases. 4. Value Which addresses the requirement for valuation of enterprise data? It is a most important V in big data. Value is main buzz for big data because it is important for IT infrastructure system, businesses to store large amount of values in database. 5. Veracity The increase in the range of values typical of a large data set. When we dealing with high volume, velocity and variety of data, the all of data are not going 100% correct, there will be dirty data. Big data and analytics technologies work with these types of data. FIGURE 1 BIG DATA PARAMETERS IV. TOOLS AND TECHNIQUES Hadoop Hadoop is an open source project of the Apache Foundation that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is a framework written in java originally developed by Doug Cutting who named it after his son s toy elephant. Hadoop uses Google s Map Reduce and Google File System technologies as its foundation. It is optimised to handle massive quantities of data which could be structured, unstructured or semi structured, using commodity hardware, that is, relatively inexpensive computers. This massive parallel processing is done with great performance. However, it is a batch operation handling massive quantities of data, so the response time is not immediate. Hadoop replicates its data across different computers, so that if one goes down, the data are processed on one of the replicated computers. Hadoop is a framework that can run applications on systems with thousands of nodes and terabytes. It distributes the file among the nodes and allows to system continue work in case of a node failure. This approach reduces the risk of catastrophic system failure. In which application is broken into smaller parts (fragments or blocks).apache Hadoop consists of the Hadoop kernel, Hadoop Distributed File System (HDFS), map reduce and related projects are zookeeper, Hbase, Apache Hive. 81

3 Hadoop Distributed File System (HDFS) consists of three Components: the Name Node, Secondary Name Node and Data Node. The Multilevel Secure (MLS) environmental problems of Hadoop by using Security Enhanced Linux (SE Linux) protocol. In which multiple sources of Hadoop applications run at different levels. This protocol is a denotation of Hadoop Distributed File System (HDFS). Hadoop is commonly used for distributed batch index building; it is desirable to optimize the index capability in close continuous. Hadoop provides components for storage and analysis for large scale processing. Now a day s Hadoop used by hundreds of companies. 1. HADOOP DISTRIBUTED FILE SYSTEM HDFS is a block-structured distributed file system that holds the large amount of Big Data. In the HDFS the data is stored in blocks that are known as chunks. FIGURE 3 HDFS ARCHITECTURE FIGURE 2 HADOOP DISTRIBUTED FILE SYSTEM The advantage of Hadoop is Distributed storage & Computational capabilities, extremely scalable, optimized for high throughput, large block sizes, tolerant of software and hardware failure. The disadvantage of Hadoop is that it is master processes are single points of failure. Hadoop does not offer storage or network level encryption, inefficient for handling small files. Hadoop is not suitable for OnLine Transaction Processing workloads where data are randomly accessed on structured data like a relational database. Also it is not suitable for OnLine Analytical Processing or Decision Support System workloads where data are sequentially accessed on structured data like a relational database, to generate reports that provide business intelligence. Hadoop is used for Big Data. It complements OnLine Transaction Processing and OnLine Analytical Processing. It is NOT a replacement for a relational database system. Hadoop mainly consists of following two components: 1. Hadoop Distributed File System (HDFS) 2. Map Reduce Hadoop thus provides: a reliable shared storage and analysis system. The storage is provided by HDFS and analysis by Map Reduce. HDFS is client-server architecture comprises of NameNode and many DataNodes.The name node stores the metadata for the NameNode.NameNodes keeps track of the state of the DataNodes. NameNode is also responsible for the file system operations etc. When Name Node fails the Hadoop doesn t support automatic recovery, but the configuration of secondary nod is possible. 2. Map Reduce MapReduce is a programming framework for distributed computing which was created by Google using the divide and conquer method to break down complex big data problems into small units of work and process them in parallel. MapReduce can be divided into two stages: FIGURE 4 MAP REDUCE ARCHITECTURE 82

4 a) Map Step The master node data is chopped up into many smaller sub problems. A worker node processes some subset of the smaller problems under the control of the JobTracker node and stores the result in the local file system where a reducer is able to access it. b) Reduce Step This step analyzes and merges input data from the map steps. There can be multiple reduce tasks to parallelize the aggregation, and these tasks are executed on the worker nodes under the control of the JobTracker. Components of Hadoop: HDFS: A highly fault tolerant distributed file system that is responsible for storing data on the clusters. MapReduce: A powerful parallel programming technique for distributed processing on clusters. HBase: A scalable, distributed database for random read/write access. Pig: A high level data processing system for analyzing data sets that occurs a high level language. Hive: A data warehousing application that provides a SQL like interface and relational model. Sqoop: A project for transferring data between relational databases and Hadoop. Avro: A system of data serialization. Oozie: A workflow for dependent Hadoop jobs. Chukwa: A Hadoop subproject as data accumulation system for monitoring distributed systems. Flume: A reliable and distributed streaming log collection. ZooKeeper: A centralized service for providing distributed synchronization and group services along with maintenance of the configuration information and records. V. APPLICATIONS McKinsey Global Institute specified the potential of big data in five main topics: Healthcare: clinical decision support systems, individual analytics applied for patient profile, personalized medicine, performance based pricing for personnel, analyze disease patterns, improve public health Public sector: creating transparency by accessible related data, discover needs, improve performance, customize actions for suitable products and services, decision making with automated systems to decrease risks, innovating new products and services Retail: in store behavior analysis, variety and price optimization, product placement design, improve performance, labor inputs optimization, distribution and logistics optimization, web based markets Manufacturing: improved demand forecasting, supply chain planning, sales support, developed production operations, web search based applications Personal location data: smart routing, geo targeted advertising or emergency response, urban planning, new business models Web provides kind of opportunities for big data too. For example; social network analysis such as understanding user intelligence for more targeted advertising, marketing campaigns and capacity planning, customer behavior and buying patterns also sentiment analytics. According to these inference firms optimization their content and recommendation engine. Some companies such as Amazon and Google publishing articles related to their work. Inspired by the writings published, developers are developing similar technologies as open source software such as Lucene, Solr, Hadoop and HBase. Social Media such as Facebook, Twitter and LinkedIn are going a step further thereby publishing open source projects for big data like Cassandra, Hive, Pig, Voldemort, Storm, IndexTank. In addition, predictive analytics on traffic flows or identify guilties and threats from different video, audio and data feeds are advantages of big data again. In 2012, Obama government announced big data initiatives of more than $200 million in research and development investments for National Science Foundation, National Institutes of Health, Department of Defense, Department of Energy and United States Geological Survey. The investments were launched to take a step forward instruments and methods for access, organize and collect findings from vast volumes of digital data. VI. CONCLUSION In this article, an overview of big data's concept, tools, techniques, applications, advantages and challenges have been reviewed. The results have given away that regardless of the fact that accessible information, tools and techniques available in the literature, there are numerous focuses to be viewed as, discussed, analyzed, developed, improved, and so on. 83

5 Although this paper obviously has not resolved the complete subject about this substantial topic, emphatically it has provided some useful discussion and we can conclude that Hadoop is possibly one of the best solutions to maintain the Big Data. REFERENCES [1] Sagiroglu, Sinanc, "Big Data: A Review", /13 IEEE. [2] Sabia, Arora, "Technologies to Handle Big Data: A Survey". [3] Shilpa, Manjit Kaur, "Big Data and Methodology A review", Volume 3, Issue 10, October [4] (last access - 10/11/2014). [5] K. Bakshi, "Considerations for Big Data: Architecture and Approach", Aerospace Conference IEEE, Big Sky Montana, March [6] D. Garlasu, V. Sandulescu, I. Halcu, G.Neculoiu, "A Big Data implementation based on Grid Computing", Grid Computing, January [7] R.D. Schneider, "Hadoop for Dummies Special Edition", John Wiley&Sons Canada, , [8] Yuri Demchenko, "The Big Data Architecture Framework (BDAF)", Outcome of the Brainstorming Session at the University of Amsterdam, 17 July [9] Aditya B. Patel, Manashvi Birla, Ushma Nair, "Addressing Big Data Problem Using Hadoop and Map Reduce", Aug, [10] how much data is on the internet and generated online every minute. [11] [12] (last access 4/11/2014). [13] -introduction-to-mapreduce/ (last access 29/10/2014). [14] Definitions of Big Data - open tracker, [15] White paper BigData-as-a service, A Market & Technology Perspective-EMC Solution group [16] [17] [18] Rathod, Chauhan, "A Survey on Big Data Analysis Techniques" IJSRD - International Journal for Scientific Research & Development Vol. 1, Issue 9, 2013 ISSN (online): [19] [20] [21] 84