1 Chapter 11 Map-Reduce, Hadoop, HDFS, Hbase, MongoDB, Apache HIVE, and Related Summary Xiangzhe Li Nowadays, there are more and more data everyday about everything. For instance, here are some of the astonishing data from the book Hadoop the Definitive Guide: The New York Stock Exchange generates about one terabyte of new trade data per day. Facebook hosts approximately 10 billion photos, taking up one petabyte of storage. Ancestry.com, the genealogy site, stores around 2.5 petabytes of data. The Internet Archive stores around 2 petabytes of data, and is growing at a rate of 20 terabytes per month. (Whites) The Large Hadron Collider near Geneva, Switzerland, will produce about 15 petabytes of data per year. In the business world, having a precise way of determining the accurate information from the big set of data is very critical and can help the company reduce the cost of information retrieval. In this chapter, we will talk about the different components of the software architecture and frameworks that process massive amount of unstructured data. Some of the topics include Map-Reduce, Hadoop, HDFS, Hbase, MongoDB, and Apache HIVE. Introduction High Level Concepts In reality, even though the technology improvement allows the storage capacities of hard drive to increase, the speed at which the data is accessed has not made significant progress. For instance, research found that One typical drive from 1990 could store 1,370 MB of data and had a transfer speed of 4.4 MB/s, you could read all the data from a full drive in around five minutes. Over 20 years later, one terabyte drives are the norm, but the transfer speed is around 100 MB/s, so it takes more than two and a half hours to read all the data off the disk. (Whites) Wow, even though the transfer speed has increased over 20 times, the storage has increased so significantly that the time for accessing the data become 30 times longer under the old processing methods. For this reason, the concept of parallel computing has brought the initial invention of big data processing with tools such as the Hadoop family. History The initial version of Hadoop was created in early 2005 by Doug Cutting and Michael Cafarella, while Cutting was working at Yahoo! at the time. The name of the project came after his son s toy elephant. The original purpose of the project was to support an open-source web search platform called Nutch. Nutch was initiated in 2002 and it was based on open source information retrieval framework called Apache Lucene and using the Java language as its backbone for the structure. After the initiate architecture of Nutch was created, Cutting and Cafarella realized that it was not able to support the billions of pages on the web. In 2003, Google published a paper about a fully functioning product called the Googles Distributed File System. Then later in 2004, following the Googles Distributed File System, Cutting and
2 Cafarella started the implementation of the Nutch Distributed File System. In the same year, Map-reduce were introduced by Google in a research paper. During 2005, the Nutch developers have completely integrated a production version of Map-reduce and Nutch Distributed File System into Nutch. By 2006, developers found out that Nutch Distributed File System and Mapreduce can be used in many other fields other than for search engine so the project was separated from Nutch and formed its own subproject of Lucene called Hadoop. Figure 1. The history of Hadoop Topics Nutch The goal of the Nutch project was to be able to realize a web-scale, crawler-based search engine. As for the current version of the project, it has two main versions and has significant difference in its purpose. The first version of Nutch 1.x is a well finished and in production. It mainly relies on the Apache Hadoop data structure and the main strengths of its functionality are for batch job processing. The second version of Nutch 2.x is build off the first version, but the main difference is that it no longer has a well-defined data structure thanks to Apache Gora. This way, objects are handled much more flexibly and one data model can be implemented to be compatible with storing all kind of data.
3 Figure 2. The architecture of Nutch Nutch is divided into two main parts: The crawler and the searcher. The crawler gets the URL lists as inputs and turns them into invert links. The searcher then uses the invert links and turns them into index to responds to user s request. Map-Reduce Map-Reduce is widely used in many big technology companies, for instance in Google, it has been reported that more than ten thousand distinct Map-Reduce programs have been implemented internally at Google over the past four years, and an average of one hundred thousand Map-Reduce jobs are executed on Google s clusters every day, processing a total of more than twenty petabytes of data per day. (Dean) One of the reasons why Map-Reduce is so popular is that programmer find it very easy to use. The program Map-Reduce is very selfexplanatory in its naming. There are two parts to this programming model. First, the map part takes a set of data and converts it into another set of data using methods specific to the domain such as filtering or sorting algorithms. The data will be also broken down into a key-value pair and get passed on to the Reduce part of the programming model. The reduce part takes the outputs from the map function and use it as input for the combine of the data. The data are combined to form even smaller set of data, usually having a count value for the reduced set of data. The figure below is a simple example of applying Map-Reduce to a set of inputted words.
4 Figure 3. The Map-Reduce example As you can see, in this Map-Reduce function, a set of inputted data is passed into the mapping part and split based on their names. Then it is passed into the reduce function and rearranged into the final set of individual key/value pair. Having explained how Map-Reduce work in higher structure, now it is the time to express the key detail of its usage in technical terms. The code for creating a Map-Reduce program requires 3 different components. It consists of a Map function, a Reduce function, and the code that runs the job. A brief touch on one of the 3 components, the map function is incorporated in a generic mapper, where Hadoop uses its own set of data type that works much more efficiently for the inputted data. The inputted text value is converted into a Java String and uses the substring function to retrieve the data we are looking for. As for the progression of the updates for Map-Reduce API, there are several new changes with the newer version. For instance, the new API preferred abstract classes over interfaces since it ease the integration part of implementing a new functions without breaking the old structure of the class. In addition, the configuration in the newer version has been combined into a centralized location. In the older version, the job configuration is set up in a JobConf object, which involves its own declaration of XML documents. But in the newer version, this specific declaration is removed and it is included with every other configuration. Hadoop Hadoop software library is an open source framework that allows the distributed computing of large amount of data using the Map-Reduce programming model. The software itself is able to detect and handle the failure during the computation. All of the components are designed so they can detect the occurrence of failure and will let the framework handle it. Some of the components of the Hadoop project are: Hadoop Common The module that contains common utilities and libraries that support the other Hadoop modules.
5 HDFS Hadoop Distributed File System, a distributed file system that stores data on common hardware and provides access to large amount of application data. Hadoop Yarn - A resource management platform that manages cluster resource and job scheduling. Hadoop MapReduce - A programming model based on Yarn for large scale data processing. Other related projects discussed in this chapter include: HBase - A scalable, distributed database built on top of HDFS that supports structured data storage for large tables. Apache HIVE - A data warehouse infrastructure that provides data summarization and analysis of large data set in HDFS. In short, Hadoop project is the entire architecture of Hadoop family and consist of all the different components that provide the capability of processing big data. Hadoop applies to many fields. For example, in finance, accurate portfolio evaluation and risk analysis require very complicated model and it will be difficult to be stored in a traditional database. This is where Hadoop comes in. It will store all the data and perform deep and computationally extensive analysis. Figure 4. The architecture of Hadoop Here is the official logo of Hadoop and the overall software architecture. As you can see, data are passed into the compute cluster and divided using HDFS and Map-Reduce. The resulting data is then well formatted and outputted. The concept of Map-reduce and HDFS are presented in their individual section. HDFS HDFS stands for Hadoop Distributed File System. When a set of data exceed the storage capacity of the system that is processing the data, the HDFS comes in to distribute the data across multiple system. When this distribution occurs, one of the biggest issues that need to be handled is having a suitable failure tolerable mechanism and recovery method within the system to ensure no data loss.
6 HDFS has several great capabilities. HDFS is designed for storing very large files; it can go up to megabytes to even terabytes in size. HDFS is also designed so it can run on commonly available hardware that is not very expensive or highly reliable. With the system designed to handle node failure the tasks can be carrying on without visible disruption to the user. On the other hand, HDFS has a few currently known issues that make it not as compatibles in some scenarios. For instance, because HDFS is optimized for processing a big amount of data, it does not work so well with applications that requires low-latency access to data. In this scenario, an additional layer of HBase on top of HDFS is a more suitable choice and it will be discussed under HBase section. When the number of files became too big, the distributed system will not be able to store them. Each of the name nodes requires memory in the system, on average, a file and directory takes about 150 bytes of memory, so even though the data within the file can be stored without problem, the number of files will go beyond the storage capacity. The structure of HDFS is a master/slave model. The HDFS cluster will have one single name node, the master server that organize the namespace and control the files that are accessed by clients. Then under the name node, there are several data nodes that manage storage attached to the nodes. They store and retrieve blocks as the name node or the clients requested and send back the set of blocks that carry those information. The blocks are stored internally in the name node and they are much larger than a normal block in a disk. The default for the block is 64MB and files are broken into block-sized chunks to be stored. There are several benefits of having a block structure for the distributed system. First, since a file can be larger than the disk in the network, the file can be divided into several blocks and to be stored on different disks. This way, the file can actually be processed in parallel. In addition, for fault tolerance and recovery, block structure is easily replicated from another disk and bring the process back to normal. Figure 5. The architecture of HDFS Since HDFS is built using the Java language, any machine that supports Java can run the name node or the data node software. There exists a variety of other interfaces that are compatible using HDFS by different methods, this include Thrift, C, FUSE, WebDAV, HTTP
7 and FTP. Usually, the other file system interfaces need additional integration in order to access HDFS. For example, for some non-java applications that have Thrift bindings, they use the Thrift API in their implementation by accessing the Thrift service and ease the interaction to Hadoop. Figure 6. Interaction of HDFS with other components As you can see in this architecture diagram, HDFS interacts with other components of Apache Hadoop to distribute files and data as requested. HBase HBase is a scalable, distributed database built on top of HDFS that supports structured data storage for large tables. It is used when the application requires real time read and write random access to large data set. HBase is designed to solve the scaling problem from a different perspective than most other solutions. It is built from scratch just by adding nodes. In comparison with the relational database systems, HBase applications are actually written in Java. For this reason, HBase is a NoSQL type database and it is neither relational nor supporting SQL. But it is capable of solving the problem a relational database management system cannot: it can store large data table on clusters made from commodity hardware. It lacks several features that are in RDBMS, for example, common functionality such as secondary indexes, triggers, typed columns, and advance query language are not part of HBase. But it also features several benefits in sacrificing those properties. Here are a few key features of HBase: since it is written in Java, it facilitates clients access through Java API. It has been designed so the base classes provide great recovery for MapReduce jobs by storing information in the HBase table. HBase table has the capability of automatically redistribute data to different regions as it grows. In addition, the architecture of HBase is constructed so reads and writes to the table are very consistent throughout the access.
8 Figure 7. The architure of HBase Similar to the structure of HDFS, the architecture of HBase is also in the form of Master/Slave relationship. HBase application typically will have a master node and multiple region servers as work horses. Each region server contains several region and data are stored in tables and these tables are then stored in each region. From a top down perspective, the architecture of HBase starts with the master node with responsibilities such as managing and monitoring the cluster and assigning regions to the region servers. Then under the master node there are the region servers that manage the regions. The region servers communicate directly with clients and handle the read and write requests accordingly. When the regions data exceed a limit that is set, the region server automatically gives order to the region and let it split into two region of the same size. Under the region servers are the regions. In this component, a set of table s row are stored within it. As the data grows larger, the region is split into two new regions of similar size. Now under regions there are tables that consist of rows and columns. Similar to RDBMS, each row has a primary key; the main differences in HBase are that the intersection of row and column coordinates is versioned and the rows are sorted. MongoDB MongoDB is one of the best examples of NoSQL database and it is widely used by many Fortune 500 companies to make their businesses more agile and scalable. MongoDB is a crossplatform document oriented database. MongoDB was originally created by 10gen in October 2007 and went open source in Since then, MongoDB has been widely used by several top websites and services as their back end component, this include Craigslist, ebay, Foursquare, SourceForge, and the New York Times. (MongoDB) It is an agile database that can change its schemas as the application evolves, while keeping the basic functionalities from the traditional
10 Figure 9. Performance Comparison for textbook insertion and query The above comparison shows the time it takes to insert textbook records in the first operation and the time it takes to query 2000 textbook records. As you can see the runtime for inserting records in MySQL exceeds MongoDB by almost three times and the querying runtime almost doubled. Apache HIVE Apache HIVE is a data warehouse solution on top of Hadoop Map-Reduce framework that provides similar functionalities to RDBMS. It was initially developed by Facebook, but later on it was also implemented and developed by Netflix and Amazon. Apache HIVE allows users to access the data stored in it the same way as how user would access them in a regular relational database. Hive provides the capability of generating tables and also has a query language called HiveQL. HiveQL is based on SQL thus it is very easy for common database users to learn and use it in practice. HiveQL currently has several capabilities similar to SQL. For instance, it has the functionality CREATE and DROP to manipulate tables and partitions. Most importantly, it features the function SELECT capable of joining tables on a mutual key, and filter data using the row selection techniques. Here is an example of HiveQL query. Figure 10. HiveQL query When a query is performed against Hive, the query is analyzed by a semantic analyzer and translated into a query execution. This process is then send to Map-Reduce framework and uses data stored in Hadoop Distributed File System as input.
11 Even though Apache HIVE provides similar capabilities to SQL, it cannot be compare with traditional system in certain perspectives. Hadoop jobs tend to have very long runtime in job submission and scheduling. For this reason, HiveQL query also tends to take long time before it can be completed. As a comparison, system such as Oracle will use much less data for analysis and can be completed in a fast pace. Hive is definitely not optimal in compare with traditionally established system but rather used for interactive data browsing, querying smaller data set, and for testing non production data. As the figure shows below, the architecture of Apache HIVE contains a few important components. Command line interface interacts with users and allows them to enter HiveQL queries. Driver is the ultimate processing tool that receives the queries and processes them with its components. The Metastore serves as storing the metadata on different tables and partitions. Compiler takes the query and metadata from Metastore to generate an execution plan. During this process, the optimizer takes the execution plan and translates it into an executable plan with multiple Map-Reduce steps. The executor then executes the plan generated by the compiler using Map-Reduce engine. Figure 11. Apache HIVE architecture Relationship between the topics To summarize, the project Nutch, an open source web search engine, was created as part of the Lucene project, the text search library. Then due to non-efficient time consumption for processing large amount of data, Hadoop project was initialized as a result of it. Hadoop Distributed File System, Map-Reduce, HBase, and MongoDB were all part of the Hadoop developing projects. HBase is built on top of Hadoop Distributed File System and it is created to satisfy applications that requires low-latency access to data. MongoDB serves as a NoSQL database in Hadoop and it makes it much more efficient for application with large data because of its object oriented structure versus traditional database. Hive is a data warehousing
12 architecture on top of Hadoop Map-Reduce framework for users to be capable of handling data the same way users would access a traditional relational database management system. Hadoop Distributed File System is the centralized file processing architecture and it is used by Hive to store the data. Hive uses Map-Reduce engine to execute the logical plan of the HiveQL query and retrieve data from HDFS. References  Tom Whites, Hadoop The Definitive Guide, O Reilly Media, 1005 Gravenstein Highway North, Sebastopol, CA 95472, 2011  Jeffrey Dean, Sanjay Ghemawat, MapReduce: Simplified Data Processing on Large Clusters, Communications of the ACM, New York, NY, USA, January 2008, Pages  An overview of the Hadoop/MapReduce/HBase framework and its current applications in bioinformatics, accessed: 04/21/2014  What Is Apache Hadoop?, accessed: 04/26/2014  Zhu Wei-ping, Using MongoDB to implement textbook management system instead of MySQL, Communication Software and Networks (ICCSN), 2011 IEEE 3rd International Conference. Page  MongoDB, accessed: 04/27/2014 Picture sources - (Other pictures are from the references documents))
Hadoop http://hadoop.apache.org/ What Is Apache Hadoop? The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using
Big Data and Apache Hadoop s MapReduce Michael Hahsler Computer Science and Engineering Southern Methodist University January 23, 2012 Michael Hahsler (SMU/CSE) Hadoop/MapReduce January 23, 2012 1 / 23
Hadoop IST 734 SS CHUNG Introduction What is Big Data?? Bulk Amount Unstructured Lots of Applications which need to handle huge amount of data (in terms of 500+ TB per day) If a regular machine need to
Hadoop Ecosystem B Y R A H I M A. History of Hadoop Hadoop was created by Doug Cutting, the creator of Apache Lucene, the widely used text search library. Hadoop has its origins in Apache Nutch, an open
Introduction to Hadoop HDFS and Ecosystems ANSHUL MITTAL Slides credits: Cloudera Academic Partners Program & Prof. De Liu, MSBA 6330 Harvesting Big Data Topics The goal of this presentation is to give
CSE 590: Special Topics Course ( Supercomputing ) Lecture 10 ( MapReduce& Hadoop) Rezaul A. Chowdhury Department of Computer Science SUNY Stony Brook Spring 2016 MapReduce MapReduce is a programming model
Open source software framework designed for storage and processing of large scale data on clusters of commodity hardware Created by Doug Cutting and Mike Carafella in 2005. Cutting named the program after
With Saurabh Singh email@example.com The Ohio State University February 11, 2016 Overview 1 2 3 Requirements Ecosystem Resilient Distributed Datasets (RDDs) Example Code vs Mapreduce 4 5 Source: [Tutorials
Hadoop implementation of MapReduce computational model Ján Vaňo What is MapReduce? A computational model published in a paper by Google in 2004 Based on distributed computation Complements Google s distributed
Large scale processing using Hadoop Ján Vaňo What is Hadoop? Software platform that lets one easily write and run applications that process vast amounts of data Includes: MapReduce offline computing engine
INTERNATIONAL JOURNAL OF PURE AND APPLIED RESEARCH IN ENGINEERING AND TECHNOLOGY A PATH FOR HORIZING YOUR INNOVATIVE WORK A REVIEW ON HIGH PERFORMANCE DATA STORAGE ARCHITECTURE OF BIGDATA USING HDFS MS.
Hadoop Distributed File System Dhruba Borthakur Apache Hadoop Project Management Committee firstname.lastname@example.org email@example.com Hadoop, Why? Need to process huge datasets on large clusters of computers
Chapter 7 Using Hadoop Cluster and MapReduce Modeling and Prototyping of RMS for QoS Oriented Grid Page 152 7. Using Hadoop Cluster and MapReduce for Big Data Problems The size of the databases used in
Big Data Rethink Algos and Architecture Scott Marsh Manager R&D Personal Lines Auto Pricing Agenda History Map Reduce Algorithms History Google talks about their solutions to their problems Map Reduce:
INTRODUCTION TO APACHE HADOOP MATTHIAS BRÄGER CERN GS-ASE AGENDA Introduction to Big Data Introduction to Hadoop HDFS file system Map/Reduce framework Hadoop utilities Summary BIG DATA FACTS In what timeframe
Hadoop Big Data for Processing Data and Performing Workload Girish T B 1, Shadik Mohammed Ghouse 2, Dr. B. R. Prasad Babu 3 1 M Tech Student, 2 Assosiate professor, 3 Professor & Head (PG), of Computer
A Brief Outline on Bigdata Hadoop Twinkle Gupta 1, Shruti Dixit 2 RGPV, Department of Computer Science and Engineering, Acropolis Institute of Technology and Research, Indore, India Abstract- Bigdata is
Hadoop & its Usage at Facebook Dhruba Borthakur Project Lead, Hadoop Distributed File System firstname.lastname@example.org Presented at the Storage Developer Conference, Santa Clara September 15, 2009 Outline Introduction
ESS event: Big Data in Official Statistics Antonino Virgillito, Istat v erbi v is 1 About me Head of Unit Web and BI Technologies, IT Directorate of Istat Project manager and technical coordinator of Web
NoSQL and Hadoop Technologies On Oracle Cloud Vatika Sharma 1, Meenu Dave 2 1 M.Tech. Scholar, Department of CSE, Jagan Nath University, Jaipur, India 2 Assistant Professor, Department of CSE, Jagan Nath
Department of Computer Science University of Cyprus EPL646 Advanced Topics in Databases Lecture 14 Big Data Management IV: Big-data Infrastructures (Background, IO, From NFS to HFDS) Chapter 14-15: Abideboul
An Industrial Perspective on the Hadoop Ecosystem Eldar Khalilov Pavel Valov agenda 03.12.2015 2 agenda Introduction 03.12.2015 2 agenda Introduction Research goals 03.12.2015 2 agenda Introduction Research
Open source large scale distributed data management with Google s MapReduce and Bigtable Ioannis Konstantinou Email: email@example.com Web: http://www.cslab.ntua.gr/~ikons Computing Systems Laboratory
Test report: Integration Big Data Edition Data processing goes big Dr. Götz Güttich Integration is a powerful set of tools to access, transform, move and synchronize data. With more than 450 connectors,
Hadoop & its Usage at Facebook Dhruba Borthakur Project Lead, Hadoop Distributed File System firstname.lastname@example.org Presented at the The Israeli Association of Grid Technologies July 15, 2009 Outline Architecture
MapReduce with Apache Hadoop Analysing Big Data April 2010 Gavin Heavyside email@example.com About Journey Dynamics Founded in 2006 to develop software technology to address the issues
CSE-E5430 Scalable Cloud Computing Lecture 2 Keijo Heljanko Department of Computer Science School of Science Aalto University firstname.lastname@example.org 14.9-2015 1/36 Google MapReduce A scalable batch processing
Workshop on Hadoop with Big Data Hadoop? Apache Hadoop is an open source framework for distributed storage and processing of large sets of data on commodity hardware. Hadoop enables businesses to quickly
Apache Hadoop FileSystem and its Usage in Facebook Dhruba Borthakur Project Lead, Apache Hadoop Distributed File System email@example.com Presented at Indian Institute of Technology November, 2010 http://www.facebook.com/hadoopfs
INTRODUCTION TO CASSANDRA This ebook provides a high level overview of Cassandra and describes some of its key strengths and applications. WHAT IS CASSANDRA? Apache Cassandra is a high performance, open
Comparison of Different Implementation of Inverted Indexes in Hadoop Hediyeh Baban, S. Kami Makki, and Stefan Andrei Department of Computer Science Lamar University Beaumont, Texas (hbaban, kami.makki,
AGENDA What is BIG DATA? What is Hadoop? Why Microsoft? The Microsoft BIG DATA story Hadoop PDW Our BIG DATA Roadmap BIG DATA? Volume 59% growth in annual WW information 1.2M Zetabytes (10 21 bytes) this
Hadoop and its Usage at Facebook Dhruba Borthakur firstname.lastname@example.org, June 22 rd, 2009 Who Am I? Hadoop Developer Core contributor since Hadoop s infancy Focussed on Hadoop Distributed File System Facebook
Introduction to Hadoop New York Oracle User Group Vikas Sawhney GENERAL AGENDA Driving Factors behind BIG-DATA NOSQL Database 2014 Database Landscape Hadoop Architecture Map/Reduce Hadoop Eco-system Hadoop
A Seminar report On Hadoop Submitted in partial fulfillment of the requirement for the award of degree of Bachelor of Technology in Computer Science SUBMITTED TO: www.studymafia.org SUBMITTED BY: www.studymafia.org
Lecture Data Warehouse Systems Eva Zangerle SS 2013 PART C: Novel Approaches in DW NoSQL and MapReduce Stonebraker on Data Warehouses Star and snowflake schemas are a good idea in the DW world C-Stores
Data-Intensive Computing with Map-Reduce and Hadoop Shamil Humbetov Department of Computer Engineering Qafqaz University Baku, Azerbaijan email@example.com Abstract Every day, we create 2.5 quintillion
Session: Big Data get familiar with Hadoop to use your unstructured data Udo Brede Dell Software 22 nd October 2013 10:00 Sesión B - DB2 LUW 1 Agenda Big Data The Technical Challenges Architecture of Hadoop
CS2510 Computer Operating Systems HADOOP Distributed File System Dr. Taieb Znati Computer Science Department University of Pittsburgh Outline HDF Design Issues HDFS Application Profile Block Abstraction
CS2510 Computer Operating Systems HADOOP Distributed File System Dr. Taieb Znati Computer Science Department University of Pittsburgh Outline HDF Design Issues HDFS Application Profile Block Abstraction
Cloudera CCD-333 Cloudera Certified Developer for Apache Hadoop Version: 5.6 QUESTION NO: 1 Cloudera CCD-333 Exam What is a SequenceFile? A. A SequenceFile contains a binary encoding of an arbitrary number
Prepared By : Manoj Kumar Joshi & Vikas Sawhney General Agenda Introduction to Hadoop Architecture Acknowledgement Thanks to all the authors who left their selfexplanatory images on the internet. Thanks
Processing of Hadoop using Highly Available NameNode 1 Akash Deshpande, 2 Shrikant Badwaik, 3 Sailee Nalawade, 4 Anjali Bote, 5 Prof. S. P. Kosbatwar Department of computer Engineering Smt. Kashibai Navale
BIG DATA TRENDS AND TECHNOLOGIES THE WORLD OF DATA IS CHANGING Cloud WHAT IS BIG DATA? Big data are datasets that grow so large that they become awkward to work with using onhand database management tools.
III Big Data Technologies Today, new technologies make it possible to realize value from Big Data. Big data technologies can replace highly customized, expensive legacy systems with a standard solution
BIG, MAPREDUCE & HADOOP LARGE SCALE DISTRIBUTED SYSTEMS By Jean-Pierre Lozi A tutorial for the LSDS class LARGE SCALE DISTRIBUTED SYSTEMS BIG, MAPREDUCE & HADOOP 1 OBJECTIVES OF THIS LAB SESSION The LSDS
American International Journal of Research in Science, Technology, Engineering & Mathematics Available online at http://www.iasir.net ISSN (Print): 2328-3491, ISSN (Online): 2328-3580, ISSN (CD-ROM): 2328-3629
Open source Google-style large scale data analysis with Hadoop Ioannis Konstantinou Email: firstname.lastname@example.org Web: http://www.cslab.ntua.gr/~ikons Computing Systems Laboratory School of Electrical
Xiaoming Gao Hui Li Thilina Gunarathne Outline HBase and Bigtable Storage HBase Use Cases HBase vs RDBMS Hands-on: Load CSV file to Hbase table with MapReduce Motivation Lots of Semi structured data Horizontal
Managing Big Data with Hadoop & Vertica A look at integration between the Cloudera distribution for Hadoop and the Vertica Analytic Database Copyright Vertica Systems, Inc. October 2009 Cloudera and Vertica
A STUDY ON HADOOP ARCHITECTURE FOR BIG DATA ANALYTICS Dr. Ananthi Sheshasayee 1, J V N Lakshmi 2 1 Head Department of Computer Science & Research, Quaid-E-Millath Govt College for Women, Chennai, (India)
Data Modeling for Big Data by Jinbao Zhu, Principal Software Engineer, and Allen Wang, Manager, Software Engineering, CA Technologies In the Internet era, the volume of data we deal with has grown to terabytes
How to properly misuse Hadoop Marcel Huntemann NERSC tutorial session 2/12/13 History Created by Doug Cutting (also creator of Apache Lucene). 2002 Origin in Apache Nutch (open source web search engine).
A REVIEW PAPER ON THE HADOOP DISTRIBUTED FILE SYSTEM Sneha D.Borkar 1, Prof.Chaitali S.Surtakar 2 Student of B.E., Information Technology, J.D.I.E.T, email@example.com Assistant Professor, Information
Lecture 32 Big Data 1. Big Data problem 2. Why the excitement about big data 3. What is MapReduce 4. What is Hadoop 5. Get started with Hadoop 1 2 Big Data Problems Data explosion Data from users on social
Big Data Processing, 2014/15 Lecture 10: HBase!! Claudia Hauff (Web Information Systems)! firstname.lastname@example.org 1 Course content Introduction Data streams 1 & 2 The MapReduce paradigm Looking behind the
BIG DATA What it is and how to use? Lauri Ilison, PhD Data Scientist 21.11.2014 Big Data definition? There is no clear definition for BIG DATA BIG DATA is more of a concept than precise term 1 21.11.14
NoSQL Data Base Basics Course Notes in Transparency Format Cloud Computing MIRI (CLC-MIRI) UPC Master in Innovation & Research in Informatics Spring- 2013 Jordi Torres, UPC - BSC www.jorditorres.eu HDFS
Introduction to Hadoop 1 What is Hadoop? the big data revolution extracting value from data cloud computing 2 Understanding MapReduce the word count problem more examples MCS 572 Lecture 24 Introduction
BIG DATA WITH HADOOP EXPERIMENTATION HARRISON CARRANZA Marist College APARICIO CARRANZA NYC College of Technology CUNY ECC Conference 2016 Poughkeepsie, NY, June 12-14, 2016 Marist College AGENDA Contents
Hadoop Distributed File System Jordan Prosch, Matt Kipps Outline - Background - Architecture - Comments & Suggestions Background What is HDFS? Part of Apache Hadoop - distributed storage What is Hadoop?
Sector vs. Hadoop A Brief Comparison Between the Two Systems Background Sector is a relatively new system that is broadly comparable to Hadoop, and people want to know what are the differences. Is Sector
Introduction to Big Data! with Apache Spark" UC#BERKELEY# This Lecture" The Big Data Problem" Hardware for Big Data" Distributing Work" Handling Failures and Slow Machines" Map Reduce and Complex Jobs"
JECET; March 2015-May 2015; Sec. B; Vol.4.No.2, 202-209. E-ISSN: 2278 179X Journal of Environmental Science, Computer Science and Engineering & Technology An International Peer Review E-3 Journal of Sciences
HBase A Comprehensive Introduction James Chin, Zikai Wang Monday, March 14, 2011 CS 227 (Topics in Database Management) CIT 367 Overview Overview: History Began as project by Powerset to process massive
Analytics March 2015 White paper Why NoSQL? Your database options in the new non-relational world 2 Why NoSQL? Contents 2 New types of apps are generating new types of data 2 A brief history of NoSQL 3
Hadoop Ecosystem Overview CMSC 491 Hadoop-Based Distributed Computing Spring 2015 Adam Shook Agenda Introduce Hadoop projects to prepare you for your group work Intimate detail will be provided in future
W H I T E P A P E R Deriving Intelligence from Large Data Using Hadoop and Applying Analytics Abstract This white paper is focused on discussing the challenges facing large scale data processing and the
Data Warehouse and Hive Presented By: Shalva Gelenidze Supervisor: Nodar Momtselidze Decision support systems Decision Support Systems allowed managers, supervisors, and executives to once again see the
Composite Data Virtualization Composite Data Virtualization And NOSQL Data Stores Composite Software October 2010 TABLE OF CONTENTS INTRODUCTION... 3 BUSINESS AND IT DRIVERS... 4 NOSQL DATA STORES LANDSCAPE...
L1: Introduction to Hadoop Feng Li email@example.com School of Statistics and Mathematics Central University of Finance and Economics Revision: December 1, 2014 Today we are going to learn... 1 General
ISSN: 2321-7782 (Online) Volume 3, Issue 4, April 2015 International Journal of Advance Research in Computer Science and Management Studies Research Article / Survey Paper / Case Study Available online
Introduction to NoSQL Databases and MapReduce Tore Risch Information Technology Uppsala University 2014-05-12 What is a NoSQL Database? 1. A key/value store Basic index manager, no complete query language
Hadoop Distributed File System T-111.5550 Seminar On Multimedia 2009-11-11 Eero Kurkela Agenda Introduction Flesh and bones of HDFS Architecture Accessing data Data replication strategy Fault tolerance
Hadoop Architecture Part 1 Node, Rack and Cluster: A node is simply a computer, typically non-enterprise, commodity hardware for nodes that contain data. Consider we have Node 1.Then we can add more nodes,
BIG DATA TECHNOLOGY Hadoop Ecosystem Agenda Background What is Big Data Solution Objective Introduction to Hadoop Hadoop Ecosystem Hybrid EDW Model Predictive Analysis using Hadoop Conclusion What is Big
WHITE PAPER Page 1 of 8 Data Mining in the Swamp Taming Unruly Data with Cloud Computing By John Brothers Business Intelligence is all about making better decisions from the data you have. However, all