Mining Large Datasets: Case of Mining Graph Data in the Cloud
|
|
|
- Bartholomew York
- 8 years ago
- Views:
Transcription
1 Mining Large Datasets: Case of Mining Graph Data in the Cloud Sabeur Aridhi PhD in Computer Science with Laurent d Orazio, Mondher Maddouri and Engelbert Mephu Nguifo 16/05/2014 Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 1 / 50
2 Context and motivations Background Application domains Computer networks, Social networks, Bioinformatics, Chemoinformatics. Graph representation Data modeling. Identifying relationship patterns and rules. Protein structure Chemical compound Social network Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 2 / 50
3 Context and motivations Background Mining graph data Graph mining aims to find patterns, hidden relations and behaviors in data. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 3 / 50
4 Context and motivations Background Mining graph data Graph mining aims to find patterns, hidden relations and behaviors in data. Mining graph goals Computing graph properties: Density, diameter, radius,... Mining substructures from graph databases. Substructures: paths, trees, subgraphs. Frequent Subgraph Mining (FSM) task. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 3 / 50
5 Context and motivations Background Availability of graph data Exponential growth in both size and number of graphs in databases. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 4 / 50
6 Context and motivations Background Availability of graph data Exponential growth in both size and number of graphs in databases. Availability of graph data sources: The protein data bank (PDB) contains of protein 3D structures. Facebook loads 60 terabytes of new data every day [Thusoo 2010]. Google processes 20 petabytes of data per day [Dean 2008]. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 4 / 50
7 Context and motivations Background Availability of graph data Exponential growth in both size and number of graphs in databases. Availability of graph data sources: The protein data bank (PDB) contains of protein 3D structures. Facebook loads 60 terabytes of new data every day [Thusoo 2010]. Google processes 20 petabytes of data per day [Dean 2008]. 3Vs of Big Data (Volume, Velocity and Variety). Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 4 / 50
![Facebook loads 60 terabytes of new data every day [Thusoo 2010].](/docs-images/44/2137691/images/page_7.jpg "Google processes 20 petabytes of data per day [Dean 2008]. 3Vs of Big Data (Volume, Velocity and Variety).")
8 Context and motivations Background Availability of graph data Exponential growth in both size and number of graphs in databases. Availability of graph data sources: The protein data bank (PDB) contains of protein 3D structures. Facebook loads 60 terabytes of new data every day [Thusoo 2010]. Google processes 20 petabytes of data per day [Dean 2008]. 3Vs of Big Data (Volume, Velocity and Variety). Availability of cloud computing environments. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 4 / 50
![Facebook loads 60 terabytes of new data every day [Thusoo 2010]. Google processes 20 petabytes of data per day [Dean 2008].](/docs-images/44/2137691/images/page_8.jpg "3Vs of Big Data (Volume, Velocity and Variety). Availability of cloud computing environments.")
9 Context and motivations Background In this work We are interested to FSM from graph databases. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 5 / 50
10 Context and motivations Background In this work We are interested to FSM from graph databases. Frequent subgraph mining algorithms Various approaches of FSM. Existing approaches are mainly: Tested on centralized computing systems. Evaluated on relatively small databases. Few works for FSM in the cloud. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 5 / 50
11 Goals Questions Distributed FSM from large graph database. Data/computation distribution. Tuning cloud parameters. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 6 / 50
12 Outline 1 Background 2 3 Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 7 / 50
13 Outline Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works 1 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works 2 3 Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 8 / 50
14 Outline Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works 1 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works 2 Distributed subgraph mining in the cloud 3 Prospects Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 9 / 50
15 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works Graph A graph is denoted as G = (V,E) where V is a set of nodes and E is a set of edges. Subgraph A graph G = (V,E ) is a subgraph of another graph G = (V,E) iff: V V, and E E (V V ). Density The density of a graph G = (V,E) is calculated by density(g) = 2 E ( V ( V 1)). Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 10 / 50
16 Outline Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works 1 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works 2 Distributed subgraph mining in the cloud 3 Prospects Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 11 / 50
17 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works Cloud computing Large number of computers that are connected via Internet. Applications delivered as services. Hardware and system software delivered as services. Pay as you go. Cloud services can be rapidly and elastically provisioned. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 12 / 50
18 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works Service models Software as a Service (SaaS). Platform as a Service (PaaS), Infrastructure as a Service (IaaS), Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 13 / 50
19 Outline Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works 1 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works 2 Distributed subgraph mining in the cloud 3 Prospects Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 14 / 50
20 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works MapReduce framework A framework for processing huge datasets. Large number of computers and task/node failures. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 15 / 50
21 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works MapReduce framework A framework for processing huge datasets. Large number of computers and task/node failures. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 15 / 50
22 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 16 / 50
23 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works SPARK framework A general engine for large-scale data processing. Combine SQL, streaming, and complex analytics. It offers several high-level operators that make it easy to build parallel applications. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 17 / 50
24 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works SHARK framework A distributed SQL query engine for Hadoop. Based on SPARK and uses the existing Hive client and metastore. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 18 / 50
25 Outline Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works 1 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works 2 Distributed subgraph mining in the cloud 3 Prospects Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 19 / 50
26 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works Cloud-based FSM techniques Cloud-based FSM approaches from: 1 Single large graphs (MRPF [Liu 2009] and Wu etal. s approach [Wu 2010]). MRPF [Liu 2009], and Wu etal. s approach [Wu 2010]. 2 Massive graph databases (Hill etal. s [Hill 2012] and Luo etal. s [Luo 2011]). Hill etal. s [Hill 2012], and Luo etal. s [Luo 2011]. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 20 / 50
27 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 21 / 50
28 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 21 / 50
29 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works In this work We focus on distributed FSM techniques from large graph databases. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 22 / 50
30 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works In this work We focus on distributed FSM techniques from large graph databases. Three crucial problems with existing approaches: Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 22 / 50
31 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works In this work We focus on distributed FSM techniques from large graph databases. Three crucial problems with existing approaches: 1 No data partitioning according to data characteristics. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 22 / 50
32 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works In this work We focus on distributed FSM techniques from large graph databases. Three crucial problems with existing approaches: 1 No data partitioning according to data characteristics. 2 Do not include the monetary aspect of cloud computing. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 22 / 50
33 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works In this work We focus on distributed FSM techniques from large graph databases. Three crucial problems with existing approaches: 1 No data partitioning according to data characteristics. 2 Do not include the monetary aspect of cloud computing. 3 Construct the final set of frequent subgraphs iteratively. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 22 / 50
34 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works In this work We focus on distributed FSM techniques from large graph databases. Three crucial problems with existing approaches: 1 No data partitioning according to data characteristics. 2 Do not include the monetary aspect of cloud computing. 3 Construct the final set of frequent subgraphs iteratively. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 22 / 50
35 Distributed subgraph mining in the cloud Outline 1 Background 2 Distributed subgraph mining in the cloud 3 Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 23 / 50
36 Distributed subgraph mining in the cloud Outline 1 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works 2 Distributed subgraph mining in the cloud 3 Prospects Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 24 / 50
37 Distributed subgraph mining in the cloud Problem formulation Notations DB ={G 1,...,G K } is a large scale graph database, SM ={M 1,...,M N } is a set of distributed machines, θ [0,1] is a minimum support threshold, Part(DB) ={Part 1 (DB),...,Part N (DB)} is a partitioning of the database over SM such that Part j (DB) DB is a non-empty subset of DB, Ni=1 {Part i (DB)} = DB,and, i j,part i (DB) Part j (DB) = /0. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 25 / 50
38 Distributed subgraph mining in the cloud Problem formulation Globally frequent subgraph For a given minimum support threshold θ [0,1], G is globally frequent subgraph if Support(G,DB) θ. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 26 / 50
39 Distributed subgraph mining in the cloud Problem formulation Globally frequent subgraph For a given minimum support threshold θ [0,1], G is globally frequent subgraph if Support(G,DB) θ. Locally frequent subgraph For a given minimum support threshold θ [0,1] and a tolerance rate τ [0,1], G is locally frequent subgraph at site i if Support(G,Part i (DB)) ((1 τ) θ). Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 26 / 50
40 Distributed subgraph mining in the cloud Problem formulation Globally frequent subgraph For a given minimum support threshold θ [0,1], G is globally frequent subgraph if Support(G,DB) θ. Locally frequent subgraph For a given minimum support threshold θ [0,1] and a tolerance rate τ [0,1], G is locally frequent subgraph at site i if Support(G,Part i (DB)) ((1 τ) θ). Loss rate Given S 1 and S 2 two sets of subgraphs with S 2 S 1 and S 1 /0, we define the loss rate in S 2 compared to S 1 by: LossRate(S 1,S 2 ) = S 1 S 2. S 1 Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 26 / 50
41 Distributed subgraph mining in the cloud System overview Approach overview Two-step approach: 1 Partitioning step, 2 Mining step. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 27 / 50
42 Distributed subgraph mining in the cloud Partitioning step Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 28 / 50
43 Distributed subgraph mining in the cloud Partitioning step Partitioning methods Many partitioning methods are possible. We consider: 1 MRGP: the default MapReduce partitioning method. 2 DGP: a density-based partitioning method. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 29 / 50
44 Distributed subgraph mining in the cloud Partitioning step Partitioning methods Many partitioning methods are possible. We consider: 1 MRGP: the default MapReduce partitioning method. 2 DGP: a density-based partitioning method. MRGP Based on the size on disk. Map-skew problems (highly variable runtimes). No data characteristics included. DGP Based on graph density. May ensures load balancing among machines. May exploit other data characteristics. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 29 / 50
45 Distributed subgraph mining in the cloud Map-Skew problems Map-skew Skew: highly variable task runtimes. Origin: Characteristics of the algorithm. Characteristics of the dataset. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 30 / 50
46 Distributed subgraph mining in the cloud Partitioning step: DGP method DGP overview Two-levels approach: 1 Dividing the graph database into B buckets, 2 Constructing the final list of partitions. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 31 / 50
47 Distributed subgraph mining in the cloud Distributed FSM step Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 32 / 50
48 Distributed subgraph mining in the cloud Distributed FSM step Distributed FSM step A single MapReduce job. Input: a set of partitions. Output: the set of globally frequent subgraphs. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 33 / 50
49 Distributed subgraph mining in the cloud Distributed FSM step Distributed FSM step A single MapReduce job. Input: a set of partitions. Output: the set of globally frequent subgraphs. In the Mapper machine We run a subgraph mining technique on each partition in parallel. Mapper i produces a set of locally frequent subgraphs. Pairs of s,support(s,part i (DB)). Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 33 / 50
50 Distributed subgraph mining in the cloud Distributed FSM step Distributed FSM step A single MapReduce job. Input: a set of partitions. Output: the set of globally frequent subgraphs. In the Mapper machine We run a subgraph mining technique on each partition in parallel. Mapper i produces a set of locally frequent subgraphs. Pairs of s,support(s,part i (DB)). In the Reducer machine We compute the set of globally frequent subgraphs Pairs of s, Support(s, DB). No false positives generated. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 33 / 50
51 Distributed subgraph mining in the cloud Distributed FSM step Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 34 / 50
52 Distributed subgraph mining in the cloud Experiments Implementation platform Hadoop release, an open source version of MapReduce. A local cluster with five nodes. A Quad-Core AMD Opteron(TM) Processor GHz CPU. 4 GB of memory. Three existing subgraph miners: gspan, FSG and Gaston. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 35 / 50
53 Distributed subgraph mining in the cloud Experiments Implementation platform Hadoop release, an open source version of MapReduce. A local cluster with five nodes. A Quad-Core AMD Opteron(TM) Processor GHz CPU. 4 GB of memory. Three existing subgraph miners: gspan, FSG and Gaston. Datasets Six datasets composed of synthetic and real ones. Different parameters such as: the number of graphs, the average size of graphs in terms of edges and the size on disk. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 35 / 50
54 Distributed subgraph mining in the cloud Experiments Table: Experimental data. Dataset Type Number of graphs Size on disk Average size DS1 Synthetic 20, MB [50-100] DS2 Synthetic 100, MB [50-70] DS3 Real 274, MB [40-50] DS4 Synthetic 500, MB [60-70] DS5 Synthetic 1,500, GB [60-70] DS6 Synthetic 100,000, GB [20-100] Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 36 / 50
55 Distributed subgraph mining in the cloud Experiments Experimental protocol Three types of experiments: 1 Quality: MRGP vs. DGP. Comparison with random sampling method. 2 Load balancing and execution time: Performance evaluation tests. Scalability tests. 3 Impact of MapReduce parameters. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 37 / 50
56 Distributed subgraph mining in the cloud Experiments: Quality gspan, θ = 30% gspan, θ = 50% Table: Number of false positives of the sampling method. Dataset DS1 DS2 DS3 Support θ (%) Number of subgraphs gspan FSG Gaston Number of Number of Number of Number of false positives subgraphs false positives subgraphs Number of false positives Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 38 / 50
57 Distributed subgraph mining in the cloud Experiments: Quality Result quality Distributed FSM vs. classic one. Low values of loss rate with DGP. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 39 / 50
58 Distributed subgraph mining in the cloud Experiments: Load balancing and execution time Runtime and workload distribution DGP enhances the performance of our approach. Balanced workload distribution over the distributed machines. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 40 / 50
59 Distributed subgraph mining in the cloud Experiments: Impact of MapReduce parameters Chunk size and replication factor High runtime values with small chunk size. The runtime is inversely proportional to the replication factor. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 41 / 50
60 Prospects Outline 1 Background 2 3 Prospects Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 42 / 50
61 Prospects Outline 1 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works 2 Distributed subgraph mining in the cloud 3 Prospects Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 43 / 50
62 Prospects At a glance A MapReduce-based framework for distributing FSM in the cloud. Many partitioning techniques of the input graph database. Many subgraph extractors. A data partitioning technique that considers data characteristics. It uses the density of graphs. Balanced computational load over the distributed machines. Experiment validation. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 44 / 50
63 Prospects Outline 1 Background Graph mining Cloud computing Frameworks for large data processing in the cloud Related works 2 Distributed subgraph mining in the cloud 3 Prospects Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 45 / 50
64 Prospects Prospects Improvements of the cloud-based FSM approach Different topological graph properties. Relation between database characteristics and the choice of the partitioning technique. Open questions What is the maximum number of buckets and/or partitions? What is the size of chunk to use in the partitioning step and in the distributed subgraph mining step? Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 46 / 50
65 Prospects Prospects Performance and scalability improvement Runtime improvement with task and node failures. Ensure minimal loss of information in the case of failures. Portability improvement Extension of our approach to SPARK, SHARK, Open Computing Language (OpenCL) and Message Passing Interface (MPI). Deployment of the approach Study the integration of our approach to recent distributed machine learning toolkits such as the Apache Mahout project and SystemML. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 47 / 50
66 Prospects Work in progress Cost models Cost models for distributing frequent pattern mining in the cloud. Application to distributed frequent subgraphs. Objective functions that consider the needs of customers: Budget limit, Response time limit, and Result quality limit. Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 48 / 50
67 Prospects Publications Journals S. Aridhi, L. d Orazio, M. Maddouri et E. Mephu Nguifo. Un partitionnement basé sur la densité de graphe pour approcher la fouille distribuée de sous-graphes fréquents. Techniques et Science Informatiques. (Accepted) S. Aridhi, L. d Orazio, M. Maddouri and E. Mephu Nguifo. Density-based data partitioning strategy to approximate large scale subgraph mining. Information Systems, Elsevier, ISSN , (In press) Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 49 / 50
68 Thank You! Sabeur Aridhi Mining Large Datasets - Big Data Forum - Lyon 50 / 50
Convex Optimization for Big Data: Lecture 2: Frameworks for Big Data Analytics
Convex Optimization for Big Data: Lecture 2: Frameworks for Big Data Analytics Sabeur Aridhi Aalto University, Finland Sabeur Aridhi Frameworks for Big Data Analytics 1 / 59 Introduction Contents 1 Introduction
CSE-E5430 Scalable Cloud Computing Lecture 2
CSE-E5430 Scalable Cloud Computing Lecture 2 Keijo Heljanko Department of Computer Science School of Science Aalto University keijo.heljanko@aalto.fi 14.9-2015 1/36 Google MapReduce A scalable batch processing
A Novel Cloud Based Elastic Framework for Big Data Preprocessing
School of Systems Engineering A Novel Cloud Based Elastic Framework for Big Data Preprocessing Omer Dawelbeit and Rachel McCrindle October 21, 2014 University of Reading 2008 www.reading.ac.uk Overview
Scalable Cloud Computing Solutions for Next Generation Sequencing Data
Scalable Cloud Computing Solutions for Next Generation Sequencing Data Matti Niemenmaa 1, Aleksi Kallio 2, André Schumacher 1, Petri Klemelä 2, Eija Korpelainen 2, and Keijo Heljanko 1 1 Department of
Large-Scale Data Processing
Large-Scale Data Processing Eiko Yoneki eiko.yoneki@cl.cam.ac.uk http://www.cl.cam.ac.uk/~ey204 Systems Research Group University of Cambridge Computer Laboratory 2010s: Big Data Why Big Data now? Increase
INTERNATIONAL JOURNAL OF PURE AND APPLIED RESEARCH IN ENGINEERING AND TECHNOLOGY
INTERNATIONAL JOURNAL OF PURE AND APPLIED RESEARCH IN ENGINEERING AND TECHNOLOGY A PATH FOR HORIZING YOUR INNOVATIVE WORK A REVIEW ON HIGH PERFORMANCE DATA STORAGE ARCHITECTURE OF BIGDATA USING HDFS MS.
Hadoop: A Framework for Data- Intensive Distributed Computing. CS561-Spring 2012 WPI, Mohamed Y. Eltabakh
1 Hadoop: A Framework for Data- Intensive Distributed Computing CS561-Spring 2012 WPI, Mohamed Y. Eltabakh 2 What is Hadoop? Hadoop is a software framework for distributed processing of large datasets
Hadoop Architecture. Part 1
Hadoop Architecture Part 1 Node, Rack and Cluster: A node is simply a computer, typically non-enterprise, commodity hardware for nodes that contain data. Consider we have Node 1.Then we can add more nodes,
Jeffrey D. Ullman slides. MapReduce for data intensive computing
Jeffrey D. Ullman slides MapReduce for data intensive computing Single-node architecture CPU Machine Learning, Statistics Memory Classical Data Mining Disk Commodity Clusters Web data sets can be very
MapReduce and Distributed Data Analysis. Sergei Vassilvitskii Google Research
MapReduce and Distributed Data Analysis Google Research 1 Dealing With Massive Data 2 2 Dealing With Massive Data Polynomial Memory Sublinear RAM Sketches External Memory Property Testing 3 3 Dealing With
Chapter 7. Using Hadoop Cluster and MapReduce
Chapter 7 Using Hadoop Cluster and MapReduce Modeling and Prototyping of RMS for QoS Oriented Grid Page 152 7. Using Hadoop Cluster and MapReduce for Big Data Problems The size of the databases used in
Log Mining Based on Hadoop s Map and Reduce Technique
Log Mining Based on Hadoop s Map and Reduce Technique ABSTRACT: Anuja Pandit Department of Computer Science, anujapandit25@gmail.com Amruta Deshpande Department of Computer Science, amrutadeshpande1991@gmail.com
Open source software framework designed for storage and processing of large scale data on clusters of commodity hardware
Open source software framework designed for storage and processing of large scale data on clusters of commodity hardware Created by Doug Cutting and Mike Carafella in 2005. Cutting named the program after
Architectures for Big Data Analytics A database perspective
Architectures for Big Data Analytics A database perspective Fernando Velez Director of Product Management Enterprise Information Management, SAP June 2013 Outline Big Data Analytics Requirements Spectrum
BENCHMARKING CLOUD DATABASES CASE STUDY on HBASE, HADOOP and CASSANDRA USING YCSB
BENCHMARKING CLOUD DATABASES CASE STUDY on HBASE, HADOOP and CASSANDRA USING YCSB Planet Size Data!? Gartner s 10 key IT trends for 2012 unstructured data will grow some 80% over the course of the next
Approaches for parallel data loading and data querying
78 Approaches for parallel data loading and data querying Approaches for parallel data loading and data querying Vlad DIACONITA The Bucharest Academy of Economic Studies diaconita.vlad@ie.ase.ro This paper
Market Basket Analysis Algorithm on Map/Reduce in AWS EC2
Market Basket Analysis Algorithm on Map/Reduce in AWS EC2 Jongwook Woo Computer Information Systems Department California State University Los Angeles jwoo5@calstatela.edu Abstract As the web, social networking,
Hadoop. http://hadoop.apache.org/ Sunday, November 25, 12
Hadoop http://hadoop.apache.org/ What Is Apache Hadoop? The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using
Hadoop Ecosystem B Y R A H I M A.
Hadoop Ecosystem B Y R A H I M A. History of Hadoop Hadoop was created by Doug Cutting, the creator of Apache Lucene, the widely used text search library. Hadoop has its origins in Apache Nutch, an open
BIG DATA What it is and how to use?
BIG DATA What it is and how to use? Lauri Ilison, PhD Data Scientist 21.11.2014 Big Data definition? There is no clear definition for BIG DATA BIG DATA is more of a concept than precise term 1 21.11.14
How To Scale Out Of A Nosql Database
Firebird meets NoSQL (Apache HBase) Case Study Firebird Conference 2011 Luxembourg 25.11.2011 26.11.2011 Thomas Steinmaurer DI +43 7236 3343 896 thomas.steinmaurer@scch.at www.scch.at Michael Zwick DI
A Brief Outline on Bigdata Hadoop
A Brief Outline on Bigdata Hadoop Twinkle Gupta 1, Shruti Dixit 2 RGPV, Department of Computer Science and Engineering, Acropolis Institute of Technology and Research, Indore, India Abstract- Bigdata is
Finding Insights & Hadoop Cluster Performance Analysis over Census Dataset Using Big-Data Analytics
Finding Insights & Hadoop Cluster Performance Analysis over Census Dataset Using Big-Data Analytics Dharmendra Agawane 1, Rohit Pawar 2, Pavankumar Purohit 3, Gangadhar Agre 4 Guide: Prof. P B Jawade 2
Advances in Natural and Applied Sciences
AENSI Journals Advances in Natural and Applied Sciences ISSN:1995-0772 EISSN: 1998-1090 Journal home page: www.aensiweb.com/anas Clustering Algorithm Based On Hadoop for Big Data 1 Jayalatchumy D. and
Surfing the Data Tsunami: A New Paradigm for Big Data Processing and Analytics
Surfing the Data Tsunami: A New Paradigm for Big Data Processing and Analytics Dr. Liangxiu Han Future Networks and Distributed Systems Group (FUNDS) School of Computing, Mathematics and Digital Technology,
Outline. High Performance Computing (HPC) Big Data meets HPC. Case Studies: Some facts about Big Data Technologies HPC and Big Data converging
Outline High Performance Computing (HPC) Towards exascale computing: a brief history Challenges in the exascale era Big Data meets HPC Some facts about Big Data Technologies HPC and Big Data converging
MapReduce and Hadoop Distributed File System V I J A Y R A O
MapReduce and Hadoop Distributed File System 1 V I J A Y R A O The Context: Big-data Man on the moon with 32KB (1969); my laptop had 2GB RAM (2009) Google collects 270PB data in a month (2007), 20000PB
MapReduce and Hadoop. Aaron Birkland Cornell Center for Advanced Computing. January 2012
MapReduce and Hadoop Aaron Birkland Cornell Center for Advanced Computing January 2012 Motivation Simple programming model for Big Data Distributed, parallel but hides this Established success at petabyte
Big Data. Lyle Ungar, University of Pennsylvania
Big Data Big data will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus. McKinsey Data Scientist: The Sexiest Job of the 21st Century -
Where We Are. References. Cloud Computing. Levels of Service. Cloud Computing History. Introduction to Data Management CSE 344
Where We Are Introduction to Data Management CSE 344 Lecture 25: DBMS-as-a-service and NoSQL We learned quite a bit about data management see course calendar Three topics left: DBMS-as-a-service and NoSQL
MapReduce and Hadoop Distributed File System
MapReduce and Hadoop Distributed File System 1 B. RAMAMURTHY Contact: Dr. Bina Ramamurthy CSE Department University at Buffalo (SUNY) bina@buffalo.edu http://www.cse.buffalo.edu/faculty/bina Partially
Analytics in the Cloud. Peter Sirota, GM Elastic MapReduce
Analytics in the Cloud Peter Sirota, GM Elastic MapReduce Data-Driven Decision Making Data is the new raw material for any business on par with capital, people, and labor. What is Big Data? Terabytes of
Parallel Databases. Parallel Architectures. Parallelism Terminology 1/4/2015. Increase performance by performing operations in parallel
Parallel Databases Increase performance by performing operations in parallel Parallel Architectures Shared memory Shared disk Shared nothing closely coupled loosely coupled Parallelism Terminology Speedup:
Big Systems, Big Data
Big Systems, Big Data When considering Big Distributed Systems, it can be noted that a major concern is dealing with data, and in particular, Big Data Have general data issues (such as latency, availability,
Architectural patterns for building real time applications with Apache HBase. Andrew Purtell Committer and PMC, Apache HBase
Architectural patterns for building real time applications with Apache HBase Andrew Purtell Committer and PMC, Apache HBase Who am I? Distributed systems engineer Principal Architect in the Big Data Platform
Hadoop IST 734 SS CHUNG
Hadoop IST 734 SS CHUNG Introduction What is Big Data?? Bulk Amount Unstructured Lots of Applications which need to handle huge amount of data (in terms of 500+ TB per day) If a regular machine need to
Scaling Out With Apache Spark. DTL Meeting 17-04-2015 Slides based on https://www.sics.se/~amir/files/download/dic/spark.pdf
Scaling Out With Apache Spark DTL Meeting 17-04-2015 Slides based on https://www.sics.se/~amir/files/download/dic/spark.pdf Your hosts Mathijs Kattenberg Technical consultant Jeroen Schot Technical consultant
Real Time Fraud Detection With Sequence Mining on Big Data Platform. Pranab Ghosh Big Data Consultant IEEE CNSV meeting, May 6 2014 Santa Clara, CA
Real Time Fraud Detection With Sequence Mining on Big Data Platform Pranab Ghosh Big Data Consultant IEEE CNSV meeting, May 6 2014 Santa Clara, CA Open Source Big Data Eco System Query (NOSQL) : Cassandra,
Big Data Systems CS 5965/6965 FALL 2015
Big Data Systems CS 5965/6965 FALL 2015 Today General course overview Expectations from this course Q&A Introduction to Big Data Assignment #1 General Course Information Course Web Page http://www.cs.utah.edu/~hari/teaching/fall2015.html
Big Data and Apache Hadoop s MapReduce
Big Data and Apache Hadoop s MapReduce Michael Hahsler Computer Science and Engineering Southern Methodist University January 23, 2012 Michael Hahsler (SMU/CSE) Hadoop/MapReduce January 23, 2012 1 / 23
Energy Efficient MapReduce
Energy Efficient MapReduce Motivation: Energy consumption is an important aspect of datacenters efficiency, the total power consumption in the united states has doubled from 2000 to 2005, representing
Viswanath Nandigam Sriram Krishnan Chaitan Baru
Viswanath Nandigam Sriram Krishnan Chaitan Baru Traditional Database Implementations for large-scale spatial data Data Partitioning Spatial Extensions Pros and Cons Cloud Computing Introduction Relevance
Big Data on Microsoft Platform
Big Data on Microsoft Platform Prepared by GJ Srinivas Corporate TEG - Microsoft Page 1 Contents 1. What is Big Data?...3 2. Characteristics of Big Data...3 3. Enter Hadoop...3 4. Microsoft Big Data Solutions...4
marlabs driving digital agility WHITEPAPER Big Data and Hadoop
marlabs driving digital agility WHITEPAPER Big Data and Hadoop Abstract This paper explains the significance of Hadoop, an emerging yet rapidly growing technology. The prime goal of this paper is to unveil
Market Basket Analysis Algorithm with Map/Reduce of Cloud Computing
Market Basket Analysis Algorithm with Map/Reduce of Cloud Computing Jongwook Woo Computer Information Systems Department California State University Los Angeles, CA Abstract Map/Reduce approach has been
A REVIEW PAPER ON THE HADOOP DISTRIBUTED FILE SYSTEM
A REVIEW PAPER ON THE HADOOP DISTRIBUTED FILE SYSTEM Sneha D.Borkar 1, Prof.Chaitali S.Surtakar 2 Student of B.E., Information Technology, J.D.I.E.T, sborkar95@gmail.com Assistant Professor, Information
BIG DATA TRENDS AND TECHNOLOGIES
BIG DATA TRENDS AND TECHNOLOGIES THE WORLD OF DATA IS CHANGING Cloud WHAT IS BIG DATA? Big data are datasets that grow so large that they become awkward to work with using onhand database management tools.
Hadoop. MPDL-Frühstück 9. Dezember 2013 MPDL INTERN
Hadoop MPDL-Frühstück 9. Dezember 2013 MPDL INTERN Understanding Hadoop Understanding Hadoop What's Hadoop about? Apache Hadoop project (started 2008) downloadable open-source software library (current
Introduction to Big Data! with Apache Spark" UC#BERKELEY#
Introduction to Big Data! with Apache Spark" UC#BERKELEY# This Lecture" The Big Data Problem" Hardware for Big Data" Distributing Work" Handling Failures and Slow Machines" Map Reduce and Complex Jobs"
Overview on Graph Datastores and Graph Computing Systems. -- Litao Deng (Cloud Computing Group) 06-08-2012
Overview on Graph Datastores and Graph Computing Systems -- Litao Deng (Cloud Computing Group) 06-08-2012 Graph - Everywhere 1: Friendship Graph 2: Food Graph 3: Internet Graph Most of the relationships
Keywords: Big Data, HDFS, Map Reduce, Hadoop
Volume 5, Issue 7, July 2015 ISSN: 2277 128X International Journal of Advanced Research in Computer Science and Software Engineering Research Paper Available online at: www.ijarcsse.com Configuration Tuning
Efficient Analysis of Big Data Using Map Reduce Framework
Efficient Analysis of Big Data Using Map Reduce Framework Dr. Siddaraju 1, Sowmya C L 2, Rashmi K 3, Rahul M 4 1 Professor & Head of Department of Computer Science & Engineering, 2,3,4 Assistant Professor,
Introduction to Hadoop HDFS and Ecosystems. Slides credits: Cloudera Academic Partners Program & Prof. De Liu, MSBA 6330 Harvesting Big Data
Introduction to Hadoop HDFS and Ecosystems ANSHUL MITTAL Slides credits: Cloudera Academic Partners Program & Prof. De Liu, MSBA 6330 Harvesting Big Data Topics The goal of this presentation is to give
# Not a part of 1Z0-061 or 1Z0-144 Certification test, but very important technology in BIG DATA Analysis
Section 9 : Case Study # Objectives of this Session The Motivation For Hadoop What problems exist with traditional large-scale computing systems What requirements an alternative approach should have How
Map Reduce & Hadoop Recommended Text:
Big Data Map Reduce & Hadoop Recommended Text:! Large datasets are becoming more common The New York Stock Exchange generates about one terabyte of new trade data per day. Facebook hosts approximately
International Journal of Innovative Research in Information Security (IJIRIS) ISSN: 2349-7017(O) Volume 1 Issue 3 (September 2014)
SURVEY ON BIG DATA PROCESSING USING HADOOP, MAP REDUCE N.Alamelu Menaka * Department of Computer Applications Dr.Jabasheela Department of Computer Applications Abstract-We are in the age of big data which
Lecture 32 Big Data. 1. Big Data problem 2. Why the excitement about big data 3. What is MapReduce 4. What is Hadoop 5. Get started with Hadoop
Lecture 32 Big Data 1. Big Data problem 2. Why the excitement about big data 3. What is MapReduce 4. What is Hadoop 5. Get started with Hadoop 1 2 Big Data Problems Data explosion Data from users on social
Map-Reduce for Machine Learning on Multicore
Map-Reduce for Machine Learning on Multicore Chu, et al. Problem The world is going multicore New computers - dual core to 12+-core Shift to more concurrent programming paradigms and languages Erlang,
GeoGrid Project and Experiences with Hadoop
GeoGrid Project and Experiences with Hadoop Gong Zhang and Ling Liu Distributed Data Intensive Systems Lab (DiSL) Center for Experimental Computer Systems Research (CERCS) Georgia Institute of Technology
Big Data Analytics Hadoop and Spark
Big Data Analytics Hadoop and Spark Shelly Garion, Ph.D. IBM Research Haifa 1 What is Big Data? 2 What is Big Data? Big data usually includes data sets with sizes beyond the ability of commonly used software
Apache Hadoop. Alexandru Costan
1 Apache Hadoop Alexandru Costan Big Data Landscape No one-size-fits-all solution: SQL, NoSQL, MapReduce, No standard, except Hadoop 2 Outline What is Hadoop? Who uses it? Architecture HDFS MapReduce Open
Cloud computing doesn t yet have a
The Case for Cloud Computing Robert L. Grossman University of Illinois at Chicago and Open Data Group To understand clouds and cloud computing, we must first understand the two different types of clouds.
A Performance Analysis of Distributed Indexing using Terrier
A Performance Analysis of Distributed Indexing using Terrier Amaury Couste Jakub Kozłowski William Martin Indexing Indexing Used by search
Cloud Computing at Google. Architecture
Cloud Computing at Google Google File System Web Systems and Algorithms Google Chris Brooks Department of Computer Science University of San Francisco Google has developed a layered system to handle webscale
NoSQL Data Base Basics
NoSQL Data Base Basics Course Notes in Transparency Format Cloud Computing MIRI (CLC-MIRI) UPC Master in Innovation & Research in Informatics Spring- 2013 Jordi Torres, UPC - BSC www.jorditorres.eu HDFS
Intro to Map/Reduce a.k.a. Hadoop
Intro to Map/Reduce a.k.a. Hadoop Based on: Mining of Massive Datasets by Ra jaraman and Ullman, Cambridge University Press, 2011 Data Mining for the masses by North, Global Text Project, 2012 Slides by
Analysis and Modeling of MapReduce s Performance on Hadoop YARN
Analysis and Modeling of MapReduce s Performance on Hadoop YARN Qiuyi Tang Dept. of Mathematics and Computer Science Denison University tang_j3@denison.edu Dr. Thomas C. Bressoud Dept. of Mathematics and
Report: Declarative Machine Learning on MapReduce (SystemML)
Report: Declarative Machine Learning on MapReduce (SystemML) Jessica Falk ETH-ID 11-947-512 May 28, 2014 1 Introduction SystemML is a system used to execute machine learning (ML) algorithms in HaDoop,
Big Data: Tools and Technologies in Big Data
Big Data: Tools and Technologies in Big Data Jaskaran Singh Student Lovely Professional University, Punjab Varun Singla Assistant Professor Lovely Professional University, Punjab ABSTRACT Big data can
Hadoop Cluster Applications
Hadoop Overview Data analytics has become a key element of the business decision process over the last decade. Classic reporting on a dataset stored in a database was sufficient until recently, but yesterday
Big Data Rethink Algos and Architecture. Scott Marsh Manager R&D Personal Lines Auto Pricing
Big Data Rethink Algos and Architecture Scott Marsh Manager R&D Personal Lines Auto Pricing Agenda History Map Reduce Algorithms History Google talks about their solutions to their problems Map Reduce:
Overview. Big Data in Apache Hadoop. - HDFS - MapReduce in Hadoop - YARN. https://hadoop.apache.org. Big Data Management and Analytics
Overview Big Data in Apache Hadoop - HDFS - MapReduce in Hadoop - YARN https://hadoop.apache.org 138 Apache Hadoop - Historical Background - 2003: Google publishes its cluster architecture & DFS (GFS)
Challenges for Data Driven Systems
Challenges for Data Driven Systems Eiko Yoneki University of Cambridge Computer Laboratory Quick History of Data Management 4000 B C Manual recording From tablets to papyrus to paper A. Payberah 2014 2
Data-Intensive Programming. Timo Aaltonen Department of Pervasive Computing
Data-Intensive Programming Timo Aaltonen Department of Pervasive Computing Data-Intensive Programming Lecturer: Timo Aaltonen University Lecturer timo.aaltonen@tut.fi Assistants: Henri Terho and Antti
International Journal of Engineering Research ISSN: 2348-4039 & Management Technology November-2015 Volume 2, Issue-6
International Journal of Engineering Research ISSN: 2348-4039 & Management Technology Email: editor@ijermt.org November-2015 Volume 2, Issue-6 www.ijermt.org Modeling Big Data Characteristics for Discovering
MapReduce on GPUs. Amit Sabne, Ahmad Mujahid Mohammed Razip, Kun Xu
1 MapReduce on GPUs Amit Sabne, Ahmad Mujahid Mohammed Razip, Kun Xu 2 MapReduce MAP Shuffle Reduce 3 Hadoop Open-source MapReduce framework from Apache, written in Java Used by Yahoo!, Facebook, Ebay,
Text Mining Approach for Big Data Analysis Using Clustering and Classification Methodologies
Text Mining Approach for Big Data Analysis Using Clustering and Classification Methodologies Somesh S Chavadi 1, Dr. Asha T 2 1 PG Student, 2 Professor, Department of Computer Science and Engineering,
Reducer Load Balancing and Lazy Initialization in Map Reduce Environment S.Mohanapriya, P.Natesan
Reducer Load Balancing and Lazy Initialization in Map Reduce Environment S.Mohanapriya, P.Natesan Abstract Big Data is revolutionizing 21st-century with increasingly huge amounts of data to store and be
Implementing Graph Pattern Mining for Big Data in the Cloud
Implementing Graph Pattern Mining for Big Data in the Cloud Chandana Ojah M.Tech in Computer Science & Engineering Department of Computer Science & Engineering, PES College of Engineering, Mandya Ojah.chandana@gmail.com
ISSN:2320-0790. Keywords: HDFS, Replication, Map-Reduce I Introduction:
ISSN:2320-0790 Dynamic Data Replication for HPC Analytics Applications in Hadoop Ragupathi T 1, Sujaudeen N 2 1 PG Scholar, Department of CSE, SSN College of Engineering, Chennai, India 2 Assistant Professor,
Role of Cloud Computing in Big Data Analytics Using MapReduce Component of Hadoop
Role of Cloud Computing in Big Data Analytics Using MapReduce Component of Hadoop Kanchan A. Khedikar Department of Computer Science & Engineering Walchand Institute of Technoloy, Solapur, Maharashtra,
Hadoop MapReduce and Spark. Giorgio Pedrazzi, CINECA-SCAI School of Data Analytics and Visualisation Milan, 10/06/2015
Hadoop MapReduce and Spark Giorgio Pedrazzi, CINECA-SCAI School of Data Analytics and Visualisation Milan, 10/06/2015 Outline Hadoop Hadoop Import data on Hadoop Spark Spark features Scala MLlib MLlib
HPC ABDS: The Case for an Integrating Apache Big Data Stack
HPC ABDS: The Case for an Integrating Apache Big Data Stack with HPC 1st JTC 1 SGBD Meeting SDSC San Diego March 19 2014 Judy Qiu Shantenu Jha (Rutgers) Geoffrey Fox gcf@indiana.edu http://www.infomall.org
Big Application Execution on Cloud using Hadoop Distributed File System
Big Application Execution on Cloud using Hadoop Distributed File System Ashkan Vates*, Upendra, Muwafaq Rahi Ali RPIIT Campus, Bastara Karnal, Haryana, India ---------------------------------------------------------------------***---------------------------------------------------------------------
Enhancing Massive Data Analytics with the Hadoop Ecosystem
www.ijecs.in International Journal Of Engineering And Computer Science ISSN:2319-7242 Volume 3, Issue 11 November, 2014 Page No. 9061-9065 Enhancing Massive Data Analytics with the Hadoop Ecosystem Misha
Unified Big Data Analytics Pipeline. 连 城 lian@databricks.com
Unified Big Data Analytics Pipeline 连 城 lian@databricks.com What is A fast and general engine for large-scale data processing An open source implementation of Resilient Distributed Datasets (RDD) Has an
A Case for Flash Memory SSD in Hadoop Applications
A Case for Flash Memory SSD in Hadoop Applications Seok-Hoon Kang, Dong-Hyun Koo, Woon-Hak Kang and Sang-Won Lee Dept of Computer Engineering, Sungkyunkwan University, Korea x860221@gmail.com, smwindy@naver.com,
Unified Big Data Processing with Apache Spark. Matei Zaharia @matei_zaharia
Unified Big Data Processing with Apache Spark Matei Zaharia @matei_zaharia What is Apache Spark? Fast & general engine for big data processing Generalizes MapReduce model to support more types of processing
Welcome to the unit of Hadoop Fundamentals on Hadoop architecture. I will begin with a terminology review and then cover the major components
Welcome to the unit of Hadoop Fundamentals on Hadoop architecture. I will begin with a terminology review and then cover the major components of Hadoop. We will see what types of nodes can exist in a Hadoop
Concept and Project Objectives
3.1 Publishable summary Concept and Project Objectives Proactive and dynamic QoS management, network intrusion detection and early detection of network congestion problems among other applications in the
Comparison of Different Implementation of Inverted Indexes in Hadoop
Comparison of Different Implementation of Inverted Indexes in Hadoop Hediyeh Baban, S. Kami Makki, and Stefan Andrei Department of Computer Science Lamar University Beaumont, Texas (hbaban, kami.makki,
Analysis of MapReduce Algorithms
Analysis of MapReduce Algorithms Harini Padmanaban Computer Science Department San Jose State University San Jose, CA 95192 408-924-1000 harini.gomadam@gmail.com ABSTRACT MapReduce is a programming model
Developing Scalable Smart Grid Infrastructure to Enable Secure Transmission System Control
Developing Scalable Smart Grid Infrastructure to Enable Secure Transmission System Control EP/K006487/1 UK PI: Prof Gareth Taylor (BU) China PI: Prof Yong-Hua Song (THU) Consortium UK Members: Brunel University
Big Data Mining Services and Knowledge Discovery Applications on Clouds
Big Data Mining Services and Knowledge Discovery Applications on Clouds Domenico Talia DIMES, Università della Calabria & DtoK Lab Italy talia@dimes.unical.it Data Availability or Data Deluge? Some decades
Enhancing MapReduce Functionality for Optimizing Workloads on Data Centers
Available Online at www.ijcsmc.com International Journal of Computer Science and Mobile Computing A Monthly Journal of Computer Science and Information Technology IJCSMC, Vol. 2, Issue. 10, October 2013,
Chapter 11 Map-Reduce, Hadoop, HDFS, Hbase, MongoDB, Apache HIVE, and Related
Chapter 11 Map-Reduce, Hadoop, HDFS, Hbase, MongoDB, Apache HIVE, and Related Summary Xiangzhe Li Nowadays, there are more and more data everyday about everything. For instance, here are some of the astonishing
LARGE-SCALE DATA PROCESSING USING MAPREDUCE IN CLOUD COMPUTING ENVIRONMENT
LARGE-SCALE DATA PROCESSING USING MAPREDUCE IN CLOUD COMPUTING ENVIRONMENT Samira Daneshyar 1 and Majid Razmjoo 2 1,2 School of Computer Science, Centre of Software Technology and Management (SOFTEM),
HiBench Introduction. Carson Wang (carson.wang@intel.com) Software & Services Group
HiBench Introduction Carson Wang (carson.wang@intel.com) Agenda Background Workloads Configurations Benchmark Report Tuning Guide Background WHY Why we need big data benchmarking systems? WHAT What is
DATA MINING WITH HADOOP AND HIVE Introduction to Architecture
DATA MINING WITH HADOOP AND HIVE Introduction to Architecture Dr. Wlodek Zadrozny (Most slides come from Prof. Akella s class in 2014) 2015-2025. Reproduction or usage prohibited without permission of
The Internet of Things and Big Data: Intro
The Internet of Things and Big Data: Intro John Berns, Solutions Architect, APAC - MapR Technologies April 22 nd, 2014 1 What This Is; What This Is Not It s not specific to IoT It s not about any specific