Getting Started with Hadoop with Amazon s Elastic MapReduce
|
|
|
- Doreen Della Ward
- 8 years ago
- Views:
Transcription
1 Getting Started with Hadoop with Amazon s Elastic MapReduce Scott Hendrickson scott@drskippy.net Boulder/Denver Hadoop Meetup 8 July 2010 Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
2 Agenda 1 Amazon Web Services 2 Interlude: Solving problems with Map and Reduce 3 Running MapReduce on Amazon Elastic MapReduce Example 1: Streaming Work Flow with AWS Management Console Example 2 - Word count (Slightly more useful) Example 3 - elastic-mapreduce command line tool 4 References and Notes Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
3 Amazon Web Services What is Amazon Web Services? For first Hadoop project on AWS, use these services: Elastic Compute Cloud (EC2) Amazon Simple Storage Service (S3) Elastic MapReduce (EMR) For future projects, AWS is much more: SimpleDB, Relational Database Services Simple Queue Service (SQS), Simple Notification Service (SNS) Alexa Mechanical Turk... Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
4 Amazon Web Services Signing up for AWS 1 Create an AWS account Sign up for EC2 cloud compute services Set up Security Credentials (under menu Account Security Credentials) - 3 kinds of credentials, you need to create an Access Key ; use it to access S3 storage 4 Sign up for S3 storage services Sign up for EMR - Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
5 Amazon Web Services What are S3 buckets? Streaming EMR projects use Simple Storage Service (S3) Buckets for data, code, logging and output. Bucket A bucket is a container for objects stored in Amazon S3. Every object is contained in a bucket. Bucket names must be globally unique. Object Entities stored in Amazon S3. Objects consist of object data and metadata. Metadata consists of key-value pairs. Object data is opaque. Objects Keys An object is uniquely identified within a bucket by a key (name) and a version ID. Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
6 Amazon Web Services Accessing objects in S3 buckets Want to: 1 Move data into and out of S3 buckets 2 Set access privileges Tools: S3 console in your AWS control panel is adequate for managing S3 buckets and objects one at a time Other browser options: good for multiple file upload/download - Firefox S3 ; or minimal - S3 plug-in for Chrome extensions/detail/appeggcmoaojledegaonmdaakfhjhchf Programmatic options: Web Services (both SOAP-y and REST-ful): wget, curl, Python, Ruby, Java... Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
7 Amazon Web Services S3 Bucket Example 1 - RESTful GET Example - Image object Bucket: bsi-test Key: image.jpg Object: JPEG structured data data from image.jpg RESTful GET access, use URL: Example - Text file object Bucket: bsi-test Key: foobar Object: text RESTful GET access, use URL: Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
8 S3 Bucket Example 2 Amazon Web Services Example - Python, Boto, Metadata from boto.s3.connection import S3Connection conn = S3Connection( key-id, secret-key ) bucket = conn.get_bucket( bsi-test ) k = bucket.get_key( image.jpg ) print "Value for key x-amz-meta-s3fox-modifiedtime is:" print k.get_metadata( s3fox-modifiedtime ) k.get_contents_to_filename( deleteme.jpg ) k = bucket.get_key( foobar ) print "Object value for key foobar is:" print k.get_contents_as_string() print "Value for key x-amz-meta-example-key is:" print k.get_metadata( example-key ) Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
9 S3 Bucket Example 2 Amazon Web Services Example - Python, Boto, Metadata - Output scott@mowgli-ubuntu:~/dropbox/hadoop$./botoexample.py Value for key x-amz-meta-s3fox-modifiedtime is: Object value for key foobar is: This is a test of S3 Value for key x-amz-meta-example-key is: This is an example value. Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
10 Amazon Web Services What is Elastic Map Reduce? Hadoop Hosted Hadoop framework running on EC2 and S3. Job Flow Processing steps EMR runs on a specified dataset using a set of Amazon EC2 instances. S3 Bucket(s) Input data, output, scripts, jars, logs. Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
11 Amazon Web Services Controlling Job Flows Want to: 1 Configure jobs 2 Start jobs 3 Check status or stop jobs Tools: AWS Management Console Command Line Tools (requires Ruby [sudo apt-get install ruby libopenssl-ruby]) jspa?externalid=2264&categoryid=262 API calls defined by the service (REST-ful and SOAP-y) Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
![com/elasticmapreduce/home Command Line Tools (requires Ruby [sudo apt-get install ruby libopenssl-ruby])](/docs-images/48/18605172/images/page_11.jpg "http://developer.amazonwebservices.com/connect/entry. jspa?")
12 Amazon Web Services EMR Example 1 - Running a simple Work Flow from the AWS Management Console EMR Example 1 Hold up a minute...! What problem are we solving? Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
13 Agenda Interlude: Solving problems with Map and Reduce 1 Amazon Web Services 2 Interlude: Solving problems with Map and Reduce 3 Running MapReduce on Amazon Elastic MapReduce Example 1: Streaming Work Flow with AWS Management Console Example 2 - Word count (Slightly more useful) Example 3 - elastic-mapreduce command line tool 4 References and Notes Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
14 Interlude: Solving problems with Map and Reduce Central MapReduce Ideas Operate on key-value pairs Data scientist provides map and reduce (input) < k1, v1 > map < k2, v2 > < k2, v2 > combine,sort < k2, v2 > < k2, v2 > reduce < k3, v3 > (output) (Optional: Combine provided in map, may significantly reduce bandwidth between workers) Efficient Sort provide by MapReduce library. Implies efficient compare(k2 a, k2 b ) Implicit parallelization - splitting and distributing data, starting maps, reduces, collecting output Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
15 Interlude: Solving problems with Map and Reduce Key components of MapReduce framework (wikipedia The frozen part of the MapReduce framework is a large distributed sort. The hot spots, which the application defines, are: 1 input reader 2 Map function 3 partition function 4 compare function 5 Reduce function 6 output writer Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
16 Interlude: Solving problems with Map and Reduce Google Tutorial View 1 MapReduce library shards the input files and starts up many copies on a cluster. 2 Master assigns work to workers. There are map and reduce tasks. 3 Workers assigned map tasks reads the contents input shard, parse key-value pairs and pass pairs to map function. Intermediate key-value pairs produced by the map function are buffered in memory. 4 Periodically, buffered pairs are written to disk, partitioned into regions. Locations of buffered pairs on the local disk are passed to the master. 5 When a reduce worker has read all intermediate data, it sorts by the intermediate keys. All occurrences a key are grouped together. 6 Reduce workers pass a key and the corresponding set of intermediate values to the reduce function. 7 Output of the reduce function is appended to a final output file for each reduce partition. Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
17 Interlude: Solving problems with Map and Reduce MapReduce Example 1 - Word Count - Data (from Apache Hadoop tutorial) Example: Word Count file1: Hello World Bye World file2: Hello Hadoop Goodbye Hadoop Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
18 Interlude: Solving problems with Map and Reduce MapReduce Example 1 - Word Count - Map Example: Word Count The first map emits: < Hello, 1> < World, 1> < Bye, 1> < World, 1> The second map emits: < Hello, 1> < Hadoop, 1> < Goodbye, 1> < Hadoop, 1> Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
19 Interlude: Solving problems with Map and Reduce MapReduce Example 1 - Word Count - Sort and Combine Example: Word Count The output of the first map: < Bye, 1> < Hello, 1> < World, 2> The output of the second map: < Goodbye, 1> < Hadoop, 2> < Hello, 1> Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
20 Interlude: Solving problems with Map and Reduce MapReduce Example 1 - Word Count - Sort and Reduce Example: Word Count The Reducer method sums up the values for each key. The output of the job is: < Bye, 1> < Goodbye, 1> < Hadoop, 2> < Hello, 2> < World, 2> Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
21 Interlude: Solving problems with Map and Reduce What problems is MapReduce good at solving? Themes: Identify, transform, aggregate, filter, count, sort... Requirement of global knowledge of data is (a) occasional (vs. cost of map) (b) confined to ordinality Discovery tasks (vs. high repetition of similar transactional tasks, many reads) Unstructured data (vs. tabular, indexes!) Continuously updated data (indexing cost) Many, many, many machines (fault tolerance) Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
22 Interlude: Solving problems with Map and Reduce What problems is MapReduce good at solving? Memes: MapReduce SQL (read the comments too) mapreduce-and-sql-aggregations.html MapReduce vs. Message Passing Interface (MPI) MPI is good for task parallelism and Hadoop is good for Data Parallelism. finite differences, finite elements, particle-in-cell... MapReduce vs. column-oriented DBs tabular data, indexes (cantankerous old farts!) database-innovation/mapreduce-a-major-step-backwards/ and database-innovation/mapreduce-ii/ MapReduce vs. relational DBs goodmath/2008/01/databases_are_hammers_mapreduc.php Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
23 Agenda Running MapReduce on Amazon Elastic MapReduce Example 1: Streaming Work Flow with AWS Management Console 1 Amazon Web Services 2 Interlude: Solving problems with Map and Reduce 3 Running MapReduce on Amazon Elastic MapReduce Example 1: Streaming Work Flow with AWS Management Console Example 2 - Word count (Slightly more useful) Example 3 - elastic-mapreduce command line tool 4 References and Notes Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
24 Running MapReduce on Amazon Elastic MapReduce Example 1 - Add up integers Example 1: Streaming Work Flow with AWS Management Console Data Map import sys for line in sys.stdin: print %s%s%d % ("sum", \t, int(line)) Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
25 Running MapReduce on Amazon Elastic MapReduce Example 1 - Add up integers Example 1: Streaming Work Flow with AWS Management Console Reduce import sys sum_of_ints = 0 for line in sys.stdin: key, value = line.split( \t ) # key is always the same try: sum_of_ints += int(value) except ValueError: pass try: print "%s%s%d" % (key, \t, sum_of_ints) except NameError: # No items processed pass Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
26 Running MapReduce on Amazon Elastic MapReduce Example 1 - Add up integers Example 1: Streaming Work Flow with AWS Management Console Shell test cat./input/ints.txt./mapper.py >./inter cat./input/ints1.txt./mapper.py >>./inter cat./input/ints2.txt./mapper.py >>./inter cat./input/ints3.txt./mapper.py >>./inter echo "Intermediate output:" cat./inter cat./inter sort \./reducer.py >./output/cmdlineoutput.txt Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
27 Running MapReduce on Amazon Elastic MapReduce Example 1 - Add up integers Example 1: Streaming Work Flow with AWS Management Console What was that comment earlier about an optional combiner? Combiner in map import sys sum_of_ints = 0 for line in sys.stdin: sum_of_ints += int(line) print %s%s%d % ("sum", \t, sum_of_ints) Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
28 Running MapReduce on Amazon Elastic MapReduce Example 1 - Add up integers Example 1: Streaming Work Flow with AWS Management Console Combiner shell test cat./input/ints.txt./mapper_combine.py >./inter cat./input/ints1.txt./mapper_combine.py >>./inter cat./input/ints2.txt./mapper_combine.py >>./inter cat./input/ints3.txt./mapper_combine.py >>./inter echo "Intermediate output:" cat./inter cat./inter sort \./reducer.py >./output/cmdlinecomboutput.txt Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
29 Running MapReduce on Amazon Elastic MapReduce Example 1: Streaming Work Flow with AWS Management Console Example 1 - Add up integers - AWS Console 1 Upload onecount directory with FFS3 2 Create a New Job Flow Name: onecount Job Flow: Run own app Job Type: Streaming 3 Input: bsi-test/onecount/input Output: bsi-test/onecount/outputconsole (must not exist) Mapper: bsi-test/onecount/mapper.py Reducer: bsi-test/onecount/reducer.py Extra Args: none Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
30 Running MapReduce on Amazon Elastic MapReduce Example 1: Streaming Work Flow with AWS Management Console Example 1 - Add up integers - AWS Console 4 Instances: 4 Type: small Keypair: No (Yes allows ssh to Hadoop master) Log: yes Log Location: bsi-test/onecount/log Hadoop Debug: no 5 No bootstrap actions 6 Start it, and wait... Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
31 Agenda Running MapReduce on Amazon Elastic MapReduce Example 2 - Word count (Slightly more useful) 1 Amazon Web Services 2 Interlude: Solving problems with Map and Reduce 3 Running MapReduce on Amazon Elastic MapReduce Example 1: Streaming Work Flow with AWS Management Console Example 2 - Word count (Slightly more useful) Example 3 - elastic-mapreduce command line tool 4 References and Notes Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
32 Running MapReduce on Amazon Elastic MapReduce Example 2 - Word count (Slightly more useful) Example 2 - Word count Map def read_input(file): for line in file: yield line.split() def main(separator= \t ): data = read_input(sys.stdin) for words in data: for word in words: lword = word.lower().strip(string.puctuation) print %s%s%d % (lword, separator, 1) Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
33 def main(separator= \t ): data = read_mapper_output(sys.stdin, separator=separator) for current_word,group in groupby(data,itemgetter(0)): try: total_count = sum(int(count) for current_word, count in group) print "%s%s%d" % (current_word, separator, total_count) except ValueError: pass Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43 Running MapReduce on Amazon Elastic MapReduce Example 2 - Word count Example 2 - Word count (Slightly more useful) Reduce def read_mapper_output(file, separator= \t ): for line in file: yield line.rstrip().split(separator, 1)
34 Running MapReduce on Amazon Elastic MapReduce Example 2 - Word count Example 2 - Word count (Slightly more useful) Shell test echo "foo foo quux labs foo bar quux"./mapper.py echo "foo foo quux labs foo bar quux"./mapper.py \ sort./reducer.py cat./input/alice.txt./mapper.py \ sort./reducer.py >./output/cmdlineoutput.txt Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
35 Running MapReduce on Amazon Elastic MapReduce Example 2 - Word count (Slightly more useful) Example 2 - Word count - AWS Console 1 Upload mywordcount directory with FFS3 2 Create a New Job Flow Name: mywordcount Job Flow: Run own app Job Type: Streaming 3 Input: bsi-test/mywordcount/input Output: bsi-test/mywordcount/outputconsole (must not exist) Mapper: bsi-test/mywordcount/mapper.py Reducer: bsi-test/mywordcount/reducer.py Extra Args: none Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
36 Running MapReduce on Amazon Elastic MapReduce Example 2 - Word count (Slightly more useful) Example 2 - Word count - AWS Console 4 Instances: 4 Type: small Keypair: No (Yes allows ssh to Hadoop master) Log: yes Log Location: bsi-test/mywordcount/log Hadoop Debug: no 5 No bootstrap actions 6 Start it, and wait... Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
37 Agenda Running MapReduce on Amazon Elastic MapReduce Example 3 - elastic-mapreduce command line tool 1 Amazon Web Services 2 Interlude: Solving problems with Map and Reduce 3 Running MapReduce on Amazon Elastic MapReduce Example 1: Streaming Work Flow with AWS Management Console Example 2 - Word count (Slightly more useful) Example 3 - elastic-mapreduce command line tool 4 References and Notes Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
38 Running MapReduce on Amazon Elastic MapReduce Example 3 - elastic-mapreduce command line tool Example 3 - elastic-mapreduce command line tool Word count (again, only better) /usr/local/emr-ruby/elastic-mapreduce --create \ --stream \ --num-instances 2 \ --name from-elastic-mapreduce \ --input s3n://bsi-test/mywordcount/input \ --output s3n://bsi-test/mywordcount/outputrubytool \ --mapper s3n://bsi-test/mywordcount/mapper.py \ --reducer s3n://bsi-test/mywordcount/reducer.py \ --log-uri s3n://bsi-test/mywordcount/log /usr/local/emr-ruby/elastic-mapreduce --list Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
39 Agenda References and Notes 1 Amazon Web Services 2 Interlude: Solving problems with Map and Reduce 3 Running MapReduce on Amazon Elastic MapReduce Example 1: Streaming Work Flow with AWS Management Console Example 2 - Word count (Slightly more useful) Example 3 - elastic-mapreduce command line tool 4 References and Notes Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
40 References and Notes MapReduce Concepts Links Google MapReduce Tutorial: http: //code.google.com/edu/parallel/mapreduce-tutorial.html Apache Hadoop tutorial: docs/current/mapred_tutorial.html Google Code University presentation on MapReduce: google.com/edu/submissions/mapreduce/listing.html MapReduce framework paper: Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
41 References and Notes Amazon Web Services Links EMR Getting Started documentation: Getting started with Amazon S3: http: //docs.amazonwebservices.com/amazons3/ /gsg/ PIG on EMR: http: //s3.amazonaws.com/awsvideos/amazonelasticmapreduce/ ElasticMapReduce-PigTutorial.html Boto Python library (multiple Amazon Services): Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
42 References and Notes Machine Learning Linear speedup (with processor number) for locally weighted linear regression (LWLR), k-means, logistic regression (LR), naive Bayes (NB), SVM, ICA, PCA, gaussian discriminant analysis (GDA), EM, and backpropagation (NN) : people/ang/papers/nips06-mapreducemulticore.pdf Mahout framework: Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
43 References and Notes Examples Links Wordcount example/tutorial: wiki/writing_an_hadoop_mapreduce_program_in_python CouchDB and MapReduce (interesting examples of MR implementations for common problems) This presentation: or presentation source, example files etc.: Scott Hendrickson (Hadoop Meetup) EMR-Hadoop 8 July / 43
A programming model in Cloud: MapReduce
A programming model in Cloud: MapReduce Programming model and implementation developed by Google for processing large data sets Users specify a map function to generate a set of intermediate key/value
A Tutorial Introduc/on to Big Data. Hands On Data Analy/cs over EMR. Robert Grossman University of Chicago Open Data Group
A Tutorial Introduc/on to Big Data Hands On Data Analy/cs over EMR Robert Grossman University of Chicago Open Data Group Collin BenneE Open Data Group November 12, 2012 1 Amazon AWS Elas/c MapReduce allows
Introduction to analyzing big data using Amazon Web Services
Introduction to analyzing big data using Amazon Web Services This tutorial accompanies the BARC seminar given at Whitehead on January 31, 2013. It contains instructions for: 1. Getting started with Amazon
Hands-on Exercises with Big Data
Hands-on Exercises with Big Data Lab Sheet 1: Getting Started with MapReduce and Hadoop The aim of this exercise is to learn how to begin creating MapReduce programs using the Hadoop Java framework. In
Cloud Computing. AWS a practical example. Hugo Pérez UPC. Mayo 2012
Cloud Computing AWS a practical example Mayo 2012 Hugo Pérez UPC -2- Index Introduction Infraestructure Development and Results Conclusions Introduction In order to know deeper about AWS services, mapreduce
MapReduce for Statistical NLP/Machine Learning
MapReduce for Statistical NLP/Machine Learning Sameer Maskey Week 7, October 17, 2012 1 Topics for Today MapReduce Running MapReduce jobs in Amazon Hadoop MapReduce for Machine Learning Naïve Bayes, K-Means,
Introduction to Hadoop on the SDSC Gordon Data Intensive Cluster"
Introduction to Hadoop on the SDSC Gordon Data Intensive Cluster" Mahidhar Tatineni SDSC Summer Institute August 06, 2013 Overview "" Hadoop framework extensively used for scalable distributed processing
Introduction To Hive
Introduction To Hive How to use Hive in Amazon EC2 CS 341: Project in Mining Massive Data Sets Hyung Jin(Evion) Kim Stanford University References: Cloudera Tutorials, CS345a session slides, Hadoop - The
CS 378 Big Data Programming
CS 378 Big Data Programming Lecture 2 Map- Reduce CS 378 - Fall 2015 Big Data Programming 1 MapReduce Large data sets are not new What characterizes a problem suitable for MR? Most or all of the data is
10605 BigML Assignment 4(a): Naive Bayes using Hadoop Streaming
10605 BigML Assignment 4(a): Naive Bayes using Hadoop Streaming Due: Friday, Feb. 21, 2014 23:59 EST via Autolab Late submission with 50% credit: Sunday, Feb. 23, 2014 23:59 EST via Autolab Policy on Collaboration
CS 378 Big Data Programming. Lecture 2 Map- Reduce
CS 378 Big Data Programming Lecture 2 Map- Reduce MapReduce Large data sets are not new What characterizes a problem suitable for MR? Most or all of the data is processed But viewed in small increments
Hadoop 只 支 援 用 Java 開 發 嘛? Is Hadoop only support Java? 總 不 能 全 部 都 重 新 設 計 吧? 如 何 與 舊 系 統 相 容? Can Hadoop work with existing software?
Hadoop 只 支 援 用 Java 開 發 嘛? Is Hadoop only support Java? 總 不 能 全 部 都 重 新 設 計 吧? 如 何 與 舊 系 統 相 容? Can Hadoop work with existing software? 可 以 跟 資 料 庫 結 合 嘛? Can Hadoop work with Databases? 開 發 者 們 有 聽 到
MapReduce and Hadoop. Aaron Birkland Cornell Center for Advanced Computing. January 2012
MapReduce and Hadoop Aaron Birkland Cornell Center for Advanced Computing January 2012 Motivation Simple programming model for Big Data Distributed, parallel but hides this Established success at petabyte
Lecture 32 Big Data. 1. Big Data problem 2. Why the excitement about big data 3. What is MapReduce 4. What is Hadoop 5. Get started with Hadoop
Lecture 32 Big Data 1. Big Data problem 2. Why the excitement about big data 3. What is MapReduce 4. What is Hadoop 5. Get started with Hadoop 1 2 Big Data Problems Data explosion Data from users on social
Map Reduce & Hadoop Recommended Text:
Big Data Map Reduce & Hadoop Recommended Text:! Large datasets are becoming more common The New York Stock Exchange generates about one terabyte of new trade data per day. Facebook hosts approximately
t] open source Hadoop Beginner's Guide ij$ data avalanche Garry Turkington Learn how to crunch big data to extract meaning from
![t] open source Hadoop Beginner's Guide ij$ data avalanche Garry Turkington Learn how to crunch big data to extract meaning from](/thumbs/26/7239192.jpg "t] open source Hadoop Beginner's Guide ij$ data avalanche Garry Turkington Learn how to crunch big data to extract meaning from")
Hadoop Beginner's Guide Learn how to crunch big data to extract meaning from data avalanche Garry Turkington [ PUBLISHING t] open source I I community experience distilled ftu\ ij$ BIRMINGHAMMUMBAI ')
Hadoop & Spark Using Amazon EMR
Hadoop & Spark Using Amazon EMR Michael Hanisch, AWS Solutions Architecture 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda Why did we build Amazon EMR? What is Amazon EMR?
MapReduce (in the cloud)
MapReduce (in the cloud) How to painlessly process terabytes of data by Irina Gordei MapReduce Presentation Outline What is MapReduce? Example How it works MapReduce in the cloud Conclusion Demo Motivation:
Cloud computing - Architecting in the cloud
Cloud computing - Architecting in the cloud anna.ruokonen@tut.fi 1 Outline Cloud computing What is? Levels of cloud computing: IaaS, PaaS, SaaS Moving to the cloud? Architecting in the cloud Best practices
Assignment 2: More MapReduce with Hadoop
Assignment 2: More MapReduce with Hadoop Jean-Pierre Lozi February 5, 2015 Provided files following URL: An archive that contains all files you will need for this assignment can be found at the http://sfu.ca/~jlozi/cmpt732/assignment2.tar.gz
CSE 344 Introduction to Data Management. Section 9: AWS, Hadoop, Pig Latin TA: Yi-Shu Wei
CSE 344 Introduction to Data Management Section 9: AWS, Hadoop, Pig Latin TA: Yi-Shu Wei Homework 8 Big Data analysis on billion triple dataset using Amazon Web Service (AWS) Billion Triple Set: contains
Tutorial: Big Data Algorithms and Applications Under Hadoop KUNPENG ZHANG SIDDHARTHA BHATTACHARYYA
Tutorial: Big Data Algorithms and Applications Under Hadoop KUNPENG ZHANG SIDDHARTHA BHATTACHARYYA http://kzhang6.people.uic.edu/tutorial/amcis2014.html August 7, 2014 Schedule I. Introduction to big data
Introduction to Parallel Programming and MapReduce
Introduction to Parallel Programming and MapReduce Audience and Pre-Requisites This tutorial covers the basics of parallel programming and the MapReduce programming model. The pre-requisites are significant
Lecture 10 - Functional programming: Hadoop and MapReduce
Lecture 10 - Functional programming: Hadoop and MapReduce Sohan Dharmaraja Sohan Dharmaraja Lecture 10 - Functional programming: Hadoop and MapReduce 1 / 41 For today Big Data and Text analytics Functional
The MapReduce Framework
The MapReduce Framework Luke Tierney Department of Statistics & Actuarial Science University of Iowa November 8, 2007 Luke Tierney (U. of Iowa) The MapReduce Framework November 8, 2007 1 / 16 Background
Last time. Today. IaaS Providers. Amazon Web Services, overview
Last time General overview, motivation, expected outcomes, other formalities, etc. Please register for course Online (if possible), or talk to Yvonne@CS Course evaluation forgotten Please assign one volunteer
Lambda Architecture for Batch and Real- Time Processing on AWS with Spark Streaming and Spark SQL. May 2015
Lambda Architecture for Batch and Real- Time Processing on AWS with Spark Streaming and Spark SQL May 2015 2015, Amazon Web Services, Inc. or its affiliates. All rights reserved. Notices This document
CONFIGURING ECLIPSE FOR AWS EMR DEVELOPMENT
CONFIGURING ECLIPSE FOR AWS EMR DEVELOPMENT With this post we thought of sharing a tutorial for configuring Eclipse IDE (Intergrated Development Environment) for Amazon AWS EMR scripting and development.
Hadoop: A Framework for Data- Intensive Distributed Computing. CS561-Spring 2012 WPI, Mohamed Y. Eltabakh
1 Hadoop: A Framework for Data- Intensive Distributed Computing CS561-Spring 2012 WPI, Mohamed Y. Eltabakh 2 What is Hadoop? Hadoop is a software framework for distributed processing of large datasets
Matchmaking in the Cloud: Amazon EC2 and Apache Hadoop at eharmony
Matchmaking in the Cloud: Amazon EC2 and Apache Hadoop at eharmony Speaker logo centered below image Steve Kuo, Software Architect Joshua Tuberville, Software Architect Goal > Leverage EC2 and Hadoop to
Introduction to Big Data! with Apache Spark" UC#BERKELEY#
Introduction to Big Data! with Apache Spark" UC#BERKELEY# This Lecture" The Big Data Problem" Hardware for Big Data" Distributing Work" Handling Failures and Slow Machines" Map Reduce and Complex Jobs"
Introduction to NoSQL Databases and MapReduce. Tore Risch Information Technology Uppsala University 2014-05-12
Introduction to NoSQL Databases and MapReduce Tore Risch Information Technology Uppsala University 2014-05-12 What is a NoSQL Database? 1. A key/value store Basic index manager, no complete query language
HiBench Introduction. Carson Wang (carson.wang@intel.com) Software & Services Group
HiBench Introduction Carson Wang (carson.wang@intel.com) Agenda Background Workloads Configurations Benchmark Report Tuning Guide Background WHY Why we need big data benchmarking systems? WHAT What is
Introduction to Hadoop
Introduction to Hadoop Miles Osborne School of Informatics University of Edinburgh miles@inf.ed.ac.uk October 28, 2010 Miles Osborne Introduction to Hadoop 1 Background Hadoop Programming Model Examples
Spark in Action. Fast Big Data Analytics using Scala. Matei Zaharia. www.spark- project.org. University of California, Berkeley UC BERKELEY
Spark in Action Fast Big Data Analytics using Scala Matei Zaharia University of California, Berkeley www.spark- project.org UC BERKELEY My Background Grad student in the AMP Lab at UC Berkeley» 50- person
Chapter 7. Using Hadoop Cluster and MapReduce
Chapter 7 Using Hadoop Cluster and MapReduce Modeling and Prototyping of RMS for QoS Oriented Grid Page 152 7. Using Hadoop Cluster and MapReduce for Big Data Problems The size of the databases used in
Introduction to Hadoop. New York Oracle User Group Vikas Sawhney
Introduction to Hadoop New York Oracle User Group Vikas Sawhney GENERAL AGENDA Driving Factors behind BIG-DATA NOSQL Database 2014 Database Landscape Hadoop Architecture Map/Reduce Hadoop Eco-system Hadoop
Cloud Computing at Google. Architecture
Cloud Computing at Google Google File System Web Systems and Algorithms Google Chris Brooks Department of Computer Science University of San Francisco Google has developed a layered system to handle webscale
Jeffrey D. Ullman slides. MapReduce for data intensive computing
Jeffrey D. Ullman slides MapReduce for data intensive computing Single-node architecture CPU Machine Learning, Statistics Memory Classical Data Mining Disk Commodity Clusters Web data sets can be very
How To Use Hadoop
Hadoop in Action Justin Quan March 15, 2011 Poll What s to come Overview of Hadoop for the uninitiated How does Hadoop work? How do I use Hadoop? How do I get started? Final Thoughts Key Take Aways Hadoop
CSE-E5430 Scalable Cloud Computing Lecture 2
CSE-E5430 Scalable Cloud Computing Lecture 2 Keijo Heljanko Department of Computer Science School of Science Aalto University keijo.heljanko@aalto.fi 14.9-2015 1/36 Google MapReduce A scalable batch processing
Spark ΕΡΓΑΣΤΗΡΙΟ 10. Prepared by George Nikolaides 4/19/2015 1
Spark ΕΡΓΑΣΤΗΡΙΟ 10 Prepared by George Nikolaides 4/19/2015 1 Introduction to Apache Spark Another cluster computing framework Developed in the AMPLab at UC Berkeley Started in 2009 Open-sourced in 2010
Cloud Computing. Chapter 8. 8.1 Hadoop
Chapter 8 Cloud Computing In cloud computing, the idea is that a large corporation that has many computers could sell time on them, for example to make profitable use of excess capacity. The typical customer
Amazon-style shopping cart analysis using MapReduce on a Hadoop cluster. Dan Şerban
Amazon-style shopping cart analysis using MapReduce on a Hadoop cluster Dan Şerban Agenda :: Introduction - Real-world uses of MapReduce - The origins of Hadoop - Hadoop facts and architecture :: Part
What We Can Do in the Cloud (2) -Tutorial for Cloud Computing Course- Mikael Fernandus Simalango WISE Research Lab Ajou University, South Korea
What We Can Do in the Cloud (2) -Tutorial for Cloud Computing Course- Mikael Fernandus Simalango WISE Research Lab Ajou University, South Korea Overview Riding Google App Engine Taming Hadoop Summary Riding
Scalable Architecture on Amazon AWS Cloud
Scalable Architecture on Amazon AWS Cloud Kalpak Shah Founder & CEO, Clogeny Technologies kalpak@clogeny.com 1 * http://www.rightscale.com/products/cloud-computing-uses/scalable-website.php 2 Architect
Open source Google-style large scale data analysis with Hadoop
Open source Google-style large scale data analysis with Hadoop Ioannis Konstantinou Email: ikons@cslab.ece.ntua.gr Web: http://www.cslab.ntua.gr/~ikons Computing Systems Laboratory School of Electrical
MapReduce, Hadoop and Amazon AWS
MapReduce, Hadoop and Amazon AWS Yasser Ganjisaffar http://www.ics.uci.edu/~yganjisa February 2011 What is Hadoop? A software framework that supports data-intensive distributed applications. It enables
Background on Elastic Compute Cloud (EC2) AMI s to choose from including servers hosted on different Linux distros
David Moses January 2014 Paper on Cloud Computing I Background on Tools and Technologies in Amazon Web Services (AWS) In this paper I will highlight the technologies from the AWS cloud which enable you
HADOOP MOCK TEST HADOOP MOCK TEST II
http://www.tutorialspoint.com HADOOP MOCK TEST Copyright tutorialspoint.com This section presents you various set of Mock Tests related to Hadoop Framework. You can download these sample mock tests at
Sriram Krishnan, Ph.D. sriram@sdsc.edu
Sriram Krishnan, Ph.D. sriram@sdsc.edu (Re-)Introduction to cloud computing Introduction to the MapReduce and Hadoop Distributed File System Programming model Examples of MapReduce Where/how to run MapReduce
Workshop on Hadoop with Big Data
Workshop on Hadoop with Big Data Hadoop? Apache Hadoop is an open source framework for distributed storage and processing of large sets of data on commodity hardware. Hadoop enables businesses to quickly
Big Data With Hadoop
With Saurabh Singh singh.903@osu.edu The Ohio State University February 11, 2016 Overview 1 2 3 Requirements Ecosystem Resilient Distributed Datasets (RDDs) Example Code vs Mapreduce 4 5 Source: [Tutorials
Open source large scale distributed data management with Google s MapReduce and Bigtable
Open source large scale distributed data management with Google s MapReduce and Bigtable Ioannis Konstantinou Email: ikons@cslab.ece.ntua.gr Web: http://www.cslab.ntua.gr/~ikons Computing Systems Laboratory
Cloud Application Development (SE808, School of Software, Sun Yat-Sen University) Yabo (Arber) Xu
Lecture 4 Introduction to Hadoop & GAE Cloud Application Development (SE808, School of Software, Sun Yat-Sen University) Yabo (Arber) Xu Outline Introduction to Hadoop The Hadoop ecosystem Related projects
Hadoop. Bioinformatics Big Data
Hadoop Bioinformatics Big Data Paolo D Onorio De Meo Mattia D Antonio p.donoriodemeo@cineca.it m.dantonio@cineca.it Big Data Too much information! Big Data Explosive data growth proliferation of data capture
Spark. Fast, Interactive, Language- Integrated Cluster Computing
Spark Fast, Interactive, Language- Integrated Cluster Computing Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael Franklin, Scott Shenker, Ion Stoica UC
CSE 590: Special Topics Course ( Supercomputing ) Lecture 10 ( MapReduce& Hadoop)
CSE 590: Special Topics Course ( Supercomputing ) Lecture 10 ( MapReduce& Hadoop) Rezaul A. Chowdhury Department of Computer Science SUNY Stony Brook Spring 2016 MapReduce MapReduce is a programming model
Amazon Elastic MapReduce. Jinesh Varia Peter Sirota Richard Cole
Amazon Elastic MapReduce Jinesh Varia Peter Sirota Richard Cole Start End From IDE Command line Web Console Notify Input Data Get Results Start End From IDE Command line Web Console AWS EC2 Instance Notify
Real-time Data Analytics mit Elasticsearch. Bernhard Pflugfelder inovex GmbH
Real-time Data Analytics mit Elasticsearch Bernhard Pflugfelder inovex GmbH Bernhard Pflugfelder Big Data Engineer @ inovex Fields of interest: search analytics big data bi Working with: Lucene Solr Elasticsearch
Challenges for Data Driven Systems
Challenges for Data Driven Systems Eiko Yoneki University of Cambridge Computer Laboratory Quick History of Data Management 4000 B C Manual recording From tablets to papyrus to paper A. Payberah 2014 2
Apache Spark : Fast and Easy Data Processing Sujee Maniyam Elephant Scale LLC sujee@elephantscale.com http://elephantscale.com
Apache Spark : Fast and Easy Data Processing Sujee Maniyam Elephant Scale LLC sujee@elephantscale.com http://elephantscale.com Spark Fast & Expressive Cluster computing engine Compatible with Hadoop Came
Big Data Course Highlights
Big Data Course Highlights The Big Data course will start with the basics of Linux which are required to get started with Big Data and then slowly progress from some of the basics of Hadoop/Big Data (like
Internals of Hadoop Application Framework and Distributed File System
International Journal of Scientific and Research Publications, Volume 5, Issue 7, July 2015 1 Internals of Hadoop Application Framework and Distributed File System Saminath.V, Sangeetha.M.S Abstract- Hadoop
COURSE CONTENT Big Data and Hadoop Training
COURSE CONTENT Big Data and Hadoop Training 1. Meet Hadoop Data! Data Storage and Analysis Comparison with Other Systems RDBMS Grid Computing Volunteer Computing A Brief History of Hadoop Apache Hadoop
Tutorial for Assignment 2.0
Tutorial for Assignment 2.0 Web Science and Web Technology Summer 2012 Slides based on last years tutorials by Chris Körner, Philipp Singer 1 Review and Motivation Agenda Assignment Information Introduction
Other Map-Reduce (ish) Frameworks. William Cohen
Other Map-Reduce (ish) Frameworks William Cohen 1 Outline More concise languages for map- reduce pipelines Abstractions built on top of map- reduce General comments Speci
Hadoop Job Oriented Training Agenda
1 Hadoop Job Oriented Training Agenda Kapil CK hdpguru@gmail.com Module 1 M o d u l e 1 Understanding Hadoop This module covers an overview of big data, Hadoop, and the Hortonworks Data Platform. 1.1 Module
Integrating Big Data into the Computing Curricula
Integrating Big Data into the Computing Curricula Yasin Silva, Suzanne Dietrich, Jason Reed, Lisa Tsosie Arizona State University http://www.public.asu.edu/~ynsilva/ibigdata/ 1 Overview Motivation Big
Chapter 9 PUBLIC CLOUD LABORATORY. Sucha Smanchat, PhD. Faculty of Information Technology. King Mongkut s University of Technology North Bangkok
CLOUD COMPUTING PRACTICE 82 Chapter 9 PUBLIC CLOUD LABORATORY Hand on laboratory based on AWS Sucha Smanchat, PhD Faculty of Information Technology King Mongkut s University of Technology North Bangkok
Hadoop Installation MapReduce Examples Jake Karnes
Big Data Management Hadoop Installation MapReduce Examples Jake Karnes These slides are based on materials / slides from Cloudera.com Amazon.com Prof. P. Zadrozny's Slides Prerequistes You must have an
Important Notice. (c) 2010-2013 Cloudera, Inc. All rights reserved.
Hue 2 User Guide Important Notice (c) 2010-2013 Cloudera, Inc. All rights reserved. Cloudera, the Cloudera logo, Cloudera Impala, and any other product or service names or slogans contained in this document
www.boost ur skills.com
www.boost ur skills.com AWS CLOUD COMPUTING WORKSHOP Write us at training@boosturskills.com BOOSTURSKILLS No 1736 1st Amrutha College Road Kasavanhalli,Off Sarjapur Road,Bangalore-35 1) Introduction &
Introduction to NoSQL Databases. Tore Risch Information Technology Uppsala University 2013-03-05
Introduction to NoSQL Databases Tore Risch Information Technology Uppsala University 2013-03-05 UDBL Tore Risch Uppsala University, Sweden Evolution of DBMS technology Distributed databases SQL 1960 1970
Eucalyptus 3.4.2 User Console Guide
Eucalyptus 3.4.2 User Console Guide 2014-02-23 Eucalyptus Systems Eucalyptus Contents 2 Contents User Console Overview...4 Install the Eucalyptus User Console...5 Install on Centos / RHEL 6.3...5 Configure
Duke University http://www.cs.duke.edu/starfish
Herodotos Herodotou, Harold Lim, Fei Dong, Shivnath Babu Duke University http://www.cs.duke.edu/starfish Practitioners of Big Data Analytics Google Yahoo! Facebook ebay Physicists Biologists Economists
Pro Apache Hadoop. Second Edition. Sameer Wadkar. Madhu Siddalingaiah
Pro Apache Hadoop Second Edition Sameer Wadkar Madhu Siddalingaiah Contents J About the Authors About the Technical Reviewer Acknowledgments Introduction xix xxi xxiii xxv Chapter 1: Motivation for Big
Machine Learning Big Data using Map Reduce
Machine Learning Big Data using Map Reduce By Michael Bowles, PhD Where Does Big Data Come From? -Web data (web logs, click histories) -e-commerce applications (purchase histories) -Retail purchase histories
Big Data and Data Science: Behind the Buzz Words
Big Data and Data Science: Behind the Buzz Words Peggy Brinkmann, FCAS, MAAA Actuary Milliman, Inc. April 1, 2014 Contents Big data: from hype to value Deconstructing data science Managing big data Analyzing
Real Time Big Data Processing
Real Time Big Data Processing Cloud Expo 2014 Ian Meyers Amazon Web Services Global Infrastructure Deployment & Administration App Services Analytics Compute Storage Database Networking AWS Global Infrastructure
Big Data. Lyle Ungar, University of Pennsylvania
Big Data Big data will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus. McKinsey Data Scientist: The Sexiest Job of the 21st Century -
How to Run Spark Application
How to Run Spark Application Junghoon Kang Contents 1 Intro 2 2 How to Install Spark on a Local Machine? 2 2.1 On Ubuntu 14.04.................................... 2 3 How to Run Spark Application on a
16.1 MAPREDUCE. For personal use only, not for distribution. 333
For personal use only, not for distribution. 333 16.1 MAPREDUCE Initially designed by the Google labs and used internally by Google, the MAPREDUCE distributed programming model is now promoted by several
Best Practices for Amazon EMR
Best Practices for Amazon EMR Parviz Deyhim August 2013 (Please consult for the latest version of this paper) Page 1 of 38 Table of Contents Abstract... 3 Introduction...
Data Intensive Computing Handout 6 Hadoop
Data Intensive Computing Handout 6 Hadoop Hadoop 1.2.1 is installed in /HADOOP directory. The JobTracker web interface is available at http://dlrc:50030, the NameNode web interface is available at http://dlrc:50070.
Hadoop MapReduce and Spark. Giorgio Pedrazzi, CINECA-SCAI School of Data Analytics and Visualisation Milan, 10/06/2015
Hadoop MapReduce and Spark Giorgio Pedrazzi, CINECA-SCAI School of Data Analytics and Visualisation Milan, 10/06/2015 Outline Hadoop Hadoop Import data on Hadoop Spark Spark features Scala MLlib MLlib
Single Node Hadoop Cluster Setup
Single Node Hadoop Cluster Setup This document describes how to create Hadoop Single Node cluster in just 30 Minutes on Amazon EC2 cloud. You will learn following topics. Click Here to watch these steps
Big Data Analytics. Lucas Rego Drumond
Big Data Analytics Big Data Analytics Lucas Rego Drumond Information Systems and Machine Learning Lab (ISMLL) Institute of Computer Science University of Hildesheim, Germany Apache Spark Apache Spark 1
Hadoop/MapReduce Workshop. Dan Mazur, McGill HPC daniel.mazur@mcgill.ca guillimin@calculquebec.ca July 10, 2014
Hadoop/MapReduce Workshop Dan Mazur, McGill HPC daniel.mazur@mcgill.ca guillimin@calculquebec.ca July 10, 2014 1 Outline Hadoop introduction and motivation Python review HDFS - The Hadoop Filesystem MapReduce
Hadoop Streaming. 2012 coreservlets.com and Dima May. 2012 coreservlets.com and Dima May
2012 coreservlets.com and Dima May Hadoop Streaming Originals of slides and source code for examples: http://www.coreservlets.com/hadoop-tutorial/ Also see the customized Hadoop training courses (onsite
FREE computing using Amazon EC2
FREE computing using Amazon EC2 Seong-Hwan Jun 1 1 Department of Statistics Univ of British Columbia Nov 1st, 2012 / Student seminar Outline Basics of servers Amazon EC2 Setup R on an EC2 instance Stat
Predictive Analytics with Storm, Hadoop, R on AWS
Douglas Moore Principal Consultant & Architect February 2013 Predictive Analytics with Storm, Hadoop, R on AWS Leading Provider Data Science and Engineering Services Accelerating Your Time to Value using
HDFS. Hadoop Distributed File System
HDFS Kevin Swingler Hadoop Distributed File System File system designed to store VERY large files Streaming data access Running across clusters of commodity hardware Resilient to node failure 1 Large files
Hadoop 101. Lars George. NoSQL- Ma4ers, Cologne April 26, 2013
Hadoop 101 Lars George NoSQL- Ma4ers, Cologne April 26, 2013 1 What s Ahead? Overview of Apache Hadoop (and related tools) What it is Why it s relevant How it works No prior experience needed Feel free
HYBRID CLOUD SUPPORT FOR LARGE SCALE ANALYTICS AND WEB PROCESSING. Navraj Chohan, Anand Gupta, Chris Bunch, Kowshik Prakasam, and Chandra Krintz
HYBRID CLOUD SUPPORT FOR LARGE SCALE ANALYTICS AND WEB PROCESSING Navraj Chohan, Anand Gupta, Chris Bunch, Kowshik Prakasam, and Chandra Krintz Overview Google App Engine (GAE) GAE Analytics Libraries
DISTRIBUTED SYSTEMS [COMP9243] Lecture 9a: Cloud Computing WHAT IS CLOUD COMPUTING? 2
![DISTRIBUTED SYSTEMS [COMP9243] Lecture 9a: Cloud Computing WHAT IS CLOUD COMPUTING? 2](/thumbs/27/10009660.jpg "DISTRIBUTED SYSTEMS [COMP9243] Lecture 9a: Cloud Computing WHAT IS CLOUD COMPUTING? 2")
DISTRIBUTED SYSTEMS [COMP9243] Lecture 9a: Cloud Computing Slide 1 Slide 3 A style of computing in which dynamically scalable and often virtualized resources are provided as a service over the Internet.
Introduction to Hadoop
Introduction to Hadoop 1 What is Hadoop? the big data revolution extracting value from data cloud computing 2 Understanding MapReduce the word count problem more examples MCS 572 Lecture 24 Introduction
Hadoop IST 734 SS CHUNG
Hadoop IST 734 SS CHUNG Introduction What is Big Data?? Bulk Amount Unstructured Lots of Applications which need to handle huge amount of data (in terms of 500+ TB per day) If a regular machine need to
Big Data and Scripting Systems build on top of Hadoop
Big Data and Scripting Systems build on top of Hadoop 1, 2, Pig/Latin high-level map reduce programming platform interactive execution of map reduce jobs Pig is the name of the system Pig Latin is the