Amazon Elastic MapReduce. Jinesh Varia Peter Sirota Richard Cole

Transcription

1 Amazon Elastic MapReduce Jinesh Varia Peter Sirota Richard Cole

2 Start End From IDE Command line Web Console Notify Input Data Get Results

3 Start End From IDE Command line Web Console AWS EC2 Instance Notify Input Data Get Results

4 Start End From IDE Command line Web Console Input dataset AWS EC2 Instance Notify Input Data Input S3 bucket Get Results Amazon S3

5 Amazon EC2 Instances Start End From IDE Command line Web Console Input dataset AWS EC2 Instance Notify Input Data Input S3 bucket Get Results Amazon S3

6 Amazon EC2 Instances Start End From IDE Command line Web Console Input dataset AWS EC2 Instance output results Notify Input Data Input S3 bucket Output S3 bucket Get Results Amazon S3

7 Start End From IDE Command line Web Console Notify Input Data Get Results

8 Use Cases Bio-informatics (Genome analysis) Data mining (Log processing, click stream analysis, similarity algorithms, etc.) Financial simulation (Monte Carlo simulation) File processing (resize jpegs) Web indexing 8

9 Why Elastic MapReduce? Large scale data processing has a lot of MUCK and we want to remove it for our customers Hard to manage compute clusters Cluster start-up and shutdown Cluster monitoring and resource management Security groups management Hard to tune performance of running clusters Dozens of difficult to understand settings can affect performance 2-3x 9

10 Amazon Elastic MapReduce Benefits Elastic Uses as many or as few EC2 instances as needed. Spin up large or small job flows in minutes. Easy to use Get up and running quickly with easy-to-use web console and sample jobs. No configuration necessary. Reliable Fault tolerant service built on top of battle-tested AWS infrastructure. Automatically retries failed tasks. Cost Effective We monitor progress of your jobs and turn off resources when job flow is done.

11 Hadoop made simple and easy

12 Job Flow Job Flow inputs: Number and Type of EC2 instances Sequence of MapReduce Steps Input Location Output Location MapReduce Algorithm Location Optional: log location, ssh keys MapReduce Step 1 Output (S3, HDFS, etc) MapReduce Step 2 Output (S3, HDFS, etc.) MapReduce Step N Output (S3) 12

13

14 EC2 EBS Merchants Data Feeds Ingestor

15 EC2 MapReduce JobStep (Parser) EC2 EC2 Instances EBS Merchants Data Feeds Ingestor Converts raw product feeds into a common format (Map-only) HDFS Solr Servers (for grouping)

16 EC2 MapReduce JobStep (Parser) EC2 EC2 Instances EBS Merchants Data Feeds Ingestor Converts raw product feeds into a common format (Map-only) HDFS Solr Servers (for grouping) Same HDFS on EC2 MapReduce JobStep (Grouper) Uses Solr to group identical products together (Map) and consolidates them into the correct groups (Reduce)

17 EC2 MapReduce JobStep (Parser) EC2 EC2 Instances EBS Merchants Data Feeds Ingestor Converts raw product feeds into a common format (Map-only) HDFS Solr Servers (for grouping) EC2 Web Servers Solr Servers (for users) Same HDFS on EC2 MapReduce JobStep (Grouper) Uses Solr to group identical products together (Map) and consolidates them into the correct groups (Reduce)

18 EC2 MapReduce JobStep (Parser) EC2 EC2 Instances EBS Merchants Data Feeds Ingestor Converts raw product feeds into a common format (Map-only) HDFS Solr Servers (for grouping) EC2 Shopper Web Servers S3 Solr Servers (for users) Same HDFS on EC2 MapReduce JobStep (Grouper) Uses Solr to group identical products together (Map) and consolidates them into the correct groups (Reduce) Product Images MapReduce Job (Image Processor)

19 Ways to use Amazon Web Management Console console.aws.amazon.com Point and click Web Services API Command Line Tools by Amazon scripts Web Service Clients Integrate from within your app

20 Demo

21 Customers 2:00 PM Applications Track Expert Company Industry John Barr YieldEx Ad inventory mgmt Ben Hardy eharmony.com MatchMaking Paco Nathan ShareThis Social Web Elias Torres Lookery Targeted Advertising Ted Dunning DeepDyve Search

22 Thank You! Jinesh Varia Peter Sirota aws.amazon.com/elasticmapreduce

23 Same Pay-as-you-go pricing Standard Amazon EC2 Instances Amazon EC2 Price per hour (On-Demand Instances) Amazon Elastic MapReduce Price per hour Small (Default) $0.10 per hour $0.015 per hour Large $0.40 per hour $0.06 per hour Extra Large $0.80 per hour $0.12 per hour High CPU Instances Amazon EC2 Price per hour (On-Demand Instances) Amazon Elastic MapReduce Price per hour Medium $0.20 per hour $0.03 per hour Extra Large $0.80 per hour $0.12 per hour