Amazon Web Services. Elastic Compute Cloud (EC2) and more...

Transcription

1 Amazon Web Services Elastic Compute Cloud (EC2) and more...

2 I don t work for Amazon I do however, have a small research grant from Amazon (in AWS$) Portions of this presentation are reproduced from slides by Amazon technology evangelist, Simone Brunozzi Disclaimer

3 Introducing Amazon Web Services What is Amazon EC2? EC2 Demo A closer look at Amazon Web Services Elastic MapReduce Demo Computational experiments on EC2 Agenda Conclusions

4 Introducing Amazon Web Services

5 Amazon.com sold first book in 1995, which shipped from Jeff Bezos garage Now sells a lot more than just books Affiliate sellers: almost anyone can sell almost anything Huge infrastructure: ~40 fulfilment centres, IT supporting millions of transactions per day What is Amazon.com?

6 Amazon s own Infrastructure-as-a-Service (IaaS) Publicly available via Amazon query (REST) and SOAP APIs over HTTP Provides storage, compute, hosting utilities, content delivery Various Software-as-a-Service on-top and supporting the infrastructure Database Demand based auto-scaling Enterprise style application messaging Infrastructure monitoring What is AWS?

7 2002: E- Commerce Service (ECS) March 06: Simple Storage Service (S3) Aug 06: Elastic Compute Cloud (EC2) beta Oct 08: EC2 enters production Evolution of AWS

8 Introducing Amazon Web Services What is Amazon EC2? EC2 Demo A closer look at Amazon Web Services Elastic MapReduce Demo Computational experiments on EC2 Agenda Conclusions

9 On demand Pay as you go Self service What is Amazon EC2?

10 Server Processors Boot disk Static IP Firewall SAN Load Balancer Monitoring Instance Elastic Compute Units Amazon Machine Image Elastic IP Security Groups Elastic Block Store Elastic Load Balancer CloudWatch EC2 Terminology

11 Try Me! EC2 Demo

12 Introducing Amazon Web Services What is Amazon EC2? EC2 Demo A closer look at Amazon Web Services Elastic MapReduce Demo Computational experiments on EC2 Agenda Conclusions

13 Compute Messaging Storage Content Delivery Elastic Compute Cloud Elastic MapReduce Auto Scaling Simple Queue Service Simple Notification Service Simple Storage Service Elastic Block Store AWS Import/Export Amazon CloudFront Monitoring Support Database Networking Amazon CloudWatch AWS Premium Support Amazon SimpleDB Relational Database Service Virtual Private Cloud Elastic Load Balancing Web traffic E-Commerce Payments/Billing Workflow Alexa Web Information Alexa Top Sites Fulfilment Web Service Flexible Payments Service Amazon DevPay Amazon Mechanical Turk More Amazon Web Services

14 Elastic MapReduce Hosted Apache Hadoop as-a-service A framework for distributed divide and conquer operations on large datasets Map function applied to each element of the dataset (in parallel) Reduce function combines output of maps Data Processing with Elastic MapReduce

15 Try Me! Elastic MapReduce Demo

16 Loose coupling with SQS

17 Producer-Consumer Problem msg msg msg msg Direct Simple Queue Service With Queue

18 Introducing Amazon Web Services What is Amazon EC2? EC2 Demo A closer look at Amazon Web Services Elastic MapReduce Demo Computational experiments on EC2 Agenda Conclusions

19 Nimrod workflows for robust design and search Vary parameters Execute programs Copy data in/out Parameter Space X, Y, Z might be: Basic datatypes; ints, floats, strings Files Random numbers to drive Monte Carlo modelling Parametric computing with the Nimrod tools X Y Z Solution Space

20 A lot of existing grid based infrastructure Grids provide services above IaaS E.g., you can build a grid on EC2 Grids provide job and data handling services, more like PaaS So, mix them together: Blair Bethwaite et al., Mixing Grids and Clouds: High-Throughput Science Using the Nimrod Tool Family, in Cloud Computing, vol. 0 (Springer London, 2010), Mixing Grids and Clouds

21 Nimrod EC2 Design

22 Introducing Amazon Web Services What is Amazon EC2? EC2 Demo A closer look at Amazon Web Services Elastic MapReduce Demo Computational experiments on EC2 Agenda Conclusions

23 Computational horsepower for parameter exploration/estimation and workflows Data processing Host a (scalable) web service/site Development platform What can I do with it?

24 Try it out, for free! Then, apply for an academic grant: What now?

25 Highlights (Per Month): 750 hours of Amazon EC2 Linux Micro Instance usage (613 MB of memory and 32- bit and 64-bit platform support) 10 GB of Amazon Elastic Block Storage, plus 1 million I/Os, 1 GB of snapshot storage, 10,000 snapshot Get Requests and 1,000 snapshot Put Requests 5 GB of Amazon S3 storage, 20,000 Get Requests, and 2,000 Put Requests AWS Free Usage Tier

26 Presentation by: Blair Bethwaite Researcher, Developer, SysAdmin Monash escience and Grid Engineering Lab Feedback/queries: Thank you!