Amazon Web Services Building in the Cloud

Amazon Web Services Building in the Cloud

Amazon has Three Parts

AWS Principles Easy to use Fast Elastic Highly available Secure Pay as you go

The Utility Model

AWS Bandwidth Growth

AWS Storage Growth

Services Overview Building block services that allow developers to innovate and make money Infrastructure As a Service Amazon Simple Storage Service Amazon Elastic Compute Cloud Amazon Simple Queue Service Amazon SimpleDB Commerce As a Service Amazon Flexible Payments Service Fulfillment Web Service People As a Service Amazon Mechanical Turk Alexa Web Services Alexa Web Information Service Alexa Top Sites Alexa Site Thumbnail Alexa Web Search Platform

Amazon S3 Simple Storage Service Highly scalable data storage in-the-cloud Programmatic access via web services API Simple to get going, simple to use Highly available and durable Pay-as-you-go: Storage: $0.15 / GB / month Data Transfer: starts at $0.17 / GB Requests: nominal charges Content Delivery Network

Amazon EC2 Elastic Compute Cloud Resizable compute capacity in the cloud Obtain and boot new server instances in minutes Quickly scale capacity, up or down, as your computing requirements change Full root access to a blank Linux or Windows machine Simple Web service management interface Changes the economics of computing

Amazon EC2 Features Elastic Block Store Persistent storage, up to 1 TB Backed by S3 Snapshots Availability Zones Separate Data Centers Increased Reliability Elastic IP Addresses Fixed IP can be quickly attached to an instance Don't need to wait for DNS propagation delays

Predictions Cost Money

Amazon Machine Images Public AMIs: Use pre-configured, template AMIs to get up and running immediately. Choose from Fedora, Ubuntu, Windows configurations and more Private AMIs: Create an Amazon Machine Image (AMI) containing your applications, libraries, data and associated configuration settings Paid AMIs: Set a price for your AMI and let others purchase and use it (Single payment and/or per hour)

Instance Types Standard High CPU Small Large Extra Large Medium Extra Large Bits 32 64 64 32 64 RAM 1.7 GB 7.5 GB 15 GB 1.7 GB 7 GB Disk 160 GB 850 GB 1690 GB 350 GB 1690 GB EC2 Compute Units 1 4 8 5 20 I/O Performa nce Medium High High High High Firewall Yes Yes Yes Yes Yes Per Hour $0.10 $0.40 $0.80 $0.20 $0.80

Amazon SQS Simple Queue Service Scalable Messaging - many application threads/servers can be used without worrying about overloading the messaging system Unlimited - any number of queues, any number of messages per queue Supports reader fail-over -messages become invisible to other readers when read the first time. Adjustable time before message is again visible to other readers

Amazon SimpleDB Simple to use - implements most common / simple database features behind a set of web service calls Flexible - doesn't require structured definition of data which allows you to add new data easily Scalable - scales easily as your application grows. Simply add more domains or data within your domains Fast - quick efficient storage and retrieval of your data Reliable - uses Amazon's high-availability data centers to provide strong and consistent performance Inexpensive - you pay only for the resources you consume, so you avoid significant up-front costs. Eventual Consistency!

Using Amazon SimpleDB item description color 123 Sweater Blue, Red 456 Dress shirt White, Blue 789 Shoes Black material Leather PUT (item, 123), (description, Sweater), (color, Blue), (color, Red) PUT (item, 456), (description, Dress shirt), (color, White), (color, Blue) PUT (item, 789), (description, Shoes), (color, Black), (material, Leather) Query Domain = MyStore [ description = Sweater ]

Leveraging AWS Application hosting - Using EC2 to provided an elastic web tier Batch processing - Using SQS, EC2 and S3 to process large amounts of data Searchable Object Store - Combine S3 and SimpleDB to store metadata data about objects and run searches Cloud-Bursting - Leveraging EC2 for excess capacity to augment your own data center

New York Times and AWS

24 hours later

Animoto scales

50 to 3500 servers in 3 days!

Using AWS with Java Web services provide language independence WSDL can be used to build an access layer Popular Open Source Java libraries include; Jets3t - Amazon S3 access typica - Amazon EC2, SQS, SimpleDB, FPS, DevPay

Jets3t An open source library for interacting with Amazon S3 and objects stored there. Supports both SOAP and REST interfaces. Well tested and very robust.

S3 API package org.jets3t.service; Class to know about; S3Service S3Bucket createbucket(string name) S3Object getobject(s3bucket bucket, String key) S3Bucket setacl(accesscontrollist acl) S3Object setcontenttype(string type) setdatainputstream(inputstream istr)

S3 Code Sample // code from a servlet that proxies data from S3 s3 = new RestS3Service(new AWSCredentials(awsAccessId, awssecretkey)); // get the object details from S3 S3Object obj = new S3Object(objectId); S3Bucket bucket = new S3Bucket(bucketName); obj = s3.getobject(bucket, objectid); // use the stream from the S3Object to move data to the servlet response stream OutputStream ostr = response.getoutputstream(); response.setcontenttype(obj.getcontenttype()); InputStream in = obj.getdatainputstream(); Copier.copyStreams(in, ostr);

Typica Typica was designed initially to support SQS via the REST interface in a very reliable and thread-safe way. The initial single digit downloads were inspiring enough to get EC2 support added. Incorporates patches from users and supports many popular services. Current popularity means hundreds of downloads for each release.

SQS API package com.xerox.amazonws.sqs; Classes to know about; QueueService messagequeue getorcreatemessagequeue(queuename) List<MessageQueue> listmessagequeues MessageQueue sendmessage(msg) String receivemessage() deletemessage(msgid) deletequeue(force) Message String getmessagebody()

SQS Sample Code QueueService qs = new QueueService(<accessId>, <secretkey>); MessageQueue queue = qs.getorcreatemessagequeue("testqueue"); queue.sendmessage("test Message"); Message msg = queue.receivemessage(); logger.debug("message received : "+msg.getmessagebody());

EC2 API package com.xerox.amazonws.ec2; Classes to know about; Jec2 List<ImageDescription> describeimages(imageids) List<ReservationDescription> describeinstances (instanceids) ReservationDescription runinstances(imageid,...) ConsoleOutput getconsoleoutput(instanceid) ReservationDescription List<Instance> getinstances();

EC2 Sample Code Jec2 ec2 = new Jec2(<accessId>, <secretkey>); ReservationDescription res = ec2.runinstances("ami-1234567", 1, 1, null, null, "dak-keypair"); String instanceid = res.getinstances().get(0).getinstanceid(); while (!res.getinstances().get(0).getstate().equals("running")) { try { Thread.sleep(5000); } catch (InterruptedException ex) {} res = ec2.describeinstances(instanceid); } logger.debug("instance running : "+instanceid); ConsoleOutput cons = ec2.getconsoleoutput(instanceid); logger.debug("console output = "+cons.getoutput()); ec2.terminateinstances(new String [] {instanceid});

SimpleDB API package com.xerox.amazonws.sdb; Classes to know about; SimpleDB ListDomainsResult listdomains() Domain createdomain(string name) Domain QueryResult listitems(string query) QueryWithAttributesResult listitemswithattributes(string query) Item List<ItemAttribute> getattributes() SDBResult putattributes(list<itemattribute> attrs)

SimpleDB Sample Code SimpleDB sdb = new SimpleDB(<accessId>, <secretkey>); Domain dom = sdb.getdomain("testdomain"); logger.info("items:"); String nexttoken = ""; while (nexttoken!= null) { ListDomainsResult result = dom.listitems(query, nexttoken, 10); List<Item> items = result.getitemlist(); for (Item item : items) { logger.info(item.getidentifier()); } nexttoken = result.getnexttoken(); }

Useful Links http://aws.amazon.com/ http://code.google.com/p/typica/ http://jets3t.s3.amazonaws.com/index.html