Cloud Computing: Amazon Web Services

Cloud Computing: Amazon Web Services overview & hands-on experience Christoph Langguth, 2009-01-15

What is AWS / Cloud Computing? Cloud Computing: No clear definition, but can be seen as a cloud of resources available to you Available in this sense means: Unlimited resources! Pay only for what you actually use AWS: Started out internally within Amazon, and with first SOAP interfaces to amazon.com for public use Is now a mixture of actual WS operations and control operations for cloud infrastructure

What does it offer? Storage (S3: Simple Storage Service) Processing capabilities (EC2: Elastic Cloud Computing)... and a bit of glue/auxiliary services: SimpleDB SQS: Simple Queue Service...

A sample use case End users submit videos to be transcoded to the website. The videos are stored in Amazon S3, and a message ( the request message ) is placed in an Amazon SQS queue ( the incoming queue ) with a pointer to the video and to the target video format in the message. The transcoding engine, running on a set of Amazon EC2 instances, reads the request message from the incoming queue, retrieves the video from Amazon S3 using the pointer, and transcodes the video into the target format. The converted video is put back into Amazon S3 and another message ( the response message ) is placed in another Amazon SQS queue ( the outgoing queue ) with a pointer to the converted video. At the same time, metadata about the video (e.g., format, date created and length) can be indexed into Amazon SimpleDB for easy querying. During this whole workflow, a dedicated Amazon EC2 instance can constantly monitor the incoming queue and, based on the number of messages in the incoming queue, is able to dynamically adjust the number of transcoding Amazon EC2 instances to meet customers response time requirements.

EC2: Pricing Instances Small: 1.7GB RAM, 160GB HDD, 1 CU*,32bit XL: 15GB RAM, 1.6TB HDD, 4x2 CUs*,64bit * $0.10 per Small Instance (m1.small) instance-hour (or partial hour) High XL: 7GB RAM, 1.6TB HDD, 8x2.5 Cus* * $0.40 per Large Instance (m1.large) instance-hour (or partial hour) (*) 1 CU =~ 1.7GHz Xeon (2006) * $0.80 per Extra Large Instance (m1.xlarge) instance-hour (or partial hour) * $0.20 per High-CPU Medium Instance (c1.medium) instance-hour (or partial hour) * $0.80 per High-CPU Extra Large Instance (c1.xlarge) instance-hour (or partial hour) Data Transfer * $0.10 per GB Internet Data Transfer - all data transfer into Amazon EC2 * $0.17 per GB Internet Data Transfer - first 10 TB / month data transfer out of Amazon EC2 * $0.13 per GB Internet Data Transfer - next 40 TB / month data transfer out of Amazon EC2 *... Storage New Pricing (effective November 1st) * $0.15 per GB first 50 TB / month of storage used * $0.14 per GB next 50 TB / month of storage used *... (Storage in EU is slightly more expensive - $0.18/GB) SQS Requests * $0.01 per 10,000 Amazon SQS Requests ($0.000001 per Request) Data transferred between Amazon EC2 instances in the same Availability Zone using private IP addresses is free of charge. Data transferred between Amazon EC2 and Amazon S3-Europe will be charged as Internet data transfer. Data transferred between Amazon EC2 and Amazon S3-US, Amazon SDB, and Amazon SQS is free of charge.

Interfaces WSDL/SOAP REST Command-line tools (Java programs using SOAP) Security: X509 certificates (EC2) Or simpler Public ID/shared secret for signing (S3) 'SHA1 digest of the concatenation of "AmazonS3" + OPERATION + Timestamp' Privacy: by using HTTPS

S3 Objects are grouped into buckets and identified by keys: http://christoph.s3.amazonaws.com/obj/ec2.png URL-friendly scheme is no coincidence: objects can be retrieved by plain HTTP (if access permissions allow) In addition, append?torrent to get a BitTorrent download May turn out cheaper if many requests SOAP: PutObject uses DIME attachments, PutObjectInline limited to 1MB :-) S3 provides redundant storage (>3 copies), automatic and dynamic placement No guarantees wrt freshness /concurrency control etc. Currently > 29 billion objects stored in S3 Petabytes of data Sensitive data should be encrypted.

EC2 All instances are virtual (probably XEN on some really fat machines) Instances are started using an imageid (image stored in S3) After an instance is terminated, all data on it is lost (but persisted across reboots) For persistent storage, use S3 or EBS (Elastic block storage) EBS is like a real block device Storage slightly cheaper than S3, but also transfers are billed ($0.10 per 1M I/O requests) Can be mounted only on one EC2 at a time Tools provided for creating custom images Start from a predefined one or from scratch Can pass up to 16K of user-defined data as parameters to the instance

EC2: hands-on! christoph@uri:~/aws$ ls ec2-api-tools-1.3-24159 ec2-api-tools.zip private.pem public.pem christoph@uri:~/aws$ export EC2_CERT=~/aws/public.pem christoph@uri:~/aws$ export EC2_PRIVATE_KEY=~/aws/private.pem christoph@uri:~/aws$ export EC2_HOME=~/aws/ec2-api-tools-1.3-24159/ christoph@uri:~/aws$ export PATH=$PATH:$EC2_HOME/bin christoph@uri:~/aws$ ec2-describe-images -o self -o amazon grep machine IMAGE ami-2c5fba45 ec2-public-images/demo-paid-ami-v1.07.manifest.xml amazon available public A79EC0DB i386 machine IMAGE ami-bd9d78d4 ec2-public-images/demo-paid-ami.manifest.xml amazon available public A79EC0DB i386 machine IMAGE ami-2f5fba46 ec2-public-images/developer-image-i386-v1.07.manifest.xml amazon available public i386 machine IMAGE ami-26b6534f ec2-public-images/developer-image.manifest.xml amazon available public i386 machine IMAGE ami-f51aff9c ec2-public-images/fedora-8-i386-base-v1.06.manifest.xml amazon available public i386 machine aki-a71cf9ce ari-a51cf9cc IMAGE ami-2b5fba42 ec2-public-images/fedora-8-i386-base-v1.07.manifest.xml amazon available public i386 machine aki-a71cf9ce ari-a51cf9cc IMAGE ami-f21aff9b ec2-public-images/fedora-8-x86_64-base-v1.06.manifest.xml amazon available public x86_64 machine aki-b51cf9dc ari- IMAGE ami-2a5fba43 ec2-public-images/fedora-8-x86_64-base-v1.07.manifest.xml amazon available public x86_64 machine aki-b51cf9dc ari- IMAGE ami-a21affcb ec2-public-images/fedora-core-6-x86_64-base-v1.06.manifest.xml amazon available public x86_64 machine aki-a53adfcc ari- IMAGE ami-2d5fba44 ec2-public-images/fedora-core-6-x86_64-base-v1.07.manifest.xml amazon available public x86_64 machine aki-a53adfcc ari- IMAGE ami-225fba4b ec2-public-images/fedora-core4-apache-mysql-v1.07.manifest.xml amazon available public i386 machine IMAGE ami-25b6534c ec2-public-images/fedora-core4-apache-mysql.manifest.xml amazon available public i386 machine IMAGE ami-2e5fba47 ec2-public-images/fedora-core4-apache-v1.07.manifest.xml amazon available public i386 machine IMAGE ami-23b6534a ec2-public-images/fedora-core4-apache.manifest.xml amazon available public i386 machine IMAGE ami-215fba48 ec2-public-images/fedora-core4-base-v1.07.manifest.xml amazon available public i386 machine IMAGE ami-20b65349 ec2-public-images/fedora-core4-base.manifest.xml amazon available public i386 machine IMAGE ami-205fba49 ec2-public-images/fedora-core4-i386-base-v1.07.manifest.xml amazon available public i386 machine aki-9b00e5f2 IMAGE ami-255fba4c ec2-public-images/fedora-core4-mysql-v1.07.manifest.xml amazon available public i386 machine IMAGE ami-22b6534b ec2-public-images/fedora-core4-mysql.manifest.xml amazon available public i386 machine IMAGE ami-36ff1a5f ec2-public-images/fedora-core6-base-x86_64.manifest.xml amazon available public x86_64 machine IMAGE ami-235fba4a ec2-public-images/getting-started-v1.07.manifest.xml amazon available public IMAGE ami-2bb65342 ec2-public-images/getting-started.manifest.xml amazon available public i386 machine

christoph@uri:~/aws$ ec2-add-keypair gsg-keypair KEYPAIR gsg-keypair ec:3a:d7:94:6c:d0:5f:dc:c9:7f:d8:12:5d:41:e4:ed:22:e3:ce:f5 -----BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEAw5WlpISExy/B5GpEmFUT02Y/RaJtc/jTMnTmvRl86hKJmBLYK29qYn7phi7X (snip) ngnsqso7ryamyjfwesaiaw9bt+encwtdsh+t1f6o9np8xify2efhdeddr6/umib0gbd8dw== -----END RSA PRIVATE KEY----- christoph@uri:~/aws$ nano -w id_rsa-gsg-keypair christoph@uri:~/aws$ ec2-run-instances ami-235fba4a -k gsg-keypair RESERVATION r-6164b808 365171125558 default INSTANCE i-4af15e23 ami-235fba4a pending gsg-keypair 0 m1.small 2008-10-14T13:33:28+0000 us-east-1b christoph@uri:~/aws$ ec2-describe-instances i-4af15e23 RESERVATION r-6164b808 365171125558 default INSTANCE i-4af15e23 ami-235fba4a ec2-67-202-34-210.compute-1.amazonaws.com domu-12-31-38-00- A0-F3.compute-1.internal running gsg-keypair 0 m1.small 2008-10-14T13:33:28+0000 us-east-1b christoph@uri:~/aws$ nmap -P0 -p22 ec2-67-202-34-210.compute-1.amazonaws.com PORT STATE SERVICE 22/tcp filtered ssh christoph@uri:~/aws$ ec2-authorize default -s 131.152.55.50/32 -p 22 GROUP default PERMISSION default ALLOWS tcp 22 22 FROM CIDR 131.152.55.50/32 christoph@uri:~/aws$ nmap -P0 -p22 ec2-67-202-34-210.compute-1.amazonaws.com PORT STATE SERVICE 22/tcp open ssh christoph@uri:~/aws$ ssh -i id_rsa-gsg-keypair root@ec2-67-202-34-210.compute-1.amazonaws.com Warning: Permanently added 'ec2-67-202-34-210.compute-1.amazonaws.com,67.202.34.210' (RSA) to the list of known hosts. _ ) Rev: 2 _ ( / \ Welcome to an EC2 Public Image :-) Getting Started c /etc/ec2/release-notes.txt [root@domu-12-31-38-00-a0-f3 ~]#

Connection to ec2-67-202-34-210.compute-1.amazonaws.com closed. christoph@uri:~/aws$ ec2-terminate-instances i-4af15e23 INSTANCE i-4af15e23 running shutting-down christoph@uri:~/aws$ ec2-describe-instances i-4af15e23 RESERVATION r-6164b808 365171125558 default INSTANCE i-4af15e23 ami-235fba4a ec2-67-202-34-210.compute-1.amazonaws.com domu-12-31-38-00-a0-f3.compute-1.internal shutting-down gsg-keypair 0 m1.small 2008-10-14T13:33:28+0000 us-east-1b christoph@uri:~/aws$ ec2-describe-instances i-4af15e23 RESERVATION r-6164b808 365171125558 default INSTANCE i-4af15e23 ami-235fba4a terminated gsg-keypair 0 m1.small 2008-10-14T13:33:28+0000

Conclusion Provides you with virtually unlimited storage and processing capabilities at the snap of your fingers Pay only for what you use Really simple and convenient to use Entire companies based on AWS for core business And an ecosystem around it, for example for scaling solutions Example given by S. Brunozzi: startup company providing video mixing experienced burst in demand System automagically scaled up from 20 instances to 6000, and down again BUT: you must completely trust Amazon, and you're sharing its fate...