Cloud Computing: Amazon Web Services

Transcription

1 Cloud Computing: Amazon Web Services overview & hands-on experience Christoph Langguth,

2 What is AWS / Cloud Computing? Cloud Computing: No clear definition, but can be seen as a cloud of resources available to you Available in this sense means: Unlimited resources! Pay only for what you actually use AWS: Started out internally within Amazon, and with first SOAP interfaces to amazon.com for public use Is now a mixture of actual WS operations and control operations for cloud infrastructure

3 What does it offer? Storage (S3: Simple Storage Service) Processing capabilities (EC2: Elastic Cloud Computing)... and a bit of glue/auxiliary services: SimpleDB SQS: Simple Queue Service...

4 A sample use case End users submit videos to be transcoded to the website. The videos are stored in Amazon S3, and a message ( the request message ) is placed in an Amazon SQS queue ( the incoming queue ) with a pointer to the video and to the target video format in the message. The transcoding engine, running on a set of Amazon EC2 instances, reads the request message from the incoming queue, retrieves the video from Amazon S3 using the pointer, and transcodes the video into the target format. The converted video is put back into Amazon S3 and another message ( the response message ) is placed in another Amazon SQS queue ( the outgoing queue ) with a pointer to the converted video. At the same time, metadata about the video (e.g., format, date created and length) can be indexed into Amazon SimpleDB for easy querying. During this whole workflow, a dedicated Amazon EC2 instance can constantly monitor the incoming queue and, based on the number of messages in the incoming queue, is able to dynamically adjust the number of transcoding Amazon EC2 instances to meet customers response time requirements.

5 EC2: Pricing Instances Small: 1.7GB RAM, 160GB HDD, 1 CU*,32bit XL: 15GB RAM, 1.6TB HDD, 4x2 CUs*,64bit * $0.10 per Small Instance (m1.small) instance-hour (or partial hour) High XL: 7GB RAM, 1.6TB HDD, 8x2.5 Cus* * $0.40 per Large Instance (m1.large) instance-hour (or partial hour) (*) 1 CU =~ 1.7GHz Xeon (2006) * $0.80 per Extra Large Instance (m1.xlarge) instance-hour (or partial hour) * $0.20 per High-CPU Medium Instance (c1.medium) instance-hour (or partial hour) * $0.80 per High-CPU Extra Large Instance (c1.xlarge) instance-hour (or partial hour) Data Transfer * $0.10 per GB Internet Data Transfer - all data transfer into Amazon EC2 * $0.17 per GB Internet Data Transfer - first 10 TB / month data transfer out of Amazon EC2 * $0.13 per GB Internet Data Transfer - next 40 TB / month data transfer out of Amazon EC2 *... Storage New Pricing (effective November 1st) * $0.15 per GB first 50 TB / month of storage used * $0.14 per GB next 50 TB / month of storage used *... (Storage in EU is slightly more expensive - $0.18/GB) SQS Requests * $0.01 per 10,000 Amazon SQS Requests ($ per Request) Data transferred between Amazon EC2 instances in the same Availability Zone using private IP addresses is free of charge. Data transferred between Amazon EC2 and Amazon S3-Europe will be charged as Internet data transfer. Data transferred between Amazon EC2 and Amazon S3-US, Amazon SDB, and Amazon SQS is free of charge.

6 Interfaces WSDL/SOAP REST Command-line tools (Java programs using SOAP) Security: X509 certificates (EC2) Or simpler Public ID/shared secret for signing (S3) 'SHA1 digest of the concatenation of "AmazonS3" + OPERATION + Timestamp' Privacy: by using HTTPS

7 S3 Objects are grouped into buckets and identified by keys: URL-friendly scheme is no coincidence: objects can be retrieved by plain HTTP (if access permissions allow) In addition, append?torrent to get a BitTorrent download May turn out cheaper if many requests SOAP: PutObject uses DIME attachments, PutObjectInline limited to 1MB :-) S3 provides redundant storage (>3 copies), automatic and dynamic placement No guarantees wrt freshness /concurrency control etc. Currently > 29 billion objects stored in S3 Petabytes of data Sensitive data should be encrypted.

8 EC2 All instances are virtual (probably XEN on some really fat machines) Instances are started using an imageid (image stored in S3) After an instance is terminated, all data on it is lost (but persisted across reboots) For persistent storage, use S3 or EBS (Elastic block storage) EBS is like a real block device Storage slightly cheaper than S3, but also transfers are billed ($0.10 per 1M I/O requests) Can be mounted only on one EC2 at a time Tools provided for creating custom images Start from a predefined one or from scratch Can pass up to 16K of user-defined data as parameters to the instance

9 EC2: hands-on! ls ec2-api-tools ec2-api-tools.zip private.pem public.pem export EC2_CERT=~/aws/public.pem export EC2_PRIVATE_KEY=~/aws/private.pem export EC2_HOME=~/aws/ec2-api-tools / export PATH=$PATH:$EC2_HOME/bin ec2-describe-images -o self -o amazon grep machine IMAGE ami-2c5fba45 ec2-public-images/demo-paid-ami-v1.07.manifest.xml amazon available public A79EC0DB i386 machine IMAGE ami-bd9d78d4 ec2-public-images/demo-paid-ami.manifest.xml amazon available public A79EC0DB i386 machine IMAGE ami-2f5fba46 ec2-public-images/developer-image-i386-v1.07.manifest.xml amazon available public i386 machine IMAGE ami-26b6534f ec2-public-images/developer-image.manifest.xml amazon available public i386 machine IMAGE ami-f51aff9c ec2-public-images/fedora-8-i386-base-v1.06.manifest.xml amazon available public i386 machine aki-a71cf9ce ari-a51cf9cc IMAGE ami-2b5fba42 ec2-public-images/fedora-8-i386-base-v1.07.manifest.xml amazon available public i386 machine aki-a71cf9ce ari-a51cf9cc IMAGE ami-f21aff9b ec2-public-images/fedora-8-x86_64-base-v1.06.manifest.xml amazon available public x86_64 machine aki-b51cf9dc ari- IMAGE ami-2a5fba43 ec2-public-images/fedora-8-x86_64-base-v1.07.manifest.xml amazon available public x86_64 machine aki-b51cf9dc ari- IMAGE ami-a21affcb ec2-public-images/fedora-core-6-x86_64-base-v1.06.manifest.xml amazon available public x86_64 machine aki-a53adfcc ari- IMAGE ami-2d5fba44 ec2-public-images/fedora-core-6-x86_64-base-v1.07.manifest.xml amazon available public x86_64 machine aki-a53adfcc ari- IMAGE ami-225fba4b ec2-public-images/fedora-core4-apache-mysql-v1.07.manifest.xml amazon available public i386 machine IMAGE ami-25b6534c ec2-public-images/fedora-core4-apache-mysql.manifest.xml amazon available public i386 machine IMAGE ami-2e5fba47 ec2-public-images/fedora-core4-apache-v1.07.manifest.xml amazon available public i386 machine IMAGE ami-23b6534a ec2-public-images/fedora-core4-apache.manifest.xml amazon available public i386 machine IMAGE ami-215fba48 ec2-public-images/fedora-core4-base-v1.07.manifest.xml amazon available public i386 machine IMAGE ami-20b65349 ec2-public-images/fedora-core4-base.manifest.xml amazon available public i386 machine IMAGE ami-205fba49 ec2-public-images/fedora-core4-i386-base-v1.07.manifest.xml amazon available public i386 machine aki-9b00e5f2 IMAGE ami-255fba4c ec2-public-images/fedora-core4-mysql-v1.07.manifest.xml amazon available public i386 machine IMAGE ami-22b6534b ec2-public-images/fedora-core4-mysql.manifest.xml amazon available public i386 machine IMAGE ami-36ff1a5f ec2-public-images/fedora-core6-base-x86_64.manifest.xml amazon available public x86_64 machine IMAGE ami-235fba4a ec2-public-images/getting-started-v1.07.manifest.xml amazon available public IMAGE ami-2bb65342 ec2-public-images/getting-started.manifest.xml amazon available public i386 machine

10 ec2-add-keypair gsg-keypair KEYPAIR gsg-keypair ec:3a:d7:94:6c:d0:5f:dc:c9:7f:d8:12:5d:41:e4:ed:22:e3:ce:f BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEAw5WlpISExy/B5GpEmFUT02Y/RaJtc/jTMnTmvRl86hKJmBLYK29qYn7phi7X (snip) ngnsqso7ryamyjfwesaiaw9bt+encwtdsh+t1f6o9np8xify2efhdeddr6/umib0gbd8dw== -----END RSA PRIVATE KEY----- nano -w id_rsa-gsg-keypair ec2-run-instances ami-235fba4a -k gsg-keypair RESERVATION r-6164b default INSTANCE i-4af15e23 ami-235fba4a pending gsg-keypair 0 m1.small T13:33: us-east-1b christoph@uri:~/aws$ ec2-describe-instances i-4af15e23 RESERVATION r-6164b default INSTANCE i-4af15e23 ami-235fba4a ec compute-1.amazonaws.com domu A0-F3.compute-1.internal running gsg-keypair 0 m1.small T13:33: us-east-1b christoph@uri:~/aws$ nmap -P0 -p22 ec compute-1.amazonaws.com PORT STATE SERVICE 22/tcp filtered ssh christoph@uri:~/aws$ ec2-authorize default -s /32 -p 22 GROUP default PERMISSION default ALLOWS tcp FROM CIDR /32 christoph@uri:~/aws$ nmap -P0 -p22 ec compute-1.amazonaws.com PORT STATE SERVICE 22/tcp open ssh christoph@uri:~/aws$ ssh -i id_rsa-gsg-keypair root@ec compute-1.amazonaws.com Warning: Permanently added 'ec compute-1.amazonaws.com, ' (RSA) to the list of known hosts. _ ) Rev: 2 _ ( / \ Welcome to an EC2 Public Image :-) Getting Started c /etc/ec2/release-notes.txt [root@domu a0-f3 ~]#

11 Connection to ec compute-1.amazonaws.com closed. ec2-terminate-instances i-4af15e23 INSTANCE i-4af15e23 running shutting-down ec2-describe-instances i-4af15e23 RESERVATION r-6164b default INSTANCE i-4af15e23 ami-235fba4a ec compute-1.amazonaws.com domu a0-f3.compute-1.internal shutting-down gsg-keypair 0 m1.small T13:33: us-east-1b christoph@uri:~/aws$ ec2-describe-instances i-4af15e23 RESERVATION r-6164b default INSTANCE i-4af15e23 ami-235fba4a terminated gsg-keypair 0 m1.small T13:33:

12 Conclusion Provides you with virtually unlimited storage and processing capabilities at the snap of your fingers Pay only for what you use Really simple and convenient to use Entire companies based on AWS for core business And an ecosystem around it, for example for scaling solutions Example given by S. Brunozzi: startup company providing video mixing experienced burst in demand System automagically scaled up from 20 instances to 6000, and down again BUT: you must completely trust Amazon, and you're sharing its fate...