Running Amazon EC2 instances. Juan Lago, July (Document )

Transcription

1 Running Amazon EC2 instances Juan Lago, July 2011 (Document ) This guide shows how to run an Amazon EC2 instance both from the command line and the Web interface, i.e, Amazon Web Service (AWS) Management Console. Prerequisites An AWS account. AWS tools. Java Runtime Environment 5 or later. A SSH client. Some bash scripting background. Creating security credentials You need security credentials before using AWS tools. Amazon provides an EC2 Getting Starting Guide for setting up an AWS account and accessing EC2 Service in the following link: First of all, I m going to create new X.509 certificate and a private key. Both, the certificate and the private key, are needed for setting up the command line tools and must be downloaded into the computer that will access AWS API. Please, sign in into the AWS Management Console ( using your login and password. Then, click the Account link located in the upper part of the Management Console Web page as shown in figure 1. 1/15

2 Figure 1. AWS Management Console. Account link highlighted. The account page will be displayed. From this page, you can manage different aspects of your account like account activity, identity and access management, payment methods, security credentials, etc. Click the link called Security Credentials and sign in again. In the Access Credentials section, choose X.509 Certificates tab and press Create a new Certificate. A new X.509 Certificate will be created as well as a new private key (see figure 2). Please download both files using the corresponding download links. In this case, I put these files in a folder called.ec2 inside my hadoop user home directory. $ cd $ pwd /home/hadoop $ mkdir.ec2 $ mv./downloads/*.pem.ec2 $ cd.ec2 $ ls -la cert-htvo5vszslljp2yw2c6qchedximomwj2.pem pk-htvo5vszslljp2yw2c6qchedximomwj2.pem 2/15

3 Figure 2. X.509 Certificate created. Private key file generated and prepared for downloading. Also, in the same Security Credentials web page you can find your AWS Account ID under the Account Identifiers section. You should use this value whenever you need to provide and EC2 user ID. The account ID is of the form XXXX-YYYY-ZZZZ. Your AWS Account ID, with the hyphens removed is your EC2 user ID. The example above would be XXXXYYYYZZZZ. Installing Amazon Web Services Tools AWS Tools must be installed in order to use the AWS API from the command line. In this guide, a Biolinux distribution (Dabian based) machine will be used for interacting with AWS API. Download the command line tools from the Amazon EC2 Resource Center: A file called ec2-api-tools.zip will be downloaded. These tools are written in Java and include shell scripts for both Windows and Linux/Mac OSX. This guide assumes you have SUN Java 6 and ssh client installed and JAVA_HOME environment variable is correctly set. For example, in Biolinux and with the hadoop user, check the Java installation: 3/15

4 $ echo $JAVA_HOME /usr/lib/jvm/java-6-sun $ java -version java version "1.6.0_26" Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) Client VM (build 20.1-b02, mixed mode, sharing) Unzip the downloaded file and move the lib and bin directories (just created after unzipping) inside.ec2 directory. $ cd $ cd Downloads $ unzip ec2-api-tools.zip $ cd ec2-api-tools $ ls drwxr-xr-x 2 hadoop hadoop drwxr-xr-x 2 hadoop hadoop -rw-r--r-- 1 hadoop hadoop -rw-r--r-- 1 hadoop hadoop -rw-r--r-- 1 hadoop hadoop $ mv bin../.ec2 $ mv lib../.ec Aug 24 01:49 bin 4096 Aug 24 01:49 lib 4852 Aug 24 01:47 license.txt 539 Aug 24 01:47 notice.txt Aug 24 01:47 THIRDPARTYLICENSE.TXT Create $EC2_HOME environment variable and add the bin directory to $PATH environment variable. For this, edit the file.bashrc for user hadoop: $ cd $ vim.bashrc export EC2_HOME=~/.ec2 export PATH=$PATH:$EC2_HOME/bin The command line tools need access to the the private key and X.509 certificate you generated after signing up for the Amazon EC2 service. It's possible to provide this information on the command line every time you invoke the tools. But it's far simpler to set up some environment variables and be done with it. Again, edit your user s.bashrc file and add the following environment variables: export EC2_PRIVATE_KEY=`ls $EC2_HOME/pk-*.pem` export EC2_CERT=`ls $EC2_HOME/cert-*.pem` Running an instance Now we are ready for running an instance from the command line. The first thing is to find a suitable AMI (Amazon Machine Image) to run. For this purpose, you can use the following command: $ ec2-describe-images -a --filter image-type=machine grep BioLinux IMAGE ami-90d32af /CloudBioLinux Ubuntu LTS 64bit 4/15

5 available public x86_64 machine aki-427d952b ebs paravirtual xen IMAGE ami-ad8e4ec /CloudBioLinux Ubuntu bit available public x86_64 machine aki-427d952b ebs paravirtual xen This example will find all AMIs of type machine containing the string BioLinux. We need the AMI ID to run the instance. This can be found in the second column of the output (in bold) from the ec2-describe-images command. In the previous example, ami-90d32af9 and ami-ad8e4ec4. See the command line reference for more details: Generating a keypair In order to login securely to a public AMI instance, you will need a public/private keypair. Amazon EC2 public images use this feature to provide secure access without passwords. There is a command in the API tools for this. You only have to choose a name for the keypair as parameter: $ ec2-add-keypair keypair_for_biolinux KEYPAIR keypair_for_biolinux32:96:68:6d:1d:7f:0e:93:df:94:9d: 59:3e:ac:f3:a2:d3:59:5b: BEGIN RSA PRIVATE KEY----- MIIEpAIBAAKCAQEA2MUcek2371xrZRVipZx+p4f4lU+8/02mLZLEQFfqVCKjajSy3ASrAFB02Ul+ SBGQ4OaX495+buyf+DNHFrXAyF8zXY/oF2o9A9AmFeuSdtXZB3lZeBBvI3//cT4L1bRQKoKZ2+tm KcqW5rcaDkWKJbKRBsCRd3EWXhUfvM8y5umwzPLgbj7st7Gfcn2g4b70OJyg604D5/8lCnYMNfxO hseaeire/4ecfmnk9dhz6xeyrdpsu82gufo45fuq3sbqcegzxqt5dwybiltcuufzrxxpfuyrx9g2 GzWJr9Siw86tm9quQzATdZBaFvTefH5YdiRm0k7tFOsyWhRxN6wrJwIDAQABAoIBADrKCpUo6s+h 8wZxLFXaAYeSGtrzy0A6YO1PNqZ0sXqsAkHVlQSSx4XITEh55CPLj81O/LayKvwOdLwtvAPuuyAc oaoowbuv9mhivwvwfcj48oluae3mfzg7ij2/xbnruyqutqiv75k59n8p/amnprbiq4jfrrrtute+ 7WKZlyaEjqh/jr/GO0pITlzCWtJ8qp1anyApCBB9Oq+fPdURE3EX0lBaXN/GKeVJ44bBHj+T82O0 binfqo05jmnqjtopw662udhyhgnzrehxmgfjgxpazad+kgw1v90q9wddh0rq/jxikawlqa1ia/nu ow+mjz5ujlbmuyvkp0nrddn/xmecgyea9bcxpqr9dtx1lhid0f0hprpxw9h9dmkkoudokedl0rov hk43i2e2czsdzxkd19azieav5thyw+q7ussf8r0ygn6cinxubeo0qrc8yqt/i/cqewrmxq9y2c3i hnbxab7vwpzd2o3bwl6xzjjp3au/rkemm01dqnev42hslmgzhpecgyea4dfcysdohukgiozhtlmv 496M+QhxM4MZJq1VeONVpHn2Vx9f2k8s0+GCaSwjZ2f5PH4By1bs7+Az9QtkNJVglaH+Dz7citlJ FxhhCXJKeI2mq9PAQwwOQo7rSrdbCiQCEBKpV5KSB0NrN6QQBP1ZFuXVCryGXcj2Obs8nQObqZcC gyad/nukm++00rshqy5sm7b7l7dm90zzbfxks6zqiec/zblnciti7nt4igiobhdlon6pgykprmkw 9bZoJBhiSTYhN9Nj4LZNs74NMz3YTAPl0gYQzDWS2I/shnIz983HKMeO1U8/npF0tdJ+mfDF3nFH YWFok8kOkz9Ywlgl0UdMAQKBgQCQBhs17gkHwr9ghQXBiRFUXh7Eyc93LEmj0rJJT6DULTsdwFJe ZFOQI6YI8C77yOC25Ry70glzyCUaRlN2/TQE+LOOgWYf3gWwhsmLPqmjS6JYvcdEerG1XHNMyLV0 OBZgraX0xPH/U81SRHw60l44LgW4vRVrRbdQhclX0G86rQKBgQCL0YFBClOz9M9oanUFGAOSrXPl TsQjqWbsiq/RxW9QZnwB4TK2g8WkrNM2Wem3n3uiU5/ixYayIXSBzjSGjtpoexF8pmQdhOdurwmI MhzTKp75fOrYth4Q+7ve1VhJQmYrsuv9kbkcmV/Jux3lefmhdGkAoKrVafLN+BGzLhxgBA== -----END RSA PRIVATE KEY /15

6 Copy the content between -----BEGIN... and -----END... in your local machine. In my case, I ve copied it in a file called id_rsa-keypair_for_biolinux in.ec2 directory. For security reasons, change this file permissions to 600. $ chmod 600 id_rsa-keypair_for_biolinux $ ls -la \bin cert-htvo5vszslljp2yw2c6qchedximomwj2.pem id_rsa-keypair_for_biolinux \lib pk-htvo5vszslljp2yw2c6qchedximomwj2.pem Running an instance Now we are going to launch a CloudBioLinux instance using an AMI ID found in our previous search: $ ec2-run-instances ami-90d32af9 -k keypair_for_biolinux -t t1.micro RESERVATION r-a1d098ce default INSTANCE i-27494b46 ami-90d32af9 pending keypair_for_biolinux 0 t1.micro t08:02: us- east-1a aki-427d952b monitoring-disabled ebs paravirtual xen sg-20d6b149 default The instance ID returned by ec2-run-instances command is an identifier and will be used later to manipulate this instance (in this example, i-27494b46) Authorizing network access to the instance Before you can reach your instance over the internet, you have to authorize traffic to it. SSH port is 22, Web access is 80, etc: $ ec2-authorize default -p 22 GROUP!! default PERMISSION! default! ALLOWS! tcp FROM CIDR /0 ingress From the AWS management console (Web interface), you can define a security group in Amazon to tell which ports are enable from specific IPs. My default security group is shown in figure 3. You can also do it from the command line with ec2-create-group command. Also, you can use ec2-describe-group to find out what groups you already have and how they are configured. 6/15

7 Figure 3. Default security group in AWS Management Console. You can think of security groups as firewall policies, sets of rules on what ports are to be allowed and not on a given group. By default every port is blocked. It s a good idea to add a security group before lunching any instances as you will have to specify a security group for the instance that you are lunching. And without specifying one you won t be able to access the instance through ssh. Connecting to the instance Now you can login as root (or any other user) and reach access to the instance as you would do with any other host, using your private key. We also need to know the instance IP or hostname. This value is returned in ec2-describe-instances: $ ec2-describe-instances i-27494b46 RESERVATION r-a1d098ce default INSTANCE i-27494b46 ami-90d32af9 ec compute-1.amazonaws.com ip ec2.internal running keypair_for_biolinux 0 t1.micro T08:02: us-east-1a aki-427d952b monitoring-disabled ebs paravirtual xen sg-20d6b149 default BLOCKDEVICE /dev/sda1 vol fc T08:02:51.000Z In this case, the hostname is ec compute-1.amazonaws.com. Because this instance is an Ubuntu platform, I will connect with the user ubuntu: 7/15

8 $ ssh -i id_rsa-keypair_for_biolinux Linux ip ec2 #24-Ubuntu SMP Fri Jan 7 18:30:50 UTC 2011 x86_64 GNU/Linux Ubuntu LTS Welcome to Ubuntu * Documentation: System information as of Fri Aug 26 08:33:34 UTC 2011 System load: 0.0 Usage of /: 45.4% of 19.69GB Memory usage: 22% Swap usage: 0% Processes: 95 Users logged in: 0 IP address for eth0: Graph this data and manage this system at At the moment, only the core of the system is installed. To tune the system to your needs, you can choose to install one or more predefined collections of software by running the following command: sudo tasksel --section server packages can be updated. 6 updates are security updates. A newer build of the Ubuntu lucid server image is available. It is named 'release' and has build serial ' '. Ubuntu LTS Welcome to Ubuntu! * Documentation: System information as of Thu Jun 2 11:16:23 UTC 2011 System load: 0.06!!! Processes: 80 Usage of /: 3.6% of 19.69GB! Users logged in: 1 Memory usage: 1%!!! IP address for eth0: Swap usage: 0% Graph this data and manage this system at At the moment, only the core of the system is installed. To tune the system to your needs, you can choose to install one or more 8/15

9 predefined collections of software by running the following command: sudo tasksel --section server packages can be updated. 0 updates are security updates. Last login: Thu Jun 2 15:21: from sph harvard.edu ubuntu@ip :~$ Terminating your instances Amazon will charged you for the time your instance is running per hour. Charging starts when you launch the instance, not when you log into. If, for instance, your instance is running during 20 minutes, you will be charge a hole hour. But you also have to consider the starting and finishing time, not only the duration. If the instance started at 9:55 am and finished at 11:03 am, you will be charge 3 hours (from 9 to 10, 10 to 11, 11 to 12). Terminating an instance can be done with ec2-terminate-instances. You need to indicate which instance to terminate with its ID: $ ec2-terminate-instances i-27494b46 INSTANCE i-27494b46 running shutting-down A useful bash script Here you are a useful bash script for performing most of the commands needed from launching to connecting an EC2 CloudBioLinux instance: #!/usr/bin/env bash # Script for launching Amazon EC2 CloudBioLinux instances # Run: $ source launch_biolinux.sh # Sets some usefull system variables (env grep MY_EC2) # Amazon CloudBioLinux instances # ami-90d32af9 - Ubuntu LTS 64bit # ami-ad8e4ec4 - Ubuntu bit INSTANCE='ami-90d32af9' #INSTANCE='ami-ad8e4ec4' echo "Launching instance..." echo ">> ec2-run-instances $INSTANCE -k keypair_for_biolinux -t t1.micro" RUN=`ec2-run-instances $INSTANCE -k keypair_for_biolinux -t t1.micro` 9/15

10 set $RUN # Now $6 has the instance ID export MY_EC2_INSTANCE=$6 echo "Instance ID = $6" echo "Retrieving $6 description..." echo ">> ec2-describe-instances $6" DESCRIBE=`ec2-describe-instances $6` set $DESCRIBE # Now $8 has the instance hostname or the string "pending" STATUS=$8 while [ $STATUS = "pending" ] do echo -n "." sleep 5 DESCRIBE=`ec2-describe-instances $6` set $DESCRIBE # Now $8 has the instance hostname STATUS=$8 done export MY_EC2_HOSTNAME=$8 export MY_EC2_CONNECT="ssh -i id_rsa-keypair_for_biolinux ubuntu@$8" export MY_EC2_TERMINATE="ec2-terminate-instances $MY_EC2_INSTANCE" Using the Web interface: AWS Management Console All the topics that we have seen for running an instance from the command line, can be done easily using the AWS Web interface, i.e., the AWS Management Console. In the following screenshots, you can see how to launch and terminate an instance. If you see a screen not showed in the screenshots, accept the default values and keep it unmodified (press continue ). 10/15

11 Figure 4. Launch instance: click 1 to access EC2 service and 2 to launch an instance. Figure 5. Choose AMI: click 1 for community AMIs and type search string in 2. Click select in your favorite AMI (3). 11/15

12 Figure 6. Select number of instances (1) and type (2). Figure 7. Choose an existing key pair (1) or create a new one (2). 12/15

13 Figure 8. Security groups: choose an existing group or create a new one. Figure 9. Review all parameters before launching the instance. 13/15

14 Figure 10. Instance in pending status while launching. Figure 11. Connect help. Inside instance actions (1) press connect. ssh command line showed in (2). 14/15

15 Figure 12. Terminating instance. In instance actions combo box, press terminate. 15/15