Starting your Amazon virtual machine

Transcription

1 Nescent Academy - Workshop 1 Starting your Amazon virtual machine Instructors: Konrad Paszkiewicz k.h.paszkiewicz at exeter ac uk Alexie Papanicolau Alexie.Papanicolaou at csiro.au Francesc Lopez francesc.lopez at yale.edu Objectives: By the end of this section you will be expected to: Log into the Amazon Console and start your instance of the Nescent AcademyVM Log in to an Amazon EC2 Linux instance using your own computer Continue with the Short read genomics tutorials at your own pace

2 Introduction For this first workshop we will provide an overview of the cloud as described by Amazon and how, as researchers, we can use this flexible resource to get work done quickly and relatively inexpensively. We will dive right into starting up the pre-prepared Virtual Machine (VM) and logging into the Amazon management console. We will give you a whirlwind tour of the features of Amazon's cloud and then start to teach you the basics of Linux (the environment), the command line and the programming language Perl. This is a tutorial designed to teach you the fundamentals of Linux in a biological context. The aim is to get you as comfortable as possible using the linux command line and basic programming so that future workshops on the analysis of data (e.g. RNA-seq etc) can focus on the actual analysis without getting bogged down in the mechanics of how to use the operating system. It is also designed to enable you to start to write your own simple programs (sometimes called scripts) to do repetitive tasks or large-scale data analysis which might otherwise prove impossible. For this tutorial I borrowed documentation from the following sites: Task 1 Tour of Amazon's Cloud In this section of the workshop we will log into Amazon's cloud (referred to as Amazon Web Services or AWS) and take a look at the various services offered by Amazon. These include: Elastic Cloud Compute (EC2): the service AWS is known for, hourly rental of Linux and Windows machines. Amazon now has special High Performance Computing nodes (HPC) and Graphical Computing nodes (GPU nodes) Simple Storage Service (S3): a storage service, not particularly fast but great for storing large buckets of data for long term storage, sharing, or temporary storage for use between instances Elastic Block Storage (EBS): similar to S3 but limited in size (max 1TB), these are virtual hard drives that you can attach and detach very quickly to and from your running instances. Think of these as the USB flash drive of the cloud computing world A ton of other services that are geared towards building highly scalable and fault-tolerant webbased services. Many can be co-opted for use in research! We have setup a general class account, the details of which Karen Cranston should supply you with. You need to know the console URL, the username, and the password.

3 Task 2 - Connecting to Your Personal VM As part of the Nescent Academy course we were extremely fortunate to have received funding from Amazon, enough to provide each student with $100 of compute and storage resources. This should be plenty to cover the costs of the class. In this section of the tutorial we will log into the Amazon console web application, setup your account and VM instance, and login to kick the tires. The Rules While $100 is very generous we ask that each student adhere to the following guidelines to ensure we have enough resources for the duration of the workshop: 1. Please only launch a single VM instance of the type specified by the instructor at the beginning of the workshop and stop the instance at the end of the workshop. Do not leave the image running overnight or over a whole week you will quickly run out of funding. 2. Please name your VM instance as I do in my demo. Including your name will make it easy to find your resources in the list of class resources 3. Please do not delete EBS volumes or terminate VM instances that don't belong to you Step 1 Logging Into the Console In addition to being extremely comprehensive, the Amazon cloud has a very easy-to-use interface for interacting with all their cloud offerings. All you have to do is log into a web application and most of the functionality of the Amazon tools are available for you and very easy to use. Username and password were given in your welcome packet along with the URL (which is repeated below). Nescent Academy has its own Amazon account and each of the students have been setup as a sub-account using something called Identity Access Manager. The nice thing about this is you have pretty much free access to the console and we can have very fine grain control on what your sub-accounts can and cannot do. To get started go to the following URL and login with your username and password. After logging in you'll be presented with a wide range of options. Most likely you've logged into the EC2 Dashboard. If you haven't, please click the drop down menu at the top and then click on Amazon Elastic Compute Cloud (EC2). On this page you'll get a summary of the EC2 state for your account. You can see mine below.

4 Step 2 Launch the AMI From here we can create computers on Amazon's 'cloud'. What this means is that we can create as many computers as we like, start them, log-in to them, do some work, transfer data to/from them or destroy them altogether. Amazon worry about the hardware, power, cooling and maintenance all we need to do is specify how powerful a computer we want (micro, small, large or extra-large). Amazon charge for each Gb stored every month and for each hour a machine is run. This can vary from a few cents per hour to a few dollars. Whilst it is very convenient if you are only doing analyses occasionally, at the moment it is still cheaper to have your own computing (although you then have the headache of maintaining it!). The reason we are using the cloud here is that most of you will only be doing occasional analyses. In the case of high-throughput sequencing data (e.g. Illumina) you will find that your desktop PC may not be powerful enough to cope with the data. As such Amazon can offer a good alternative. It also means that you can start and stop your AMI from home and continue to work through the tutorial from there. Once logged into the console we can select an Amazon Machine Image (AMI) Virtual Machine and launch it. Click on the Launch Instance button in the center. Then select Launch Classic Wizard.

5 Now, make sure you click on the 'My AMIs' tab: The AMI is a snapshot of the VM we prepared for you for this class. To select the VM we have prepared for this workshop do this type the ID ami-02ce6b6b into the search box. This should come up with Nescent v4. Please do not use any other AMI for this workshop! Click Select. The next page is Instance Details where your first choice is the number and type of instances. For this class your default choice should be micro instance. Depending on how many CPUs and how much memory you need you can choose a different instance if you were running this outside the class. See for information about the instance types and for information about pricing per hour. Tip: The Availability Zone option can be any of the options, but you should stick to it for the class. If you create any S3 volumes (see the additional information in the supplementary tutorial) they are specific to a given availability zone so you'll need to have your instances and EBS volumes in the same zone.

6 Your next choice is Launch Instances or Request Spot Instances. The latter lets you bid on how much you want to spend on your EC2 node. This is a fantastic innovation, it works really well if you have tasks that need to get done but not urgently so you can afford to wait and, therefore, will only run your jobs during cheap off-peak hours. But, for this workshop we'll use the regular Launch Instances. Use a Large instance and pick either us-east-1a, us-east-1b or us-east-1c as the Availability Zone. Click Continue. You can use the defaults on the next page, click Continue again. On the next page you'll be given an opportunity to name your VM. Make sure you give this VM a name that includes your username so you can later find it in the list of instances. For example StudentKonrad since the username is StudentKonrad.

7 The next step lets you select the keypair used to let you log into this machine. This keypair is a file you download and supply to your SSH client in order to connect to the running server. By default most machines on Amazon don't use passwords as this can be a security risk. However, to make life easier, our machine does use passwords. As such you can click on 'Proceed without a key-pair' If you ever want to use a different machine image then you should either create a new key pair or use an existing one you have made. For the purposes of this workshop, you will not need it.

8 In the next step Configure Firewall select Choose one or more of your existing security groups. Select 'Default'. Select Continue. This will open the following ports: 22 (ssh) 80 (web server) Your instructor may ask you to alter these settings depending on the tools being used. The next step is just a review. If it looks OK hit Launch. At this point you can watch your instance boot under the Instances section in the EC2 tab.

9 At this point you wait just a couple minutes for the AMI instance to come online. Above you can see the instance is running, give it a couple minutes to finish its boot cycle. It's booting somewhere on a virtualized cluster node in Virginia! If you click on Instances on the left-hand panel you'll get a list of your running VMs (you'll see yours listed along with the other students, did you give it a name you can recognize?).

10 Once it turns green, you should click on the AMI and copy the Public DNS address to the clipboard (shown below is an example where the Public DNS is ec compute-1.amazonaws.com). We will need it in a moment. Step 3 Log into the Running VM's Remote Desktop with NX If you need a windowing environment rather than just an SSH terminal this is possible with CloudBioLinux! This is a great option if you want to use a GUI application (Graphical User Interface like Apollo or Artemis). It's very cool to see a remote desktop with Firefox and every other GUI application rendered quick and snappy over the Internet! Here are the steps to get remote NX login working, take a look at the document for more information. Note that these instructions will only work for this particular VM AMI. Many AMIs will not have the NX server installed and therefore you will not be able to connect using the NX-client. In these cases you will have to look at Step 4 and use SSH. 1 - Download an NX Client See for the various client installers. You should download the NX client specific to your machine (i.e. Windows, Linux or Mac). Make sure you download the NX Client and not the NX-server. Note that Mac OSX is incompatible with NX client 3.5. You should download NX client preview version 4 if you have Mac OSX.

11 2 - Launch the Client and Connect Once it is installed, start the NX Connection Wizard on your laptop. On Windows the start menu the icon will look like: At the next screen you should name the connection Nescent Academy and set the host-name to be the public DNS address of your Amazon AMI. Note, if you ever stop or restart your AMI, the Public DNS address will change. Make sure you update your NX client with the new DNS!

12 Click Next > and at the next screen ensure you change KDE to GNOME Click Next > and Next > again on the next screen. Double click on the NX Nescent Academy shortcut which will be on your Desktop. Finally enter the login username ubuntu and password ubuntu (without the quotation marks). Once connected you should get a nice desktop with links to the Terminal and to the Firefox web-browser. If a window pops up asking if you would like to run any updates, just close it. One final task is to change your password from ubuntu to something less easy to guess. Double-click on the 'Terminal' icon and type the command 'passwd' (without the quote marks). Enter 'ubuntu' as the current password (again without the quotes) and then type in a new password of your choice. Note that for security, no characters will appear as you type. Please make it difficult for someone to guess your new password! You will then need to re-enter it to confirm. Every time you log in via NX after this you will need to enter 'ubuntu' as the username and the password you have just entered. Make sure you remember it!

13 To close the connection just close the window, you can choose Disconnect if you want to log back in later and pick up where you left off. However, if you stop or restart the AMI via the console, any unsaved information will be lost. If you do stop or restart, remember to check and change the Public DNS address using the AWS console and change the DNS address in NX. Optional Step 4 Log into the Running VM via SSH Note this is for advanced users who may want to access the server via SSH using a key-pair. Do NOT try this during this tutorial. To connect over SSH you need to get the server IP address. Right click on the running instance and select connect. That gives you the command for Linux or Mac, open a terminal and cut and paste this command. Note, you should have the key file you downloaded present in the same directory that you execute the command from. For example, the command might look like the following and this assumes key- StudentKonrad.pem is in the same directory: $> ssh -i key-studentkonrad.pem ubuntu@ec compute-1.amazonaws.com Linux/Mac Tip: When you do the above command it may complain and say permissions are too loose on the.pem file. If this happens use chmod to make the file read/write only to you (it's supposed to be private): chmod a-rwx key-studentkonrad.pem

14 chmod u+rw key-studentkonrad.pem And try the SSH command again. (You ll learn exactly what these commands do during the next tutorial) Windows Tip: If you ever use a different AMI from the one used in this workshop, the chances are you will need an SSH client to connect to the instance. Mac and Linux have this built in, just open a terminal and you're ready to execute the command above. For Windows you should download the Putty program ( which gives you a very easy-to-use SSH program for Windows. When you launch this program look for the following setting, you'll need to provide the program with the path to your.pem file that you downloaded when launching your cluster node. See the Private key file for authentication option in the screenshot below. The username above is ubuntu and the server is ec compute-1.amazonaws.com. You'll use the server name that's assigned to the one you launched.

15 At this point you've logged into the machine. Other instructors will expect you to do these steps (launch a VM and connect via SSH) for their workshops. Step 5 Stopping and starting the Instance When you're done with the workshop it's very important to turn off the VM to avoid unnecessary charges. Just log back into the console, right click the instance and select Terminate. It will ask you to confirm. You can then watch the status change from shutting down to terminated. Note this will destroy all work done to date. If you wish to keep your data, use the 'Stop' option instead. In this state you will not be charged for computing time, but will still be charged for storage. Very important! If you stop and then start you instance your Public DNS address may change. If this happens you will need to use the new DNS address with the NX-client or SSH. Although much of what we have just done may not make much sense yet, most of you will feel totally comfortable and confident working on an EC2 node running Linux within a few hours. It's really amazing how quickly the fact that this is a remote computer will fade away. It may be hundreds of miles away but it will act just like a local computer, especially if you connect via NX. Move on to the Linux tutorial! 15

16 Additional Information Please do not attempt this part of the tutorial until you have completed the the Linux tutorial! Moving Data into and out of the Cloud We'll start with a comparison of S3 and EBS then move on to creating your own EBS volume which you can attach, format, then save data on. Get data into EC2 via scp (or Fugu for Mac WinSCP or something similar for Windows) Create an EBS volume, attach, format, and mount Get data into EC2 via wget from a public website Move data into and out of S3 Getting Data on the VM To get data onto the Virtual Machine you can post it to a web server then use wget or Firefox running on the VM over NX to pull the data down to one of the directories above. It s a little lowtech but it saves you having to mess around with scp. wget -O - -q google.com > index.html If you want to push data directly to the VM instance you can use scp (or a GUI tool such as fugu on the Mac or any windows program that supports transfers over ssh). For the GUI tools take a look at the tutorial located at: It covers the setup of WinSCP and Putty. If anyone has problem setting up these programs let me know and we can work out the issues after the workshop. The command line utility (run on your local machine) would look like the following, replace the server address with your running VM's address: scp -i key-<username> <localfile> ubuntu@ec compute- 1.amazonaws.com:/<remote_dir>/ 16

17 Creating an EBS Snapshot You will generally want to write files to your EBS volume so they will persist upon system termination. Think of the EBS volume as the USB flash drive of the cloud! This section of the tutorial was taken verbatim from (written by Rosangela Canino-Koning) since it provides an excellent tutorial on setting up an EBS. If you want to save your data across instances that is, if you want to have persistent data Amazon can do that for you, too. You need to use the Amazon Elastic Block Storage service, which creates a virtual hard drive that you can (virtually) attach to your EC2 instance. Note that EBS charges are based partly on how much disk space you use per month, rather than on connectivity or anything else, so even if you don t have an EC2 machine running, you ll be paying Amazon for data storage. Prerequisites This tutorial assumes you ve already set up an account on Amazon Web Services, and that you ve completed the EC2 tutorial to set up an Amazon instances. Ask Amazon to create a new Elastic Block Storage Volume for you At the AWS Management Console, on the left menu bar, click Volumes. Click Create Volume. 17

18 Enter the desired size, and select the zone in which your instance is running. The volume and instance must be in the same zone. Otherwise, the volume cannot be attached to your instance. Make sure you select the same availability zone as your running instance! Then click Create. Wait for your volume to finish being created, then click Attach Volume. 18

19 Select the desired running instance. You may leave the Device at the default: /dev/sdf. This is the name that your EC2 instance can use to connect to the disk space. If it gives you an error, just try another device name until you find one that's free. Make sure you remember the name. Click Attach. When attachment is complete, connect to your instances via SSH. If the volume is newly created, you must format the volume. WARNING: ONLY DO THIS ONCE, WHEN YOU FIRST CREATE THE VOLUME. OTHERWISE, YOU WILL LOSE ALL YOUR DATA. %% sudo mkfs -t ext3 /dev/sdf It will ask you if you want to use the entire device say y for yes. Then, mount the volume. You ll do this every time you attach the volume to an instance: %% sudo mkdir /working %% sudo mount /dev/sdf /working %% sudo chmod a+rwx /working Your drive is now ready to use it will be available under /working. Shutting down your instance Any volumes you have attached will automatically detach when you shut down the instance. You can also stop all processes that are using the volume, change out of the directory, and type %% sudo umount /working and then detach the volume via the AWS Web site. 19

20 Reading and Writing to S3 EBS volumes are great but 1) they can't be used across availability zones, 2) they are limited in size, and 3) they can't be used by multiple instances at the same time. The S3 services gets around these issues. It's basically a bucket services where you give a key (the bucket name plus file path) and byte data as a value. Like the EBS volumes you get charged for the space you use at a monthly rate. It's robust, available across availability zones, can handle many simultaneous connections, and provides higher limitations than EBS volumes: files can be up to 5TB you can have up to 100 buckets you can have an unlimited number of files in the bucket Creating the Bucket You can do this step either with a command line tool, API or you can use the console. Click on the S3 tab in the console and choose the Create Bucket button. Call it <username>bucket, so mine would be StudentKonradBucket. Then click Create. You should now see the bucket on the left. Uploading to the Bucket You can use the AWS console to upload to or download from the bucket, the Amazon Import/Export server to read or write data from hard drives into/out of S3, or you can use a program/api to read/write data. Here we'll use the very simple aws program ( to push and pull data into S3. To upload a file (test.txt) to my new bucket I would first setup my ~/.awssecret file to include two lines, first my access key ID and second my secret access key. For example, the file might look like: 1B5JYHPQCXW13GWKHAG2 2GAHKWG3+1wxcqyhpj5b1Ggqc0TIxj21DKkidjfz That file allows the aws command to connect to Amazon. Next, take my test.txt file and put it into my S3 bucket. %% aws put StudentKonradBucket/test.txt test.txt 20

21 You should now see the file in your bucket. Downloading from the Bucket Downloading is just as easy: %% aws get StudentKonradBucket/test.txt test.txt Remember, if you need to move large amounts of data into or out of S3 you should seriously consider using the Import/Export service rather than transferring data over the public Internet. Making an S3 Item Public You can share items in S3 with the public (or control their access very tightly for specific users, see the S3 documentation for details). To simply make something public, right click the item in the console and select Make Public. You can then see the public URL in the properties panel that can be shared with others. You can use this technique to serve entire websites directly out of S3 if you wanted! 21

22 The Analysis Lifecycle on Amazon's Cloud Putting together the examples and tutorials above a pattern for using Amazon's Cloud starts to emerge: Pick you AMI and instance type, guestimate your costs by using the Amazon calculator: Launch your instance Attach your EBS volumes, make some new ones, delete some old ones Log into your instance over ssh, install some software Copy data from a URL, scp, or S3 to your local instance EBS volume Run your analysis, get a coffee, check your result Write your results to an EBS volume or S3, share with the world via an EBS snapshot or public S3 bucket Turn off your instance, don't pay for time you don't use! Wash, rinse, repeat! HPC and Amazon So what we've done so far has been fun and there's a lot of potential for just starting up a compute node and, within a few minutes, getting work done. However, for large amounts of computation (think whole human genome sequencing) you're going to need more than a single compute node, no matter how powerful. There are a few really excellent tools available to you on Amazon, this section covers what they are and how you might use them. Cluster compute instances: nodes with very low latency, 10Gb interconnects that are wellsuited for ad hoc cluster creation and distributed computing. Take a look at the StarCluster project which will let you create SGE clusters on EC2 in just a few minutes: GPU compute instances: Amazon now offers GPU compute nodes if your algorithm can be written using OpenCL or CUDA Data Import/Export: this is a key for dealing with large (multi-terabyte) datasetsets such as NGS whole genomes. It allows you to prepare a drive and ship it off to Amazon. They import it into S3 and you can pull that data into your running instances. Files can be up to 5TB in size in S3. This is a great option since it's much faster to move large files around by FedEx than the public Internet! Programmatic APIs: everything I showed you can be done via a programmatic API in your favourite language. This means you can script cluster creation and teardown, data persistence and sharing, and pretty much any other feature in the Amazon Cloud. This functionality really takes the cloud to a whole new level and gives you massive amounts of control and flexibility to get your work done. This is really just the tip of the iceberg when it comes to cloud computing. There are many other more exotic features of Amazon's cloud that may be useful such as the Simple Queue Service (SQS), Elastic MapReduce, and SimpleDB to name just a few. Also, everything (and I mean everything) we covered in these tutorials can be scripted or interacted with programmatically. You 22

23 can launch instances, create EBS volumes, insert data in S3, and trigger exports to hard drives all via a series of web services operated by Amazon. Even better, whatever your favourite language, there will be an API specifically written for that language that exposes these services in an easy to use way. There are popular APIs in Java, PHP, Python, Ruby, Windows/.NET and even one for Android and ios cellphones! For HPC computing on EC2 I hope the last section gives you some ideas about how you can use the Amazon Cloud for research.