Launching and Connecting

Transcription

1 Module 7 Launching and Connecting Introduction In this module we are going to learn how to launch an instance and how to connect to it. We are going to discuss two ways to connect: (a) with the command line and (b) to a remote desktop, with the benefits of having a graphical interface. We will briefly take a look at how to run graphical applications via the command line (instead of the remote desktop), and we will also set up a web server to publish a very simple website. Finally, we will set up the different cloud storage types, so we can access the storage remotely and directly from our instance. Videos The following videos go through most of the content in this module and offer a less indepth description of the subject than the documentation does QNQsZ4 General notes What is an instance? In Module 5 we have already discussed in more detail what an instance is, but let s do a little recap at this point. In the NeCTAR Research Cloud, an instance is a virtual machine. Instances running inside the Research Cloud are just like real life computers, except that they run remotely, in the Cloud. Here and when learning about them on the web, you will often find them referred to as instances.

2 Instance vs local computer In this document we will often talk about the local computer as opposed to the instance. The local computer is the one you are probably sitting in front of right now: It is your home or office computer that you use to connect to the instance. The instance is your virtual machine that is running in the NeCTAR cloud. You may be familiar with connecting to remote computers already. You may have connected to a remote desktop, or used a command line. The same tools and techniques apply when connecting to running instances. Your instance has a public IP address and can be reached and controlled with any remote access tools you wish to use, after you have set it up for remote access. For example, you can use an SSH terminal, or alternatively connect to your remote desktop. In this module we will create and launch an instance (a virtual machine, short: VM) and go through the two options on how you can connect to your instance (terminal and remote desktop). Use of Terminals Throughout the exercises, we will use a Terminal to type commands. To prepare, make sure you know how to open your Terminal by following instructions below for your Operating System. On Windows, you will use two terminals: The PuTTY terminal to connect to and type commands for the instance. You will need to download PuTTY: You will need the files PuTTYgen.exe and the file Putty.exe from the official PuTTY website. This Terminal will be referred to as the ssh terminal. The Windows command line on your computer. You can open it by typing cmd into the search field (on Windows < 8 on Start ; on Windows 8 in the App search field). This Terminal will be referred to as your local Terminal. On Mac and Linux, there are built in terminals available. You can access them as follows: On Linux Systems, depending on your desktop environment, opening a terminal is usually achieved by clicking on the terminal symbol, or by right click on the desktop and select Open Terminal. On a Mac, you can open a terminal as follows: 1. Bring up Spotlight Search by holding down the Command and Spacebar

3 keys simultaneously. 2. In the search box that appears, type Terminal and press the Return key. Alternatively, in the Finder, go to Applications > Utilities and you will find the Terminal app there. Conventions The notation throughout the training documents can be interpreted as follows: Words in italics are used for names and terminology, e.g. name of a software, or name of a computing concept. It may also just emphasise a word in the traditional way. Quotations are also written in italics and are put in between quotatioin marks. Words in bold are used to highlight words which identify important concepts of a paragraph, to make it easier for users to skim through the text to find a paragraph which explains a certain idea, concept or technology. Additional information which is optional to read is displayed in info boxes like this one. Important information is displayed in boxes like this one. Definition of terms are displayed in boxes of this style. Possibly specific prerequisites for reading a particular section are contained in this type of box at the beginning of a section. The use of command line tools is part of this course. In a Terminal, you will be directed to type in commands. Commands are formatted as follows: command-name argument1 argument2... argumentn Throughout the exercises, when you see a word in pointed brackets, like <this word>, it means that you have to replace everything inside the brackets, and including the brackets, with whatever is described within the brackets.

4 For example, if you are prompted to run a command named the command which takes two arguments: the-command -f <yourfile> Then you have to replace the second argument, <yourfile>, with the file name that is being referenced in the exercise. For example the-command -f thefile.txt When editing a file, the contents of it will be displayed in a different font and with background colour as follows: The content of the file The next line in this file Output on the command line terminal is printed in boxes formatted as follows: NectarInstance:~ Jennifer$ whoami Jennifer Create a ssh key First, we will need to create a ssh key pair for connecting to our instances. An ssh key pair is required to encrypt the data which is transferred to communicate with the instance. The pair consists of two keys: A private key, and a public key. The public key is called public because that s the one which is placed on the server and which you may upload online without worrying about security issues. It is useless without its counterpart, the private key. The private key is the secret key which you keep on your local computer so that you may connect to the instance. You should keep the private key very safe and never share it with anyone. Take care not to lose it, otherwise you won t be able to connect to your machine. The connection to the machine only works with both keys.

5 The image above shows two computers connected through their ports, with an ssh server and client encrypting the connection. Applications can use this secure ssh connection to exchange data. We will first have to generate this key pair before we can proceed with launching an instance. The ssh keys will have a name. You should choose a name that you can associate with your instance, for example Nectar_Key. As a general rule of thumb, file names should consist of alphanumeric characters and underscores ( _ ) only. You should avoid white spaces. They makes working with files on Unix systems a bit clumsy. Note: The process differs between Unix/Mac Systems and Windows. Follow the steps described below for your platform, and then continue to the next section. Mac and Linux SSH key pairs can be generated with the ssh keygen command. If you choose Nectar_Key as your key name, you will have two files after the generation of keys: The private key, called Nectar_Key and the public key, called Nectar_Key.pub. 1. Open your Unix or Mac Terminal. You will start off in your home directory. If you already had the terminal open before, make sure you are in the home directory, by simply typing cd Now, check if you already have a directory called.ssh, by typing ls -a A list of files in your home directory will be printed. If you don t see the name.ssh in this list, create the directory (if you see the directory, skip this command): mkdir.ssh and change to this directory: cd.ssh Now generate the key with ssh-keygen -t rsa -f Nectar_Key This generates a pair of keys in the directory.ssh, which is the default location for ssh keys. You will be asked to specify a passphrase. Choose a secure passphrase! But make sure that you store your password somewhere safe you will not be able to connect to your instance if you forget the password! 2. You will have generated (1) a key you keep private, named Nectar_Key and (2) a

6 public key, named Nectar_Key.pub. If you type ls Then you should see both files listed. 3. Open the public key file with your favourite text editor. If you don t have a favourite editor, here are some suggestions: On a Mac, type open -e Nectar_Key.pub which should open the key in TextEdit. On a Linux gnome environment (e.g. Unity), type gedit Nectar_Key or if you are using KDE, type kate Nectar_Key If you are using another desktop environment we assume you are familiar with your favourite text editor. 4. Copy all the text which is contained in the file Nectar_Key.pub into the clipboard. With CTRL+A you can mark all text, and with CTRL+C copy it (on a Mac it is Command ( ) instead of CTRL). You will need the contents of the public key on the keyboard in one of the next steps. Windows Open the PuTTYgen program by clicking on PuTTYgen.exe which you downloaded in the beginning of this tutorial. 1. For Type of key to generate, select SSH 2 RSA. 2. Click on Generate. 3. Move your mouse in the area below the progress bar. When the progress bar is full, PuTTYgen generates your key pair. 1. Type a passphrase in the Key passphrase field. Choose a secure passphrase! Type the same passphrase in the Confirm passphrase field.

7 2. Right click in the text field Public key for pasting into OpenSSH authorized_keys file and choose Select All. Right click again in the same text field and choose Copy. This copies the content onto the clipboard, we will need to paste it somewhere else in one of the next steps. 3. Click the Save private key button to save the private key. It is important that you save the key, because you will need the file later. So save it in a folder where you can find it again. In this exercise, we will name the file Nectar_Key.ppk. 4. Save your public key as well by clicking the Save public key button. As a file name, type Nectar_Key.pub.txt. Launching an instance We will now create an instance and choose the resources we require. For this tutorial, we don t need a lot of resources. You may require more for your research. For more details about how to request an allocation, see Module 5. Security note Before we start, a brief note about security: It is important that we make our instance secure from the very moment we create it. In order to enable the right security settings, we will select the instance to be part of one or several security groups. A Security Group opens up access to the instance from outside. Essentially, a security group is a collection of firewall rules, which open up different Ports to access the machine from outside. Module 5 has already introduced the concept of Ports. Now we need to free up some Ports in order to be able to communicate with our instance. For example, we will first connect to our instance with a SSH shell from our terminal (Reminder: SSH encrypts our communication with the server). SSH uses Port number 22, so this has to be freed up.

8 Note: All the pre defined security groups mentioned in the instructions are available in Project Trials only. To learn how to create your own security group (or re create the trial groups), refer to the Appendix of this module. The following steps will explain how to create an instance and how to make sure it has been assigned to the security groups we need to connect to it. Step 1: Logging on First, we will have to log on to the Dashboard, which is our interface for managing our virtual machines. The Dashboard can be accessed with any browser, e.g. Firefox, Safari or Windows Internet Explorer. Open your browser of choice (we suggest Firefox), and go to: Click Log in on the first window. Then, select your organisation from the dropdown menu (you may also type in the first characters of your organisation name in order to find it quicker). You will be redirected to a page where you can type in your username and password. This should in most cases be the username and password you usually use to access your account at your University or Research Organisation. Click OK for all terms and conditions if logging in the first time (you will not have to do this every time). Step 2: Open the Launch Dialogue Go to Dashboard > Instances. A list of your running instances will be listed. If this is the first time you are doing this, the list will be empty. Now click on Launch instance on the top right of the overview.

9 Step 3: Selecting resources Choose the name of your instance and the flavor. Different flavors allocate different amounts of memory and disk space for you. Select the flavors in the dropdown box and information about the resources are displayed on the right side of the window. You may select a few flavors to see the differences between the flavors. For this tutorial, choose the flavor m1.small, which offers 30GB of ephemeral storage (we will need this in a later section. You could launch several instances of the same flavor, but for now we will launch only 1, so leave Instance Count at 1. Leave Instance Boot Source on Boot from Image. Choose the most recent version of NeCTAR Ubuntu from the Image name list. NOTE: At the time of writing the image NeCTAR Ubuntu (Utopic) amd64 was used. If you choose a version much newer than this, there may be some differences to the descriptions given in this tutorial. In general, a newer version should work fine with the exercises below, you may only see different text (phrasing) in the terminals command line output. Don t click Launch just yet we have to set the security settings first!

10 Step 4: Ensuring security Now, we will choose the security settings. Go to the tab Access & Security. We can choose which Security Groups the instance will belong to (see above for explanation about security groups). We will connect to the instance with ssh. Ssh uses port number 22, so this has to be freed up. Therefore, tick the ssh box. We will also publish a little website in a later exercise, which requires http, so tick the http box also. To test whether your instance is reachable, we will also need icmp. Note: We can also change the security groups of the instance later on, but it s convenient to do this now. Next, we will specify the key for connecting to this instance (that s the one we just generated). 1. You should have the contents of your public key still in your clipboard, unless you have copied and pasted something else since we did this above. If you have, copy the contents of Nectar_Key.pub again. 2. Click on the plus ( + ) to add a new key. 3. Specify the Key Pair Name. This can be any name, but for simplicity choose Nectar_Key also. 4. Paste the contents into the large text box labelled Public Key (use CTRL+V, or right click in the box and Paste ). 5. Now, click Import key pair. This key will now be available for future use. Next time you launch an instance you can select this key from the drop down list directly, without having to copy and paste it again. The window closes and you are back to the window to configure your instance. Step 5 (optional): Choose your Availability zone

11 If you need access to specific facilities (e.g. Volume storage), you can manually choose the location of your instance in the Availability Zone tab. The Availability Zone is where your instance will be running physically. Unless you know that you need access to specific services, it is recommended to leave this in the default setting, because then the location with the most available resources is chosen. Step 5: Launching the instance You can now launch the instance by clicking on the Launch button. You will get back to the overview of your instances. You will see the status of your instance in the overview. You need to wait a few minutes. The Instance will normally pass from Status Build to Active. You will see an IP address for your instance (4 numbers separated by a dot). Write it down, or keep this window open so you can see it, as we will need it in the next step to connect to your instance. Congratulations!! Your instance is now reachable on this IP address. Your applications can connect to the Ports that you have allowed earlier in the Security Groups. Connect to the instance To connect to your instance, you will need its IP address. You should have just written this down, or you can look it up on the Dashboard (Compute > Instances). In the following text, your IP address is described as NNN.NNN.NNN.NNN, which you will have to replace with your own instance s IP address. To get a secure connection, we need to log on to the instance via ssh. Connect via ssh Step 1: Test connectivity First, let s do a little test to see if your instance is reachable. On Linux/Mac, you can use the command line shell that you have just opened. On Windows, open your Windows command line.

12 Use the terminal to try to ping your instance (checking that you can reach it): ping NNN.NNN.NNN.NNN replacing the N s with the IP address of your NeCTAR instance you noted above. The ping command just sends a simple signal to your instance and checks whether it responds. If the ping does not work, then most likely neither will any of the following steps. The ping has worked if you get a response every few seconds, e.g. PING ( ) 56(84) bytes of data. 64 bytes from : icmp_seq=1 ttl=49 time=354 ms 64 bytes from : icmp_seq=2 ttl=49 time=358 ms Step 2: Logging in We will now log in to our instance using ssh with the private key (the username might be different depending on the image you launched). Follow the instructions below for the operating system of your local computer. Note: If the ping has worked but the SSH connection does not work after following instructions below, follow the troubleshooting steps in the Appendix. Linux/Mac We will need a command line shell (a Terminal ) in this exercise. On Linux and Mac, you have a built in terminal. Open it now (if you are unsure how, refer to the notes in the introduction). Go to your command line and type ssh -i Nectar_Key [email protected] using your own IP address instead of NNN.NNN.NNN.NNN. The command line will ask you to establish the authenticity of the server. Type yes. You will then be asked for your passphrase (that s the one you chose when generating the SSH Key). Note: If you named your key other than Nectar_Key, you should replace this with your own private key file name here. If you saved the private key in another folder, you will have to specify the full path to it (e.g. ssh i /home/john/yourkeyfolder/nectar_key NNN.NNN.NNN.NNN). On a Mac:

13 You may have to specify the full path to the key. If the above does not work, try instead: ssh -i $HOME/.ssh/Nectar_Key Our instance has a default username that we need to specify when we log in. In this tutorial, we chose a Ubuntu Image when launching the instance. At the time of writing, the user name for Ubuntu is ubuntu. Other NeCTAR images may expect different usernames. Try using root instead of ubuntu ( [email protected] ) for other Operating Systems. For Fedora, the user name may be ec2 user. The right user name for the image you chose is documented at the NeCTAR Image_Catalog. Well done! Now, you should be connected to your instance and your ssh terminal command line is ready to use. Windows On a windows system, we will use the PuTTY terminal to connect to the instance. We first have to set up our PuTTY session to include the details of our instance, including the keys we are using. Double click on the file PuTTY.exe which you downloaded when you created the SSH key. In your PuTTY Window, type your instance s IP address into the Host field (you should have taken a note about the IP address earlier, if not you can find it on the Dashboard: Go to Compute > Instances).

14 Then, select the Data category in the navigation window. Here, we will have to specify the username. Choose ubuntu as your username. Leave the other fields on default settings. Our instance has a default username that we need to specify when we log in. In this tutorial, we chose a Ubuntu Image when launching the instance. At the time of writing, the user name for Ubuntu is ubuntu. Other NeCTAR images may expect different usernames. Try using root instead of ubuntu for other Operating Systems. For Fedora, the user name may be ec2 user. The right user name for the image you chose is documented at the NeCTAR Image_Catalog. Next, click on the SSH > Auth sub category. There is a field labelled Private key file for authentication. Click on the Browse button:

15 In the browse window that comes up, select the private SSH key file that you saved on your computer earlier (we named it Nectar_Key) Note: If you have generated your ssh key on a Linux system and now need to convert it to Windows, see how to do this in the Appendix. Instead of converting keys, you can also simply create a separate set of keys to use with PuTTY. In this case, you have to add the new key as an authorized key on the instance. See the Appendix for instructions on how to do this. Now, return to the Session screen in the left navigation window. This is the screen we started at. We will save our settings (the session ), so that in future we don t need to type in all the details again. In the Saved Sessions section, choose any name for this session so that you can remember it, for example MyNeCTARInstance. Then, click on the Save button.

16 Now, connect to the instance by clicking the Open button. The first time that you connect, you will be asked to verify the identity of the remote server. This is expected the first time you connect to a new server, so you can select Yes to continue. You will be prompted for a passphrase. Type in the passphrase that you used to create the SSH key earlier on. Every time you connect to the instance via Putty, you will have to type in your passphrase. This may get a little cumbersome, so note that there is a tool called Pageant that you can get on the same page where you downloaded PuTTY. Pageant is an SSH authentication agent. It holds your private keys in memory, already decoded, so that you can use them often without needing to type the passphrase every time. It also eliminates the need to explicitly specify the relevant key to each user account, if you use more than one account to log into a server. We will not use pageant in this tutorial, but you may be interested to read up on how to use it. You can find instruction in the Appendix. Well done! Now, a black Terminal window should open and your command line is ready to use. Configuration and control Welcome to your new virtual machine! First of all, we want to make sure our system is updated. On Debian and its derivatives, like Ubuntu, you can update your packages with the commands: sudo apt-get update sudo apt-get upgrade The first command refreshes the package information, and the second upgrades your system. There are several ways to upgrade your system. The command above (sudo apt get upgrade) simply updates the existing packages. You can also update more intelligently. Here s a description of the various upgrade commands (extracts from the apt get manual): apt get upgrade: upgrade is used to install the newest versions of all

17 packages currently installed on the system [..]. Packages currently installed with new versions available are retrieved and upgraded; under no circumstances are currently installed packages removed, or packages not already installed retrieved and installed. New versions of currently installed packages that cannot be upgraded without changing the install status of another package will be left at their current version. apt get dist upgrade: dist upgrade in addition to performing the function of upgrade, also intelligently handles changing dependencies with new versions of packages; apt get has a smart conflict resolution system, and it will attempt to upgrade the most important packages at the expense of less important ones if necessary. So, dist upgrade command may remove some packages. apt get full upgrade: full upgrade performs the function of upgrade but may also remove installed packages if that is required in order to resolve a package conflict. You may also want to change the password for the user ubuntu: passwd Note: You will not be prompted for your ubuntu user password when you connect through ssh. This is because in the default configuration, the ssh key is regarded sufficient for authentication purposes. Therefore, each user on the virtual machine should have their own ssh key. More importantly, you should change the password for root access (administrator rights), especially if you are planning to add more users to the virtual machine: sudo passwd root and type in the new password twice. If you do not change the root password, and later add other users to the VM, any user will be able to log in as root and perform administrator tasks (using the command su ). When you create new user accounts, and you do not want them to have root access, make sure you do not include them in the user group sudo. Any user which is member of this group can execute administration tasks.

18 Having problems connecting? The Appendix includes a section on troubleshooting. Using a command line text file editor Sometimes, we will have to edit files on the instance via the command line. Because this is a command line, we cannot use programs like Word or inote to edit a file. Instead, we have to use a command line editor. There is a large number of command line editors available. A simple and easy to use editor is nano. Vim is also popular but a bit more difficult to use for beginners. If you want to edit a file, type nano <path-to-your-filename> The terminal changes into a simple editor. You can edit the file and then exit the editor with Ctrl+X. A prompt will come up asking if you want to save the file, type Y to confirm this. We will create a file to test this. Type nano MyFile.txt and hit enter to confirm the command. Write a little story or anything you are just thinking of into the file. Then, hit Ctrl+X to exit, and confirm with Y that you want to save the file. Now, type ls and hit enter to confirm the command. You will see the list of files that are there. The list should contain a file MyFile.txt. You may print the contents of the text file onto the terminal by typing: cat MyFile.txt and you should be able to read your message in the terminal. Exit again with Ctrl+X. Restarting your instance Sometimes, you may want to restart your instance. For example, to make sure everything works correctly after you installed some packages. You can do this in the Dashboard. Step 1. Reboot on the Dashboard. Go to the Dashboard and click on Instances on the navigation panel on the left to see

19 your instance. On the right side of your instance, you can open up a drop down box. click on Soft Reboot Instance (a soft reboot is what we know as a reboot. A hard reboot forces the reboot in case the soft reboot does not react). Step 2. Re connect ssh client. Your ssh session, or the PuTTY session if you are on Windows, will have disconnected now, so you have to re connect. On Windows PuTTY, you should have your login session still saved so you just need to load these settings up. On Mac and Linux, simply re run the ssh command (Tip: you can browse the command line history with the up/down arrow keys in your ssh terminal. Your ssh command should be in there so you don t have to type it again). Tip for Unix/Mac You can make your life a bit easier by saving your configuration for the ssh connection. This will avoid always having to type your IP address. There is two options to do this: a) specifying all details or b) creating an alias for your IP address. Then, instead of typing in the full ssh command every time, you will just do the following: Option 1: ssh MyNectarInstance Option 2: ssh -i Nectar_Key ubuntu@mynectarinstance Option 1 You need to edit a file on your local computer (not on the instance!). Open a new terminal on your computer. Then, open up the file ~/.ssh/config using your favourite editor, for example:

20 nano ~/.ssh/config and add the following contents: Host MyNectarInstance HostName NNN.NNN.NNN.NNN User ubuntu IdentityFile ~/.ssh/nectar_key Again, replacing the N s with your IP address. You can choose any other alias name instead of MyNectarInstance. Save the file and close it. From now on, you can connect with the command: ssh MyNectarInstance or, if you chose another name, replace MyNectarInstance with it. Option 2 You may also just alias your IP address in the system s host file. On Unix systems and on Mac 10.2 and newer, the file to edit is usually the file /etc/hosts. On Mac 9 and earlier you can find it in the Preferences or Systems folder. Open this file in your favourite editor, for example: sudo nano /etc/hosts and add the following line: NNN.NNN.NNN.NNN MyNectarInstance replacing the N s by the IP address of your instance. You may also choose another name instead of MyNectarInstance. Next time you ssh into the instance, you can connect with: ssh -i <path-to-your-keyfile> ubuntu@mynectarinstance Note this is the same command we used earlier, but you can type MyNectarInstance instead of your IP address. You will be asked to accept the authenticity of the host again. Answer with yes. Connect to a remote desktop Using the command line may not feel very intuitive, especially for beginners. Aside

21 from that, sometimes we need to run software that has a graphical interface. As an alternative to connecting with a ssh terminal, we can also connect to the desktop of our instance and get a graphical interface as we know it from our own computers. A remote desktop login allows users to connect to their desktop while being physically away from their computer. You can also connect to your own home computer s desktop while you are away, if you set up your home computer accordingly with remote access. This may be a bit difficult to do from your organisation or from your home, as you need access to the computer via the internet from outside, which may require the permission of firewalls used on the premises, and you need a fixed IP address. However on your NeCTAR instance, it is easy to set up a remote desktop, because the instance is already accessible from outside. Some remote desktop solutions allow only one user to see the desktop at the same time. Other software also allows desktop sharing in which several users can see a desktop simultaneously. Such a shared desktop can be used as a platform for real time collaboration. This mostly requires a separate setup, which is not part of this tutorial. To connect to a remote desktop, the remote machine (your instance) has to have a remote desktop server installed and running. On your home or office computer, you have to install a remote desktop client (sometimes called viewer ) which connects to the remote desktop server on your instance. The client then displays the desktop of your remote machine. It is like a window into your instances desktop; The desktop environment itself is running on the instance. Terminals on remote desktop vs SSH terminal We have previously connected to the instance with via a ssh terminal. When we get a window into the desktop of our instance, we can open a terminal on the desktop as well. The image below shows both (1) a ssh terminal (left) and (2) a desktop with an open terminal (right):

22 So what is the difference between the two terminals? When you type commands in either of them, it will have the same effect. They are both terminals that you can use to control your instance. However, the difference is that 1. The ssh terminal is set up through a secure ssh communication line with your instance. If we use a ssh terminal, only text is being transferred (the commands you type and the responses you get). This is very fast, but you cannot run graphical applications, because you can only communicate with the instance via text. Note: it actually is possible to run graphical applications via the terminal as well, with some disadvantages. This is discussed later in this module, but this is very slow and usually not recommended. 2. The terminal on the remote desktop is connected securely as well (at least the way we will set it up in this tutorial). But instead, the communication includes graphical data and events (e.g. your mouse clicks, and the windows displayed). This is slower than using the terminal only, but you may find it is still fast enough to work with. You can run graphical applications and use the OS menu to find programs and change settings, which may be easier for many of us. Remote desktop protocols When you are connected to a remote desktop, everything you do to interact with the system, for example clicking the mouse, moving a window or typing on your keyboard, is communicated to the remote machine (in this case, your instance). For this kind of communication, computers use protocols which are designed for this purpose. The type of protocol determines properties of the remote desktop, for example how secure it is, how fast it works and what kind of desktop environments it supports. In

23 this tutorial we will use two different protocols to connect to our remote desktop: NX and VNC. Some wide spread protocols include: RDP (Remote Desktop Protocol) Originally developed by Microsoft to connect to Windows machines. The protocol is not very secure, therefore setting up tunneling through a secure connection is required. NX NX compresses the data to minimize the amount of information transmitted. NX also caches data to make the session as responsive as possible. For example the first time a menu is opened it may take a few seconds, but on each subsequent opening the menu will appear almost instantly. Therefore, NX allows fast speed even with slow internet connections. NX uses the SSH protocol to send its data and is therefore secure. VNC (Virtual Network Computing) The VNC Protocol (called the RFB protocol) is very simple. It is based on graphic primitives from server to client ( Put a rectangle of pixel data at the specified X,Y position ) and event messages from client to server. This makes VNC a bit slow. By default, this is no secure protocol. Therefore, you need to tunnel the connection via a secure SSH connection to make it secure. This may seem a bit complicated to achieve for beginners. Alternatively, some clients like UltraVNC) support the use of an encryption plugin which encrypts the entire VNC session. Because VNC is quite popular, the client software options are pretty good for Unix and Windows platforms. VNC allows multiple users to see (share) the same desktop. To transfer files, printing or audio requires separate services. The VNC only handles displays and inputs, such as interaction with keyboard and mouse. Users may access desktops with VNC through their Web browser, without needing to install separate viewer applications. VNC displays a desktop within a browser window by running a Java applet, and this facility should work on any system that has a JVM installed. VNC does not support the newer 3D desktops (e.g. the newer GNOME/Unity or KDE desktops environments. The desktop environments have to be run in classical 2D mode for it to work (e.g. GNOME Flashback).

24 To learn more about VNC, here is a good article Free remote desktop client and server software includes: Vnc4server the main VNC server software for Linux. UltraVNC, RealVNC, TightVNC popular clients for VNC sessions X2Go An open source product which is pretty fast. It uses the NX protocol and is operates through a secure SSH connection Supports remote audio, printing, clipboard, file sharing etc. Can share a user s session remotely, but it has to be set up separately. It is fairly easy to set up There is even a readily available browser plugin to make remotely controlling another PC possible from a web browser. But: Port forwarding and Dynamic DNS are recommended if you re using this over the Web. Due to the use of older NX libraries, it has problems with the newer 3D Desktops, e.g. Gnome/Unity and newer KDE desktop environments. Only the older environments are supported. No Android / ios client available, unless using the web interface NX Server, free edition Easy to set up Not very fast Only free to 3 users Can t share a user s session remotely. FreeNX Server A Server for the NX protocol, similar to NX Server, but free. No real good documentation, not easy to install Can t share a user s session remotely. NoMachine) A client to connect to an NX server (e.g. FreeNX). Not free for commercial use (only private) Simultaneous connections from only 2 users. NeatX (outdated) Google has released NeatX, a complete independent implementation of the NX protocol. Neatx was developed as part of an internal project which has now finished. It had no releases and is not being actively developed.

25 Can t share a user s session remotely. xrdp uses the RDP protocol (which is not very secure) Servers are also available for Unix and Mac. Commercial products may be better and easier to set up than the free software listed above, but they come at a cost (at least for commercial use). Popular software includes: TeamViewer supports Windows, OS X, Linux, Android, and ios free for personal use setup very easy pretty expensive for commercial / business use Splashtop supports Windows, OS X, Linux, Android, and ios allows you to stream audio and video across computers with minimal latency very easy to use and set up RealVNC The Official VNC software Offers its client and server apps for Windows, OS X, Linux, Android, ios, Chrome free for basic individual private use, but added features and support come at an extra cost. Wikipedia offers a good comparison of remote desktop software. The options which are easiest to set up and provide a fairly good user experience are X2GO (using the NX protocol) and VNC. In the following exercises, we will see how we can set up both of these options using freely available software. VNC vs X2Go X2Go uses the NX protocol, and VNC the RFB protocol. This makes for different properties and advantages of each approach. The VNC protocol is pixel based. This can make it a bit slow. One advantage is that it is widely used, and several available remote desktop products use this network protocol. This means that you may use any VNC viewer (client) to connect to a system that offers remote access with the VNC protocol.

26 The NX protocol (used by X2Go) has a better understanding of the underlying graphic layout than VNC (which is pixel based). Therefore, it is often more efficient (faster). In summary: VNC: inherent support of different platforms: you can install it on Unix and Windows servers is widely used, hence there is a large number of tools is slow is inherently insecure, it takes a bit more work to set up a secure connection NX (X2Go): is fast easily integrates with security plugins there are not many well maintained servers and clients available (X2Go is one of the few) Servers can only be installed on Unix servers, for Windows/Mac there are only clients (no servers) On Mac computers, installation of the X2Go client is a bit more work than on Windows and Linux, and usage can also be a little slower. Install a desktop environment First of all, for either of the options (VNC or X2Go), we need to install a desktop environment on our instance (Remember: The desktop actually runs on the instance, you only get an interactive window to it on your local computer. So we need to install a desktop on your instance). A desktop environment is what we use to graphically interact with a system (it provides the desktop, windows that open up and symbols you can click on to start a program). Windows and Mac OS have only one desktop environment, but in most Unix based systems you can choose from several environments. Popular choices are Unity, Gnome and KDE. There are also popular simple and light weight environments called LXDE, XFCE, Mate and more. You may install several desktop environments, but you only need to choose one to start with. The following desktop environments are being discussed in this section:

27 LXDE XFCE Gnome KDE If you are unsure which desktop environment to choose, we recommend you start with LXDE. Unfortunately, there is no remote desktop solution that supports all desktop environments. The desktop environments which are currently not supported for remote connections are the ones using 3D desktop acceleration. This affects e.g. Unity and Gnome3. All traditional remote desktop approaches (VNC, NX etc.) fail to work with this. Some versions of Gnome can be run in flashback Mode (with the 3D acceleration switched off), but that can be a bit tricky to set up and does not necessarily work. We will therefore discuss only the desktop environments which do work. Note: This tutorial is assuming you are using Ubuntu (UTopic), however later versions may also work with the same instructions. When in doubt, links to external documentation are given at the relevant places. LXDE This light weight environment is easy to install and works well for remote connections. Install the LXDE environment on your instance by typing into your ssh terminal: sudo apt-get install lxde

28 XFCE Install the XFCE environment by typing into your ssh terminal sudo apt-get install xfce4 GNOME Gnome will have to be run in Flashback mode to disable the 3D acceleration which is not supported in VNC or X2Go (see for example this official documentation and this one on ubuntuusers.de). X2Go is only compatible with Gnome versions earlier than 3.12 (see x2go documentation), and ideally earlier than The Ubuntu version we use in this tutorial has Gnome , so you cannot use Gnome with X2Go with the Ubuntu version used in this tutorial. However, you may still get Gnome working with VNC by following the instructions below. You can find out your Gnome version (after you installed it with following instructions) by typing in your ssh command line: gnome-shell --version If you are using a version older than 3.12, you cannot use Gnome with X2Go (you may run into trouble with versions older than 3.08 too). However, you may still get it working with VNC. To install gnome in flashback mode, follow these instructions: 1. Install the gnome environment by typing into your ssh terminal: sudo apt-get install gnome-core Alternatively, if you want to have some apps like Gimp, LibreOffice etc. preinstalled, instead of the command above, type: sudo apt-get install gnome (this will take longer as more packages are downloaded). Confirm the installation of the package with Y.

29 Hit [Enter] to confirm the message that comes up. Then, you have to select the default display manager. It does not really matter which one you choose. The correct display manager will be started by your remote client anyway. LightDM or GDM are display managers for Gnome. If unsure, just select LightDM. 2. To switch on flashback mode, we have to install a package and disable the newer scrollbars. Type: sudo apt-get install gnome-session-flashback 3. [This step is not necessary on the Ubuntu version used in this tutorial] sudo apt-get purge overlay-scrollbar gsettings set org.gnome.desktop.interface ubuntu-overlay-scrollbars false 4. [This step is not necessary on the Ubuntu version in this tutorial] You might also need the following package on older Ubuntu versions (if in doubt, install it anyway, it should already be installed and then it does nothing) sudo apt-get install xfonts-base KDE Note: Only versions earlier than KDE 5 are supported in X2Go. With the ubuntu version used in this course, you may use KDE. You can find out your version of KDE (after you installed it with following instructions) with: kde4-config -v Install the desktop environment by typing into your ssh terminal: sudo apt-get install kde-plasma-desktop Alternatively, if you want to have some more apps pre installed (this will take longer and take up more space), instead of the command above, type: sudo apt-get install kde-full Confirm the installation of the package with Y. This may take a while as a lot of packages need to be downloaded. Hit [Enter] to confirm the message about installing KDE. Then, you have to select the default display manager. It does not really matter which one you choose if there are several options. The correct display manager will be started by your remote destkop

30 client anyway. KDM is the display manager for KDE, so if unsure, just select this one. Other desktop environments There are also other desktop environments, for example MATE, which we will not discuss further in this tutorial. Connect with X2Go We will first install the X2Go Server on your instance, and afterwards the X2Go client on your local computer. Then you can access the virtual machine s desktop with the X2Go client. Server installation In this tutorial we are using Ubuntu Version on our instance and the following instructions are going to assume this. If you are using an earlier version of Ubuntu, or another Linux distribution, you may need to do other steps. This will be commented upon in the following instructions at the relevant places. See the official X2Go documentation for details on the steps to perform. 1. Add the X2Go repository and refresh the software package system by typing into your ssh terminal: sudo add-apt-repository ppa:x2go/stable and confirm this with [Enter]. Then, type: sudo apt-get update

31 2. Install the X2Go server. Type into your ssh terminal: sudo apt-get install x2goserver x2goserver-xsession Confirm the question on whether to use up the said disk space with Y 3. We now have to install extra bindings for our desktop environment which add extra features for X2Go (e.g. file sharing). Follow the instructions for the environment you want to use (if it is not listed, see official documentation) LXFE: In your ssh terminal, type sudo aptitude install lxde sudo apt-get install x2golxdebindings Note: If the installer package complains about unmet dependencies and suggests to remove lxde, say yes to this. Afterwards, re install lxde again with the command GNOME: GNOME only works for versions earlier than gnome 3.10, better earlier than gnome 3.08, as pointed out earlier). In your ssh terminal, type: apt-get install x2gognomebindings KDE: In your ssh terminal type: sudo apt-get install plasma-widget-x2go 4. The X2Go server should now be running. You can check this with typing into your ssh terminal: service --status-all grep x2go The result should show a line like this: [ + ] x2goserver If the service is running. If it isn t, you can start the server by typing: service x2goserver start If in doubt, or anything is not working, try restarting your instance before you try anything else (we have already restarted the instance here). Client installation You will now install the X2Go client on your local computer. Depending on which OS you are using, instructions will differ. Go to the instructions below for your OS. If you have a platform not listed here, you can find additional documentation on how to install the client on the official X2Go website. A good instruction can also be found here. Ubuntu: Install the client by typing into a terminal on your local computer: apt-get install x2goclient

32 Windows: Download and install the newest windows client from the X2Go website. Follow the installation wizard s steps, you may keep all default options. Max OS X: The X2Go Mac client uses the OS X X11 server called XQuartz. If you don t have Xquartz yet, you can get it on the XQuartz website. Download and install it. Note: there is also a download of the file on download.cnet.com. Search there for XQuartz. If the download speed on the original website is painfully slow, you may download XQuartz from here. Just make sure you download the newest version. At the end of the installation, it will point out that you will have to log out and log back in again for changes to take effect. You can also do this after we have changed some settings in the next step. You have to enable a few settings for Mac. Start the XQuartz server. You can find it in the spotlight search (open it with Cmd+Space), and type in xquartz to find it. Click on it to start. In the Preferences, go to the Security tab and enable allow connections from network clients. NOTE: If you start XQuartz after already having logged out and in again, you can access the settings (or exit full screen mode) with CMD +, (command and comma) Important: Because above instructions may change any time, please refer to the most current X2Go documentation Download and install the newest X2Go client from the X2Go Website. Connecting the X2Go client We will now start the X2Go client application that you installed in the last step and connect it to the instance. Step 1: Start the client On Linux, you simply type into your local computer s terminal x2goclient

33 On Windows and Mac, start the application as you do with any other program (find the x2goclient application). E.g. on Mac, bring up the spotlight (Cmd + Space) and search for x2goclient. Step 2: Start a new session The session settings should come up right away if you are using the client for the first time. If you are not, click on New session, which is the top left button that looks like a blank sheet of paper. Hint for Mac users: the X2Go windows may be sized too large for your screen initially, so you cannot see the usual OK and Cancel buttons in the dialogue windows. To resolve this, you simply have to select the window by clicking on the top window bar and holding the left mouse button to move the window a little bit. When you release the mouse button, the window should be automatically resized to fit your screen. The session preferences window should look similar to this: 1. Specify your session name in the first field (In the screenshot, we called it NeCTAR Tutorial session ). 2. In the host field, type in your IP address 3. In the Login field, specify the username (in this tutorial, it is ubuntu ).

34 4. Select your private SSH key in the field Use RSA/DSA key for ssh connection. You may click on the little folder symbol behind the field to browse for your private ssh key file. Note for Mac/Linux: earlier in this tutorial, we have put the key in the.ssh folder in your home directory. If you cannot see this directory, you have to create a shortcut to it. Open a terminal on your local computer, and type in your local computer s terminal: ln -s.ssh ssh You should now have a folder called ssh in your home directory which you can see from the select file dialogue. 5. As a Session type, select the Desktop environment you installed earlier (e.g. LXDE, Gnome, KDE). You may also have installed several desktop environments, in this case select which one you wish to start. 6. Click OK to get to the X2Go Start Window. If asked whether to trust the host, click Yes. Step 3: Connect to the desktop We will now connect to our remote desktop. You should now be seeing the X2Go Start Window: On the right side of the Window, you can see a shortcut to your session. Note that you can quickly change the desktop environment and screen resolution in the box. Don t do this now however, let s first try to connect with the default. You can change this next time you log in. Click on the box with your session name, and the client is going to try to connect to your instance s remote desktop. You will have to type in your ssh key passphrase.

35 This may take a while, especially if it is the first time you connect. Don t get impatient. This is a good time to grab a cup of coffee. A window will then come up which shows your desktop. Below is an image of an example when XFCE was used: You can increase the screen size and the resolution of your desktop should adapt to it. Resizing the window can sometimes be a bit slow, therefore you may try different screen resolutions in the login box (which displays your session name) so you don t have to resize the window every time. To close your remote desktop, you should log off. This is in most desktop environments a power button somewhere on the desktop, or it is accessible with a right click on the desktop. Choose logout as an option, not restart or shutdown. If in doubt, just close the window in which the remote desktop is displayed (this is the dirty option, but it will also work). When you log back in, you don t have to specify a password for the ubuntu user, as we are using your ssh key instead. Important Note if you are using KDE: KDE does not adapt the screen size that we set in the client. Therefore, we have to disable the KScreen KDED module. We will do this right away. 1. On your remote KDE desktop, go to system settings. You can right click on the desktop and then select Run command from the drop down. 2. In the text input field that comes up, type System Settings and the dropdown should show the System Settings which you can click on.

36 3. Go to System Settings > Startup and Shutdown > Service Manager, and uncheck KScreen Log out (Right click on Desktop > Leave > Log out) and log back in again (You don t have to specify a password for the ubuntu user when you log back in, as we are using your ssh key instead). You should now see your full desktop. Your screen size should adapt to what you have set in the session settings of your X2Go client (default is 800x600). File sharing with X2Go One nice thing about X2Go is that you can do file sharing between your local computer and your instance easily on the desktop. Let s now enable file sharing so we can drag and drop files between our computer to the instance s desktop. Step 1. Enable file sharing We have to enable file sharing for our instance s user ubuntu. For this, you have to add the user to the group fuse on your instance. Go to your ssh terminal (or open a terminal on your remote desktop) and type: sudo adduser ubuntu fuse Important if are using a Mac: You also have to enable remote login to support file sharing. To enable file sharing, you also have to change your system preferences. Go to the Apple (on top left of your screen) > System preferences, and then select Sharing : Activate the checkbox for Remote login :

37 Should you have problems connecting in the following steps, try restarting your Mac computer for the changes to take effect. Now, we have to configure the X2Go client session preferences to enable file sharing. First, create a folder on the desktop (or elsewhere if you like) which you are going to use as our shared folder. Name it for example x2gosharedfolder. Think of this folder as a place where you can put files which you can then access from within your instance also. You can Drag & Drop files into this folder, and it will be easy to access them from your instances remote desktop. Conversely, you can Drag & Drop files from within your remote desktop into this folder, and they will appear on your local computer as well. Close your remote desktop by logging off (or just shutting the window in which your remote desktop is displayed). Then, go back to the X2Go Start Window in the X2Go client (If you aren t there already, click cancel button in the login field). You should now see this X2Go Start Window: Via the little arrow on the right bottom of your session box, select Session

38 preferences as shown in the screenshot above. Here you can change preferences of your session. Go to the Tab Shared folders, and click on the little folder symbol to select a folder of your choice which you want to use for sharing (earlier we have named this x2gosharedfolder on our desktop): When you have selected the folder, don t forget to click on the Add button next to the path to your file. Next to the new element added to the box, enable the Automount check box. Finally, click OK to close the window. Step 2. Logging in. Now you can open your remote desktop by clicking on your session box and logging in again. You don t have to specify a password for the ubuntu user, as we are using the ssh key instead. You will typically find the shared folder of your computer as a link on the remote desktop (see comments below for your desktop environment). Step 3. Remaining configurations for individual desktops. Some desktop environments require some extra settings to make file sharing work.

39 Refer to the documentation below for your desktop environment. If you are using LXDE: No extra settings required. Your shared folder should be on the desktop now. If you are using KDE: Your shared folder will be visible on your remote machine by adding the X2Go plasmoid applet to the desktop. To add the widget, right click on the desktop and select Add widgets. Note: You will need to have installed the plasma widget, which we did in a previous step with the command *sudo apt get install plasma widget x2go. In the search window that comes up, type x2go and the x2goplasmoid should come up in the list. Double click on it to add it to the desktop. You can now access your shared folder on your local computer via the link on the plasmoid. After you open the folder (double click), you should then see something like this on your desktop: You can change the size of the widget by dragging it if it looks graphically wrong. You need to hover over the right hand menu of the widget and click with your mouse on the little box in the top:

40 You can then drag it out to the right side: If you are using XFCE: File sharing has to be set up manually. The shared folder can most likely be found in the folder /tmp/.x2go ubuntu/media/disk. Check what the name of your folder may be. Type into your ssh terminal: ls /tmp/.x2go-ubuntu/media/disk you should recognize your folder name, which is named after the path where it is located on your computer (if you named your folder x2gosharedfolder, the name should end with this). Then, type ls /tmp/.x2go-ubuntu/media/disk/<your-folder-name> to see all the files you have in your shared folder (if your folder is empty, no files will be listed, of course). Right click on the desktop and select Open Terminal Here. Then, type ln -s /tmp/.x2go-ubuntu/media/disk/<your-folder-name> MySharedFolder replacing <your folder name>> with the name you found above. Be careful with spelling (it s case sensitive) if you don t get the path to the folder right, it will not work. You can also change the name MySharedFolder in the command to something else. When you click on the desktop again, the link to the shared folder should appear.

41 You can verify where your shared folder has been made available by looking into the file /var/log/syslog on your instance. You should see a line that contains the following:* successfully to /tmp/.x2go ubuntu/media/disk/* Step 4: Exchange files Your shared folder can now be accessed both from your instance and your local computer. First, on your local computer, create a text file in your shared folder (x2gosharedfolder), write some text in it, and save it to the shared folder. For example on a Mac: 1. Open TextEdit (Cmd+Space, type textedit and open the TextEdit program) 2. Write a message into the window 3. Click File > Save and select the shared folder as a place to save the file to. Name it for example x2gotest.

42 Now, go to your remote desktop. Open your shared folder (you should have found where it is in the last step). For example, when you are using the LXDE desktop environment, your shared folder will be on your desktop. Now if you open the shared folder, you should see the text file you have just created. You may open it to see what s in there:

43 Depending on which text editor you used to create your file, it may not look the same on the remote machine. This is because it uses a Linux Operating system, and the default editor on your instance may not open the file with the program which supports this file format. For example, in the screenshot above, you can see that the RTF file we created on a Mac contains formatting instructions as well (you can see it is an RTF file because it is called x2gotest.rtf). Because Linux opens it in a plain text editor, you can see the formatting instructions as well. This should not worry you at this point, because we are just testing how to share files with your remote desktop, but this is a good place to notice some differences between operating systems and text files. Now, we are going to do it the other way round: We are going to create a text file on our remote desktop, copy it to the shared folder, and see that we can access it in our shared folder on our local computer as well. On your remote desktop, right click on your desktop, and select Create new > empty file (This may be called different in your desktop environment, but it should not be too hard to find how to create a new file. If there is no such command on the desktop, open your file explorer, go to your Desktop in the navigation pane on the left, and try to do it there). Name the file TestOnVM.txt.

44 Open the file by double clicking on it. A text editor will open. Write a message into the file, save it (File > Save), then close it. Now, drag the file from the desktop to your shared folder. A message will appear that it is being copied, and then you should see it in your shared folder:

45 Now, you can open your shared folder on your local machine (the x2gosharedfolder on your desktop). It will have the new file in it! You may double click on it to open it and find the message you typed on your remote desktop inside. Troubleshooting If you ever run into trouble with the desktop environment, try first to log out and log back in again. If that does not work, you may have to restart the X2Go server. You can do this in the ssh command line of the instance (you may also open a terminal on your remote desktop to do this): sudo service x2goserver restart or sudo /etc/init.d/x2goserver restart Resources X2Go documentation: Installation of the X2Go server

46 A good article about installing X2Go server on Ubuntu Connect with VNC Connecting with VNC We will now connect to our remote desktop using VNC. We will first install the VNC Server on the instance, and then use a VNC Client (also called viewer ) on our local computer to connect to the remote desktop. Install the VNC Server on your instance In this tutorial, we will use the VNC Server called vnc4server on our instance. For an overview of other servers, refer to the Ubuntu help page. Step 1. Install VNC server To install the server, type into your ssh terminal: sudo apt-get install vnc4server Confirm installation of the package with [Enter]. Step 2. Configure the VNC server We will now have to edit a file in order to start up the right desktop environment. The following instructions have been tested for Ubuntu and If you have another Ubuntu version on your instance, you may have to look online if the following instructions don t work. Some more information is given at this link. First, we have to start up the VNC server once, in order to create a password and generate the configuration file. In your ssh terminal, type: vncserver and you will be prompted to choose a password. This is a password which is used for the VNC server only it is not the same as your ssh key (although you may of course choose the same passphrase)! After you finished typing and confirming your password, we will shut down the vnc server again and change some settings. To shut it down, type:

47 vncserver -kill :1 Then, open the settings file: nano.vnc/xstartup In the editor that comes up, make the following changes: Uncomment the following lines, by putting a # in front of them: # x terminal emulator geometry 80x ls title "$VNCDESKTOP Desktop" & # x window manager & Additionally, make the following changes to the file, depending on your desktop environment: Gnome : Add the following lines at the bottom of the file: metacity & gnome settings daemon & gnome panel & KDE: Add the following line at the bottom of the file: startkde & LXDE: Add the following lines at the bottom of the file: lxterminal & /usr/bin/lxsession s LXDE & XFCE Add the following lines at the bottom of the file: xrdb $HOME/.Xresources startxfce4 & Step 3. Start the server To start the vnc server, you need to configure the resolution which your screen should have (You can change this to another resolution in the command below). To start with, choose a low resolution which will definitely fit into the screen on your local computer with some margin: vncserver -geometry 1024x768 -depth 24

48 The result of the command will display a line similar to the following. New '*hostname*:#* (*username*)' desktop is *hostname*:**#** where hostname and username are replaced by your instances values, and a number is displayed instead of the hash #. Take note of this number following the colon (:). This will be part of the port number you will need in the SSH tunnel setup, which we will do in the next step. This will also be the number you will need to use to eventually terminate the server when you are finished using it (if you want to shut it down at all). If you want to shut down the VNC server, type into your ssh terminal (use the number you have taken note of instead of the #): vncserver -kill :# For example, if your port number before was 1, type vncserver -kill :1 Whenever you want to change your desktop environment (e.g. from LXDE to KDE), you have to shut down ( kill ) the vnc server with these commands. Then, you have to edit your.vnc/startup file as discussed in the last step, following the instructions for your desktop environment. Then, restart the VNC server again, as you have done earlier. Connect to the remote desktop If you tried to use a VNC client right now, it would not be accessible from outside, because our instance is protected by a firewall. When we launched our instance (as we did in this section), we have only allowed secure ssh connections on Port 22 (and insecure connections to web services on Port 80). VNC servers usually provide their services on Port 5900 (or 5901, ). This Port is still being blocked by our firewall! One way to get access to our VNC server would be to open up this port in our firewall and allow an insecure connection (remember: VNC does not have in build encryption). This is not desirable we want a secure connection, of course. The safe way to do this is to tunnel all our VNC communications through SSH, which then encrypts all communication. The following steps will explain how this is done. Step 1: Set up ssh tunneling Be sure you have the number (#) you were given in the above vncserver command.

49 We will use this number to determine the Destination Port. Use your number and add it to For example, if your number is 1, then 5901 will be you destination port. Follow the steps below for your local computer s operating system: Linux / Mac In a terminal on your local computer, create a tunnel from your local machine (Port 5900) to your instance (Port 5901) with this command: ssh -i <path-to-your-keyfile> -L 5900:localhost:5901 -N -f -l ubuntu NNN.NNN.NNN.NNN where you should replace the NNN.NNN.NNN.NNN with yours instance s IP address, and <path to your keyfile> is the path to where you have saved your ssh key. Hint for Linux users: If you copy your private key to the /home//.ssh folder, as we have done in this tutorial, you won't have to specify the key every time and can skip the i option. The f N options in the above command cause the tunnel to run in the background: once you have typed in the password, the command line prompt returns to your terminal. If you want to close your ssh connection, you have to kill it. In your terminal on your local computer, type: ps aux grep ssh In the printed result, find your ssh tunneling command and note the number on the left, which is displayed after your user name. One line of output for example looks like this: ubuntu In this example, is the process number kill <the-ssh-tunnel-process-number> If you don t use the N f options, you will be logged into an ssh terminal to your instance, and can use it as usual to type commands on your instance. If you type logout or close the terminal, the ssh tunnel will close as well. Windows 1. Launch PuTTY and load your previously saved configuration by selecting the

50 session name and clicking Load. (If you have Putty open, close it and then reopen it to get to this window). 2. Select SSH > Tunnels in the left pane. Enter these values: Source port: 5900 Destination: localhost:5901 Then, click the Add button. Your tunnel will now appear in the list. 3. You can save the tunneling along with your session by selecting Session in the left pane. Click on your session name in the Saved Sessions pane, then click Save (or create a new name instead by typing in a new name, e.g. called TunneledNectarSession ). 4. Now you can click Open to start the SSH terminal. The tunnel will now be active as long as the PuTTY terminal window stays open. You can use this SSH terminal

51 as a shell into the instance as well (as we have been doing the whole time), while simultaneously having your remote desktop open. 5. The next time you run PuTTY, you must open these saved settings by selecting the name in the Saved Sessions pane and clicking Load, then clicking Open. Or, you can simply double click the session name to load and run the saved session. See also the section about SSH Tunnelling in the Appendix for more information. Step 2: Install the VNC Client on your computer We can connect to our desktop using a VNC Client (also often referred to as VNC Viewer ). You may download any VNC viewer you want. Here, we ll suggest the following, depending on which OS your home Computer runs: Ubuntu You can use the pre installed Remmina client. Alternatively, install vinagre with the command sudo apt-get install vinagre. You can use the free RealVNC Viewer. Continue with the instructions below (Mac/Windows/Linux alternative client: RealVNC) if you would like to use RealVNC. We recommend you use the default Remmina client to start with. Windows You can use the free RealVNC Viewer. Continue with the instructions below (Mac/Windows/Linux alternative client: RealVNC) if you would like to use RealVNC. Alternatively, you can also use UltraVNC or TightVNC ( Note for TightVNC: Choose the custom installation mode and then select to only install the client, you may have to unselect the server). Choose the client you want and install it. We suggest you use RealVNC for this tutorial. Mac OS X OS X has a VNC client built in. It s been installed in every version of OS X since It s called Screen Sharing and it s located in your CoreServices folder, a difficult to find folder which is part of OS X. The easiest way to find screen sharing via the spotlight search : Type Cmd+Space to

52 bring it up. Type screen sharing and the app should be found. Or you can also find the application in this path: /System/Library/CoreServices/Screen Sharing.app or, as of OS X Yosemite the containing folder is now located in /System/Library/CoreServices/Applications You can access it through the terminal (Applications Folder > Utilities > Terminal.app). In the terminal, type: open /System/Library/CoreServices or on OS X greater than 10.10: open /System/Library/CoreServices/Applications That will open a Finder window where you will see Screen Sharing and you can open it or create an alias if you want. You can use the free RealVNC Viewer. Continue with the instructions below (Mac/Windows/Linux alternative client: RealVNC). Mac/Windows/Linux alternative client: RealVNC Download the client (the viewer) from the RealVNC website. When downloading, it may ask for your E Mail, but you can specify some random details, the viewer will still work. When installing, choose to only install the client (the viewer): When you connect (we will do this in a moment), it asks you whether you accept an insecure connection. Say yes (it will in fact be secure via our tunnel).

53 RealVNC by default compresses the colours to save bandwidth. To see your screen in full colour, you need to go to Options in the initial window, then disable The checkbox for *Adapt to network speed, and push up the slider to Best quality. Step 3: Connect to the VNC server Open your VNC client ( viewer ) of choice. This will generally be a very simple interface with just a few fields to fill. The following instructions are generic for all clients. There are also some more detailed instructions given below for some examples. Choose VNC as a protocol, if the viewer asks for it. As server (or host), specify localhost. If a port specification is required, choose If you cannot find a port specification, you probably have to type localhost:5900 instead of just localhost in the server (or host ) field. Now connect, and you should get a password prompt, in which you type the password you just chose for the VNC Server (not your ssh passphrase, unless you chose the same!). Then, you should get to the remote desktop. Do not use any ssh or security options that clients may offer, because we have just done the ssh tunnel separately ourselves. Your connection will be secure. If you have followed the instructions above, your connection is secure, even if the vnc viewer says it is not (The VNC viewer does actually not realise it is running through the ssh tunnel). Example on the image above: Connecting with TightVNC on a Windows machine. The

54 Putty Terminal is displayed on the left. Mac Example client: Screen sharing To connect with the in built Screen Sharing app, open it (Apple+Space and type screen sharing and open it up). Into the text field, type *localhost:5900 *and then click connect : You will be prompted for your password. That s the password you selected when you set up the VNC Server in the previous step. Now a window should open to the remote desktop. Example client: RealVNC Start the RealVNC client (on a Mac, you can find it via the spotlight search: Cmd+Space and then type realvnc. It will open a folder in which you find the executable). To launch a connection, type localhost:5900 into the text field. Leave the encryption setting on Let VNC Server choose and click connect. When it asks you whether you accept an insecure connection, say yes (it will in fact be secure via our ssh tunnel).

55 You will now have to type in your password. That s the password you chose when setting up the VNC server on your instance in the last step. RealVNC by default compresses the colours to save bandwidth. To see your screen in full colour, you need to go to the realvnc Options. You can either do this in the initial login window, or when your remote desktop is open, mouse over the top of the window and select options. In the options, disable The checkbox for Adapt to network speed, and push up the slider to Best quality (better quality will be slower but mostly you may want it anyway). Also enable the checkbox Scale to window size if you have issues with the window being too large for your desktop. General troubleshooting

56 Your colours look a bit off? Some clients (e.g. RealVNC and Remmina) use fewer colours and higher compression in order to reduce latency of your remote desktop. You have to change this settings in the client each client is different so you might have to find out where you can change this. In Remmina, you can create a saved session (by clicking on new ) instead of using the default Quick Connect. You can change the settings for the colour quality there. RealVNC by default compresses the colours to save bandwidth. To see your screen in full colour, you need to go to Options in the initial login window, then disable the checkbox Adapt to network speed, and push up the slider to Best quality. Problems connecting? Make sure your ssh tunnel is open and you have set the right port numbers. On Windows, your PuTTY session must be open and you must have configured the tunnel as described above. Automatic start up of the VNC server With the current settings, you will have to manually re start the Vnc server on your instance every time after you reboot the instance. We can automate this to save you this work, and start up the Vnc server automatically. The easiest way is to add an execution command in a file. Log into your instance (via ssh terminal) and follow these instructions: 1. Verify the location of your VNC program by typing: which vncserver Most likely you will see /usr/bin/vncserver as result. We will assume this in the following. If you get something else, you will have to use your own path instead of this one. 2. Create a file startvnc.sh (or call it whatever you want) in directory /etc/init.d: sudo nano /etc/init.d/startvnc.sh 3. The editor will open. Put the following contents in the file (Note: If your username is other than ubuntu, you have to replace it in the 3rd line): #!/bin/bash PATH="$PATH:/usr/bin/" export USER="ubuntu" su ${USER} c "/usr/bin/vncserver geometry 1024x768 depth 24"

57 Note: If your vncserver is located somewhere else, as you have found out in step 1, you have to use your own path here instead of /usr/bin/vncserver 4. Close the editor with Ctrl + X (and confirm saving the file with yes ). 5. Now we have to make this file executable sudo chmod +x /etc/init.d/startvnc.sh 6. Add this file to be executed at startup: sudo update-rc.d startvnc.sh defaults Optional: You may also set up a service out of the vnc server. After you have set it up, it allows you to start the vnc server any time with the following command: service startvnc.sh start (or shut it down with stop instead of start). This is probably the more proper way to do it, but the file startvnc.sh is a bit more complex: #!/bin/bash PATH="$PATH:/usr/bin/" export USER="ubuntu" DISPLAY="1" DEPTH="32" GEOMETRY="1024x768" OPTIONS=" depth ${DEPTH} geometry ${GEOMETRY} :${DISPLAY} localhost". /lib/lsb/init functions case "$1" in start) log_action_begin_msg "Starting vncserver for user '${USER}' on localhost:${display}" su ${USER} c "/usr/bin/vncserver ${OPTIONS}" ;; stop) log_action_begin_msg "Stopping vncserver for user '${USER}' on localhost:${display}" su ${USER} c "/usr/bin/vncserver kill :${DISPLAY}" ;; restart) $0 stop $0 start ;; esac exit 0 Reconnect to the VNC afresh Of course, you won t have to do all the above steps every day when you want to

58 connect to your remote VNC desktop. The only things you need to do each time are: 1. Start the ssh tunnel, as described above in step 1. On Windows, if you are using PuTTY, you may have created a saved session which you can start easily. 2. Open your VNC client and connect, as described in step 3 above. Let s do this now to see how easy this is. Close all your applications (all your terminals / PuTTY, and your open VNC viewer). If on Unix/Mac, also kill your ssh tunnel (for instructions, see the notes about SSH tunneling in the Appendix). Now, do the above steps 1. and 2. above to get back to your remote desktop. Troubleshooting on Windows/Putty: If you are Windows and PuTTY, check whether you have properly saved your session last time. When you click on your session and then click Load, you can check whether all the settings we have configured earlier ( Connect to the instance, and Step 1: Set up ssh tunneling ) are still there. If you do not have all the details saved, change them now and save your session again (click on Session in the navigation panel on the left, select your session in the Sessions pane, and click Save). Then, try again. Resources Wikipedia page on VNC Ubuntu Help page on VNC Ubuntu Help page on VNC Servers An article on VNC on Linux Article on how to install an configure VNC on Ubuntu Another article on how to install a VNC server on Ubuntu Running several desktop environments It is possible to have several desktop environments installed and use them interchangeably. You will have to install each desktop environment you want to use on your instance (as we have done in this section earlier. Note that with VNC, you have to change your desktop environment on the server side

59 (on your instance), and everyone who connects to the instance will automatically use the desktop which has been configured for this instance. To change the desktop environment, you have to edit the xstartup file and restart the vnc server, as described in the last section. With X2Go, you can decide on the X2Go client which desktop to use in your x2goclient application, you can choose from the drop down box which desktop environment you want to run. Graphical Interfaces on the ssh terminal If you want to run a single graphical application, you can also do this without setting up a remote desktop. You can do this via the ssh command line and start the application directly. The advantage is that you don t need to set up a remote desktop server on your instance and install a client on your local machine. The downsides are (1) it introduces some security issues, which will not be discussed in detail here, and are usually not a big worry (see security note below for a few details). And (2), this is (perhaps surprisingly) a lot slower than a remote desktop solution. This is because of the way the communication is encoded. Note: This technique allows no desktop sharing, just remote application running (However you could use XNest, not discussed further at this point). The security issue arising with running graphical applications via the ssh connection is that we have to enable X11 Forwarding. X11 is the base for the Linux graphical applications (e.g. opening windows). With X11 forwarding, the server is likely to gain shell access to the client (the ssh terminal you are running on your computer): the server can send X11 commands back to the client. X11 was not designed with security in mind, it was designed with the idea that all programs that you re displaying are run by you and therefore trusted anyway. However ssh also has a SECURITY extension which disables some obvious attacks such as keyboard grabs and key injection. However it still allows other security issues like focus stealing. In simple terms: Let s say you open a SSH connection to <some server>, with X11 forwarding enabled. The primary risk is that if <some server> is malicious, then <some server> can do all sorts of things to the windows/applications you have open on your own computer. For instance, <some server> can open up windows on your computer, can

60 close other windows you have open, can spy on the contents of other windows you have open and on the keys you type into other windows. It can inject spoofed keystrokes and mouse events into other windows you have open, and generally just mess with every other window you have open on my machine even if some of those other windows are local applications running locally. In general, because you log on to your own remote machine and not just <some server>, this security risk is not much of a worry only if your instance was infected by malicious software, this would be a potential security lead. Step 1. Check that your Instance allows X11 Forwarding We have to enable X11 forwarding. It may already be enabled, but make sure it is. Go to your ssh terminal and type: sudo nano /etc/ssh/sshd_config to edit the configuration file. Scroll down until you find the line X11Forwarding yes (it may be a no instead of the yes). If this line starts with a #, remove the #. Make sure it is set to yes. If you cannot find the line, add it at the end of the file, just as it is shown above. Scroll further to find the following line: X11DisplayOffset 10 and make sure it is set to 10. Exit the editor with Ctrl+X (and confirm saving the file with Y ). If you did not have to change the file, you can jump to the next step. Otherwise, you now have to restart ssh on your instance. You can do by typing: sudo /etc/init.d/ssh restart Step 2. Log in to your instance via ssh Linux

61 Open up a terminal. We will connect to the instance with the ssh command that we used in an earlier section, but this time you have to add the X option to the command, to enable graphics mode. ssh -i <your-private-key-file> -X [email protected] Mac You first need to install XQuartz on your Mac. If you don t have it already, download from the XQuartz website Note: If you have previously followed the steps to install the X2Go Client, you will already have XQuartz installed! After you installed XQuartz, open up a Terminal on your local computer. We will connect to the instance with the ssh command we used in an earlier section, but this time you have to add the X option to the command, to enable graphics mode. ssh -i <your-private-key-file> -X [email protected] Windows If you use Windows you will need to install a X11 server locally on your windows computer. You can use for example Xming. Open your PuTTY session and load your settings that you previously saved (to do this, select Session in the navigation pane on the left, select your saved session from the list, and click Load). Then, go to Connection > SSH > X11 in the navigation pane. You need to check Enable X11 forwarding and specify the X display location. If you did not change anything in the default configuration of Xming, then set the display location to localhost:0. Otherwise, if you change the Xming configuration using the tool Programs > Xming > XLaunch, make sure to set the parameters for the display accordingly.

62 Step 3. Install X Windowing system on your instance Your instance still needs a concept of graphical interfaces, so we need to install it. If you already have previously installed a desktop environment (e.g. LXDE, Gnome, KDE, XFCE) as we have done in an earlier section, then you can skip this step, because the windowing system is already installed. To install the very basic windowing system, go to your ssh session into your instance and type: sudo apt-get install xauth sudo apt-get install x11-apps It is likely the xauth package (first command) already installed, but we are doing this just in case. If it is installed, the first command will not re install any packages. Step 4. Run a graphical application We will now test if this works. Go to your ssh terminal (or on Windows, the PuTTY terminal) and type: xcalc & This should open up a simple calculator application that is installed by default. You

63 may have to wait quite a while until it opens up, as said earlier, this way to run graphical applications pretty slow. Note: The & after the command causes the calculator to open up and detach from the terminal. If you skip it, the calculator will open, but you won t be able to type other commands in the terminal. If this worked, try a larger application, like firefox. Go to your ssh terminal (or PuTTY on Windows) and type: sudo apt-get install firefox (It may already be installed) Then, type firefox & and wait a while until firefox opens. You may notice the amount of time you have to wait is much longer than when you have a remote desktop open (as an earlier section described). To start with, ignore some warnings and error messages that may come up it really takes forever for firefox to come up. Troubleshooting on Unix/Mac If you receive errors trying to run graphical applications, try the option Y (instead of X) in your ssh command (in Step 2 above). Install a web server In this section we are going to enable a web server and create our first simple web site. To install the web server, go to the ssh terminal and type: sudo apt-get install apache2 Confirm installation of the package with Y. Now, open the web browser on your local computer, and type your instance s IP address in the URL field. We will see the default apache website of the browser. Let s now start our own simple website. In your ssh terminal, type: nano /var/www/html/index.html

64 the directory /var/www/html is where your website lives by default, and the file index.html is the start page which we want to edit. In the editor that comes up, remove all the text inside and replace it with the text below. Tip: Copy and paste will not work with many terminals. If the traditional copy & paste does not work, you can instead try the following: copy the text as usual; then click with your mouse on the place you want to insert the text in your ssh terminal editor; Then, click the right or middle mouse button this will work on many systems to paste the contents of your clipboard. <!DOCTYPE html PUBLIC " //IETF//DTD HTML 2.0//EN"> <HTML> <HEAD> <TITLE> A Small Hello </TITLE> </HEAD> <BODY> <H1>Hi</H1> <P>This is very minimal "hello world" HTML document.</p> </BODY> </HTML> Now, go to your browser again and refresh the page. You should now see a very crude website displaying Hi, this is a very minimal hello world HTML document. Managing storage Prerequisite: You should know what types of storage are available on the NeCTAR cloud. This was discussed in Module 6. You should also be familiar with connecting to your instance via an ssh terminal, as described in an earlier section. In this section, we will go through the steps to set up various types of storage on our instance. A brief recap of the available storage types: Object storage: These are single files which you can upload on the server and which you can access from anywhere via a simple web interface or a suitable client application (of which this section will introduce a few).

65 Volume Storage (also called Block Storage) and Ephemeral Storage (also called On Instance storage) are directly attached to your instance. Ephemeral storage is tightly attached to your instance and is deleted when you terminate (delete) your instance. You cannot recover the data after you terminate the instance, unless you have backed it up. Volume Storage can be attached and detached from your instance(s). After you terminate the instance, the data is not lost. You can attach it to a new instance after you launch it. Requesting storage The NeCTAR trial account only includes object storage and ephemeral storage. You will have to file an allocation request to get access to Volume storage. The allocation process was described in Module 5, and more information can also be found on the NeCTAR support website. If you don t have an allocation yet, you can complete the next section, On Instance storage, and section Object storage. You will have to wait for your allocation until you can follow instructions for Volume Storage. However you can still read through the Volume storage instructions to see that it is easy, and in parts very similar to handling the secondary disk with the On Instance storage. On instance storage On Instance storage appears as two separate hard disks. For example, in the flavor m1.small (you selected the flavor when launching the instance), this is 1. a primary disk (10GB) which is saved with snapshots you make of the instance (see Module 9 for details about snapshots), and 2. a secondary disk (30GB) which appears as a second hard disk on your instance, and is not backed up with snapshots. You will have to back up your data separately, which is a topic we will discuss later in Module 9. The primary disk (1) is usually used for the operating system, you should not use it primarily to store data. Your home directory is also located within this disk space (when you log on with ssh, you start off in your home directory /home/ubuntu/). To store large and important files, you should use the secondary disk (2). The advantage

66 of this disk, as opposed to the volume storage, is that you don t have to create it separately: it already comes with your instance. Depending on the flavor you choose, you can get up to 480GB of storage for the secondary disk. To get access to the secondary disk, we will have to mount it first. Mounting is a technique which essentially makes your hard drive accessible to you at a certain location (in a folder). Usually, the secondary disk is available on your system path /dev/vdb. However this is a device file, so you cannot read files there as if it was a folder. For this, we have to mount /dev/vdb to a folder first. On some images, e.g. on the Ubuntu image we use, the device /dev/vdb will already have been mounted for you, most likely in the folder /mnt. However we will still go through the steps of how to mount a device to another folder (name it for example /data), in order to learn all steps involved in preparing storage. You will benefit from this knowledge when setting up the Volume storage. To see where your secondary disk is mounted (or if it is mounted), go to your ssh terminal and type: mount grep vdb You may get output that looks similar to this: ubuntu@myfirstinstance:~$ mount grep vdb /dev/vdb on /mnt type ext3 (rw) This means that vdb is mounted on mnt and has the ext3 file system with read (r) and write (w) permissions. If you don t get an output at all, vdb is not mounted. Alternatively, try this command to check the mount point and size of your disks: sudo lsblk -l This should give you an output similar to this: ubuntu@myfirstinstance:~$ sudo lsblk l NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 10G 0 disk vda1 253:1 0 10G 0 part / vdb 253: G 0 disk /mnt This tells us that we have two hard drives, vda and vdb, of sizes 10G and 30G

67 respectively. The primary disk (vda) is further partitioned into a sub disk called vda1 (using the full 10G) and this is mounted under the mountpoint /. In Linux terms, this means this disk is used for the operating system. The secondary disk (vdb) has 30G and is mounted on /mnt. Many OS, for example Ubuntu, already have mounted the secondary disk, but it may also not be mounted yet, in which case nothing is specified under MOUNTPOINT. If your disk vdb has already been mounted, take note of what is specified as mount point for it. Then, unmount the disk using the command: sudo umount <mount point> replacing <mount point> with your mount point. For example, if your disk was mounted on /mnt, do sudo umount /mnt You may also be interested in the file system type which your secondary drive is currently using. A file system is used to control how how the storage of data (files, folders, etc.) is organized on a computer disk (hard disk, floppy disk, CDROM, etc.). The large amounts of data on a hard drive needs to be organized, similar to a library of books which needs to be catalogued, otherwise books cannot be found efficiently. Libraries have developed standards to organise their books, and similarly, computer systems have. Windows uses the vfat or the NTFS systems, while Linux has others, e.g. ext3 and ext4 are current standards. File systems have different properties, e.g. maximum file size or partition size that is supported, and whether they journal or not. A journaling file system is more reliable when it comes to data storage: it prevents inconsistency, and is much faster at file system checks than non journaled file systems. For more information about the different Linux file systems, refer to this Ubuntu help page. To find out which file system type your drives use, type: sudo lsblk -f which will print a different view of the information we have seen in the last command

68 (because we use the f option instead of l). It should look similar to this: ubuntu@myfirstinstance:~$ sudo lsblk f NAME FSTYPE LABEL UUID MOUNTPOINT vda vda1 ext4 1b85cad6... / vdb ext3 ephemeral0 24cf7b From this, we can see that vda1 is a partition of vda. We can also see that vda1 uses the file system type ext4, while vdb is formatted with ext3. In the above example, the disk devices are available on the paths /dev/vda and /dev/vdb (you have to put the /dev/ before the disk name to get the path). Follow these instructions below to set up your secondary hard disk. You should do this every time you start up a new instance and want to prepare the secondary disk for usage. Step 1. Formatting The first time you start up an instance, you may want to format the hard drive of your secondary disk to make sure you have the right file system type and start from a clean state. Formatting erases all data that may already be on the hard disk. If you are working with a new instance, nothing of importance will be on it, e.g. only a default system Lost+Found folder, or something similar. You should only be careful with this step if you have used the disk before and have data on it. Then, you should back up this data or skip this step (if you skip this step, you will have to keep the file system which is already used on the disk). One way to format the drive with the ext4 file system type is: sudo mkfs.ext4 /dev/vdb (if your secondary drive is at another path than /dev/vdb, which you have just found out before, you have to replace this here). Confirm the formatting with y. You should see the status message Writing superblocks and filesystem accounting information: Wait a while until it says done., and then your hard drive is formatted.

69 Now, you may want to verify that you have formatted the drive with the ext4 file system. Again, use the command sudo lsblk -f which prints information similar to this: sudo lsblk f NAME FSTYPE LABEL UUID MOUNTPOINT vda vda1 ext4 1b85cad6... / vdb ext4 24cf7b You should now see that vdb is formatted with ext4. Only vda is mounted on /, and this should stay this way, because the OS is saved under /. We will mount the secondary drive in the next step. Step 2. Mounting You can now mount your hard drive, so you can access it. First, create a folder which will be used to access your data. For example, we can name it /data: sudo mkdir /data Then, mount it: sudo mount /dev/vdb /data This will mount the disk if the type of file system can be guessed. Otherwise you may have to specify the file system type with the t option, e.g. sudo mount /dev/vdb /data -t ext4. If you did not create and format the Volume then you can look up the filesystem type: sudo parted -l This maps the device dev/vdb to the folder /data. Now, your disk is mounted at /data, and you can add, remove and access files on your secondary disk at the path /data. You can list the contents of this directory to see what is on your disk (alternatively, if you are using a remote desktop, you can open it in your file explorer. It should be displayed as a disk symbol in the left navigation panel of your file browser).

70 ls /data You may see that there is a Lost+Found folder (this is a default system folder). If nothing is printed, you have no files there yet. Step 3. Write permissions By default, you do not have write permissions on the secondary drive. You can try to copy a file onto /data, but you will get an error message about write permissions (Note: This is not due to our folder not being located in our home directory). So we need to add our write permissions now. First, find out which user group you belong to is. Type id ubuntu and find what is written behind gid=. This may be gid=1000(ubuntu). So your group name is also ubuntu and has the ID Now, there are two options: You can 1) enable write permission only for yourself (your username ubuntu), or 2) for the user group you are in. This way, you can share the drive with other users that you may give access to your instance and who you put in the same user group. The instructions below detail what you need to do for both alternatives. Option 1) sudo chown ubuntu /data Option 2) sudo chgrp ubuntu /data sudo chmod g+rwx /data This should do it. You can now write and execute files from your secondary drive. You will only need to do this once for the secondary drive on this instance. Step 4. Auto mounting Every time you reboot your machine, you have to mount (not format!) the secondary drive again (so only do Step 2 above, not Step 1). We can avoid this by auto mounting it. For this, we have to edit a file called fstab: sudo nano /etc/fstab

71 Look for a line which starts with /dev/vdb. You may find one that looks like this: /dev/vdb /mnt auto defaults,nobootwait,comment=cloudconfig 0 2 If you find such a line, this means your secondary hard disk is already automatically mounted at system start. On the Ubuntu distribution we are using in this course, this is the case: it is auto mounted to the folder /mnt, as we can see in this line. If you find this line, you can edit it and just change the /mnt *to */data. If you cannot find this line, just add this line to the file at the bottom: /dev/vdb /data auto defaults,nobootwait 0 2 Exit the nano file editor with Ctrl+X and confirm saving of the file with Y. Next time you reboot your instance, you should already find your secondary drive in your folder /data, and you won t need to do any of the Steps described here to access it. Object storage You can connect to your object store with a variety of clients, as long as they support OpenStack Swift or Amazon S3. We will discuss some example clients which support Swift in this section. Security Warning Swift does NOT provide encryption of the data it stores. If you have sensitive data that requires encryption you must encrypt the data files before upload them to your object store. Access to your object store requires your password, but this password may be saved in a file that people who have access to your local computer can read. You must be careful to not give anyone access to your computer similarly, they would get access to your private ssh key that you save on your computer to log in to your instance, so be careful with giving people access to your computer. The NeCTAR implementation of Swift is geodistributed across Nodes of the Research Cloud so that availability is not reliant on any one datacentre or network infrastructure. Each collection of Swift nodes/hardware is known as a region, which may or may not include a Swift proxy server (the Internet facing and serving component of Swift). With some Swift clients/apis users can explicitly choose which proxy to connect to, this might be useful e.g. for

72 speeding up writes to object storage by choosing the nearest proxy. Due to NeCTARs Swift having multiple regions, some clients/apis require explicit configuration of a default region, which should be Melbourne for most users (see e.g. the Cyberduck profile below). Get your OpenStack credentials Before you start, you should get your login details. Note that you will not need to do this if you just want to upload files via the Dashboard (as described in the next section). But if you want to use other tools, and to complete this tutorial, you should take note of your credentials now. You can download your credentials from the Dashboard 1. Your tenant ID is the name of your project as reported in the left hand panel of the dashboard (e.g. pt 12345) 2. Your username is reported at the top of the dashboard on the right (e.g, logged in as: [email protected]) 3. Open the Dashboard API tab: get there via Compute > Access & Security > API Access. 4. Look up what is specified for the Identity Service in the Dashboard API tab. It will be a URL like Take note of the text between and the colon (keystone.rc.nectar.org.au in the example). This will be the Server. Take note of the number (5000 in the example). This will be the Port. You will need the Keystone Server and Port to authenticate your access. Also take note of the keystone version which you are using, in this case its v Get your credential file (this is called the OpenStack RC file for OpenStack, and EC2 file for Amazon). You can download by clicking on the buttons on the top right on the same window: Download OpenStack RC File. Download it and save it somewhere on your computer where you can easily find it later. 6. You will also need your OpenStack password. This is not the same password that you use to log in to the Dashboard! If you have never used the OpenStack password before, you need to generate it first. You can do this in the Settings of the Dashboard: click next to your user name (your e mail) on the top right and a drop down menu will open. Click on Settings. You need to reset your password by

73 clicking Reset Password on the left panel of the settings. A long combination of numbers and letters will be displayed. This is your OpenStack password. Copy the text and save it somewhere safe. Your OpenStack password will be a long combination of numbers. At the time of writing, unfortunately this password cannot be changed yet, but this should be changed in a future update of OpenStack on NeCTAR. You will not need your OpenStack RC file in this Module, but you will need it later in Module 10. So while you are on the Dashboard API page already, you may download it now, and remember for later where you put it. All platforms: Access via the Dashboard We will upload some files to our Object Store using the Dashboard. This is probably the easiest method to upload files. First, select any file on your computer some image, text file, or anything not so important. If you are unsure which to choose, just create a new text file, write something in it, and save it to the Desktop (or anywhere else where you can find it. In this example, we are using a file called SampleObject.txt with the text I am a sample object file in it, saved on the Desktop. Step 1: Create a container Go to the Dashboard and then to Object Store > Containers. You will get to a page which shows the overview of your containers. First of all, you will have to create a Container. A Container is like a folder in which you can put files. The difference to a normal folder is that a Container is intended for a flat hierarchy, so you

74 cannot create containers within a container however you may create Pseudo Containers within the Container, so you can still organize your files hierarchically. Click on Create Container. As a Name, type in MyObjectContainer (or you may use your own container name, in which case you have to consider this in the instructions below). Keep the access on private. Click on Create Container. Your Container MyObjectContainer will then be listed on your Dashboard: Step 2: Upload a file Click on Upload Object on the right side of your container overview. A window will open in which you can Browse for your file. Note: If you cannot see the Upload Object field, you have to click on your container MyObjectContainer to select it.

75 You may also rename it in the given text field if you wish, for now we are going to keep the name SampleObject.txt. Click Upload Object to close the window and upload the file to your object store. It will now be displayed on the right side of the overview. We can see that it has 27 Bytes. If you wanted to download this file (e.g. when accessing the store from another computer), you can do this here by clicking on Download. Windows/Mac: Access with Cyberduck 1. Download the newest version possible of the application Cyberduck (you need at least version 4.4): 2. Install Cyberduck. On a Mac, you will download a.zip file which contains the app. Save it to the Desktop (or any other folder). Extract the zip file by double clicking on it. On Windows, you will download an installer. You may keep the default options, if unsure. You don t need to install Bonjour, if you are asked. 3. Create a text file on your local computer (e.g. save it on the Desktop) with the following contents. Name it NeCTARObjectStore.cyberduckprofile (or any other name, it just has to end with.cyberduckprofile). <!DOCTYPE plist PUBLIC " //Apple//DTD PLIST 1.0//EN" " Protocol swift Vendor NeCTAR Swift (Keystone) Context /v2.0/tokens Region

76 Melbourne Description NeCTAR Swift Melbourne (Keystone) Username Placeholder tenantid:username Password Placeholder password </pre></div> 1. Double click on your file NeCTARObjectStore.cyberduckprofile. 2. A Dialogue should open where you can specify Server etc. If it does not, open Cyberduck and go to Bookmarks > New Bookmark. 3. Choose NeCTAR Swift Melbourne (Keystone) from the drop down box, if it is not selected yet. If it does not exist, something has gone wrong in importing the NeCTAR profile (your file NeCTARObjectSTore.cyberduckprofile). The image shows how it looks on a Mac, on Windows your interface will look a bit different, but the fields are the same. 4. Nickname: give the connection a name of your choice, e.g. NeCTAR Object Store 5. As Server and Port, fill in what you took note of before when looking up the details in the dashboard. (This is probably keystone.rc.nectar.org.au and 5000) 6. TenantID:Access ID (or username on a Mac): Type in your tenant ID and username (you took note of this before) separated by a colon, for example pt 12345:[email protected]. 7. Expand the window by clicking on More Options. You can choose a Download Folder to which your files will be downloaded by default. As Path you can (but do not have to) specify the version number of keystone, e.g. /v2.0/. 8. Leave all other options in the default setting, and close the window (by clicking on the window cross)

77 9. Double click on the symbol of your connection and Cyberduck will try to connect to your object store. You will be asked to type your OpenStack password (not the password you use to login to the Dashboard!) 10. You can now drag and drop files into your object store container (drag in on the folder symbol), and from the object store container to your local computer (onto your desktop or into an open folder in your file explorer). You can also right click and Download (it will download into the folder you selected as Download Folder in the connection dialogue). 11. You can synchronize files to a folder on your local computer. Right click on the container you want to synchronize, and select Synchronize. You will be prompted to select a folder to synchronize into. Attention! Do not select the same folder which you have specified as Download Folder in the connection dialogue! This will create duplicate and nested directories. Select your folder, and it will start to synchronize your files. You can then access your object store files in this folder, edit them from there, synchronize with Cyberduck again (you have to select the same folder again), and your Object Store will be up to date! 12. You can edit files directly by right clicking on them and select Edit. Then you may choose a program to edit it with. Resources

78 NeCTAR support website Cyberduck and OpenStack HowTo Windows only: Access with Cloudberry 1. If you do not already have CloudBerry Explorer then download and install it on your computer from the official website. 2. Open CloudBerry Explorer 3. Click File > OpenStack > New Account. Then, click Add. 4. Display name: use a meaningful name to this account (e.g., the name you supplied as your container in the step above) 5. User name: your username, which is your e mail (e.g., [email protected]) 6. API key: your OpenStack password 7. Authentication Service: This is the server, port and version of keystone that you took note of with your credentials, e.g Tick Use Keystone authentication 9. Select the Tenant name option from the drop down menu and insert your tenant ID (e.g., pt 12345) 10. Close the Window with OK, and close the next Window with Close. 11. On the right window, select your display name from the drop down box. CloudBerry will connect to your object store. On the left window, you see your files of your local computer.

79 12. You can drag and drop files between both windows. 13. You can directly edit a file by Right click > Open. Linux and Mac OS X: Using any FTP client Prerequisite: You should know how to use an FTP client. You can use any FTP client, for example Filezilla. For this, we will have to run a program that acts like a FTP server on our own local machine. This FTP server will then communicate with the SWIFT object storage on our cloud and provide the file exchange via the FTP service. The following instructions are for a Debian based distribution, e.g. Ubuntu. For more information, refer to the official project page. Open a Terminal on your local computer to type in the following commands. 1. Install the ftp server package Linux sudo apt-get install ftp-cloudfs or use your other package manager if you are on another distribution than ubuntu. Mac OS X Install python s pip (unless you already have it) using the homebrew package manager: brew install python Alternative: If you don t have homebrew, you may use the easy_install command, however there may be problems with running the ftp cloudfs command with this. You may try anyway, and if

80 it does not work, then uninstall pip again (sudo pip uninstall pythonswiftclient) and get homebrew instead. sudo easy_install pip Upgrade setuptools sudo pip install --upgrade setuptools Install python swiftclient sudo pip install python-swiftclient sudo pip install python-keystoneclient Install ftp cloudfs pip install ftp-cloudfs 2. Start the local FTP server ftpcloudfs --foreground --keystone-auth --auth-url --keystone-tenantseparator=: & 3. Start your favourite FTP client, for example FileZilla. If you don t have FileZilla, you can install it with sudo apt-get install filezilla Your login details for the FTP client are as follows: Server (or Host): This IP address is the one of your local computer (not an IP address you can use to access your computer from outside though) Remember, you are connecting to the FTP server we just launched on our local machine. User: <tenant>:<username> Replace <tenant> and <username> with your OpenStack credentials. Password: Your OpenStack password Port: You can now browse your files and drag and drop them between both windows, as you are used to when using the FTP client. On the left side, you can see the file system of your local computer, and on the right side, you can see your object store.

81 You will have to do steps 2 and 3 every time you restart your local computer. Linux [Advanced]: Access with CloudFuse CloudFuse is a lightweight open source application that may be installed on Linux servers to create a local mount point to a customers existing Cloud Files container. You can find more information and documentation about cloudfuse on the cloudfuse github page To set up cloudfuse for access to your object store, launch a terminal on your local computer and follow the steps below. 1. Install necessary packages sudo apt-get install build-essential libcurl4-openssl-dev libxml2-dev libssl-dev libfuse-dev git 2. Clone the CloudFuse source cd git clone git://github.com/redbo/cloudfuse 3. Compile the source code and install it cd cloudfuse./configure make sudo make install modprobe fuse

82 4. Create a file called $HOME/.cloudfuse sudo nano $HOME/.cloudfuse 5. Put the following contents into your file (replace <your tenant name>, <yourusername> and <your password> with your own credentials): authurl= tenant= username= api_key= cache_timeout=60 6. Set the right permissions: chmod 600 $HOME/.cloudfuse 7. Mount your Swift storage with the following commands sudo mkdir /mnt/cloudfiles/ cloudfuse /mnt/cloudfiles/ Of course you may choose another name instead of cloudfiles, and/or create the directory somewhere else. 8. You should now be able to access your files on the directory /mnt/cloudfiles just as if they were on your local computer. ls -l /mnt/cloudfiles/ Or open your file explorer to find your mounted drive there (see image to the right). You may even drag and drop files to and from your object storage, as if it was a regular hard drive or folder on your computer. 9. Every time you restart your computer, you will have to re connect the object store with the command: cloudfuse /mnt/cloudfiles/ Troubleshooting: Your username (of your local computer) may need to be part of the fuse user group. This can be accomplished with the following command in your computer s terminal:

83 sudo usermod -a -G fuse <username> All platforms: Access with the openstack command You may also use the openstack command to list the files in our object storage that we just created. This will be discussed in Module 10. Access via APIs We will discuss more options on how to connect to the object store in Module 10. Volume storage This section is going to go through the steps required to use Volume storage from your instance. You are going to 1. Create a Volume storage 2. Attach the Volume to the instance 3. Prepare the Volume for use within your instance You need to submit an allocation request in order to be able to use volumes. The trial account does not include quota for volumes. You can only use volume storage from instances running in the same Availability Zone which you have also chosen with your allocation. Therefore, you need to select the same Availability Zone when launching your instance. If you haven t done so when launching the instance, you ll have to launch a new one, as described in this earlier section, and select your allocation zone in the Availability zone tab. There are two options how to manage volumes: (1) on the Dashboard or (2) via command line. In this module, we will only cover managing volumes via the Dashboard. Module 10 will show how we can manage volumes with commands. Step 1: Creating the volume. Before you can use a volume, you have to create it. Go to Compute > Volumes and click on the + Create Volume button on the top right.

84 A window opens in which you can give your Volume a name, and add a description. There are also other options which you can specify: Volume Source: You can either choose to create an empty volume, or create a volume from an image. The latter is useful if you want to create a volume which already has specific data on it. It is possible to save (or back up) a volume by creating an image of it we will see how to do this in Module 9. You may want to restore a volume that you previously have saved as an image, or someone else may have sent you an image with data on it which you should have

85 access to. When you create a volume from an image, your volume will have all the data on it that was on the volume when the image was created of it. In this tutorial, we will create an empty volume. Type: This field isn t currently used, it s there for future expansion. Leave this at No volume type. Size(GB): Type in the desired size (in GB) of your volume. Availability zone: Select your availability zone. It must be the same in which your instance is running. Creation of the volume will only work if you have a volume allocation at this availability zone. Click Create volume and you will see your new volume in the list of volumes. The Status should switch to Available after it has been created. This can take a few minutes. Step 2: Attaching / Detaching volumes. Before you can access the volume, you have to attach it to an instance. On the Dashboard, go to Compute Volumes in the left navigation pane if you are not already on the page which shows the list of your volumes. On the right side next to your volume there is a drop down box. Select Edit attachments. A window will open in which you can select your instance. If you don t see any instance, you have no instance running in the same Availability Zone. In this case, create one as described in this earlier section.

86 Select your instance and click Attach Volume. The status of your instance will switch to Attaching, and when it has been attached successfully, it should display In Use. You should also see the Server and Device that the volume is attached to. In the example below, it is attached to MyFirstInstance and to the device /dev/vdc. In order to detach the volume again, you would go to the drop down next to your volume again and select Edit attachments. A window opens in which you may detach the volume again. Your data won t be lost, it will only be detached from the particular instance. You can re attach it later again to any instance. Step 3: Preparing your volume for use. After you have created your Volume and attached it to your instance you will have to take a few extra steps to prepare that volume for use: you will have to (1) format it and (2) mount it.

87 The section about On Instance Storage has more information about formatting and mounting drives. Log in to your instance via ssh (or use a terminal on your remote desktop) to type in the following commands. Step 3.1. Look up where your volume has been created On your Dashboard, go to Compute > Volumes and you should see your volume. In the column Attached to it should display something like Attached to <YourInstanceName> on /dev/vdc It may also be attached to /dev/vdb, /dev/vdd or something else. Take note of the device name that it has been attached to and use this instead of /dev/vdc in the following. Step 3.2: Format the volume. Formatting erases all data. All your data on this volume will be lost. You should do this only the first time you use the volume, or if you are sure that you want to erase all data. If you don t want to erase the data, skip to the next step. A new Volume may not have a file system (depending on how it was created) and you need to create one before mounting. The exact command syntax is depends on the instance s operating system and the type of filesystem you require. In this example we will create an ext4 file system (a popular file system on linux) for an Ubuntu instance. To format the drive, type: sudo mkfs.ext4 /dev/vdc Display information about your disks with sudo lsblk -f or sudo parted -l

88 This should now show that /dev/vdc has the ext4 file system. Step 3.3: Mount the volume. Mounting a disk has already been discussed in detail for the secondary drive of the Ephemeral storage, so at this point, only a summary is provided: To mount your volume, you will have to create a folder first to which the device will be mounted, e.g.: mkdir ~/MyVolumeStore Then, mount this folder: sudo mount /dev/vdc ~/MyVolumeStore This will mount the disk if the type of file system can be guessed. Otherwise you may have to specify the file system type with the t option, e.g. sudo mount /dev/vdc ~/MyVolumeStore -t ext4. If you did not create and format the Volume then you can look up the filesystem type: sudo parted -l You can now access your volume on ~/MyVolumeStore (MyVolumeStore in your home directory). You may not have write permission on the Volume yet, you can allow write access for the user ubuntu with: sudo chown ubuntu ~/MyVolumeStore Alternatively, you may grant write access to the <user group> (replace <user group> with the name of the user group, e.g. ubuntu): sudo chgrp <user group> ~/MyVolumeStore sudo chmod g+rwx ~/MyVolumeStore Copying files across You may want to easily access files on your ephemeral or volume storage from your local office computer. We will discuss two ways to do this: 1. Using a SFTP client

89 2. On the command line, using a secure copy command Option 1: Using a SFTP client You may use any FTP client which also supports the secure SFTP protocol, which uses ssh to secure the connection. There are many clients you may use. In this tutorial, we are going to describe the setup for the popular FileZilla FTP client. Step 1. Install FileZilla Windows and Mac: Download the FileZilla client (not the server!) for your OS on the FileZilla website. Follow the installation instructions for your platform. Linux: It is strongly recommended that you use the package manager of your distribution. E.g. on Ubuntu, you can install with: sudo apt-get install filezilla Alternatively, you can also find FileZilla in the Ubuntu Software Center. Step 2. Enable ssh agent to handle your private key. It is possible to add a key file to the FileZilla client (On Edit > Settings, or a Mac, this is FileZilla > Settings, and then add the key file on Connection > SFTP), and then use the Interactive logon type. However, when adding the key directly to FileZilla, it is converted to an unprotected one. This is not recommended. So we will use another method: We will use a SSH authentication agent, which holds your private keys in memory, already decoded, so that you can use them without needing to type a passphrase. Windows: You willl need to download another tool from PuTTY called Pageant (the PuTTY SSH authentication agent). You can find instructions on how to install Pageant and load your key in the Appendix. Don t forget to close Pageant when you re done! Otherwise any user with access to your computer can simply open FileZilla and access your files without having to type the ssh key passphrase. To close pageant, go to the system tray icon, right click and Exit pageant. Linux and Mac OS X: Open up a terminal on your local computer. Add your ssh key to the agent:

90 ssh-add ~/.ssh/nectar_key It will ask you for the passphrase. Type this in to unlock the key and add it to the agent. Don t forget to remove the key from the agent when you are done! Otherwise any user with access to your computer can simply open FileZilla and access your files without having to type the ssh key passphrase. Remove the key with: ssh-add -d ~/.ssh/nectar_key To make sure your key is not loaded, type: ssh-add -l and the Nectar_Key should not be listed. Step 3. Configure your site in FileZilla Open FileZilla and go to File > Site Manager. Specify the following details: Host: Your IP address Port: 22 Protocol: SFTP SSH File Transfer Protocol Logon type: Normal User: ubuntu password: The password for ubuntu user, however it will not be used because you are using the SSH key instead, so you can leave this empty.

91 On the left hand panel, you may rename your site to something more meaningful, e.g. NeCTAR. Click Connect to connect to your instance and close the site manager. Whenever you want to reconnect, you have to open the Site Manager again. You can also access it via the symbol on the top right of the window. Then, you will be asked whether you want to trust the host. You can answer with Yes. You should now be connected to your instance and see your files: in the left hand pane, your local files are listed on the right hand pane, you will see the files on your instance. You can drag and drop between them or drag and drop to/from file windows on your computer. When you are done, close the window to disconnect. Option 2: Use the command line Instructions vary between Linux/Mac and Windows. Follow the instructions for your OS

92 below. Windows You may use the PuTTY command line tool pscp. First, open the Windows Command prompt: On Windows earlier than 8, click on the Start button locate the Search filed. On newer Windows systems, click Search in the right hand pane. Type cmd in the search field and your command line prompt will open. Download pscp.exe from the PuTTY website and save it in your C:\WINDOWS\system32 folder. Saving pscp.exe in the system32 folder is the easiest way to do it. If you cannot access your system32 folder, you may add the folder where you downloaded pscp.exe to the systems PATH environment variable. Say you have saved pscp.exe for example in D:\Documents\pscp.exe. Go to your Windows command line that you just opened, and type: set PATH=%PATH%;"D:\Documents\pscp.exe" Now go to your Windows command prompt. The pscp.exe command takes two paths as arguments: A path we shall symbolically name <local file path> is a path on your local computer. If you copy a file from your instance to your local computer, this will be the path to the folder on your local computer where you want to copy the file to. If you copy a file from your local computer to the instance, this will be the path on your local computer where the file is located. A path we shall symbolically name <instance file path> is a path on your instance. If you are copying a file from your local computer to the instance, this is the path to the folder where you want to copy the file to. If you are copying a file from the instance to your local computer, this is the path where the file you want to copy is located on your instance. If you have another username than ubuntu, replace ubuntu in the following commands with your user name.

93 To copy a file to your instance, go to your windows command prompt and type: pscp.exe -i <path-to-ssh-key> <local-file-path> [email protected]:<instance-file-path> replacing the N s with your IP address. You need to specify the path where you have saved your nectar ssh key in <path to ssh key>. Note that if you are using Pageant (as described in the Appendix), you can skip the option i <path to ssh key> in all pscp commands. For example, to copy the local file c:\documents\foo.txt to your instance and save it there in the folder /data, if your private key is saved in c:\documents\nectar_key.ppk, you would type: pscp -i c:\documents\nectar_key.ppk c:\documents\foo.txt [email protected]:/data Note that the local path to c:\documents\foo.txt has to be given in Windows syntax, using backslash \ instead of forward slash /. The <instance filepath> on your instance however uses the Linux notation with the forward slash / separator. Note: If you leave <instance file path> empty, it will copy your file into your ubuntu users home directory. To copy a file from your instance to your local computer, you have to swap the order of the <local file path> and [email protected]:<instance file path> in the command above: pscp.exe -i <path-to-ssh-key> [email protected]:<instancefile-path> <local-file-path> Note: If you use. as <local file path>, it will copy your file into your current working directory, which is the folder your PuTTY terminal is currently operating from. For example, if you want to copy the file /data/foo.txt to your C:\documents folder on your local computer, and your private ssh key is located in C:\documents\Nectar_Key.ppk: pscp.exe -i C:\documents\Nectar_Key.ppk

94 C:\documents If the pscp command is not working, make sure you have no error in the paths to your ssh key or your OpenRC file. On Windows, white spaces are problematic. If you path has white spaces, put the whole path in quotation marks, e.g. D:\My Files\pscp.exe. Also, note that you have to use the backslash \ in windows, instead of the forward slash / on Linux systems. Linux or Mac OS X You can use the scp command to copy files across. The scp command takes two paths as arguments: A path we shall symbolically name <local file path> is a path on your local computer. If you copy a file from your instance to your local computer, this will be the path to the folder on your local computer where you want to copy the file to. If you copy a file from your local computer to the instance, this will be the path on your local computer where the file is located. A path we shall symbolically name <instance file path> is a path on your instance. If you are copying a file from your local computer to the instance, this is the path to the folder where you want to copy the file to. If you are copying a file from the instance to your local computer, this is the path where the file you want to copy is located on your instance. If you have another username than ubuntu, replace ubuntu in the following commands with your user name. To copy a file to your instance: scp -i ~/.ssh/nectar_key <local-file-path> [email protected]: <instance-file-path> replacing the N s with your IP address. Note: If you leave <instance file path> empty, it will copy your file into your ubuntu users home directory. For example, if you want to copy the file /home/username/foo.txt to your /data folder on the instance:

95 scp -i ~/.ssh/nectar_key /home/username/foo.txt To copy a file from your instance to your local computer, you have to swap the order of the <local file path> and [email protected]:<instance file path> in the command above: scp -i ~/.ssh/nectar_key [email protected]:<instance-filepath> <local-file-path> Note: If you use. as <local file path>, it will copy your file into your current working directory. For example, if you want to copy the file /data/foo.txt to your /home/username/ folder on your local computer: scp -i ~/.ssh/nectar_key [email protected]:/data/foo.txt /home/username/ Summary Well done!! You have completed the probably longest and hardest Module and are on your way to becoming an experienced Cloud user. This module has provided hands on instructions on how to launch an instance and connect to it. Two ways to connect have been presented: (a) with the command line (ssh) and (b) to a remote desktop. After having connected, we have gone through the first configurations necessary (e.g. updating the system) and then we went on to instructions on how to set up the storage so it can be accessed from the instance and remotely from the office computer. Topics have covered: Registering on the Dashboard Creating SSH Keys Launching an instance Connecting via ssh Installing and connecting to a remote desktop Running graphical interfaces via the SSH shell Installing a web server and creating a first simple website Managing storage Format and mount secondary ephemeral drive and volumes

96 Access the object store via the Dashboard and via other clients Copying files across using FTP or scp You may be interested in taking a look at the Appendix which provides some more information and instructions for some troubleshooting. You are now ready to go on with Module 8 which will discuss a few security related topics. Appendix A number of topics are covered in the appendices. Scroll through the document to find what you may be looking for. Create a new security group On the Dashboard: 1. Go to Access and Security 2. Create Security Group 3. Name it something meaningful, and add any description 4. Click Create Security Group. You will see the new group in the list now. 5. Click Manage rules at the right of your group. 6. At the top right of the page, click Add rule 7. You probably want an Incoming TCP rule. 8. If you want to allow access from any computer, type /0 in the CIDR The part before the slash / is the address for localhost and the part after the slash specifies the subnet mask you don t need to worry about the meaning in this tutorial, but look it up online, e.g. google, if you want to restrict access to computers from only a certain range of IP addresses, e.g. only computers from within a university. After you have created the security group, you can add it to the instance. Go to Compute > Instances and in the right hand drop down menu for your instance, select Edit Security Groups.

97 Troubleshooting Linux and Mac: Problems connecting Check which ports are open on your system: Using nmap to check with ports are open: On Linux: sudo apt-get install nmap On Mac OS X: brew install nmap The nmap command will show you which ports of the computer are open, i.e., there is a server or client using it. sudo nmap -st -O localhost For example, if you are tyring to connect to ssh, type the above commands into your local terminal, and you should see the line: 22/tcp open ssh Or, if you are trying to get VNC going, you can type the commands in your instances ssh terminal, and the output should show something like: 5900/tcp open vnc Using netstat (Linux): sudo netstat -tulpn Using nc (netcat) to check whether there is something listening on the other side: nc -v -w 5 NNN.NNN.NNN.NNN -z <port> ( v verbose, w timeout, z just scan for daemons) If you get WARNING: UNPROTECTED PRIVATE Permissions 0664 for '<yourkey>' are too open. It is required that your private key files are NOT accessible by others. This private key will be ignored. bad permissions: ignore key: <yourkey> Permission denied (publickey). Then you have changed the access rights to the key while copying it around. Fix this with: chmod go-rw <yourkey> Converting ssh keys from Linux for PuTTY If you have generated your ssh key on a Linux system and now want to use it in Putty,

98 you have to convert the keys first. If you have generated your key file with putty, you don t need to do this. To convert the key, follow these steps: 1. Start PuTTYGen. 2. Click on the menu Conversions and select Import key. 3. Select your private key and enter the passphrase. 4. Click on the button Save private key and save your key file. You may also want to simply create a separate set of keys for PuTTY (see this section). In this case, you have to add the new key as an authorized key on the instance. This is explained here in this Appendix. Windows: Using Pageant to handle your ssh key You may use a SSH authentication agent, which holds your private keys in memory, already decoded, so that you can use them without needing to type a passphrase. PuTTY also offers a tool called Pageant, which is the PuTTY SSH authentication agent. Download it on the PuTTY website. If you have not generated your ssh key with PuTTY in the first place, you will also need to convert it to the PuTTY format. You can do this using the tool PuTTYgen (you can find instructions on how to do this in this Appendix). To load your ssh key with Pageant: Step 1. Start up Pageant (by clicking on pageant.exe which you downloaded). In your system tray, you ll see the Pageant icon appear: A computer symbol with a hat on. Step 2. Right click the icon and select Add Key and select your private key (PPK) file. Enter your pass phrase. Now, your key is loaded in the Background and you can use FileZilla via the secure FTP protocol SFTP in the next step. Don t forget to close Pageant when you re done! Otherwise any user with access to your computer can easily gain access with your ssh key without having to type the ssh key passphrase. To close pageant, go to the system tray icon, right click and Exit pageant.

99 See also the pageant tutorial on digitalocean. Creating a new user account If you want to create a new user account on your VM, type in the following command in your ssh terminal: sudo adduser <username> replacing <username> with the username you would like to create. It will prompt you to create a password for this user, and type in their name and contact details. If you would like to just create the user without specifying password and details, use sudo useradd <username> instead. You then can set the user password with: sudo passwd <username> It is recommended you use a random password initially, and then give it to the user, with the strong recommendation of changing it (a user cannot set their own initial password themselves, hence this initial step is required). Users can change their own password with: passwd You have now created a user with minimal privileges. For example, the user cannot perform administration tasks which require the sudo command prefix. If you would like to add privileges for this user, you can add it to certain privileges groups, e.g. the sudo group: sudo adduser <username> sudo The user can then use the sudo command to perform administrative tasks. They will have to type in their own password to execute sudo commands (this is a security measure in case they left their keyboard unattended). To check the right groups the user is assigned to, type id <username>

100 Each user should have their own ssh keys. Ask them to create it, using their passphrase of choice, and then send you their public key. Add their public key to your VM by following instructions here in this appendix. Adding a new ssh key If you want to use a new key to connect to the instance, you have to add it to the file authorized_keys on the user s account. If you are adding another key for the ubuntu user, the <username> in the command below is ubuntu, otherwise it is the user name you want to add a key for. First, create the.ssh directory for the user, if it does not exist yet (if it exists, this command will return an error): sudo mkdir /home/<username>/.ssh Then, edit the file authorized_keys sudo nano /home/<username>/.ssh/authorized_keys Paste the contents of your public key as one line into this file. Tip: Copy the content of the public key by opening it with a text editor, marking all text and copying with CTRL+C. Then, go to your PuTTY Window which has the editor open for authorized_keys. Pasting the content will not work with CTRL+V on PuTTY, but instead, clicking the right or middle mouse button should paste the content. Similarly, if you want to remove a key, you have to delete the line for that key from the authorized_key file on your instance. Make sure that the directories and files have the correct ownership and access: sudo chown -R <username> /home/<username>/.ssh sudo chgrp -R <username> /home/<username>/.ssh sudo chmod 700 /home/<username>/.ssh sudo chmod 500 /home/<username>/.ssh/authorized_keys SSH Tunnelling explained SSH Tunnelling is best explained with an example. Imagine you re on a private

101 network with very conservative firewall rules which don t allow connections to a specific server. For example, blockedsite.com is not accessible from your network. You can of course ask your system administrator to free up this site for you, but you may have to wait for the approval. To get around this firewall rule we can redirect our connection via a port that does not block this website: We can create a tunnel through a server which is not on our network and which can access blockedsite.com. We call this server the gateway. Let s say blockedsite.com uses the http protocol (as most websites do) through Port 80. Instead of connecting our port 80 to the blocksite.com s port 80 (which would not work), we will redirect all traffic from our port 80 through the secure ssh port 22. This one will not be blocked by the firewall, and nobody will be able to decipher the traffic going through it (unless they get access to your private key and passphrase of course). On the gateway, our connection will come in at Port 22, and the ssh server will then redirect the traffic to port 80 on blockedsite.com. We call this technique local port forwarding. Of course, the connection between the gateway and the server of blockedsite.com is not encrypted; only the tunnel between our computer and the gateway is. The technique of local port forwarding through an ssh tunnel can also be used to create a secure connection all the way to the destination server, so that there is no point at which the connection is not encrypted. In this case, the gateway is also the destination server itself. As an example, we are going to discuss the setup for ssh tunneling on port 5900 on our instance (the default VNC port), using the gateway localhost. The example is for Mac and Linux. Observe that the gateway is localhost, because when we specify the destination server, it is also the gateway in this case (this can be a bit confusing, because you would guess we specify the gateway first but actually we first specify the destination server, and then the gateway through which we want to create the tunnel. From the gateway s point of view, the destination server is localhost). The tunnelled service will be available on our localhost, port 6000 (we use different numbers so you can distinguish it in the text below). We could do this for any other port numbers.

102 1. On your machine, create a tunnel from your machine (port 5900) to the other machine (port 5900) ssh -i <your-key-name> -L 6000:localhost:5900 -N -f -l ubuntu NNN.NNN.NNN.NNN replacing NNN.NNN.NNN.NNN with the IP address of the instance. 2. On the instance: Install and start any service that will listen on port 5900 (e.g. the VNC Server). 3. A quick test should show you that you cannot actually directly connect to port 5900 on the instance: telnet This will try connect you to the remote port (without any useful application, but it s a good way to test whether the port is open). The shell should just say trying to connect but nothing happens: Because we have not allowed access to port 5900 from outside only via the SSH tunnel! And the ssh tunnel is accessible via our localhost. End the connect trying with Ctrl+C. Instead, try: telnet localhost 6000 or nc -v -w 5 localhost -z 6000 And you should see a command prompt (no error). So now you are talking to the port and can specify commands specific to the server, but we don t really want to do this now. To quit the telnet command prompt: Hit Ctrl+] (control and the right bracket). That will return you to a telnet command prompt. From there you can enter the quit command (typing q is sufficient) and you will be returned to your usual Windows command prompt. If this still does not work, make sure there is a server listening on port Check this with: sudo netstat -tulpn There should be your service listed as Listening on the port you have meant it to provide its service. If your service is not listening on that port, it won t be possible to connect to it either. The ssh L options are explained as follows. Syntax: L <local port to listen>:<remote host>:<remote port> <gateway> N : Do not execute a remote command i.e. just forward ports. f: Requests ssh to go to background just before command execution. Once

103 password supplied it will go to background and you can use prompt for type commands on local system. l ubuntu: ubuntu is the user to log in as on the remote machine (sshserver.mydomain.com). Note: The f N options in the above command cause the tunnel to run in the background: once you have typed in the password, the command line prompt returns to your local shell. In this case, if you ever want to close your ssh connection, you have to kill the connection: ps aux grep ssh Then, find your ssh tunneling command and note the number on the left, after your user name. To kill (close) the ssh tunnel, type: kill <the-ssh-tunnel-process-number> If you don t use the N f options, you will be logged into an ssh terminal to your instance, and can use it as usual to type commands on your instance. If you type logout or close the terminal, the ssh tunnel will close as well. This is the end of Module 7. Continue with Module 8