place for practical guides on cloud computing BUILDING A HYBRID CLOUD WITH OpenNebula OpenNebula connecting with Private cloud [KVM hypervisor]and Public cloud [Amazon EC2] Table of Contents Hardware/software requirements :...2 Setup OpenNebula Host...2 Setup VM Host...3 Configure the OpenNebula host:...4 Configure the VM host:...5 Install OpenNebula in Ubuntu server [OpenNebula host]:...6 Prepare Amazon EC2 :...9 Install EC2 API Library and tools in OpenNebula and configure OpenNebula...9 Create EC2 Host and VM in OpenNebula...12 Connecting to your new EC2 instance :...13 Creating a instance in Private cloud:...14 Terminate an EC2 instance through OpenNebula...16 ANIL KUMAR A N Chennai cloud.b.lab@zoho.com admin@cloud-b-lab.co.in
Hardware/software requirements : Hybrid cloud setup with OpenNebula OpenNebula Frontend [OneHost] VM Host [nodekvm] Network setup A VT enabled hardware with Ubuntu 11.04 or 11.10 server installed Hostname : OneHost IP address [eth0] : 192.168.1.104 Gateway : 192.168.1.1 username : localadmin A VT enabled hardware with Ubuntu 11.04 or 11.10 server installed Hostname : nodekvm IP address[br0] : 192.168.1.105 Gateway : 192.168.1.1 username : localadmin A router with minimum two Ethernet ports and internet connectivity Setup OpenNebula Host Before installation of OpenNebula in Ubuntu Server, just check the following to ensure that you have a proper setup. /etc/network/interfaces # The loopback network interface auto lo iface lo inet loopback # The primary network interface auto eth0 iface eth0 inet static address 192.168.1.104 netmask 255.255.255.0 network 192.168.1.0 broadcast 192.168.1.255 gateway 192.168.1.1 # dns-* options are implemented by the resolvconf package, if installed dns-nameservers 192.168.1.1 dns-search example.com /etc/hosts 127.0.0.1 localhost 192.168.1.104 OneHost.example.com OneHost /etc/resolv.conf search example.com nameserver 192.168.1.1 Setup VM Host /etc/network/interfaces # The loopback network interface auto lo iface lo inet loopback
# The primary network interface auto eth0 iface eth0 inet manual auto br0 iface br0 inet static address 192.168.1.105 netmask 255.255.255.0 network 192.168.1.0 broadcast 192.168.1.255 gateway 192.168.1.1 # dns-* options are implemented by the resolvconf package, if installed dns-search westel.com bridge_ports eth0 bridge_fd 9 bridge_hello 2 bridge_maxage 12 bridge_stp off
Configure the OpenNebula host: Create a folder "cloud" and create a group named "cloud" sudo mkdir -p /srv/cloud/ sudo groupadd -g 10000 cloud Create a user "oneadmin", add user to group "cloud" and have /srv/cloud/one as home folder. sudo useradd -u 10000 -m oneadmin -d /srv/cloud/one -s /bin/bash -g cloud Setup password for "oneadmin" and make oneadmin owner of "/srv/cloud" sudo passwd oneadmin sudo chown -R oneadmin:cloud /srv/cloud/ - I chose redhat123 as the password Test by logging as user "oneadmin" and exit su -l oneadmin exit Install Network file Server [NFS] Not a must for EC2 integration sudo apt-get install nfs-kernel-server edit /etc/exports and add the following line to make folder /srv/cloud/one/var shareable with ESXi server. We will configure ESXi server later. /srv/cloud/one 192.168.1.105(rw,fsid=0,nohide,sync,root_squash,no_subtree_check) Restart NFS server sudo /etc/init.d/nfs-kernel-server start create a SSH key for oneadmin and disable host key checking else make all hostkeys known on the OpenNebula node. su -l oneadmin ssh-keygen {Note - all defaults, and no passphrase.} cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys nano ~/.ssh/config [add below two lines to SSH config file] Host * StrictHostKeyChecking no exit
Configure the VM host: The VM hosts need to be able to access the home directory of the OpenNebula user. Therefore install NFS common sudo apt-get install nfs-common Add an NFS entry for /srv/cloud/one in /etc/fstab (change 192.169.1.104 to the local IP of your OpenNebula host): sudo nano /etc/fstab 192.168.1.104:/srv/cloud/one /srv/cloud/one nfs defaults 0 0 Now create and mount the directory: sudo mkdir -p /srv/cloud/one sudo mount /srv/cloud/one Create oneadmin user: sudo groupadd -g 10000 cloud sudo useradd -u 10000 -g cloud -m oneadmin -s /bin/bash sudo usermod -d /srv/cloud/one oneadmin sudo passwd oneadmin sudo chown oneadmin:cloud /srv/cloud/ Install packages for the KVM hypervisor sudo apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils ruby Configure libvirt to allow users from the group cloud to manage virtual machines sudo nano /etc/libvirt/libvirtd.conf unix_sock_group = "cloud" Configure VNC to be accessible from the outside. For a production environment, VNC should be secured by TLS certificates (see below): sudo vi /etc/libvirt/qemu.conf vnc_listen = "0.0.0.0" sudo service libvirt-bin restart Change group ownership for /var/run/libvirt/libvirt-sock to allow access by users of the cloud group sudo chown :cloud /var/run/libvirt/libvirt-sock
Install OpenNebula in Ubuntu server [OpenNebula host]: Login to 192.168.1.98[OneHost] and download OpenNebula Release 3.2.1 su -l oneadmin Download stable opennebula release for ubuntu [opennebula-3.2.1.tar.gz] from http://downloads.opennebula.org/ and save it in /srv/cloud/one folder. Also add oneadmin to sudoers file with permissions same as root, Un-tar the build tar xzf opennebula-3.2.1.tar.gz cd opennebula-3.2.1// Before installing OpenNebula, install all pre-requisite packages sudo apt-get install libsqlite3-dev libxmlrpc-c3-dev g++ ruby libopenssl-ruby libssl-dev ruby-dev sudo apt-get install libxml2-dev libmysqlclient-dev libmysql++-dev libsqlite3-ruby libexpat1-dev sudo apt-get install libc6 libgcc1 libmysqlclient16 libpassword-ruby libsequel-ruby libsqlite3-0 libssl0.9.8 libstdc++6 libxml2 libxmlrpc-c3-0 libxmlrpc-core-c3-0 sudo apt-get install ruby rubygems libmysql-ruby libsqlite3-ruby libamazonec2-ruby sudo apt-get install rake rubygems libxml-parser-ruby1.8 libxslt1-dev genisoimage scons sudo gem install nokogiri rake xmlparser sudo apt-get install opennebula-common [optional] sudo apt-get install mysql-server [ set the password when asked. I normally give mygreatsecret as the pwd] configure MYSql: <refer below screen shot in case of any doubt) mysql -uroot -pmygreatsecret CREATE USER 'oneadmin'@'localhost' IDENTIFIED BY 'oneadmin'; CREATE DATABASE opennebula; GRANT ALL PRIVILEGES ON opennebula.* TO 'oneadmin' IDENTIFIED BY 'oneadmin'; quit; <Screen shot as shown below>
Before installing OpenNebula, configure mysql support as oneadmin user cd ~/opennebula-3.2.1 scons sqlite=no mysql=yes [change your folder to opennebula source] Install openebula in /srv/cloud/one accessible by group cloud and as user "oneadmin"./install.sh -u oneadmin -g cloud -d /srv/cloud/one Create a profile file[~/.bash_profile] to set ENVIRONMENT VARIABLES required to start and use services rendered by "one" nano ~/.bash_profile export ONE_LOCATION=/srv/cloud/one export ONE_AUTH=$ONE_LOCATION/.one/one_auth export ONE_XMLRPC=http://localhost:2633/RPC2 export PATH=$ONE_LOCATION/bin:/usr/local/bin:/var/lib/gems/1.8/bin/:/var/lib/gems/1.8/:$PATH execute the profile file and set the environment variables source ~/.bash_profile [Note: Anytime you open a new SSH window for OneHost, change user to oneadmin and source ~/.bash_profile before issuing any one command] Create and store OpenNebula user and password in a file. Substitute <THE_PASSWORD> with value mkdir ~/.one echo "oneadmin:<the_password>" > ~/.one/one_auth Make required changes in OpenNebula configuration file ~/etc/oned.conf nano ~/etc/oned.conf a. comment following line # Line 58 or near by [c hange if your password for oneadmin is some different] #DB = [ backend = "sqlite" ] b. Set SQL as MYSQL-uncomment #lines 61 through 66 or near by DB = [ backend = "mysql", server = "localhost", port = 0, user = "oneadmin", passwd = "oneadmin", db_name = "opennebula" ]
Start OpenNebula one start { Note: it should start with no error messages} Now You can test OpenNebula services by typing onevm list on the $ prompt. The command should execute with no error.the list will be empty for now.
Prepare Amazon EC2 : Create an Amazon account Go to http://aws.amazon.com, and click Create an AWS Account. Follow the on-screen instructions. Part of the sign-up procedure involves receiving a phone call and entering a PIN using the phone keypad. Refer: http://docs.amazonwebservices.com/awsec2/latest/gettingstartedguide/signup.html Create credentials make a folder ec2 with in /srv/cloud/one/ After signing up, you should end up at the EC2 console Create a key pair and download the private key Click Key Pairs under Networking and Security in the Navigation pane and then click the Create Key Pair button (save it in e.g. ~/.ec2/ec2.pem). This private key is for making SSH connections to newly created instances. You will also need to set up your Amazon API credentials. Go to Account->Security Credentials click X.509 Certificates tab Create a new Certificate Download the private key and the certificate (save them in e.g. ~/.ec2/cert- XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem and ~/.ec2/pk- XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem). Never share these files. Make your credential files private: chmod go-rwx ~/.ec2/*.pem 5. Scroll to the bottom of the page and note your account ID (a number of the form XXXX-XXXX- XXXX). Create a Ubuntu instance in EC2 [http://docs.amazonwebservices.com/awsec2/latest/gettingstartedguide/launchinstance.html] Note the AMI- ID,as we need to specify it in onevm template later Create an Elastic IP using NETWORK and SECURITY section and note the IP. Install EC2 API Library and tools in OpenNebula and configure OpenNebula Install EC2 API tools in Openenbula frontend machine Login to opennebula Front-end machine [onehost] as root user and perform the following steps apt-get update Install the Amazon EC2 Query API library: gem install amazon-ec2 Install the Sinatra web framework and the thin web server: [if you get Invalid gemspec in [/var/lib/gems/1.8/specifications/tilt-1.3.3.gemspec]: invalid date format in specification: "2011-08-25 00:00:00.000000000Z"
Type below on the command prompt as root user: sudo sed -i 's/ 00:00:00.000000000Z//' /var/lib/gems/1.8/specifications/* ] gem install sinatra gem install thin gem install uuid gem install curb apt-get install libsqlite3-ruby apt-get install libcurl4-gnutls-dev apt-get install openssl libruby1.8 gem install sqlite3-ruby apt-get install openjdk-6-jdk The service is configured through the $ONE_LOCATION/etc/econe.conf file Edit $ONE_LOCATION/etc/econe.conf and update the following :SERVER=<IP of onehost/opennebula frontend> :PORT=4567 Check if multiverse is enabled in Ubuntu 11:10 server sudo nano /etc/apt/sources.list if commented, uncomment the following deb http://in.archive.ubuntu.com/ubuntu/ oneiric multiverse deb-src http://in.archive.ubuntu.com/ubuntu/ oneiric multiverse deb http://in.archive.ubuntu.com/ubuntu/ oneiric-updates multiverse deb-src http://in.archive.ubuntu.com/ubuntu/ oneiric-updates multiverse Login as oneadmin user Make sure you have the following environment variables set up in your shell profile. This is accomplished by adding the following lines to your ~/.bash_profile : sudo nano ~/.bash_profile export EC2_KEYPAIR=<your keypair name> # name only, not the file name export EC2_URL=https://ec2.<your region>.amazonaws.com # example: ec2.us-east-1.amazonaws.com export EC2_PRIVATE_KEY=<PATH>/<where your private key is>/pk-xxxxxxxxxxxxxxxxxxxxxxxxxxxx.pem # The value of the EC2_PRIVATE_KEY environment variable export EC2_CERT=<PATH>/<where your certificate is>/cert-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.pem # The X.509 certificate to use when constructing requests to Amazon EC2 export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/ Load the changes into the current shell environment: source ~/.bash_profile Check to see if it's working by running the following command: ec2-describe-images -o self -o amazon If above step fails due to "Client.AuthFailure" then ensure you have signed up for both AWS and ec2 with amazon.com, and have provided valid payment details. Also double check that the EC2_PRIVATE_KEY and EC2_CERT point to the correct
locations Create a AMI using a running/stopped EC2 instance ec2-create-image <instance id> --name <name for new instance> -K <EC2_PRIVATE_KEY> -C <EC2_CERT> -v -U <EC2_URL> example: I am using the already sourced environment values: ec2-create-image i-5ccd593b --name ubuntu1110x64 -K $EC2_PRIVATE_KEY -C $EC2_CERT -v -U $EC2_URL Edit File: /srv/cloud/one/etc/vmm_ec2/vmm_ec2rc and add below lines nano /srv/cloud/one/etc/vmm_ec2/vmm_ec2rc EC2_HOME=/usr # find the installed location of ec2-api-tools and provide the path here. For me it is /usr {/bin will appended by the program itself } EC2_URL=https://ec2.us-east-1.amazonaws.com EC2_PRIVATE_KEY=<PATH>/<where your private key is>/pk-xxxxxxxxxxxxxxxxxxxxxxxxxxxx.pem EC2_CERT=<PATH>/<where your certificate is>/cert-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.pem JAVA_HOME=/usr/lib/jvm/java-6-openjdk/ Edit File: /srv/cloud/one/etc/oned.conf and make the following changes # KVM Information Driver Manager Configuration IM_MAD = [ name = "im_kvm", executable = "one_im_ssh", arguments = "-r 0 -t 15 kvm" ] # EC2 Information Driver Manager Configuration IM_MAD = [ name = "im_ec2", executable = "one_im_ec2", arguments = "im_ec2/im_ec2.conf" ] # KVM Virtualization Driver Manager Configuration VM_MAD = [ name = "vmm_kvm", executable = "one_vmm_exec", arguments = "-t 15 -r 0 kvm", default = "vmm_exec/vmm_exec_kvm.conf", type = "kvm" ] # EC2 Virtualization Driver Manager Configuration VM_MAD = [ name = "vmm_ec2", executable = "one_vmm_ec2", arguments = "-u https://ec2.us-east-1.amazonaws.com -k <PATH>/<where your private key is>/pk- XXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem -c <PATH>/<where your certificate is>/cert-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.pem -h <path where ec2-api tools are installed> vmm_ec2/vmm_ec2.conf", type = "xml" ] # Dummy Transfer Manager Driver Configuration
TM_MAD = [ name = "tm_dummy", executable = "one_tm", arguments = "tm_dummy/tm_dummy.conf" ] # SHARED Transfer Manager Driver Configuration TM_MAD = [ name = "tm_nfs", executable = "one_tm", arguments = "tm_shared/tm_shared.conf" ] Create EC2 Host and VM in OpenNebula Start one server as oneadmin user source ~/.bash_profile one start Add EC2 host to OpenNebula onehost create ec2 im_ec2 vmm_ec2 tm_dummy dummy You will get an output like ID NAME RVM TCPU FCPU ACPU TMEM FMEM AMEM STAT 0 ec2 0 500 500 500 8.5G 8.5G 8.5G on Create a OpenNebula template file to launch an Instance in Amazon EC2 nano ~/ec2.one Add the following content to ec2.one, save and exit CPU = 0.5 MEMORY = 128 # KVM template - this will be usedwhen submitting this VM to local resource[private cloud] # NAME = ttylinux #DISK = [ source = "/srv/cloud/one/ec2/ttylinux.img", target = "hda", readonly = "no" ] #NIC = [ NETWORK = "Small network" ] #FEATURES=[ acpi="no" ] #EC2 template to submit a VM to EC2 [public cloud] NAME = EC2 EC2 = [ AMI="<ami id of ubuntu instance>", KEYPAIR="<Name of keypair>", ELASTICIP="<Generated Elastic IP>", AUTHORIZED_PORTS="22", INSTANCETYPE=m1.small] #Add this if you want to use only EC2 cloud #REQUIREMENTS = 'NAME = "ec2"'
Launch the Instance in Amazon EC2 onevm create ec2.one Monitor the log files in ~/var/oned.log and ~/var/<vmid>/vm.log oned.log: Mon Apr 16 01:03:51 2012 [DiM][I]: New VM state is ACTIVE. Mon Apr 16 01:03:51 2012 [LCM][I]: New VM state is PROLOG. Mon Apr 16 01:03:51 2012 [VM][I]: Virtual Machine has no context Mon Apr 16 01:03:51 2012 [DiM][D]: Deploying VM 1 Mon Apr 16 01:03:51 2012 [LCM][I]: New VM state is BOOT Mon Apr 16 01:14:16 2012 [VMM][D]: Message received: LOG I 1 ExitCode: 0 Mon Apr 16 01:14:10 2012 [VMM][I]: Monitoring VM 1. Mon Apr 16 01:14:16 2012 [VMM][D]: Message received: POLL SUCCESS 1 USEDMEMORY=0 USEDCPU=0 NETTX=0 NETRX=0 STATE=a IP=ec2-23-23-200-8.compute-1.amazonaws.com vm.log: Mon Apr 16 01:03:51 2012 [TM][D]: Message received: TRANSFER SUCCESS 1 - Mon Apr 16 01:04:04 2012 [VMM][D]: Message received: LOG I 1 ExitCode: 0 Mon Apr 16 01:03:51 2012 [VMM][I]: Generating deployment file: /srv/cloud/one/var/1/deployment.0 Mon Apr 16 01:04:04 2012 [VMM][D]: Message received: DEPLOY SUCCESS 1 i-5ab7263d Mon Apr 16 01:03:58 2012 [VMM][I]: ExitCode: 0 Mon Apr 16 01:04:04 2012 [VMM][I]: ExitCode: 0 Mon Apr 16 01:04:04 2012 [LCM][I]: New VM state is RUNNING Mon Apr 16 01:14:16 2012 [VMM][W]: Adding custom monitoring attribute: IP ec2-23-23-200-8.compute-1.amazonaws.com Mon Apr 16 01:14:16 2012 [VMM][D]: Monitor Information: Just note the monitoring attribute: IP {ec2-23-23-200-8.compute-1.amazonaws.com }from the log, [You can get it from EC2 console also]. We need it to SSH to the instance On successful creation you will get following output onevm list ID USER GROUP NAME STAT CPU MEM HOSTNAME TIME 4 oneadmin oneadmin one-1 runn 0 0K ec2 00 00:02:21 Login in AMAZON EC2 management console and check the status of the instance. It should be running in awhile Connecting to your new EC2 instance : In order to log in to your instance, you will need to authorize network access to the ssh port for the default security group: {if you have created a different security group and already added SSH to it, just skip below
step} ec2-authorize default -p 22 You may then log in to the instance using ssh: ssh -i <private SSH key file> ubuntu@<external-host-name> example : ssh -i ~/.ec2/ec2.pem ubuntu@ec2-23-23-200-8.compute-1.amazonaws.com The <private SSH key file> is the filename of the private SSH key that corresponds to the Amazon Key Pair that you specified in the ec2-run-instances command. The <external-host-name> can be found using the ec2-describe-instances command. An example SSH command: Once you have logged in, you may begin to set up and use the instance just like any other Ubuntu machine. Creating a instance in Private cloud: ADD a KVM host to OpenNebula onehost create nodekvm im_kvm vmm_kvm tm_nfs dummy onehost list [you will see both EC2 host and the newly added nodekvm] ID NAME RVM TCPU FCPU ACPU TMEM FMEM AMEM STAT 0 ec2 1 500 500 450 8.5G 8.5G 8.3G on 3 nodekvm 1 800 800 750 7.8G 7.6G 7.6G on Create virtual network using a template SmallNetwork.net Sudo nano Small network.net NAME = "Small network" TYPE = FIXED #Now we'll use the cluster private network (physical) BRIDGE =br0 LEASES = [ IP="192.168.1.50"] LEASES = [ IP="192.168.1.60"] Create a onevet onevnet create SmallNetwork.net Download ttylinux.img from opennebula.org and untar it in ~/ec2 folder $ wget http://dev.opennebula.org/attachments/download/170/ttylinux.tar.gz $ tar xvzf ttylinux.tar.gz
Uncomment the KVM Template portion in EC2.one to create an instance in nodekvm #EC2 template machine, this will be use wen submitting this VM to EC2 CPU = 0.5 MEMORY = 256 NAME = ttylinux DISK = [ source = "/srv/cloud/one/ec2/ttylinux.img", target = "hda", readonly = "no" ] NIC = [ NETWORK = "Small network" ] FEATURES=[ acpi="no" ] #EC2 template to submit a VM to EC2 [public cloud] NAME = EC2 EC2 = [ AMI="<ami id of ubuntu instance>", KEYPAIR="<Name of keypair>", ELASTICIP="<Generated Elastic IP>", AUTHORIZED_PORTS="22", INSTANCETYPE=m1.small] #Add this if you want to use only EC2 cloud #REQUIREMENTS = 'NAME = "ec2"' Create an instance in nodekvm onevm create ec2.one List the running VMs onevm list oneadmin@onehost:~/ec2$ onevm list ID USER GROUP NAME STAT CPU MEM HOSTNAME TIME 4 oneadmin oneadmin one-1 runn 0 0K ec2 00 00:02:21 5 oneadmin oneadmin ttylinux runn 3 256M nodekvm 00 01:53:31 Get IP details of the running VMs oneadmin@onehost:~/ec2$ onevm show 4 grep IP ELASTICIP=23.23.200.8, IP=ec2-23-23-200-8.compute-1.amazonaws.com oneadmin@onehost:~/ec2$ oneadmin@onehost:~/ec2$ onevm show 5 grep IP ELASTICIP=23.23.200.8, IP=192.168.1.50, Connect to both EC2 and ttylinux instances ssh -i keypair.pem ubuntu@ec2-23-23-200-8.compute-1.amazonaws.com
oneadmin@onehost:~/ec2$ ssh -i cloudblab1.pem ubuntu@ec2-23-23-200-8.compute-1.amazonaws.com Welcome to Ubuntu 11.10 (GNU/Linux 3.0.0-16-virtual x86_64) * Documentation: https://help.ubuntu.com/ System information as of Mon Apr 16 10:13:04 UTC 2012 System load: 0.0 Processes: 54 Usage of /: 13.5% of 7.87GB Users logged in: 0 Memory usage: 2% IP address for eth0: 10.60.9.106 Swap usage: 0% Graph this data and manage this system at https://landscape.canonical.com/ Get cloud support with Ubuntu Advantage Cloud Guest http://www.ubuntu.com/business/services/cloud ubuntu@ip-10-60-9-106:~$ logout Connection to ec2-23-23-200-8.compute-1.amazonaws.com closed. ssh root@ 192.168.1.50 oneadmin@onehost:~/ec2$ ssh root@192.168.1.50 root@192.168.1.50's password: Chop wood, carry water. # Use "logout" to leave the shell. # Use "logout" to leave the shell. # exit logout Connection to 192.168.1.50 closed. onevm shutdown <vmid> Terminate an EC2 instance through OpenNebula That's it. You have successfully implemented a hybrid cloud using OpenNebula. Refer my other tutorials and try Contextualization, running a windows VM etc If you liked this tutorial, just post your comments to ANIL KUMAR A N cloud.b.lab@zoho.com admin@cloud-b-lab.co.in