Virtual Machine Dynamic Provision and Image Management for UniCloud Pangfeng Liu April 10, 2010 1 Related work 1.1 Virtual Machine 1.1.1 Application performance isolation in virtualization Performance isolation is that performance of one VM should not affect performance of another VM in the same computer, which application providers concerned about because there are several kinds of applications in different virtual machine on a same physical machine. Therefore, Somani and Chaudhary presented the good isolation with credit scheduler of Xen, and the other scheduler called SEDF has shown the better performance than credit scheduler in [14]. 1.1.2 Live Migration of Virtual Machines Clark et al. compared original migration (stop-and-copy) with live migration (pre-copy), and observed performance under VM migration with bandwidth and the number of pre-copy iterations. The result in [9]is that pre-copy is always better than stop-and-copy. In addition, more pre-copy iterations and more bandwidth can effective reduce downtime. In these year, xen and vmware has supported the live migration. 1.1.3 Post-Copy Based Live Virtual Machine Migration Using Adaptive Pre-Paging and Dynamic Self-Ballooning Post-Copy is a new method of migration compared with Pre-Copy. Post-Copy migration transfer VM s processor state to target host first, and then send memory contents [10]. As the result, all the memory content will be transferred at most once, which Pre-Copy migration scheme will iterative transfer the dirty pages. As soon as processor state arrives, then dealing the page faults through network with Pre-Paging strategy. Furthermore, Hines and Gopalan eliminate the transfer of unallocated memory pages through Dynamic Self-Ballooning. 1
1.2 Cloud Infrastructure and Management 1.2.1 Memory Buddies: Exploiting Page Sharing for Smart Colocation in Virtualized Data Centers Memory Buddies [15] is a memory sharing-aware placement system for virtual machines. The goal is to maximize the number of VMs in a data center. It uses the VMware ESX [5] technique content-based page sharing to share pages from different VMs in the same host. The algorithm uses fingerprint to predict what kind of combination saving the most memory pages. 1.2.2 Virtual Organization Cluster Murphy et al. proposed the idea of using one cluster to simulate several virtual clusters to support virtual organizations [11]. The virtual cluster for individual virtual organization is called virtual organization cluster. Every virtual organization provides VM images they needed. Cluster administrators run a virtual cluster supporting each virtual organization. 1.2.3 Dynamic Provisioning of Virtual Organization Clusters Based on virtual organization cluster [11], Murphy et al. further refine the virtue machine framework so that it can dynamically adjust the number of VMs to support virtual organization [12]. They use condor system and maintain a job queue for system to detect the load of each virtual organization. Using this information, they can decide VM numbers for different virtual organization in different situation. 1.2.4 An Efficient Resource Management System For On-line Virtual Cluster Provision Chen et al. studied how to reduce image preparation overhead by VM image caching and multicasting VM image to physical machines [8]. Also the Load-Aware Mapping,a resource mapping strategy, can further reduce deploying overhead by awaring VM image cache information and make resource used efficiently. 1.2.5 Dynamic Provisioning of Resources in Data Centers Simmons et al. proposed the idea of how to allocate the resources in the data center [13]. In this paper, it describes some policies that we should concern to make th balance between the datacenter and the customers. Finally, decisions are based on the model which combines all the policies. 2
1.3 Tools 1.3.1 Xen Xen [7] is an open source virtual machine monitor. It can execute sereval guest OS on the same machine. Xen systems include Xen hypervisor as the lowest and most privileged layer to schedule different guest OS across the physical CPUs. It needs modified Unix-like OS to be domain 0 and domain U. 1.3.2 Hadoop Hadoop [2] is a open source software framework which support data-intensive distributed applications. Its capable for thousands of nodes and petabytes of data for applications. Hadoop also include a distribute file system inspired by Googles GFS and the framework is by Googles MapReduce. 1.3.3 MPI MPI [3], which stand for message passing interface, allows many computers communicate with one another. MPI specify a communication protocol for parallel computing. It is a message-passing application interface for programmers. MPI is common used in high-performance computing. 1.3.4 GlusterFS GlusterFS is a application level distributed share file system [1]. it has client and server components. Server side store the data, And client side using TCP/IP to access data on servers. GlusterFS has two modes to store data. Those modes are file-based striping and file-based mirroring,respectively. 1.3.5 OpenNebula OpenNebula can provide an interface to manage vitualinfrastructure including storage, network [4]. And it also provide virtualization technologies to enable migration and deploying VM on distributed system. 1.3.6 XCP- Xen Cloud Platfrom XCP [6]is a platfrom for the cloud enviroment. It collects several tools in the previous products, and combine the xen machines to a resource pool. Therefore, it can monitor VMs, do load balancing and migrate automatically the pool. 2 progress report 2.1 Xen We install xen 4.0/3.2 [7] on our machines. We have 8 machines running Debian system. First, we install xen 3.2 but there exist a problem. The problem is that 3
there are some machines can not run VM. These machines is grid01,grid02,grid04 and grid05. Now we test xen 4.0 on grid04 and grid05. It works! Although xen 4.0 is unstable, it fixes many problems in xen 3.2. Now, we describe how we install xen 4.0/3.2 and how to create a VM image. 2.1.1 Install Xen 3.2 We use debian packages to install xen 3.2. We just type apt-get install to install packages. It needs packages xen-linux-system-2.6.26-2-xen-686,libc6-xen and bridge-utils. Make sure that menu.lst is correct. It must boot by xen hypervisor. And modify /etc/xen/xend-config.sxp file if need. For example, if you want to do live migration, the line #(xend-relocation-hosts-allow ) must deletes the # sign. 2.1.2 Install Xen 4.0/4.1 unstable version Type hg clone http://xenbits.xen.org/xen-unstable.hgs to download the source code. Enter the directory. Type make xen; make tools; make install-xen; make install-tools. At this step, you may need to install some packages like bin86,dev86,etc. Go to http://git.kernel.org/ to download modified kernel supporting xen. Compile new kernel for booting. Modify menu.lst and xendconfig.sxp. Boot in new kernel. 2.1.3 Create VM images First, we use dd to create two images, one for disk and one for swap. And we use mkfs.ext3 and mkswap commands to format this two images. In order to install a basic Debian system in our image, we mount the image and run debootstrap. For example, we type mount -o loop disk /mnt; debootstrap arch i386 lenny /mnt http://ftp2.de.debian.org/debian. In this step, debootstrap will ask us to set up root password. Next, we set up our VM environment like network, fstab, hostname, etc. Modify VM configure file. There is an example: 4
2.2 OpenNEbula 2.2.1 description kernel = /boot/vmlinuz-2.6.26-2-xen-686 ramdisk = /boot/initrd.img-2.6.26-2-xen-686 memory = 512 root = /dev/hda1 ro disk = [ file:/testvm/swap.img,hda2,w, file:/testvm/disk.img,hda1,w, ] name = testvm1 vif = [ ] on poweroff = destroy on reboot = restart on crash = restart extra = console=hvc0 xencons=tty we use OpenNebula(ONE) [4] v1.4 to build private cloud on ubuntu 8.04. 2.2.2 before-install We need install ruby, sqlite3, xmlrpc-c, openssl, ssh, scons, g++, flex, blson on ONE server and Install ruby on all cluster nodes. All cluster nodes should have ssh server running and hypervisor working properly configured and ssh-key exchange to ONE server. Build a share file system which can be accessed by all nodes in order to live migration VM. in our case we use NFS or GlusterFS [1] to build share file system. 2.2.3 install Download ONE and install it on share file system. 2.2.4 configure put all VM images on share filesystem. configure etc/one.conf as follow(let ONE use share image mode) TM MAD = [ name = tm nfs, executable = one tm, arguments = tm ssh/tm nfs.conf, default = tm ssh/tm nfs.conf ] add cluster nodes to ONE server. build a VM configure file for example. 5
NAME = xen hdp CPU = 1 MEMORY = 2048 OS = [kernel=/boot/vmlinuz-2.6.24-27-xen,initrd=/boot/initrd.img-2.6.24-27-xen,root=hda2 ] DISK = [source= /srv/cloud/xen hdp/disk.img,target=hda2,readonly= no ] DISK = [source= /srv/cloud/xen hdp/swap.img,type =swap,size =512,target=hda1,readonly= no ] NIC = [ip= ] GRAPHICS = [type= vnc,listen= 127.0.0.1,port= 5900 ] we can use this VM configure file to create VM. And then we can depoly VM on spicific cluster node. Finally we can live-migrate VM between cluster nodes by ONE. 2.3 GlusterFS 2.3.1 description we use GlusterFS [1] to store our VM images. 2.3.2 before-install install sshfs, build-essential, flex, bison, byacc, fuse on all node. make directory /export/sdb1 on all server nodes. 2.3.3 install install glusterfs on all nodes. 2.3.4 configure Execute glusterfs-volgen to generate glusterfs configure file. For example make 192.168.50.70 and 192.168.48.35 2 node as server. glusterfs-volgen name test2 192.168.50.70:/export/sdb1 192.168.48.35:/export/sdb1; copy this configure file to all server node /etc/glusterfs/glusterfsd.vol. start glusterfs server on all server nodes. /etc/init.d/glusterfsd start; copy the configure file to client nodes and make directory for mounting. mounting glusterfs on specific directory without direct IO (direct IO is fast but xen can not boot sucessfuly if image is on direct IO mode filesystem). glusterfs disable-direct-io-mode -f /usr/src/glusterfs-2.0.9/test2- tcp.vol /mnt/glusterfs; 6
2.3.5 potential problem GlusterFS dose not synchronize clients cache data. VM images may not sync when VM migration. 2.4 Hadoop Performance We first consider how virtualization affects on hadoop performance, and then find opportunities of improve performance running hadoop under virtual machines. We setup a hadoop cluster with three nodes each on heterogeneous machine, say native cluster. Then build virtual machine on each computer to run virtual cluster, which using the same settings with native cluster, and we can obtain their performance. The following table describes our hardware. Intel Core 2 CPU E2180 @ 2.00GHz Processor Architecture Intel Core 2 CPU E2180 @ 2.00GHz AMD Athlon 64 Processor 3500+ 2.20GHz 4GB Memory 4GB 4GB 58GB Disk 63GB 500GB Network Connection 1G Host Os Ubuntu 8.04 Guest OS Ubuntu 8.04 Xen version Xen 3.2 Memory to Guest 512MB Disk (Guest) 2G No. of VCPUs to all OSs 2 We first run a RandomWrite generate 30maps, 10MB per maps and measure the job complete time. The virtual cluster has 71.52% performance compare with native cluster in average. Then we run Sort job to sort the random data generated before, which is 66.73% of native. The following are result data. RandomWrite sec Native VM 1 47 66 2 46 61 3 44 64 4 44 64 5 45 61 Avg 45.20 63.20 Performance 71.52% 7
Sort sec Native VM 1 77 133 2 74 95 3 77 109 4 72 120 5 75 105 Avg 75.00 112.40 Performance 66.73% 2.5 MPI 2.5.1 Benchmark Intel MPI Benchmarks 3.2 The IMB 3.2 package consists of 3 parts: IMB-MPI1 2 MPI-2 functionality parts IMB-EXT (One-sided Communications benchmarks), and IMB-IO (I/O benchmarks). http://software.intel.com/en-us/articles/intel-mpi-benchmarks 2.5.2 Machine The three real machines and VM configuration are as follows: 8
Intel Core 2 CPU E2180 @ 2.00GHz Processor Architecture Intel Core 2 CPU E6400 @ 2.13GHz AMD Athlon 64 Processor 3500+ 2.20GHz 4GB Memory 4GB 4GB 63GB Disk 20GB 500GB Network Connection 1G Host OS Ubuntu 8.04 Guest OS Ubuntu 8.04 Xen version Xen 3.2 Memory to Guest 512MB Disk (Guest) 2G No. of VCPUs to all OSs 2 2.5.3 Performance We set the number of MPI processes to run IMB be three and one virtual machine per real computer and put the image files on the local, nfs, and the gluster file systems [1]. IMB-MPI1 test1 test2 average local 127m 43.4950s 135m 46.8430s 131m 45.1690s nfs 162m 0.0770s 157m 18.9650s 159m 39.5210s glusterfs 159m 9.9650s 162m 19.1160s 160m 44.5405s The time taken to complete IMB-MPI1 for nfs and glusterfs is nearly same and both of them are greater than the time for local. IMB-IO test1 test2 average local 21m 22.9640s 30m 45.2870s 26m 4.1255s nfs 75m 39.0970s 80m 14.8100s 77m 56.9535s glusterfs 28m 9.9880s 27m 8.4790s 27m 48.7335s Unlike IMB-MPI1, the taken time of IMB-IO for glusterfs is close to the time for local, but the time for nfs is still longer than others. References [1] Gluster file system. http://www.gluster.org/. [2] Hadoop. http://hadoop.apache.org. 9
[3] Mpi. http://www.mcs.anl.gov/research/projects/mpi. [4] Opennebula. http://www.opennebula.org/. [5] Vmware esx. http://www.vmware.com. [6] Xcp. http://www.xen.org/products/cloudxen.html. [7] Xen. http://www.xen.org/. [8] Y. Chen, T. Wo, and J. Li. An efficient resource management system for on-line virtual cluster provision. Cloud Computing, IEEE International Conference on, 0:72 79, 2009. [9] C. Clark, K. Fraser, S. Hand, J. Hansen, E. Jul, C. Limpach, I. Pratt, and A. Warfield. Live migration of virtual machines. In NSDI 05: Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation, pages 273 286, Berkeley, CA, USA, 2005. USENIX Association. [10] M.R. Hines and K. Gopalan. Post-copy based live virtual machine migration using adaptive pre-paging and dynamic self-ballooning. In VEE 09: Proceedings of the 2009 ACM SIGPLAN/SIGOPS international conference on Virtual execution environments, pages 51 60, New York, NY, USA, 2009. ACM. [11] M. A. Murphy, M. Fenn, and S. Goasguen. Virtual organization clusters. In PDP 09: Proceedings of the 2009 17th Euromicro International Conference on Parallel, Distributed and Network-based Processing, pages 401 408, Washington, DC, USA, 2009. IEEE Computer Society. [12] M. A. Murphy, B. Kagey, M. Fenn, and S. Goasguen. Dynamic provisioning of virtual organization clusters. Cluster Computing and the Grid, IEEE International Symposium on, 0:364 371, 2009. [13] B. Simmons, A. McCloskey, and H. Lutfiyya. Dynamic provisioning of resources in data centers. In ICAS 07: Proceedings of the Third International Conference on Autonomic and Autonomous Systems, page 40, Washington, DC, USA, 2007. IEEE Computer Society. [14] G. Somani and S. Chaudhary. Application performance isolation in virtualization. Cloud Computing, IEEE International Conference on, 0:41 48, 2009. [15] T. Wood, G. Tarasuk-Levin, P. Shenoy, P. Desnoyers, and M. D. Corner E. Cecchet. Memory buddies: exploiting page sharing for smart colocation in virtualized data centers. SIGOPS Oper. Syst. Rev., 43(3):27 36, 2009. 10