Virtualization analysis

of 15 Virtualization analysis CSD Fall 2011 Project owner Björn Pehrson Project Coaches Bruce Zamaere Erik Eliasson HervéNtareme SirajRathore Team members Bowei Dai daib@kth.se 15 credits Elis Kullberg elisk@kth.se 18 credits Gurpreet Singh Sambhy sambhy@kth.se 24 credits Hannes Junnila haju@kth.se 15 credits Nur Mohammad Rashed nmrashed@kth.se 15 credits Siddharth Madan smadan@kth.se 15 credits Vasily Prokopov Prokopov@kth.se 18 credits

of 15 Table of Contents 1 Introduction... 4 1.1 Purpose of this document... 4 1.2 Scope of this document... 4 1.3 Audience of this document... 4 1.4 Introduction for virtualization... 4 2 Overview of XEN, KVM and LXC... 6 2.1 XEN... 7 2.2 KVM... 7 2.3 LXC... 8 3 Comparisons among XEN, KVM and LXC... 9 3.1Macro-benchmarks:... 9 3.2 Scalability:... 9 3.3 Security... 10 4 Advantages and disadvantages of XEN, KVM and LXC... 11 4.1 XEN... 11 4.2 KVM... 11 4.3 LXC... 11 5. Practical aspects of container security... 13 6 Conclusions... 14 Reference... 15

of 15 Overview of changes: Version Changes 0.1 Initial document

of 15 1 Introduction 1.1 Purpose of this document The purpose of this document is to give a brief introduction of different types of virtualization technologies and make comparisons among them in order to choose the best for Bifrost router. 1.2 Scope of this document The scope of this document is to give a brief introduction and comparison of three main types of virtualization which are XEN, KVM and LXC. 1.3 Audience of this document The audience of this document mainly target at coaches and teams of CareNet. 1.4 Introduction for virtualization Virtualization, in computing, is the creation of a virtual (rather than actual) version of something, such as a hardware platform, operating system, a storage device or network resources.[1] There are several types of virtualizations: hardware virtualization, desktop virtualization, software virtualization and memory, storage and so on. In this document, we mainly focus on hardware virtualization. Hardware virtualization or platform virtualization refers to the creation of a virtual machine that acts like a real computer with an operating system. Software executed on these virtual machines is separated from the underlying hardware resources. For example, a computer that is running Microsoft Windows may host a virtual machine that looks like a computer with Ubuntu Linux operating system. Subsequently, Ubuntu-based software can be run on that virtual machine.[1][2] There are two main techniques for hardware virtualization: Virtual containers share only one kernel on the OS-level. Many virtual operation systems with different kernels run on a real physical machine. A supervisor is needed in order to manage these operation systems on one physical machine; however, there have already been supervisors inside each virtual operation systems. So we have a new name for the supervisor of the real machine: hypervisor (also called VMM) which means the supervisor of

of 15 supervisors. In more detail, the hypervisor virtualization can be divided into two categories. Full Virtualization (FV) and Para virtualization (PV), which can be both combined with hardware-assisted virtualization. Full virtualization uses binary translation to run arbitrary, unmodified operating systems on top of the hypervisor. VMware is a good example of this. While there may be an important cost when using the guest system which emulates the real system s resources. This cost can be mitigated by using Hardware-assisted virtualization. Para virtualization also modifies the guest operating system to optimize the interplay between virtual machine monitor and the virtual machine itself. It is also based on a hypervisor, but the devices are not emulated. The aim of the modified interface is to decrease the execution time.

of 15 2 Overview of XEN, KVM and LXC Table1: comparisons among Xen, KVM and LXC. [3]

of 15 2.1 XEN Ian Pratt started the research project of XEN at University of Cambridge and released the first public edition in October 2003. After that, Ian created Xensource Company and main release 2.0 and 3.0 were delivered in 2004 and 2005. Xen is a hypervisor and can run guest systems which are called domains. There are two types of domains. DomU is a kind of unprivileged domain and Dom0 is a special guest system with privileged functions which contain the applications to control other guest systems. Dom0 uses a modified kernel and is running on Xen hypervisor. It is the only domain interacts with hardware through linux kernel drivers. DomU rely on its virtual drivers to interact with hardware devices. 2.2 KVM KVM (Kernel based Virtual Machine) is an open source Linux kernel virtualization infrastructure which relies on the hardware virtualization technologies. Its first version is linux 2.6.20 which is released in February 2007. KVM developers had an original idea: instead of creating kernels themselves, they choose to use linux kernel itself as a basis for hypervisor. Thus, KVM is currently implemented as loadable kernel modules. kvm.ko, that provides the core virtualization infrastructure and a processor specific module, kvm-intel.ko or kvm-amd.ko. This original approach brings several benefits. The virtualized environment takes advantage of all the ongoing work made on the Linux kernel itself. Using KVM, each virtual machine is a regular linux process which is scheduled by the linux scheduler. KVM emulates virtual devices, such as network interfaces or hard disks. In order to improve performance, recent KVM versions propose a hybrid approach called virtio [4]. Virtio is a kernel API that improves the performance of communications between guest systems and the host system by providing a simpler and faster interface than the emulated devices from QEMU. Virtiobased devices exist both for network interfaces and hard disks.

of 15 2.3 LXC LXC is the user space control package for Linux Containers, a lightweight virtual system mechanism sometimes described as chroot on steroids. LXC builds up from chroot to implement complete virtual systems, adding resource management and isolation mechanisms to Linux s existing process management infrastructure. Linux Containers (lxc) implement: Resource management via process control groups (implemented via the cgroup filesystem) Resource isolation via new flags to the clone system call (capable of create several types of new namespaces for things like PIDs and network routing) Several additional isolation mechanisms (such as the -o newinstance flag to the devpts file system).

of 15 3 Comparisons among XEN, KVM and LXC In this section, we evaluate the different virtualization solutions with a set of benchmarks. 3.1 Macro-benchmarks: Table2: comparisons among native Linux, Xen and KVM in CPU, Kernel compile, Disk I/O. [5] We can get some distinct results from this table: Xen and KVM had similar CPU performance: Xen: 0.999, KVM: 0.993 Xen was better than KVM on kernel compile: Xen: 0.487, KVM: 0.384 KVM was better on disk I/O:Write Xen: 0.855, KVM: 0.934 Read Xen: 0.852, KVM: 0.994 3.2 Scalability: Picture1: compile time and number of guests that run to completion. [5] Results we get from this picture: Xen scaled linearly with respect to number of guests KVM had many guest crashes

of 15 4 guests: 1 crashed guest 8 guests: 4 crashed guests 16 guests: 7 crashed guests 30 guests: system crashed during compile 3.3 Security XEN and KVM usually allow you to run any operating system, since the emulation platform actually gets right down to emulating the hardware. While LXC uses cgroups to create a restricted view of the host operating system. Within the LXC guest environment, you can only see what the admin allows you to see of the host system; you can have a separate process space, for example and also create a separate file system for the guest.

of 15 4 Advantages and disadvantages of XEN, KVM and LXC 4.1 XEN Advantages of XEN: Concerning on micro benchmarks, XEN has an excellent performance which has been show above in table 2. XEN uses a very good management tool which is xm. XEN already has a large market share. Some vendors have supported XEN and XEN users and developers are very active. Disadvantages: XEN requires too many interrupts and hops between kernel and user space 4.2 KVM Advantages: It uses linux kernel as its hypervisor and does not duplicate scheduler and memory management code which means KVM is simpler and that Linux is capable of being a good hypervisor. RedHat supports KVM camp now and will be pushing it as the virtualization platform of choice. KVM ships as an official kernel module which means less maintenance for the distro creators Disadvantages: It does not work on CPUs that don t have hardware virtualization support Not very stable yet. Real mode evaluation does not work perfectly on Intel machines. 4.3 LXC Linux Containers take a completely different approach than system virtualization technologies such as KVM and Xen, which started by booting separate virtual systems on emulated hardware and then attempted to lower their overhead via Paravirtualization and related mechanisms. Instead of retrofitting efficiency onto full isolation, LXC started out with an efficient mechanism (existing Linux process management) and added isolation, resulting in a system virtualization mechanism as scalable and portable as

of 15 chroot, capable of simultaneously supporting thousands of emulated systems on a single server while also providing lightweight virtualization options to routers and smart phones. The LXC is small enough to easily manage a container with simple command lines and complete enough to be used for other purposes. It has virtually no overhead, and it provides a degree of flexibility because of its ability to share resources between different LXC guests. Also, LXC supports not only virtualzing a running instance of an operating system but also individual applications, for which devoting an entire virtual machine is overkill. Advantages: [6] Better isolation as compared to a chroot (chroot jail). Low overhead. LXC uses minimal resources in terms of RAM and hard drive space without the overhead of installing a guest OS in a virtual machine (VMWare / VirtualBox / KVM ). Applications and services (servers) run at native speed. There is support for Linux containers in libvirt. Linux containers work well with btrfs. No special hardware is required, runs on 32 and 64 bit processors. Linux containers are Open source. Unlike XEN or OpenVZ, no patch is required to the kernel. Disadvantages: Linux containers run Linux processes on a Linux kernel. This means you can run Linux (Fedora container on an Ubuntu host) but not other operating systems (Not BSD / OSX / Windows). There are no GUI (graphical) interfaces to configure or manage the containers. There is a paucity of documentation on how to install and configure a container. Configuring a container requires a modest technical knowledge and skill (and a large grain of patience).

of 15 5. Practical aspects of container security 5.1 Overview For the time being, it seems that container based virtualization is the only option for the residential gateways. The main issue regarding container based virtualization solutions is security. In this section we provide an overview of the most important security related aspects of Linux Containers. These guidelines should provide full guest-to-host isolation. This is needed to assure custodians that administrators with container access cannot view the internet traffic on unrelated ports of the residential gateway. 5.2 Filesystem When setting up the fstab of the guest operating system, it is possible to simply mount elements of the host s filesystem to the guest. This decreases redundancy of for example library files, but leads to security issues in terms of privacy. More importantly though mounting the /dev/ folder or /proc/ folder will enable a root user inside a container to reboot the entire host using for example: echo 1 > /proc/sys/kernel/sysrq echo b > /proc/sysrq-trigger which could be a security issue. On the other hand, it is quite useful in the CareNet case since it enables administrators to change the host kernel settings (using the /proc/ filesystem) from the guest. 5.3 Cgroups Every new container needs a separate cgroups filesystem that provides a data structure for storing most control information. 5.4 Networking There exists multiple ways of connecting network devices to a host, with different demands for hardware and different security considerations. On lxc the following four are implemented: phys: A network device is dedicated for a container. This is the most secure one, as other containers and the host userspace can t access the network device. An issue, however, is that dedicated hardware is

of 15 needed for running multiple hosts. This makes it not only more expensive, but less efficient, if the cards wouldn t be needed otherwise. vlan: The host is connected to a virtual lan on a physical device on the host machine. From a security standpoint this would be similar to the above. However, this comes with the need of the routers/swithces for the network to be able to do vlan tagging. veth: A virtual hub is created, to that the host can share one physical network device. This is the most widely used, as it doesn t need any extra hardware, and enables easy communication between the host computer and the containers. However, it comes with one big security issue, which is that all the containers and the host machine can listen to all communication through the network device. This allows eavesdropping on other hosts, which is especially a concern if the hosts are not supposed to have knowledge of each other, such as at commercial provider. macvlan: This method allows the kernel to create virtual lans based on the mac-address of the client, so that the physical device corresponds to multiple mac-addresses and can separate the traffic going to any of the virtual interfaces on the client machines. This gives a good separation between the host and the different clients, as for them it looks like they would be connected to a switch forwarding the traffic to only the actual destination of the traffic, which disables eavesdropping on the other clients and the host. [7] 6 Conclusions Considering the aspects of overload and speed, LXC is much better than the other two options. Furthermore, only static files can be run on Bifrost and in these three alternatives, only LXC support this kind of files which means LXC is the most suitable choice for Bifrost. If configured correctly LXC does provide a secure guest environment. For the time being, namespace isolation is the only option for CareNet since the residential gateways do not feature the VT-Z (or similar) CPU instruction set extensions needed for full virtualization.

of 15 7 References 1 Turban, E; King, D; Lee, J; Viehland, D (2008). "Chapter 19: Building E-Commerce Applications and Infrastructure". Electronic Commerce A Managerial Perspective (5th ed.). Prentice-Hall. pp. 27. 2 "Virtualization in education". IBM. October 2007. Retrieved 6 July 2010. "A virtual computer is a logical representation of a computer in software. By decoupling the physical hardware from the operating system, virtualization provides more operational flexibility and increases the utilization rate of the underlying physical hardware." 3 http://virt.kernelnewbies.org/techcomparison 4 Rusty Russell. virtio: towards a de-facto standard for virtual I/O devices. SIGOPS Oper. Syst. Rev.,42(5):95 103, 2008. 5 Quantitative Comparison of Xen and KVM, Todd Deshane, Ph.D. Student, Clarkson University Xen Summit, June 23-24, 2008, Boston, MA, USA. 6 http://blog.bodhizazen.net/linux/lxc-linux-containers/ 7 http://blog.flameeyes.eu/2010/09/04/linux-containers-and-networking