Virtualisation Without a Hypervisor in Cloud Infrastructures: An Initial Analysis

Virtualisation Without a Hypervisor in Cloud Infrastructures: An Initial Analysis William A. R. de Souza and Allan Tomlinson Information Security Group Royal Holloway, University of London Egham Hill, Egham, United Kingdom William.DeSouza.2012@live.rhul.ac.uk, Allan.Tomlinson@rhul.ac.uk Abstract Virtualisation is a fundamental technology for data centres and cloud architectures. The central component of virtualisation is the hypervisor, which may be considered as a virtual machine with high privileges in the system that plays a fundamental role in the virtualised environment. In order to perform this role, a hypervisor is built as a complex and large piece of software. Because of this, it has a great surface of attack and it is a main target for attackers in virtualised environment. Many approaches have been presented to mitigate the threats against hypervisors, e.g. minimise its code, add extra code to verify its integrity and harden it. A NoHype architecture is a new approach to this problem that proposes simply eliminate the hypervisor. In this paper we do an initial analysis of this approach. We show that, although it is a feasible architecture that can be implemented with today's commodity hardware, it does not mitigate all the threats to the hypervisor, it introduces new threats and restrains scalability of cloud architecture. Keywords Virtualisation; hypervisor; cloud computing; security. I. INTRODUCTION Virtualisation is a technique used to simulate one or more computers in a single physical machine. By physical machine we mean a hardware device, such as a PC, a server or mobile device. It allows us to run several different environments with multiple operating systems (guest OS) on the same physical machine (host hardware). Two main components of a virtualised environment are the virtual machine (VM) and the virtual machine monitor (VMM) or hypervisor. A virtual machine is a software implementation that provides an isolated software container where an OS and its applications can run [1]. The same resources available in the underlying physical machine (although not necessarily the same amount) are available to the virtual machine. A hypervisor is a high privileged VM that manages all other virtual machines in the same virtualised environment [1]. It consists as a layer between the VMs and the hardware and controls the guest OSs access to the machine resources. Virtualisation is a central technology in data centres and has laid the foundation for advances in this kind of computer facility, enabling the cloud infrastructure and cloud computing [1]. The three main categories of virtualisation are: Full virtualisation, Paravirtualisation and Hardware-assisted virtualisation [1][18][19]. Full Virtualisation provides a complete abstraction of the guest OS, simulating the underlying hardware in such a manner that the guest OS is not aware about the virtualisation and has the impression that all hardware resources are allocated to it. This is achieved by a combination of binary translation and direct execution [18]. Binary translation is a technique that replaces non-virtualisable instructions with new sequences of instructions. In order to improve the performance, user level code is directly executed on the processor. No modifications are necessary neither the guest OS nor the underlying hardware. VMware s virtualization products and Microsoft Virtual Server are examples of full virtualisation [18]. Paravirtualisation address the non-virtualisable instructions problem by modifying the guest OS kernel and replacing these instructions with hypercalls that communicate directly with the virtualisation layer, which provides hypercall interfaces for other critical kernel operations [18]. Thus, the hypercall plays the same role in Paravirtualisation that binary translation plays in the Full virtualisation techniques. For instance, the instruction IOINSR, used when the guest software attempted to execute an I/O instruction, should be replace with a new sequence of instructions in binary translation or should be transformed in a hypercall if Paravirtualisation is being used. A hypercall is similar to a system call used in a OS, that is why in order to use Paravirtualisation it is necessary modify the guest OS. In this case, commodity OS cannot be used. The Citrix XenServer is a example of Paravirtualisation. Some OS as Ubuntu and Red Hat also offer support for Paravirtualisation. Hardware-assisted virtualisation is a set of features developed by hardware vendors in order to provide hardware mechanisms to simplify the use of virtualisation. It targets privileged instructions and includes a new CPU feature that allows the hypervisor to run in a new root mode below ring 0. Thus privileged and sensitive calls are set to automatically trap to the hypervisor, eliminating the need for either binary translation or hypercalls. Examples of this technology include Intel Virtualization Technology (VT-x) and AMD s AMD-V [18]. ISBN: 978-1-902560-27-4 2013 PGNet

A cloud infrastructure provides a set of resources for customers to run their applications and store their information. By taking advantage of virtualisation, the cloud infrastructure allows several virtual machines, from different customers, to exist in the same physical machine permitting economies of scale and providing a dynamic and scalable set of resources at a cost affordable for customers. So, customers can purchase the amount of resources that they need for their applications. That main feature of cloud infrastructure is also the main concern for customers, since the sharing environment is suitable for a malicious party to attack the assets in the infrastructure [2] [3]. Thus, a malicious VM can attack another VM running on the same server, the hypervisor or the hardware infrastructure, potentially exploiting a wide range of vulnerabilities [4]. Some work has shown that this is possible, as in the case of a successful execution of code on the host from a guest OS in a VMware environment [5] and an exploitation of the Xen hypervisor that allows to include a backdoor functionality inside of it [6]. There are many approaches to mitigate those threats. One approach is adding extra code to the hypervisor in order to verify its integrity [7] [8]. Conversely, another approach is minimising the hypervisor [9] [10] in order to diminish the attack surface, leaving just the essential functionality in the hypervisor. The most common approach is hardening the hypervisor [11] [12] [13]. The 'no hypervisor' strategy [14] [15] proposes a radical new approach. Rather than defending the hypervisor, they remove the attack surface by getting rid of the hypervisor, but preserving the semantics of virtualisation. In those works, the authors present an architecture called NoHype that is focused on Cloud Computing. From this point, we will use the terms NoHype architecture or NoHype system, interchangeably. The NoHype architecture is built upon the Full virtualisation technique. Paravirtualisation and Hardwareassisted virtualisation are not directly considered in NoHype. However although not explicit in [14][15], it is clear that since NoHype eliminates the hypervisor then binary translation, or any other type of technique to address non-virtualisable instructions, is not needed after the disengagement stage. In this context, this work does an initial analysis of how much the no hypervisor strategy can help in a cloud computing infrastructure. Thus, we discuss how safe it can be, the threats that it can mitigate, the limitations of this model and most importantly, if it is introducing new threats in the cloud infrastructure. As we will see from the analysis, virtualisation without a hypervisor is a feasible architecture and can be implemented with today's commodities hardware. But, it does not mitigate all the threats posed by a hypervisor and introduces new threats. Besides, it restrains one important feature of cloud architecture: scalability. The remainder of this work is organised as follows. In Section 2 we discuss background on hypervisors. In Section 3 we explain the hypervisor attack surface. In Section 4 we present the no hypervisor architecture. We conduct a brief analysis of the no hypervisor architecture in Section 5 and present a conclusion and work in progress in Section 6. II. THE HYPERVISOR The hypervisor is a VM with elevated privileges and plays a main role in the virtualised environment. Among its tasks, we have management of VMs (the guest OSs), scheduling, memory management, maintaining VM state, creating partition to VM with isolation and so on. Some required features for a hypervisor are security, since it is a main target for attacks, and resource scalability on-the-fly, i.e., the hypervisor should be able to allocate more resources from the host system without stopping the VM that needs the resource. The performance and scalability of a hypervisor contributes to the quality of the virtualisation in a cloud infrastructure. There are two main types of hypervisors: bare-metal hypervisor (or type 1) and hosted hypervisor (type 2) [1]. The bare-metal hypervisor runs directly on the hardware platform and it is a kind of thin OS. It controls and handles the resources of the hardware, scheduling VM and the access to resources. Besides, it monitors the guests OS. The type 1 is preferred in environments that require high efficiency. Some examples of commercial type 1 hypervisors are VMware ESX, Citrix XenServer e Microsoft Hyper-V. The hosted hypervisor runs on top of an OS environment, as a process. In this sense, it manages and controls resources presented by the underlying OS. It is normally used in systems that require a variety of input/output devices and efficiency is not a critical factor. Some examples of commercial type 2 hypervisors are Parallels workstation, Microsoft virtual server, VMware server and VMware workstation. III. UNDERSTANDING THE HYPERVISOR ATTACK SURFACE A VM exit is a kind of trap-and-emulate virtualisation implementation and occurs when the guest VM s code tries executing a privileged instruction. This is an error since the VM is in user mode. So the guest VM s code is interrupted (trap) and the hypervisor code begins to execute to handle the privileged instruction (emulate) [1]. VM exits are rather frequent. For instance, in an idle VM running on top of Xen 4.0, the VM exits occur 600 times/s [15]. In [15] we can see a more detailed experiment on VM exits with Xen 4.0 hypervisor. This is the major entry point for attacks in the hypervisor, since a malicious VM could force a VM exit to occur, trying to simulate an execution of privileged instructions, and inject malicious code or cause a malfunction in the hypervisor. By Injecting code, a malicious VM can violate confidentiality, integrity and availability of other VMs and of the hypervisor. IV. NO HYPERVISOR ARCHITECTURE The main idea of the no hypervisor (NoHype) architecture, as proposed in [14] and [15], is eliminating the hypervisor attack surface altogether. In this way, there is no more need for the virtual machines to interact with the hypervisor when they are executing. However, in NoHype architecture the semantic of virtualisation is preserved, since it is possible to run and manage virtual machines as is done in the cloud infrastructures.

No special hardware is necessary to do this, so today s commodity hardware can be used to host NoHype architecture. Thus, getting rid of the hypervisor, one can get rid of the attacks that a hypervisor would be vulnerable. A. The Threat Model for NoHype Architecture The objective of NoHype is protecting the cloud infrastructure against attacks perpetrated through or against the hypervisor by the guest VMs. The idea is eliminating the interaction between VMs and hypervisor, preventing attacks. The threat model [16] of the NoHype architecture is shown in Fig. 1 as a Data Flow Diagram (DFD). In the threat model we can see that the cloud infrastructure provider, the cloud management software and the modified guest OS (key idea 3, below) are assumed not to be malicious and they are included in a "trust" boundary. The cloud provider modifies the guest OS, accordingly with NoHype requirements and make it available for customers. The process "Modify OS for NoHype" is responsible for this task. The cloud management software offers an interface for customers manage their VMs. The multiple process "Enable services on VMs" allows start, stop, migrate and all others services related to a VM. A VM starts a VM exit, as described in section 3, so the hypervisor identifies the exit type and executes the suitable action by means of the process "Emulate instructions". After completing the action, the hypervisor return the control of execution for the VM. It makes no assumption about the customer, other than the responsibility for protecting their applications in a VM. So, the customers are included in an "unknown" boundary. B. NoHype Key Ideas NoHype considers the main roles of a hypervisor in today's cloud infrastructure and provides the same functionality by another means; capitalizing on the cloud model and on the resources available in commodity hardware. Comparatively, the main resources managed by the hypervisor (its main role) are processor cores, memory, I/O devices and interrupts and timers. The key ideas of NoHype infrastructure are to preallocate memory and cores, use only virtualised I/O devices, short-circuit the system discovery process and avoid indirection. Fig. 1. DFD for the threat model for NoHype architecture. The key ideas are detailed below [14] [15]: 1) Key idea 1: Pre-allocate memory and cores. The hypervisor dynamically manages the memory and processor cores, so VMs can be promised more resources than are actually physically available. Since in cloud the customer specifies the resources needed before a VM is created, NoHype can pre-allocate processor cores and memory, enforcing memory isolation by means of hardware paging mechanisms. 2) Key idea 2: Use only virtualised I/O devices. Virtualisation software emulates I/O devices. NoHype dedicates I/O devices to the guest VM, since the devices themselves are virtualised and there is just a few devices needed in the cloud infrastructure, as network connection (NIC), storage, and graphics card. 3) Key idea 3: Short-circuit the system discovery process. In order to run in a different kind of platform, an OS tries to discover the configuration of the host system. NoHype uses a temporary hypervisor and needs a modified guest OS (provided by the cloud infrastructure) in order to allow

hardware discovery only during the bootup and caching the system configuration data for later use. 4) Key idea 4: Avoid indirection. Hypervisors need to map the virtual view to real hardware (indirection). NoHype dedicates processor cores to a VM. So, a guest VM can access the real processor ID, eliminating the need for indirection. V. A INITIAL ANALYSIS OF THE NO HYPERVISOR ARCHITECTURE In [15] there is a security analysis of the NoHype architecture. We complement this analysis and add some new points of view related to the security and the operation of the cloud infrastructure using NoHype system, describing item by item identified, as showed below. A. System Management Software After implementing a NoHype system in a server, we need the system management software to perform some of the hypervisor tasks, as start, stop and migrates VMs. This is assumed to be secure in NoHype. However, since NoHype is intended to be used in a real world scenario, in a cloud environment using NoHype architecture, the system management software is still an important entry point and target for attacks, since it runs in privileged mode. It is noteworthy that there is no interaction between the system management software and the VM guest code in NoHype. The threats to the system management could come from the interaction with other components of the cloud infrastructure, such as the cloud management software, the cloud provider or the modified guest OS, which could be previously compromised by a malicious party. B. Temporary Hypervisor NoHype architecture proposes altogether to get rid of the hypervisor. But it still needs the hypervisor, even though it is just in the bootup process. Although the VM guest is disengaged from the temporary hypervisor before VM guest is able to execute its code, if the temporary hypervisor is compromised from a previous attack, all proposed security for NoHype system could be compromised. Moreover, the temporary hypervisor stays active for the whole lifetime of the system: this means that it remains a prime target for attacks. C. Data Cached from the System Discovery Process NoHype utilises a temporary hypervisor and a modified guest OS to perform the system discovery tasks. This is an important feature of OSs that allows them to run in different hardware platforms. In order to avoid the need for a hypervisor during the lifetime of the guest VM, the data collected during the system discovery process is cached; then a guest VM can query the data as often as it needs. The problem is to guarantee that this data cached will not be modified by another malicious VM. It is probably done by the same memory protection mechanisms, as EPT, depends on the memory region where the data is cached. D. Kill VM Routine The kill VM routine is a NoHype piece of code that is triggered any time a VM does some illegal action and consequently causes a VM exit, as implemented in NoHype system. Thus, a VM exit is illegal in the system. But, since NoHype needs a temporary hypervisor, a VM exit is not always illegal. As soon as the guest VM is disengaged, a flag in the memory is set, indicating the illegal condition of the VM exits. Since is not clear where in the memory this flag is and what part of the architecture is responsible for managing this flag, we cannot know how hard is, for an attacker, to change this flag. So, an attacker could just change this flag to allow VM exit and compromise the system. Also, the kill VM routine must be protected itself. E. Denial of Service by Means of IPI In the NoHype system, a VM can send interprocessor interupts (IPI) to other cores, as much as it wishes. As a consequence, a malicious VM can send several IPI for a core where a target VM is running or for the core 0, where the system management software is running, as defined by NoHype system. In order to mitigate this threat, NoHype system uses a flag (for each type of IPI) in a shared region in the memory in a manner that a VM sending an IPI can set the flag and the VM receiving the IPI can check and clear the flag. So, a VM receiving an IPI can ignore this IPI, if the flag is not set. Allegedly the security of this process is based on the fact that "no VM can access memory of another VM", so an attacker will not be able to set the flags. But, it is not clear how a VM can differentiate a legal IPI from an IPI sent by an attacker and since an attacker can be a (malicious) VM, he can set his own flags and the VM receiving the IPI can do nothing about this, but receive the IPI. Besides, since the region in memory that holds the flag is shared, why wouldn't the attacker (a malicious VM) have access to it? An experiment performed by the authors in [15] has indicated that this is not a serious problem, since the ability to send IPI by a VM could not stop its targets. F. Hardware Dependency for Isolation The isolation in the NoHype system is dependent on hardware mechanisms. Especially in the case of memory, isolation heavily depends on the correct functioning and implementation of the extended page table (EPT) in order to guarantee confidentiality and integrity. Thus, the EPT is a critical point in the system and beyond the control of NoHype. It must be considered in the threat model and mitigated. G. Infrastructure Mapping Since NoHype eliminates the hypervisor, VMs are closer to the underlying hardware. Thus, a malicious VM could map the underlying hardware infrastructure and perform sidechannel attacks, as described in [17]. In this way, a malicious VM could identify where its target VM is performing in order to perpetrate an attack against it. Singly, it cannot be a problem in NoHype system. But, considering the possibility of previous attacks being successful, as suggested in the previous

sections, discovering where the target VM is running is an important task performed by an attacker in order to advance in his exploitation. H. Pre-allocation x scalability NoHype capitalises on the fact that customers need to determine the amount of resources they want for their applications, in a cloud environment. So, NoHype can assign cores and portions of memory for these customers in advance (pre-allocation), which is the basis of its operation. But, the ability to dynamically increase the amount of hardware resources for a customer is one of the main features in a cloud infrastructure and therefore a serious problem in the NoHype architecture. VI. CONCLUSION AND WORK IN PROGRESS The NoHype architecture proposes a radical new approach to address the matter of security involving the hypervisor: get rid of it! It presents four key ideas: pre-allocate memory and cores, use only virtualised I/O devices, short-circuit the system discovery process and avoid indirection. Basically, NoHype identifies the main roles of a hypervisor and searches for some other manner to do the same thing, in order to eliminate the hypervisor. Although the NoHype system is a feasible architecture that can be implemented with today's commodity hardware, it has a set of issues that could lead a cloud provider to stay with a traditional hypervisor infrastructure. For instance, it does not mitigate all the threats posed by a hypervisor, it introduces new threats to the virtualised environment and it restrains scalability, which is one important feature of cloud architecture. Our work is establishing an extended threat model for the NoHype architecture and identifying entry points in the architecture which can be attacked or can enable attacks. Using Data Flow Diagrams in deeper levels, we will investigate the data flows within the system, the processes transforming these data, the components involved and the types of boundaries in NoHype and in the cloud. Ultimately our goal is to accurately identify the entry points and threats to this architecture with a view to mitigating any vulnerabilities. REFERENCES [1] A. Silberschatz, P. Galvin, and G. Gagne, Operating System Concepts, 9th ed. Hoboken, NJ: Wiley, 2013. [2] M. Christodorescu, R. Sailer, D.L. Schales, D. Sgandurra, and D. Zamboni, D, "Cloud security is not (just) virtualization security: a short paper," Proceedings of the 2009 ACM workshop on Cloud computing security, ACM, 2009, 97-102. [3] A.S. Ibrahim, J.H. Harris, and J. Grundy, "Emerging Security Challenges of Cloud Virtual Infrastructure," in Proceedings of APSEC 2010, Cloud Workshop, Sidney, Australia, 20 nov2010. [4] D. Shackleford, Virtualization Security: Protecting Virtualized Environments. Indianapolis, IN: Sybex, 2013. [5] K. Kortchinsky, Hacking 3D (and Breaking out of VMWare), BlackHat USA, 2009. [6] R. Wojtczuk, Subverting the Xen hypervisor, BlackHat USA, 2008. [7] A.M, Azab, P. Ning, E.C. Sezer, and X. Zhang, "HIMA: A Hypervisor- Based Integrity Measurement Agent," Proceedings of the 2009 Annual Computer Security Applications Conference, IEEE Computer Society, 2009, 461-470. [8] A. M. Azab, P. Ning, Z. Wang, X. Jiang, X. Zhang, and N. C. Skalsky, "HyperSentry: Enabling stealthy in-context measurement of hypervisor integrity," in ACM Conference on Computer and Communications Security (CCS), pages 38 49, October 2010. [9] A. Seshadri, M. Luk, N. Qu, and A. Perrig, "SecVisor: A tiny hypervisor to provide lifetime kernel code integrity for commodity OSes," SIGOPS Oper. Syst. Rev., 41(6):335 350, December 2007. [10] J. M. McCune, Y. Li, N. Qu, Z. Zhou, A. Datta, V. Gligor, and A. Perrig, "TrustVisor: Efficient TCB reduction and attestation," in IEEE Symposium on Security and Privacy, pages 143 158, May 2010. [11] C. Li, A. Raghunathan, and N. K. Jha, "Secure virtual machine execution under an untrusted management OS," in Proceedings of the Conference on Cloud Computing (CLOUD), July 2010. [12] R. Sailer, E. Valdez, T. Jaeger, R. Perez, L. V. Doorn, J. L. Griffin and G. S. Berger, "shype: Secure hypervisor approach to trusted virtualized systems," Technical Report RC23511, IBM Research, 2005. [13] U. Steinberg and B. Kauer, "NOVA: A microhypervisor-based secure virtualization architecture," in European Conference on Computer Systems, April 2010. [14] E. Keller, J. Szefer, J. Rexford, and R. B. Lee, "NoHype: Virtualized cloud infrastructure without the virtualization," in International Symposium on Computer Architecture (ISCA), June 2010. [15] J. Szefer, E. Keller, R.B. Lee, and J. Rexford, "Eliminating the hypervisor attack surface for a more secure cloud," in Proceedings of the 18th ACM conference on Computer and communications security, ACM, 2011, 401-412. [16] F. Swiderski and W. Snyder, Threat Modeling. Redmond, WA: Microsoft Press, 2004. [17] T. Ristenpart, E. Tromer, H. Shacham, and S. Savage, "Hey, you, get off of my cloud: Exploring information leakage in third-party compute clouds," in ACM Conference on Computer and Communications Security (CCS), November 2009. [18] VMware. Understanding Full Virtualization, Paravirtualization, and Hardware Assist. 2007. http://www.vmware.com/files/pdf/vmware_paravirtualization.pdf. Accessed in 20/03/2013. [19] Shackleford, D. Virtualization Security: Protecting Virtualized Environments Sybex, 2013.