Virtualization Security Edward Ray, CISSP NetSec Design & Consulting, Inc. 826 North Red Robin Street Orange, CA 92869 001 714 381 6508 eray@netsecdesign.com Eugene Schultz, Ph.D., CISSP Emagined Security 2816 San Simeon Way San Carlos, CA 94070 001 888 235 1906 eeschultz@sbcglobal.net ABSTRACT Many organization fail to take into account the security of virtual servers, which can result in potential loss of data from internal and external threats. Virtualization has now become commonplace throughout the world; however few if any organizations know the risks associated with running multiple machines on the same physical hardware. The purpose of this paper is to provide an overview of both the benefits and risks associated with virtualization, and steps that should be taken to minimize risks associated with deployment. 1. INTRODUCTION Few issues in the IT arena are regarded with more interest and passion than virtualization. Virtualization refers to technologies designed to provide a layer of abstraction between computer hardware systems and the software running on them. By providing a logical rather than a physical view of computing resources, virtualization solutions make several very useful functions possible. Most fundamentally, they in essence make an operating system recognize a group of servers is a single pool of computing resources. They can allow running multiple operating systems simultaneously on a single machine. Virtualization has its roots in partitioning, which divides a single physical server into multiple logical servers. Once the physical server is divided, each logical server can run an operating system and applications independently. In the 1990s, virtualization was used primarily to re-create end-user environments on a single piece of mainframe hardware. IT administrators who wanted to roll out new software but wanted see how it would work on a Windows NT or a Linux machine used virtualization technologies to create the various user environments. But with the advent of the x86 architecture and inexpensive PCs, virtualization faded and seemed to be little more than a fad of the mainframe era. The recent rebirth of virtualization on x86 platforms is to the credit of the current market leader, VMware. VMware developed the first hypervisor (a special type of virtual machine monitor) for the x86 architecture in the 1990s, planting Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Conference 04, Month 1 2, 2004, City, State, Country. Copyright 2004 ACM 1-58113-000-0/00/0004 $5.00. the seeds for the current virtualization boom. 2. TYPES OF VIRTUALIZATION There are three basic categories of virtualization: 1. Storage virtualization, which melds physical storage from multiple network storage devices so that they appear to be a single storage device. 2. Network virtualization, which combines computing resources in a network by splitting the available bandwidth into independent channels that can be assigned to a particular server or device in real-time. 3. Server virtualization, which hides the physical nature of server resources, including the number and identity of individual servers, processors and operating systems, from the software running on them. This last category is far and away the most common application of the technology today, and it is widely considered the primary driver of the market. When most people use the term "virtualization," they are most likely referring to server virtualization. 3. BENEFITS OF VIRTUALIZATION The industry buzz around virtualization is just short of deafening. This must-have capability has fast become gonna-get-it technology, as new vendors enter the market, and enterprise software providers weave it into the latest versions of their product lines. The reason is that the more virtualization is used; it continues to demonstrate additional tangible benefits, thereby broadening its value. Server consolidation is definitely the sweet spot in this market. Virtualization has become the cornerstone of just about every organization's favorite money-saving initiative. Estimates show that between 60 and 80 percent of IT departments are pursuing server consolidation projects. The reasons why are obvious by reducing the numbers and types of servers that support their business applications, organizations are looking at significant cost savings.
Another major benefit of virtualization is dynamic load balancing capacity across multiple file systems and machines. Applications slow down or even come to a halt when processing bottlenecks occur on conventional machines. Dynamic load balancing helps ensure that such bottlenecks do not occur, thereby enabling applications to run continuously and without disruption. This is especially important for business-critical applications. Still another benefit of virtualization is lowered power consumption, both from the servers themselves and the facilities' cooling systems. Fuller use of existing, underutilized computing resources translate into a longer life for the data center and a fatter bottom line. Additionally, a smaller server footprint is simpler to manage. Virtualization s benefits go far beyond efficiency, functionality and continuity, however, in that virtualization also offers much for information security. VMs can be used to isolate processes from attackers and malware, making systems and applications more difficult to successfully attack or infect. User access to applications can be tightly controlled in that virtualization allows special applications to be isolated from end-user applications, making unauthorized access to the former very difficult. Even if a system or application that runs in a virtualized environment is successfully attacked, any impact resulting from the attack is almost always attenuated. The ability of attackers and malicious code to spread attacks (particularly malware-based attacks) is thereby reduced. A good example of the usefulness of virtualization in the information security arena is the way Java applets run in a sandbox environment in the Java VM. The sandbox restricts capabilities such as reading or writing to files on each local computer, starting or calling programs on each local computer, and obtaining network connectivity to the same computer from which applets have been loaded. Other significant benefits of virtualization include failover functionality, ability to maintain systems without taking them down, the ability to pool computing resources, the ability to have custom virtual machines (VMs), each of which serves as a container for application delivery, and many others. Interactive virtualization-related risks, e.g., when there is a virtualized server and a virtualized network, are also a critical security issue. In this case, the total risk exceeds the sum of the individual risks. Orthogonal to interactive virtualization-related risks are risks in the host environment--the originally installed OS that serves as the host to everything else on a hardware platform. Any vulnerability in any virtualized OS or application can be the weak link that causes multiple compromises in virtualized components. Another security-related risk is hyperjacking, in which an attacker crafts and then runs a very thin hypervisor that takes complete control of the underlying operating system. A good example of how this risk might present itself is the Blue Pill rootkit developed by security researcher Joanna Rutkowska. A rootkit is a Trojan program designed to hide all evidence of its existence from system administrators and others who look for anomalies and security breaches in systems. The Blue Pill rootkit bypasses the Vista integrity-checking process for loading unsigned code into the Vista operating system s kernel. This code uses AMD s secure VM, designed to boost security, to masquerade itself from detection, and becomes a hypervisor, taking control of the operating system without system administrators and others detecting its presence. Additionally, even in virtualized environments it is possible to capture data from layer 2 of the network by configuring a network interface card in a certain manner. Furthermore, virtualized environments are typically characterized by great diversity, something that can interfere with IT standardization and compliance efforts. Consider, for example, virtualization in the Java applet environment. Although Java applets are typically run as part of a Web page, they can be downloaded and then run locally as a file independently of the sandbox s restrictions. The sandbox does not always function as intended, either. Applets can, for example, send information from computers on which they execute to other network-connected systems, thereby substantially raising the risk of unauthorized disclosure or theft of stored data and programs. There has been a startling growth in types of attacks directed at virtual servers. For example: 4. VIRTUALIZATION RELATED SECURITY RISKS Secure isolation, confining a program to a virtualized environment should guarantee that any action performed inside the VM cannot interfere with the system that hosts it, is basic to virtualization. Consequently, VMs have seen rapid adoption in situations in which separation from a hostile or hazardous program is critical. If the physical host server's security becomes compromised, however, all of the VMs and applications residing on that particular host server are impacted. And a compromised virtual machine might also wreak havoc on the physical server, which may then have an adverse effect on all of the other VMs running on that same machine. At the February, 2008 Black Hat Security conference in Washington D.C., a researcher demonstrated that an attacker could take control of the VMware and Xen virtualization software when moving a virtual machine from one physical computer to another. A tool was released that allows an attacker to take control of VM's hypervisor, a virtualization engine that permits multiple operating systems and applications to run on a host computer at the same time. The attacker could then download sensitive data from the live virtual machines (VMs). Data moves in clear-text format during a VM migration, permitting an attacker to perform a man-inthe-middle attack on a virtual machine's hypervisor that would allow stealing data in transit, Malware authors can now detect VM software and
adjust their code to not reveal what it would do on a real machine. Malicious code that runs in virtualized environments is getting smarter. The previously discussed Blue Pill rootkit is one of the best examples. In September 2007 Microsoft fixed vulnerability (MS07-049) in its Virtual PC software that allowed an attacker to escape the virtual Operating System (OS) to access the physical OS in Microsoft s Virtual PC software. In October 2007, VMware released updates that fix a number of vulnerabilities. This vendor announced the details on a mailing list, but glossed over the problem on its own website. See: http://www.vmware.com/support/ws6/doc/releasenotes_ws6.html http://www.vmware.com/support/server/doc/releasenotes_server.h tml These updates fix quite a few vulnerabilities, the more serious ones being: 1. CVE-2007-4496 A privileged user in a guest OS can execute arbitrary code on the host OS. expedient to analyze the true risks that may present themselves in virtualized environments and also to avoid having a false sense of security with respect to virtualization. Organizations that buy more redundant hardware and run multiple VMs together on a shared hardware platform also need to be especially cautious concerning the particular types of servers that reside on a single physical machine. For example, it would be a bad idea to put the firewall, an intrusion detection system, a public Web server, and database server all on one shared physical machine. In the VM world, if one VM is compromised, all VMs on the same physical machine can be more readily compromised. In fact, it would be easier to compromise multiple VMs in this hypothetical case, because the hardware that each VM uses is on the same platform. Even if all the VMs are equally secure against attacks, risk is nevertheless escalated due to the fact the VMs can talk among themselves without passing information through the network layer. The bottom line is that it is prudent to be careful about the architecture used and the VMs that are mixed together on the same physical platform. Finally, if an organization that has air-gapped networks that carry differently classified information (e.g., proprietary and nonproprietary information), migrating the machines that store this information to a virtualized environment all for the sake of making it easier for users to access both types of information would be very unwise from a risk management perspective. It would be far better to instead use a KVM (Keyboard, Video and Mouse) switch. 2. CVE-2007-4497 A user on the guest OS can cause denial of service on both the host and the guest OS. Travis Ormandy of Google wrote a paper that analyzed security flaws in the implementation of several vendors virtualization products. Exploitation of flaws resulted in buffer overflows, ability to access and change power utilization code, and more. Although the names of the two products are withheld to avoid shedding negative light on certain vendors products, it is not terribly difficult to guess which vendor platforms were analyzed in Ormandy s study. 5. SECURING VIRTUALIZED ENVIRONMENTS Securing virtualized environments and, in particular, VMs must start before VMs are deployed, and ideally, before vendors and products are selected. The reason is that security and securability must be factored into the evaluation and selection process; otherwise, security in virtualized environments must be retrofitted, something that is likely to not only lead to unidentified risks, but also to practical difficulties and escalating cost over time. Questions to be answered before deployment include: With all of the previously mentioned developments, determining the best and safest -way to leverage security virtualization may seem daunting. So how does an organization that uses virtualization mitigate virtualization-related risk? As with everything else in information security, risks have to be weighed against benefits. Using Parallels on a Macintosh to run Windows applications in a VM environment is normally very justifiable from a security risk perspective because the benefits far outweigh the risk. Running a VM that has known vulnerabilities to show how easy it is for real attackers to attack a system and how little skill is required to execute a program that gives an attacker complete control of the target system is perfectly acceptable in the context of teaching, but not in the context of mainstream IT operations. In malicious code research, analyzing the risk to benefit ratio is not nearly as easy as it might seem. Malicious code can break out of the VM and compromise the physical machine s OS. Attackers could then start to build malicious code capable of breaking out of the segregated environment. It is thus extremely Where and how do you use virtualized environments in your organization? Do you have a patch management policy in place for the virtual machine operating systems? Do you have a patch management system in place for the virtual machine software? What policies do you have regarding the use of virtualized environments? Is your organization aware of the risks associated with deploying virtual environments?
Is your organization interested in reducing these risks so that the benefits of a virtualized environment can be safely realized? During this process of evaluating and selecting security controls, organizations must consider the following security issues: Virtualization software, such as hypervisors, represents a new layer of privileged software that will be attacked and must thus be protected. The loss of separation of duties for administrative tasks, which can lead to a breakdown of both the least privilege and defense in-depth principles. Patching, signature updates, and protection from tampering for offline VM and VM "appliance" images. Patching and secure confirmation management of VM appliances where the underlying OS and configuration are not accessible. The fact that in the process of finding vulnerabilities and assessing correct configuration there will be limited visibility into the host OS and virtual network. The fact that there will be a restricted view into inter- VM traffic for inspection by intrusion prevention systems (IPSs). Mobile VMs will require security policy and settings to migrate with them. Security and system and network management tools tend to be immature and incomplete in the first place; using them in virtualized environments only compounds these problems. Security of the VM is dependent on the operating system and should follow the same processes already developed by the information security practice for these operating systems as if each VM were a physical host. From a security perspective a VM and a physical server do not differ. Besides using the service console s access to the VM File System (VMFS), the only other way to access another VM is through its network connections. Therefore securing the network is of primary importance. Due to the fact that the Console Operating System (COS) hypervisor can access the VM disk files, securing the service console is even more important. In most organizations the approach taken for securing virtual environments is to use current configuration standards and tools that were used in the past for securing any OS, network device, or application. Although this approach has some merits; it fails to address the security ramifications of having multiple platforms on the same physical machine. Simply applying the same controls used in securing physical servers will not, for example, provide sufficient protections for VMs. Securing VMs must start before the VMs are deployed, and ideally, before vendors and products are selected, so that security and securability can be factored into the evaluation and selection process. The first focus should thus be physical security. All the logical protections that a virtualized environment can have will be in vain if anyone can walk into a datacenter and steal disk drives from any machine. This scenario can easily happen if a service console is not afforded strong levels of physical protection. Organizations will in all likelihood have to update their information security standards (and possibly also their information security policy) to help ensure that individual VMs and the COS are properly protected. Once the necessary changes to standards (and possibly also the information security policy) are made, approved and implemented, technical control measures should be selected and implemented. Achieving suitable levels of security in virtualized environments requires securing all of the following: 1. The VM OS: The VM OS must be secured using the same best practices that the information security organization dictates for the OS in question. Failing to secure the VM OS can make compromising it trivial, but can also substantially elevate the probability of a network compromise. The reason is that once compromised, the VM OS can readily serve as a springboard for attacks against the network. 2. VM Networks: All externally initiated VM network connections should be shielded by a properly configured well maintained firewall. Additionally, as just mentioned the OS in the VM must be properly secured. 3. Securing the VMKernel: The VMkernel is by its nature extremely secure. With no public Application Programming Interfaces (APIs), possible ways of hacking or cracking this crucial software dwindle to almost none. This does not mean that it is impossible to compromise this software, however. Best practices for this component include regularly patching the VMKernel with vendor updates in accordance with an organization s information security policy. 4. Securing VM Server to VM Server traffic and VMkernel traffic: Communication between servers (i.e. VMware VMotion) passes memory data between VM servers to help manage each VM host and to facilitate performance balancing. The VMkernel network is also used to perform NFS mounts and Internet Small Computer System Interface (iscsi) access. The data are passed unencrypted; access to the network that passes these data should thus be adequately controlled at all times. 5. Console Operating System: The COS has access to everything that the VMkernel will expose, namely
hardware as well as the data stores for the VM disk files. The COS is thus another crucial security consideration. At a minimum, non-administrator VMs should have no access to the COS network this will limit possible attack origin points. 6. VM Deployment: There are many different tools for deploying VMs that use the COS network connection. Typically these tools send unencrypted information to the target VM Server. Securing the deployment network by ensuring that such traffic is encrypted with strong encryption (e.g., Advanced Encryption Standard encryption) is thus a necessary part of any defense-in-depth solution. 7. VM Backup: Typically VMs are backed up in one of two ways. The first is to backup from within the VM, which uses the network connections of the VM. The second method is externally via the COS (i.e., VMware Consolidated Backup). In either case, backup data traversing the network should be encrypted with strong encryption and the backup server should be adequately secure in terms of access authentication and file protection. 8. Data. Data in virtualized environments need to be protected in the same manner a datacenter should be protected. Although each VM is separate and distinct, the service COS is part of the VM Server and it has access to critical data. Properly configured access control lists (ACLs) and applying the least privilege principle will both minimize the likelihood of data security breaches. Additionally, placing the service console properly within an organization s network is a must. At a minimum, keep the service console out of a Demilitarized Zone (DMZ) where it can more easily fall prey to externally initiated attacks. Placing it at a point within a network where a firewall shields it from such attacks is far better from a security perspective. Finally, it is important to ensure that sufficient auditing (coupled with procedures that require regular and systematic inspection of audit log output) is enabled and continuously running on the VM server service console. Network monitoring is also necessary. The ultimate goal should thus be to provide as much preventative protection as needed while at the same time to allow for auditing and monitoring the OS with minimal impact on the operation of the system applications. Additionally, understanding that eventually a wider variety of attacks, some of which may be successful, will surface in virtualized environments is imperative; necessary adjustments must be made and necessary additional security controls must continually be considered and, if justified in terms of costs versus benefits, implemented to minimize the likelihood of successful attacks in these environments. Candidate controls include (but are not limited to) network firewalls, application firewalls, strong authentication, anti-virus/antispyware tools, denial of service (DoS) protection through fault tolerance or other mechanisms, forensic tools, remote logging, periodic vulnerability scans, and patch management. 6. CONCLUSION Information security professionals and others need to thoroughly understand virtualization and its advantages and disadvantages from security, information technology, and business viewpoints. They also need to keep up with changes in virtualization that not only have occurred in the past, but also that will continue to occur in the future. With the ever-increasing popularity of virtualization, one thing is certain virtualization and computing will continue to converge well into the future. Unfortunately, virtualization is also likely to provide a disproportionately increasing number of targets for attackers and malicious code. It thus behooves information security professionals to be as proactive as possible in their approach to managing virtualization-related risks 7. REFERENCES [1] Haletky, Edward L. VMware ESX Server in the Enterprise: Planning and Securing Virtualization Servers, Prentice Hall, 2008 [2] Ormandy, Travis. An Empirical Study into the Security Exposure to Hosts of Hostile Virtualized Environments. September 2, 2008, http://taviso.decsystem.org/virtsec.pdf [3] Waters, John K. ABC: An Introduction to Virtualization, CIO, March 15, 2008, http://www.cio.com/article/40701/abc_an_introduction_to _Virtualization