Models For Modeling and Measuring the Performance of a Xen Virtual Server

Transcription

1 Measuring and Modeling the Performance of the Xen VMM Jie Lu, Lev Makhlis, Jianjiun Chen BMC Software Inc. Waltham, MA 2451 Server virtualization technology provides an alternative for server consolidation by creating a set of logical resources that share underlying physical resources. Xen virtual machine monitor, a popular virtualization solution in Linux world, supports execution of multiple guest operating systems with unprecedented levels of performance and resource isolation. Performance modeling of virtual servers faces challenges of obtaining meaningful measures as the operating system deals with virtual resources. This paper presents a practical approach for measuring and modeling the performance of Xen. 1. Background Server virtualization technology provides an alternative for server consolidation by creating a set of logical resources that share underlying physical resources. Virtual servers enable dynamic tuning through dynamically moving resources, make better use of the resources through sharing resources, provide high availability through isolating guest operating systems, reduce overall costs through higher utilization, and increased flexibility through rapid provisioning. There are many server virtualization approaches. Table 1 compares the most popular commercial virtualization products. All these approaches enable the execution of multiple guest operating systems on the same computer hardware (except Solaris Container, which is a single OS image). IBM micro-partition (SPLPAR) employs hardware-based hypervisor provided by its Power 5 server. VMware ESX Server provides hardware abstraction using a proprietary kernel extension. Microsoft Virtual Server provides virtual machines by simulating the complete hardware. Solaris Container is a single OS image, which provides application isolation with its zone. Xen provides software-based hypervisor using the technique of paravirtualization. The term of hypervisor came from the mainframe world, which first introduced the virtualization concept. It is a mechanism that allows multiple operating systems to share the underline hardware resources With the virtualization techniques, a business needs to plan for effective consolidation of business services, minimize risk in deploying virtual servers, identify and resolve performance issues before they impact Product Type Hypervisor IBM micro-partition (SPLPAR) Hardware platform System Firmware Power 5 VMware ESX Server Full Software x86 Xen Para, Full Software x86 MS Virtual Server Emulation No x86 Solaris Container OS No SPARC and x86 Guest OS AIX, Linux, i5/os and and Linux Linux and and Linux Solaris Table 1. Comparison of popular commercial virtualization products Modify guest OS Yes No Yes (for Para) No N/A

2 business users, and assure that applications perform well when using virtualized resources. All these needs The major challenge that we are facing is to obtain meaningful performance measures. However, the most critical performance measures, such as utilization and I/O rate, obtained from standard OSprovided facilities are often virtual-based. Therefore, it could be misleading by using conventional tools that are based on physical measures. Many papers have been published discussing performance modeling and capacity planning issues for virtualization from various aspects. In this paper, we are in particular focusing on measuring the performance from a Xen virtual server and applying the metrics to performance modeling and capacity planning. In Section 2, we give a brief introduction on the architecture of Xen. We then discuss how to obtain performance measures from Xen in Section 3. Finally, we present our performance modeling method via a series of experiments in Section Xen introduction Xen is an open source virtual machine monitor (VMM) developed by the Computer Laboratory at University of Cambridge. It is designed for x86 to support execution of multiple guest operating systems with unprecedented levels of performance and resource isolation [1]. The traditional x86 architecture lacks the technology to support full virtualization. Solutions like VMware s ESX server implement full virtualization at the cost of sacrificing performance. In contrast with full virtualization, paravirtualization offers better performance by presenting a virtual machine abstraction that is similar but not identical to the underlying hardware. Therefore, it requires modifications to the guest operating system. However, it virtualizes all architectural features required by the existing standard application binary interface (ABI), and hence no modifications are required to guest applications. In order to protect the hypervisor from OS misbehavior, guest OSes must be modified to run at a lower privilege level. Xen operates at a higher privilege level than the supervisor code of the guest OSes, thus itself is the hypervisor. Privileged instructions are paravirtualized by requiring them to be validated and executed within Xen. Any guest OS attempt to directly execute a privileged instruction is failed by the processor, either silently or by taking a fault, since only Xen executes at a sufficiently privileged level. In Xen, the guest OSes are responsible for allocating call for effective performance modeling and capacity planning tools and methods for virtual servers. and managing the hardware page tables. They have direct read access to hardware page tables, but updates are batched and validated by Xen. Guest OSes install fast handler for system calls, to allow direct calls from an application into its guest OS and avoid indirecting through Xen on every call. Each guest OS registers a descriptor table for exception handlers. Hardware interrupts are replaced with a lightweight event system. Each guest OS has a timer interface and is aware of both real and virtual time. Xen provides a high-performance communication mechanism for passing buffer information vertically through the system, while allowing Xen to efficiently perform validation checks. It exposes a set of clean and simple device abstractions. I/O data is transferred to and from each domain via Xen, using sharedmemory, asynchronous buffer-description rings. Figure 1 from [2] illustrates the overall system structure of a Xen virtual server. The initial domain, Domain, is created at boot time. It is responsible for hosting the application level management software. Domain is a virtual machine, same as other guest domains, which runs on virtual s and virtual memory. However, only Domain is permitted to use the privileged control interface to hypervisor. The control interface provides the ability to create and terminate other domains. It also controls the scheduling parameters, physical memory allocations, physical disks access, and network devices access. Paravirtualization requires the modifications on the guest OS kernel. Xen provides patched Linux kernel (XenLinux) for both version 2.4 and 2.6. Many Linux distributions have already included or will include Xen into their packages, such as RHEL 5 and SLES 1, as Xen becomes more popular. Xen also implemented support of full virtualization for processors with Intel VT or AMD Pacifica technology. In this configuration, it does not require modifying the guest OS, which makes it possible to host proprietary operating systems, such as. 3. Data visibility Traditional performance measures are obtained from the operating system. Xen is hosting multiple guest OSes. Each OS only has a partial view of the physical system. To each guest, it appears that it is the only operating system using the hardware. Effective

3 VM Device Manager & Control s/w VM1 User Software VM2 User Software VM3 User Software AGP ACPI PCI (XenLinux) Back-End Native Device Drivers (XenLinux) Front-End Device Drivers (XenLinux) SMP Front-End Device Drivers (WinXP)) Front-End Device Drivers VT-x x86_32 x86_64 IA64 Control IF Safe HW IF Event Channel Virtual Virtual MMU Xen Virtual Machine Monitor Hardware (SMP, MMU, physical memory, Ethernet, SCSI/IDE) Figure 1. Xen 3. architecture performance modeling requires a picture of the whole server. Moreover, all guests and Domain within a Xen virtual server run on virtual s and virtual memory. Therefore the guest operating systems are not aware of the actual physical configurations of the server. For example, the virtual s can be configured to a different number than the available physical (or logical if Hyper-Threading is enabled) s in the server. With full virtualization, the time reported by standard Linux facilities is based on virtual time and thus cannot be used for performance modeling and capacity planning. In performance modeling, the layer closest to the physical hardware resources has the best view of system activity. The hypervisor keeps track of the actual usage and other resources used by the individual guests. These metrics are the most reliable statistics for measuring system activity. At user level, Xen provides the xm tool as the primary management software running in Domain to access the data at the hypervisor layer. At low level, there are three basic approaches to obtain the data. We discuss them in the following sections. 3.1 Xen daemon The Xen daemon (Xend) can be used to obtain the detailed configuration data. Xend is the node control daemon that performs system management functions related to virtual machines. It forms a central point of virtualized resources control and must be running in order to start and manage virtual machines. Xend must be run as root because it needs access to privileged system management functions. An HTTP interface is provided for communicating with Xend that allows one to pass commands to the daemon. There are four ways to connect to Xend: SXP over a UNIX domain socket (/var/lib/xend/xend-socket); SXP over TCP (port 8); XML-RPC over a UNIX domain socket (/var/run/xend/xmlrpc.sock); XML-RPC over TCP (port 85). These services can be enabled or disabled in the configuration file /etc/xen/xend-config.sxp. As of Xen 3..2, XML-RPC over UDS is the only service that is enabled by default. In the first two cases above, Xend responds with data in S-expression format [3].

4 3.2 XenStore The XenStore is an information storage space shared between domains. It is meant for configuration and status information rather than for large data transfers. Each domain gets its own path in the store. The appropriate drivers are notified when values are changed in the store. XenStore can be accessed in one of the following ways: Through procfs, using the file /proc/xen/xenbus; Through xenstored, using UNIX domain socket /var/run/xenstored/socket or /var/run/xenstored/socket_ro. The protocol for both methods is the same, which is based on read() and write() calls. It is encapsulated in the xenstore library, which is a standard Xen component. Utilities such as xenstore-list and xenstore-read use the library to provide command line access to XenStore data. 3.3 Hypervisor calls Xen also provides hypervisor calls to access the physical configurations as well as the usage based on real time. A user space program running on Domain can perform hypervisor calls as ioctl commands on the special file /proc/xen/privcmd. This can also be accomplished using the xenctrl library, a standard component of Xen. The end result of performance modeling is individual workload response time. The response time is driven by two factors: service time and usage. An application running in a full-virtualized guest OS may still have acceptable response time even if it reports high usage. In this case, it is the physical usage reported by the hypervisor that drives the response time, not the usage reported by the guest OS. Therefore, it is important to collect data from Domain, which is the only domain that has access to the global view of the host. The overall system configuration can be obtained via the hypervisor call DOM_PHYSINFO. It provides the following critical information for capacity planning: topology: number of threads per core, number of cores per socket, number of sockets per node, and number of nodes in the box; speed in KHz; Memory: total pages and free pages. Using the hypervisor call DOM_GETDOMAININFOLIST, Domain can get a list of all domains with the following parameter: Numeric ID; Status (dying shut down paused blocked running); Memory pages allocated to the domain, and its memory limit; Number of virtual s (Vs); Aggregate usage, in nanoseconds. It is also possible to get the usage for each individual by using the hypervisor call DOM_GETVINFO. The usage reported for the domain is calculated as sum over that domain s Vs. 3.4 Device data Individual PCI devices can be assigned to a given domain, which is called driver domain, to allow that domain direct access to the PCI hardware. Normally, Domain is the only driver domain. Driver domain is the only domain that sees physical block devices. Other guest domains see virtual block devices (VBDs) instead. VBDs can be backed by an entire physical device (either disk or disk partition), by a logical volume, or by a file. Such information is part of the domain configuration data. Standard Linux facilities will report virtual disk I/O statistics for VBDs in guest domains other than the driver domain. The physical disk I/O statistics for physical block devices are reported in the driver domain only. Similarly, driver domain is the only domain that sees both virtual and physical network adapters. Other guest domains see only virtual adapters (VIFs). In each domain, virtual adapters are given names eth, eth1, etc., like real Ethernet adapters would be. Each virtual adapter is connected to a peer virtual adapter as follows: ethx in domain #Y is connected to vify.x in Domain. This includes eth in Domain, which is connected to vif.. Physical adapters are visible in driver domain as peth, peth1, etc. However, their MAC addresses are visible in ethx, instead of pethx. Virtual adapters in guest domains are assigned virtual MAC addresses that are part of the domain configuration. This enables the correlation of data collected from within a guest OS to data collected from hypervisor. 4. Performance analysis and modeling 4.1 Data collecting

5 Traditional performance analysis collects performance metrics from the operating system. Data collector is running as an application within the OS. In our experiments, we run regular data collector within each OS image. It collects performance metrics for system configuration and statistics, disk configuration and statistics, network configuration and statistics, and process statistics. In addition to regular metrics from OS, we also collect Xen specific metrics via hypervisor calls in Domain. It includes the configuration and statistics of each domain hosted by Xen. The configuration data is sampled for every hour, while the statistics data is sampled for every 1 seconds. All data is summarized into 15 minute spills. 4.2 Baseline We start our study with a controlled workload running on a dedicated Linux system, and obtain Utilization, as shown in Figure 2. Utilization of Workloads on Standalone Linux Response Time (sec) Response Time of Workload "work" : 12:15 12:3 12:45 13: 13:15 13:3 13:45 Figure 3. Response time of standalone Linux We first run the same workload in one of the guests, and leave the other idle. Both the utilization and throughput remain pretty much the same as in a standalone system although the response time increases a little bit. We then run the same workload in both guests. With the added competition, the utilization decreases and the response time increases significantly as illustrated in Figures 4 and Utilization (%) : 12:15 12:3 12:45 13: 13:15 13:3 13:45 Utilization (%) Utilization of VM1 with Competition : 18:3 19: 19:3 Others PD-Perform-Agent work Others PD-Perform-Agent work Figure 2. utilization of standalone Linux Figure 4. utilization of VM1 with competition We use BMC Performance Assurance for Servers in our study. The estimated response time well matches the response time measured by the workload itself, as illustrated in Figure 3. This demonstrates that the traditional performance analysis and modeling tool for regular Linux (or UNIX in general) works well for standalone systems. 4.3 Traditional method does not work Next we setup a Xen virtual server using Fedora Core 5, a preview of RHEL 5. It is configured with two guest domains in addition to Domain. We start with a simple configuration that can be easily extended to other configurations Response 4. Time (sec) Response Time Comparison Standalone Without Comp With Comp Figure 5. Response time comparison When we use the same analysis method as was used on the standalone system, the results are way off from

6 the actual measurement, especially when there is competition among guests. Figures 6 and 7 demonstrate the difference between estimated and measured response time. Response Time (%) Response Time of work@vm1 without Competition : 2:15 2:3 2:45 21: 21:15 21:3 21:45 Figure 6. Response time of VM1 without competition Response Time (sec) Response Time of work@vm1 with Competition : 18:15 18:3 18:45 19: 19:15 19:3 19:45 Figure 7. Response time of VM1 with competition Obviously, traditional performance analysis and modeling method for regular system does not work for the virtual server environment. 4.4 Consolidating workloads In older versions of XenLinux, the standard Linux accounting was based on virtual time, and therefore could not be used for performance modeling. As of 3..2, Xen implements steal (involuntary wait) time accounting in Linux under the paravirtualization configuration. As a result, the standard Linux accounting in the Xen kernel tracks real time instead of virtual time. Thus, the performance data collected from within the guest OS may be used for performance modeling. In this case, the involuntary wait time becomes an important metric because it keeps track of the time when there is a process in a ready state but the is not available. It indicates the performance impact due to the competition from other domains. Figure 8 shows the relationship between the time and the involuntary wait time. 4% 3% 2% 1% % Involuntary Wait Time vs. Time 18: 18:15 18:3 18:45 19: 19:15 19:3 19:45 VM1 VM1 IV wait VM2 VM2 IV wait Domain Figure 8. Involuntary wait time vs. time As a modeling workaround, we may build a performance model for each individual guest using existing tools. Next, we revise the configuration of Domain to match the capacity of the entire physical server. We then move all workloads from each individual guest to the updated system of Domain. In this way, all workloads from various isolated guests are competing together. This method produces reasonable results as shown in Figure 9. Response Time of work@vm1 with Competition Response 4 Time (sec) : 18:15 18:3 18:45 19: 19:15 19:3 19:45 Figure 9. Response time using workaround 4.5 Modeling at hypervisor level The previous workaround does not apply to fullyvirtualized domains that run unmodified kernels and will see virtual time only. Moreover, it requires a performance model for every guest running on the virtual server. Thus, a data collector has to run on each guest. This would incur considerable overhead when there are a large number of guests running together. Since the most commonly used configuration of virtual server is to have each application running on an isolated virtual machine, there is not much need to have workload characterization at the process level. Rather, each virtual machine (guest) can be seen as a workload running on the physical server. Hence, it does not require one to instrument every guest OS.

7 Therefore, one only needs to collect performance data at the hypervisor level via Domain. When we run the workloads in the Xen guests for the experiment in the previous section, we have Xen specific data collected as well. Figure 1 illustrates the utilization for each domain. Although the data matches the data collected within each guest, they are from totally different sources. Furthermore, we still have accurate accounting for fully-virtualized domains, while the accounting from OS would be virtual. 7% 6% 5% 4% 3% 2% 1% Overall Utilization from Hypervisor % 18: 18:3 19: 19:3 2: 2:3 21: 21:3 Domain VM1 VM2 Figure 1. Overall utilization from hypervisor Xen is mostly comparable with VMware ESX server, especially under full-virtualization. There are some existing performance analysis and modeling tools for VMware ESX server that are proven and working well. With the configuration and statistics data for each domain hosted by Xen, we use BMC Performance Assurance for Virtual Servers to model Xen. Figures 11 and 12 show the breakdown of degradation of the two guests based on the performance model built with hypervisor level data. These metrics provide the quantified indicator on the performance impact among guests within the same virtual server. Degradation (%) Degradation of VM1 18: 18:3 19: 19:3 2: 2:3 21: 21: Degradation : 18:3 19: 19:3 2: 2:3 21: 21:3 5. Conclusion degradation Degradation of VM2 degradation by other guests Figure 12. degradation of VM2 Xen provides an alternate virtualization solution. We have demonstrated that traditional performance analysis and modeling methods do not work in a virtualized environment. Then we presented a practical approach for measuring and modeling the performance of Xen virtual server. Correlating the data from hypervisor and data from within a guest OS provides both top down and bottom up views of performance aspects on Xen server. References [1] Paul Barham, Boris Dragovic, Keir Fraser, Steven Hand, Tim Harris, Alex Ho, Rolf Neugebauer, Ian Pratt and Andrew Warfield, Xen and the Art of Virtualization, Proceedings of the ACM Symposium on Operating Systems Principles, 23. [2] Ian Pratt, Keir Fraser, Steven Hand, Christian Limpach and Andrew Warfield, Xen 3. and the Art of Virtualization, The 6th Free and Open source Software Developers' European Meeting, Brussels, 26. [3] S-expression, [4] The Xen virtual machine monitor, [5] XenSource, degradation degradation by other guests Figure 11. degradation of VM1