VMware ESX Server Multiple Workload Performance on Dell PowerEdge 2850 and PowerEdge 6850 Servers

VMware ESX Server Multiple Workload Performance on Dell PowerEdge 2850 and PowerEdge 6850 Servers Enterprise Product Group (EPG) Dell White Paper By Todd Muirhead, Dave Jaffe and Scott Stanford July 2005

Contents Executive Summary... 4 Introduction... 5 VMWare ESX Server Software... 5 Hardware Configuration and Setup... 5 Four VM Testing Workloads... 7 Microsoft SQL Server 2000... 7 SuSE Linux LAMP DVD Store... 8 NetBench 7.03... 8 Exchange 2003 with LoadSim 2003... 9 ESX on PowerEdge 2850 Servers... 10 Hardware Overview... 10 Testing Results... 11 Reasons to Choose PowerEdge 2850... 11 ESX on PowerEdge 6850 Servers... 12 Hardware Overview... 12 Testing Results... 13 Reasons to Choose PowerEdge 6850... 13 Mixed Workload Efficiency... 14 Results... 14 Conclusions... 15 Tables Table 1 - Dell EMC Storage Configuration... 6 Table 2 - Configuration of Virtual Machines used in testing... 7 Table 3 - PowerEdge 2850 configuration for system as used in testing.... 10 Table 4 - Multiple workload testing results for PowerEdge 2850... 11 Table 5 - PowerEdge 6850 server configuration for server used in testing.... 12 July 2005 Page 2 Dell Enterprise Product Group

Table 6 - Multiple Workload testing results for PowerEdge 6850 with the two processor types used.... 13 Table 7 - Mixed workload testing results.... 14 July 2005 Page 3 Dell Enterprise Product Group

Section 1 Executive Summary VMWare ESX Server Software on Dell PowerEdge 2850 and PowerEdge 6850 servers provide a virtualization platform that is capable of supporting a wide variety of applications and operating systems. Choosing the right PowerEdge Server depends on many factors that mostly depend on what the intended number, size, performance requirements, and availability requirements of the virtualized environments to be run. The virtualization capability of ESX Server Software allows for many Virtual Machines (VMs) to run on the same physical hardware at the same time. Each VM is configured and managed by ESX so that it has its own slice or partition of the server. ESX manages these VMs dynamically so that server resources are allocated to the VMs that are busy and then reallocated as workload changes. Dell PowerEdge 2850 and PowerEdge 6850 servers feature the latest Intel Xeon processors, PCI Express I/O slots, and ECC DDR-2 SDRAM memory, thus providing a robust, scalable, and expandable platform that is ideal for virtualization environments. Once the requirements of the VMs are understood the correct PowerEdge server can be chosen. In this paper, four different workloads were run on two Dell servers, the PowerEdge 2850 and PowerEdge 6850, to show how well ESX server software manages multiple workloads and to illustrate when to pick which type of server.. July 2005 Page 4 Dell Enterprise Product Group

Section 2 Introduction VMWare ESX Server Software VMWare ESX Server allows for many different applications to run on the same hardware at the same time. ESX Server provides the virtualization layer that resides just above the hardware to create the Virtual Machine (VM) containers. Each VM runs its own operating system (OS) which in turn has its own set of applications and services. Because each VM is isolated by ESX Server from the other VMs in the same way that physical systems are isolated from each other, there is a great amount of flexibility in using ESX Server to run multiple types of applications and operating systems at the same time. Each individual VM can be rebooted or powered off without having any impact on other VMs running on the same physical server. This allows one application on one VM to be patched or upgraded and that VM rebooted without all of the other applications running on that physical system to have any downtime because each application and operating system runs in its own VM. In addition to this, VMWare ESX Server supports a feature called VMotion which allows for a running VM to be moved from one physical server to another physical server without any downtime on the VM. A pool of servers running ESX Server software and using VMotion allows for VMs to be moved around for load balancing or maintenance reasons without impacting end users because there is no downtime during the movement of VMs. To enable this feature all servers involved in a move must be connected to the same Storage Area Network (SAN) shared storage. During a VMotion move, only the RAM contents of the VM or VMs being moved are actually copied across the network because the VM s disk is on the shared storage and is already accessible by both servers. When using VMWare ESX Server farms it will often be the case that many different operating systems and applications will be running on the same physical server at the same time. In order to measure the performance and scalability of ESX Server on Dell 8G PowerEdge Servers, it makes sense to model and test a variety of workloads. Hardware Configuration and Setup The Dell PowerEdge Servers used in testing, the PowerEdge 2850 and PowerEdge 6850, were both installed with VMWare ESX Server 2.5.1 instead of a traditional operating system such as Microsoft Windows Server 2003 or Red Hat Enterprise Linux 4. The top processors available for each system at the time of testing in July 2005 were used. The 2850 used two 3.6Ghz Intel Xeon Processors DP with 2MB L2 cache and the 6850 was configured with four of the 3.66 Ghz Intel Xeon Processors MP with 1MB L2 cache and another 6850 with four of the 3.33 Ghz Intel Xeon Processors MP with 8MB L3 cache. Each server was connected to the SAN with two Qlogic 2340 HBAs. ESX Server provides failover across the multiple paths to LUNs through both HBAs. SAN storage was provided July 2005 Page 5 Dell Enterprise Product Group

by a Dell/EMC CX700. The VMs used in the testing were spread across 48 73GB 10K rpm fibre channel disks located on the CX700. Eight five disk (4 +1) RAID5 LUNs were created. Four of these LUNs were used for the SQL Server, NetBench, LAMP, and Exchange VMs. Additionally, each of the four Exchange VMs used one of these LUNs for the mail store data drives. The final eight drives were used for four two-disk RAID1 transaction log volumes for the Exchange VMs. Controller 1 Dell/EMC CX700 Disk Enclosures 4 DAE2 Disks 48 x 73 GB/ 10K RPM LUNs 8 x 5 Disk RAID 5 4 x 2 Disk RAID 1 Software Navisphere Manager Access Logix Table 1 - Dell EMC Storage Configuration July 2005 Page 6 Dell Enterprise Product Group

Section 3 Four VM Testing Workloads To simulate the way that customers run applications in VMs on ESX Server, the systems were tested by increasing the number of VMs until CPU utilization for the entire server reached 85 percent. A utilization level of 85 was chosen as a reasonably high level of server usage that might be reached in production but is well below the max 100 percent utilization that is used in many industry standard benchmarks and is hopefully not reached in production. The four different workloads used to test virtualization on the servers were SQL Server 2000 with an OLTP workload, Microsoft Exchange 2003 Server with LoadSim 7.03, SuSE Linux with LAMP stack, and Windows 2003 Server with NetBench 7.03. Each of these workloads was setup in multiple VMs and run under the same load at the same time. By keeping all settings on the VM and driver systems the same and then observing how many VMs could be run at the same time a measure of how many VMs the server could support was obtained. Table 2 shows the configuration of the VMs used in all four tests. Workload RAM Disk NIC Virtual CPUs SQL Server 2000 512MB 10GB Vmxnet 1 SuSE LAMP 1024MB 10GB Vlance 1 NetBench 512MB 10GB Vmxnet 1 Exchange 2003 512MB 10GB 130GB Data 10GB - Logs Vmxnet 1 Table 2 - Configuration of Virtual Machines used in testing. Microsoft SQL Server 2000 The SQL Server 2000 virtual machines used in testing were the same VMs used in two previous studies ( The Scalable Enterprise: VMWare ESX Server on the Dell PowerEdge 6650 and The Scalable Enterprise: Price/Performance Benefits of VMWare ESX Server on the Dell PowerEdge 6650 vs the IBM xseries 445, both available at www.dell.com/vmware ). These VMs have an online transaction processing (OLTP) database simulating the database backend of a simple web-based store front. The database size of about 1GB is small and representative of a database used for development or testing. To simulate a load against the VM, a C program was created to make connections to the database and to simulate the actions of a user, but with a very small think time between actions. This small think time allows for a single driver system to simulate the workload of July 2005 Page 7 Dell Enterprise Product Group

a large amount of users with a small number of connections. The stress and connectivity patterns found in the combination of small think times and the single driver system is similar to how the database would be used if a middle tier application server was used to pool the connections to the database. Each SQL Server 2000 VM was driven by a single thread of the driver application with a 20 millisecond delay. SuSE Linux LAMP DVD Store NetBench 7.03 To add some variation to the VMware testing and to underscore that VMware ESX Server can support Linux virtual machines as well as Windows VMs, the DVD Store database and web based application was built on the Linux operating system, the Apache web server, the MySQL database server, and the PHP web application language. All of the components of this stack (popularly known as a LAMP stack for Linux/Apache/MySQL/PHP) are available through Open Source. While in this test all of the layers of the stack are contained in a single VM, it is easy to move the Apache/PHP web layer to one or more VMs as well as run the MySQL database on a Master VM and one or more Slave VMs using MySQL replication. The DVD store uses additional database features such as foreign keys, transactions, and full text searching that are not used in the SQL Server VM workload. Using these features causes the overall orders per minute to decrease in relation to the additional work that is done by the database for each order due to the additional CPU overhead needed for these features. The driver for the LAMP stack differs from the driver used in the SQL Server testing in that it emits HTTP requests and receives HTML code returned from the Apache/PHP layer, whereas the SQL Server driver communicates directly with the database. However it measures the same parameters: the total orders per minute handled by the application and the average response time as seen by the simulated customers. Each SuSE LAMP VM was driven by a single thread of the driver program with a 30 millisecond delay for this test. The LAMP stack has been fully documented in MySQL Network and the Dell PowerEdge 2800: Capacity Sizing and Performance Tuning Guide for Transactional Applications, available at http://www.dell.com/downloads/global/solutions/mysql_network_2800.pdf. The complete DVD Store application code, including a SQL Server and LAMP version, is freely available for public use under the GPL license at http://www.linux.dell.com/dvdstore. NetBench 7.03 from PC Magazine is a benchmark tool that is designed to simulate the type of load that a file server experiences. A set of files is created and accessed according to predefined scripts. Typically NetBench is run with an increasing number of client engines running against a single server to measure how much throughput in MB/s can be achieved with a given number of connections. In order to test the number of VMs on an ESX server, the number of VMs and the number of client engines were increased at the same rate, until the CPU utilization on the ESX server reached 85 percent. NetBench 7.03 with the included standard DiskMix script was used with a.6 second think time to connect two client engines to each VM. 1 This simulates a number of file servers hosted on the same ESX server similar to a file server consolidation scenario. The driver July 2005 Page 8 Dell Enterprise Product Group

systems where the client engines ran had mapped drives to all of the VMs in the test. In NetBench the test directories path file was modified so that as successive client engines were added they would use the next drive letter which corresponded to the next VM. Exchange 2003 with LoadSim 2003 Similar to the other tests, the objective with the Exchange 2003 test was to scale the number of simulated Outlook mail user connections that an ESX system host could support by scaling the number of concurrently running Exchange 2003 Server VMs. LoadSim 2003 simulates Outlook MAPI based user profiles and actions that are found in typical corporate messaging environments. LoadSim 2003 is a very effective tool that is used to better understand how a backend Exchange 2003 Server will perform under typical workday messaging loads. Since the I/O generated by LoadSim 2003 mimics real world Outlook user activities, the Exchange 2003 Server VM CPU, memory, network, and disk I/O subsystems are all subjected to moderately heavy random workloads. These workloads thereby stress and utilize all of the ESX host based virtualized hardware components, and the scaling results are a good indicator of how a messaging server consolidation strategy scales on an ESX virtualized PowerEdge server configuration. More information about LoadSim 2003 can be found in MMB3 Comparative Analysis available at http://www.dell.com/downloads/global/solutions/mmb2_comparison_with_mmb3.doc. With the other three workloads a larger number of smaller VMs were used, but with Exchange it made more sense to model a smaller number of larger VMs. Given the typical memory and storage requirements for backend Exchange 2003 servers, it seemed unlikely that a large number of small Exchange VMs would be used in the same way that there could be large numbers of file servers, small SQL Server databases, or LAMP stack applications. Using large VMs in one test also provided another dimension to the testing by including VMs with greater disk, RAM, and processor requirements per VM. July 2005 Page 9 Dell Enterprise Product Group

Section 4 ESX on PowerEdge 2850 Servers Hardware Overview The Dell PowerEdge 2850 is a 2 U (3.5 inches) dual processor server with an 800Mhz front side bus that can use up to 12GB of RAM (16GB in the future). The PowerEdge 2850 can support six internal disks, offers three PCI slots, and includes dual onboard gigabit NICs. Two PCI slots were used for Qlogic 2340 HBAs to provide fibre channel connectivity to the SAN. An Intel gigabit NIC was used in the final slot so that with the two onboard gigabit NICs the total number of NICs in the system was three. This allowed for 1 NIC to be used for the ESX service console, one for the VMs, and one for VMotion. The 2850 used for testing was configured with 8GB of RAM. The Dell PowerEdge 1850 and PowerEdge 1855 blade servers also support dual processors and up to 12GB of RAM, but the differences between these platforms have to do with the size of each. The PowerEdge 1850 is a 1U server that supports two disks and two PCI slots. The PowerEdge 1855 blade server runs in a chassis that is 7U high and has slots for 10 PowerEdge 1855 blade servers making it the most rack dense solution of the three systems when the chassis is fully populated. The PowerEdge 1855 also has two internal drives, but has no PCI slots. The PowerEdge 1855 does have a daughter card socket that currently has Ethernet and fibre channel HBA options available. Virtualization Software CPU Memory Internal Disks NICs Disk Controller Fibre Channel Host Bus Adapter Dell PowerEdge 2850 VMware ESX Server 2.5.1 2x 3.6 GHz Intel Xeon Processors DP w/ 2MB L3 cache 8 GB 2x 73 GB 2x 10/100/1000 Mb/s (internal) 1x Intel Pro 1000XT Gb PERC4ei (Embedded Integrated) QLogic 2340 Height 2 Rack Units (2U) or 3.5-inches Table 3 - PowerEdge 2850 configuration for system as used in testing. July 2005 Page 10 Dell Enterprise Product Group

Testing Results. The SQL Server, SuSE LAMP, and NetBench 7.03 tests all showed that approximately the same number of VMs could be supported at approximately the same ESX Server CPU utilization. Even though each of these three workloads was quite different the number of VMs that could be run was about the same. Table 4 has the results for all four workloads running on the PowerEdge 2850. In the case of the SuSE LAMP VMs, the ESX Server was overcommitted for RAM. Each of the 10 SuSE LAMP VMs was assigned 1 GB of RAM, but the 2850 was configured with only 8GB. There is a swap file that ESX Server has available if the over-commitment of memory at the VM level causes the server to actually run out of memory, but due to the way that ESX shares common memory pages between VMs and manages active memory there was no swapping during the testing. The Exchange test is quite different in that each VM was supporting 1,000 LoadSim 2003 users and using 2GB of RAM. In order to support the I/0 requirements of these users an additional data LUN was added, whereas the other tests were able to support their I/O requirements with the same disk that was used for the OS. The large size of the Exchange VMs also did not lend itself to a level of fine granularity in getting the ESX CPU utilization to the desired 85% level. With four Exchange VMs, the ESX Server CPU utilization was at 100 percent on the PowerEdge 2850. Workload Number of VMs on ESX CPU Utilization PowerEdge 2850 SQL Server 2000 11 86% SuSE LAMP 10 87% NetBench 11 85% Exchange 3 63% Table 4 - Multiple workload testing results for PowerEdge 2850. Reasons to Choose PowerEdge 2850 Testing shows that the PowerEdge 2850 is able to handle different types of workloads equally well in a virtualized environment. Additionally the high performance of the dual processor PowerEdge 2850 makes it a great system to scale out VMWare ESX Server farms. Using the PowerEdge 2850 as the building block of an ESX Server farm would provide a cost effective method for adding capacity due to its low acquisition cost. The excellent performance results of the PowerEdge 2850 make it the choice for virtualization environments that do not require more than three PCI slots for connectivity (NICs and fibre channel) and do not have large RAM requirements (greater than 12GB per system). July 2005 Page 11 Dell Enterprise Product Group

Section 5 ESX on PowerEdge 6850 Servers Hardware Overview The Dell PowerEdge 6850 is a 4 U (7 inch) server that supports up to four Intel Xeon Processors MP and 32GB of RAM. The processor cache sizes available on the 6850 are 1MB L2 and 8MB L3. The larger 8MB cache provides the better performance while the 1MB cache processors are less expensive. The PowerEdge 6850 has seven I/O slots and an optional Qlogic 2362 fibre channel daughter card. Four of the slots are PCI Express and three are PCI X. The large number of slots provides room for additional NICs which are often needed for ESX Server environments with high network bandwidth requirements or a large number of isolated networks. If using the Qlogic daughter card for fibre channel connectivity, all slots are left free for NICs or other connectivity expansion. Hot plug PCI-Express and Memory RAID features on the 6850 also raise its overall availability by further reducing the scenarios that could result in downtime. The PowerEdge 6850 used for testing was configured with 16GB of RAM and two PCI-X Intel gigabit NICs in addition to the two internal gigabit NICs on the system. Additionally two Qlogic PCI-X based Qlogic 2340s were used for fibre channel connectivity. Virtualization Software CPU Memory Internal Disks NICs Disk Controller Fibre Channel Host Bus Adapter Height Dell PowerEdge 6850 VMware ESX Server 2.5.1 4x 3.66 Ghz Intel Xeon Processors MP w/1mb L2 Cache and 4x 3.33 Ghz Intel Xeon Processors w/8mb L3 Cache 16 GB 2 x 73 GB 2x 10/100/1000 Mb/s (internal) 2x Intel Pro 1000XT Gb PERC 4/DC QLogic 2340 4 Rack Units (4U) or 7-inches Table 5 - PowerEdge 6850 server configuration for server used in testing. July 2005 Page 12 Dell Enterprise Product Group

Testing Results The PowerEdge 6850 provides excellent performance in a single system and provides high capacity in terms of I/O and RAM. ESX running with all four workloads performed better on the 6850 with 3.33Ghz processors with 8MB L3 cache, but the less expensive 3.66Ghz processors with 1MB cache were able to host at least 75% of the number of VMs as the 8MB cache processors. Table 6 shows the complete results of the PowerEdge 6850 testing under multiple workloads. Exchange testing on the PowerEdge 6850 with the 3.33Ghz 8MB cache processors encountered different challenges in getting the CPU utilization up to the 85% level compared to the less resource intensive VM stacks. In an effort to remove any bottlenecks the data disk was upgraded from a 5 disk RAID5 to a 10 disk RAID1/0, the number of LoadSim 2003 drivers was doubled, and the number of LoadSim 2003 users was increased to 4,400. None of these things significantly raised the CPU utilization, although LoadSim 2003 latency time was reduced. The additional CPU headroom that is available could be used to support additional VMs and is not wasted as it would be on a traditional nonvirtualized server. PowerEdge 6850 with 3.66Ghz 1MB Cache PowerEdge 6850 with 3.33Ghz 8MB Cache Workload Number of VMs ESX CPU Utilization Number of VMs ESX CPU Utilization SQL Server 2000 12 85% 16 87% SuSE LAMP 14 85% 16 85% NetBench 14 89% 17 86% Exchange 4 81% 4 66% Table 6 - Multiple Workload testing results for PowerEdge 6850 with the two processor types used. Reasons to Choose PowerEdge 6850 The PowerEdge 6850 provides an excellent platform for virtualization with support for up to 32GB of RAM, seven I/O slots, and additional high availability features. The large amount of RAM lends itself to supporting a large number of VMs that have significant memory requirements. The seven I/O slots allow for a large number of NICs to be installed to support VMs that have significant network bandwidth requirements. The number of I/O slots is additionally enhanced by the Qlogic 2362 daughter card, which provides dual port fibre channel connectivity without taking up a PCI slot. The high availability features of hot-plug PCI and Memory RAID increase hardware uptime by limiting downtime. In order to decide which processor to use on the PowerEdge 6850 the performance requirements must be weighed against cost savings. The larger 8MB cache Xeon processors provide higher performance and would be the best choice in performance sensitive environments. To host a large number of VMs with lower performance requirements a 6850 with 32GB of RAM and the 1MB cache Xeon processors would be a good solution. July 2005 Page 13 Dell Enterprise Product Group

Section 6 Mixed Workload Efficiency Results In order to test how efficient VMWare ESX server is at running multiple workloads at the same time, in a manner very similar to how VMs are run on ESX servers in a server consolidation environment, all four workloads were run at the same time. As close as possible to one fourth of each of the four workloads were run simultaneously on both the 2850 and 6850. To measure the efficiency, CPU utilization was calculated based on the utilizations of each of the individual workload tests and then compared with the actual CPU utilization measured during the mixed workload test. The results of the mixed workload testing show that ESX on both the 2850 and 6850 was able to handle the multiple workloads at the same time efficiently. The 2850 actual utilization was only 5% higher than the calculated value, while the 6850 was slightly under the calculated total. The four processor 6850 was better at handling the mixed workload than the two processor 2850, but both were within five percent of the calculated value, showing that there is not a significant penalty for running multiple workloads on the same server at the same time. This means that with a farm of ESX servers it is not necessary to attempt to keep VMs running the same workload type on the same physical server due to the excellent efficiency provided by ESX. PowerEdge 2850 PowerEdge 6850 with 3.33Ghz 8MB Cache Workload Number of VMs CPU Utilization Number of VMs CPU Utilization SQL Server 2000 3 23.4% 4 21.75% SuSE LAMP 2 17.4% 4 21.5% NetBench 3 23.18% 4 21.25% Exchange Server 2003 1 (750 users) 15.75% 1 (1000 users) 16.5% Calculated Mixed CPU Utilization Actual Mixed CPU Utilization Table 7 - Mixed workload testing results. 80% 81% 85% 80% July 2005 Page 14 Dell Enterprise Product Group

Section 7 Conclusions The Dell PowerEdge 2850 and PowerEdge 6850 servers provide excellent platforms for running ESX Server virtualization software. By testing multiple workloads and a mixed workload environment, the ability of ESX Server to manage a high number of VMs with a range of I/O and memory usage patterns was demonstrated to be consistently good. All four workloads showed basically the same results on each server meaning that any of these workloads is probably good for overall VM capacity sizing. Further testing with a mix of these workloads running at the same time showed that ESX Server was able to efficiently manage all of these different I/O patterns and memory usage. In choosing what server is best for an ESX Server environment, performance and capacity requirements must be considered. The dual processor PowerEdge 2850 offers a cost effective high performance solution well suited to building out a farm of ESX Servers. The PowerEdge 6850 provides high performance and high capacity solutions with support for up to 32GB of RAM, 7 I/O slots, and four processors. By determining what is the most critical factor for the virtualization environment the correct PowerEdge server can be selected. THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. Dell, PowerEdge, and PowerConnect are trademarks of Dell Inc. Intel and Xeon are registered trademarks of Intel Corp. VMware, the VMware ʺboxesʺ logo, ESX Server, GSX Server, and VMotion are trademarks of VMware, Inc. NetBench is a registered trademark of Ziff Davis Media Inc. or its affiliates in the U.S. and other countries. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell disclaims proprietary interest in the marks and names of others. 1 The NetBench client driver systems were two Dell PowerEdge 6650 servers with four 2.8 Ghz Intel processors installed with Microsoft Windows Server 2003 with 8GB of RAM and using Intel Gigabit Ethernet adapters. NetBench Client driver systems and ESX servers used in test were connected to Dell PowerConnect 5224 Gigabit Ethernet switch. The NetBench Controller was running Windows 2003 Server Enterprise Edition. 2 The use of Microsoft SQL Server in this test does not indicate that Dell or Microsoft has tested or certified SQL Server installations over VMware virtualization software. Furthermore, as described in Microsoft Knowledge Base article # 273508, Microsoft does not support issues that occur in Microsoft operating systems or programs that run in a virtual machine until it is determined that the same issue can be reproduced outside the virtual machine environment. In addition, the use of Microsoft Exchange Server 2003 in this test does not indicate that Dell or Microsoft has tested or certified Microsoft Exchange Server 2003 installations over VMware virtualization software. Copyright 2005 Dell Inc. All rights reserved. Reproduction in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell. Information in this document is subject to change without notice. July 2005 Page 15 Dell Enterprise Product Group