October 2006 www.veritest.com info@veritest.com AMD Opteron Processor vs. Intel Woodcrest Performance Analysis with VMware ESX 3.0 Test report commissioned by AMD. Executive summary Advanced Micro Devices, Inc. ( AMD ) commissioned VeriTest, the service of Lionbridge Technologies, Inc. to conduct a competitive analysis and performance testing of commercially available Tier 1 AMD and Intel platforms on VMware ESX 3.0. The platforms chosen were an AMD Opteron 2218 (2.6GHz) processor-based Tier 1 platform and an Intel 5150 (2.66GHz) Woodcrest Tier 1 platform. VMware software virtualizes servers so that multiple operating systems, each encapsulated in an independent virtual machine, can run on the same physical server. Key findings The AMD Opteron 2218 processor-based Tier 1 platform exhibited a 10% advantage over the Intel 5150 Woodcrest Tier 1 platform for the light IO workload and increased this advantage as the IO workload increased. The AMD Opteron 2218 processor-basedtier 1 platform exhibited a 26% advantage over the Intel 5150 Woodcrest Tier 1 platform for the heavy IO workload. The AMD Opteron 2218 processor-based Tier 1 platform maintained a slight edge in power consumption for all workloads, giving the Opteron platform as much as a 26% performance/watt advantage. The purpose of this test was two-fold: 1. Determine how the AMD Opteron 2218 processor-based Tier 1 platform compares to the Intel 5150 Woodcrest Tier 1 platform across a standard mixed workload of DB Hammer, WebBench, and NetBench 2. Determine how both platforms scale as the IO workload is increased The standard WebBench, NetBench, and DB Hammer workloads do not stress the IO bandwidth capabilities of platforms like many real world workloads. For this testing, it was discovered that only about 50 MB/sec of IO traffic was generated when loading the platform with 12 VMs of DB Hammer, WebBench, and NetBench (4 VMs of each). Real world web and file servers often generate hundreds of MB/sec of IO traffic on 2-socket dual-core platforms. To increase the IO load in order to more closely emulate real world server use, the file sizes of the WebBench and NetBench workloads were incremented to generate more IO. Every file within WebBench and NetBench were incremented by the same percentage in order to maintain the same relative distribution of file sizes. The original WebBench and NetBench file sizes are referred to as the Light workload and generate approximately 50 MB/sec of traffic on each of the platforms tested. The file sizes were then increased to generate IO workloads of approximately 75 MB/sec, 100MB/sec, and 150 MB/sec. These are referred to as the Medium1, Medium2, and Heavy workloads respectively.
VeriTest conducted the study by setting up a test bed that consisted of the AMD or Intel-based server being tested, a storage-area-network (SAN), 12 physical clients, a dual-port 2 Gb fibre channel adapter, and a gigabit Ethernet switch. The AMD or Intel-based servers under test were configured to use VMware ESX Server 3.0. Virtual machines and depending on the application were configured to run 64-bit Windows 2003 Server Enterprise Edition, 32-bit Windows 2003 Server Enterprise Edition, or 64-bit SuSE Linux Enterprise 9.3, SP3. In addition, the test bed had pre-defined workloads that included Symmetric Multi-Processor system (SMP) 64-bit SQL (single VM connected to, but not pinned to, two CPU cores), Uni-Processor system (UP) 32-bit SQL (single VM connected to, but not pinned to, a single CPU core), 64-bit WebBench, and 64-bit NetBench. Besides the 64-bit or 32-bit difference between the SQL virtual machines, the 64-bit version of SQL operated as a SMP and the 32-bit version of SQL operated as a UP. In order to determine the relative performance between these two systems, 12 virtual machine workloads were created and then the Light, Medium1, Medium2, and Heavy workloads comprised the four test runs. During each test run, all 12 virtual machines ran at the same time. Overall, there were 2 SMP 64-bit SQL2005 workloads, 2 UP 32-bit SQL2005 workloads, 4 64-bit UP WebBench workloads, and 4 64-bit UP NetBench workloads. The SQL workload used a combination of 2 64-bit SMP and 2 32-bit UP workloads to represent a range of database work. The 2 SMP SQL workloads were configured to generate 90 instances every 100ms by DBHammer, while the 2 UP workloads were configured to generate 50 instances every 100ms. For a constant workload, CPU utilization can be used as a metric for performance but the actual batch requests/sec also needs to be monitored as that indicator can vary. The CPU utilization was captured using VMware s ESXTOP utility. The PerfMon tool was used to collect the SQL batch requests/sec. Testing Methodology As stated in the Executive Summary, AMD commissioned VeriTest, a division of Lionbridge technologies, to conduct a structured sequence of tests in order to evaluate a Tier 1 AMD Opteron processor-based platform against a Tier 1 Intel-based server platform using VMware s ESX 3.0 virtualization software. Due to logistical and scheduling requirements, the testing was executed onsite at AMD s facility in Austin, TX. A complete audit of the system setup and configuration was completed on all client systems and servers under test by a VeriTest engineer to confirm and validate usage of default settings. Each server had the following components in common: 2 dual-core Processors o AMD Opteron 2218 (2.6GHz w/ 2x1MB L2) o Intel 5150 (2.6GHz/1333MHz with 4MB L2) 16GB of Memory o AMD: PC2-4200 DDR2 RDIMM (2GB DIMMs) Reg ECC o Intel: PC2-5300 DDR2 FBDIMM (2GB DIMMs) REG ECC 2 Intel PRO/1000 PT Dual Port Server Adapters Dual-port 2Gb Fibre Channel QLogic QLE2462 63GB 15k rpm SAS Drive with ESX 3.0 1 Power Supply VMware ESX Server 3.0.0 AMD supplied the server hardware for both the AMD and the Intel-based systems. Test Environment The test bed consisted of one of the previously mentioned servers under test, one 16-port SMC smart Gigabit Ethernet switches, two HP Modular Storage Arrays dedicated to each server being tested, and 12 physical clients. A complete description of the test bed is revealed in Appendix A, with a graphical representation shown. The test tools used for this study consisted of the following: AMD Virtualization Competitive Performance Study 2
VMware ESX Server 3.0.0- ESX Server 3.0.0 is virtualization software that allows for the partitioning of a single physical server into multiple virtual servers. Each of these virtual servers has its own virtualized hardware that includes CPUs, system memory, disk drives, and networking interfaces. In addition, each of these virtual servers can and will run its own operating system. ESX Server 3.0 maps the virtualized hardware for each of these virtual servers to the actual physical server hardware. ESX Server 3.0.0 64-bit guest support is currently experimental per VMware s published documentation. SQL DBHammer- DBHammer is a software testing utility that provides database loads in order to test Microsoft SQL server. In this study, DBHammer was configured to generate workloads that would allow SQL batch commands to be tracked. WebBench- WebBench is a software testing utility that uses physical client systems to simulate users with web browsers that are requesting static or dynamic web page content. In this test, WebBench was configured to request static content only. NetBench- NetBench is a software testing utility that measures how well a file server handles file input/output requests from Windows clients. In this study, NetBench was utilized to generate workloads that would allow network throughput to be measured. Virtual Machine Workload Configurations In all tests, the SQL workload was generated using the DBHammer utility, driven from external clients. The first test leveraged the Light workload files for WebBench and NetBench. These files represented the original WebBench and NetBench file sizes and generated approximately 50 MB/sec of traffic. The second test leveraged the Medium1 workload files for WebBench and NetBench. These files represented the original WebBench and NetBench file sizes increased to generate approximately 75 MB/sec of traffic. The third test leveraged the Medium2 workload files for WebBench and NetBench. These files represented the original WebBench and NetBench file sizes increased to generate approximately 100MB/sec of traffic. The fourth and final test leveraged the Heavy workload files for WebBench and NetBench. These files represented the original WebBench and NetBench file sizes increased to generate approximately 150 MB/sec of traffic. Testing Process The testing process consisted of utilizing pre-configured VM workloads and applying a specified quantity and mix of them for four testing passes. In each subsequent testing pass the WebBench and NetBench workloads were increased in order to generate a larger overall test load. Here follows a brief description of each test pass: Test Run #1: Light This test run consisted of loading the 12 VM workloads and the Light WebBench and NetBench workload files. AMD Virtualization Competitive Performance Study 3
Test Run #2: Medium1 This test run consisted of loading the 12 VM workloads and the Medium1 WebBench and NetBench workload files. Test Run #3: Medium2 This test run consisted of loading the 12 VM workloads and the Medium2 WebBench and NetBench workload files. Test Run #4: Heavy This test run consisted of loading the 12 VM workloads and the Heavy WebBench and NetBench workload files. Test results The goal of the testing was to demonstrate performance of the two systems under different types of load. Figure 1 shows the IO load and IO performance of both platforms. As can be seen from the chart, as the WebBench and NetBench file sizes were incremented for the four workload categories, the IO loading for the Light, Medium1, Medium2, and Heavy workloads produced approximately 50MB/s, 75MB/s, 100MB/s and 150MB/s loads respectively. The AMD Opteron 2218 processor-based platform was capable of performing more work, thus resulting in higher levels of IO traffic than the Intel 5150 Woodcrest platform. The red line indicates the percentage of IO that the AMD platform generated relative to the Intel platform. For the heavy workload, the AMD Opteron 2218 platform generated 24% more IO traffic (total disk and network traffic) than the Intel platform. This correlates well with the 26% advantage seen from the geometric mean of the overall benchmark scores for the heavy workload (see Figure 3). AMD/Intel IO Performance on Vmware ESX 3.0 (Total Disk + NW Traffic) 180.0 140% Intel Disk+NW 160.0 AMD Disk+NW AMD/Intel Relative % 150 MB/s 130% 140.0 124% 120% 120.0 114% 110% MB/sec 100.0 80.0 108% 75 MB/s 108% 100 MB/s 100% 90% 60.0 50 MB/s 80% 40.0 70% 20.0 60% 0.0 Light Medium1 Medium2 Heavy 50% Figure 1: I/O Performance AMD Virtualization Competitive Performance Study 4
Figure 2 below shows the individual breakdown of each workload type under the four levels of load. In this chart, the WebBench scores drop as we progress from the Light to Heavy workloads this is due to the WebBench score being measured in requests/sec and each request requiring more data as the file sizes are increased. Likewise, the NetBench scores change as the workload is increased, but since NetBench is scored in Mbits/sec, its score increases as the IO requirements increase. Finally, the DBHammer score trends downward across the four workload types due to heavier demands from the WebBench and NetBench VMs. In addition to these trends, the chart shows that AMD platform outperformed the Intel platform for all 3 benchmarks on all 4 workload types, with the WebBench results showing the greatest relative performance difference. Workload Scores (Sums of Individual VMs) (DBH and WB in rqsts/sec, NB in Mb/sec) 3000.00 2500.00 2000.00 1500.00 DBHammer Intel DBHammer AMD WebBench Intel WebBench AMD NetBench Intel NetBench AMD 1000.00 500.00 0.00 Light Medium1 Medium2 Heavy Figure 2: Workload Scores Figure 3 summarizes the overall benchmark performance showing the performance of the AMD platform relative to the Intel platform. This chart is the same data as that shown in Figure 2, but shown in a manner that assists in visualizing the performance differences of the two platforms. As can be seen from Figure 3, the AMD platform was able to maintain an average performance advantage in each of the 12 VMs simultaneously. Performing a geometric mean of the 3 major benchmarks (composed of scores from 4 VMs each), we see that the AMD platform begins with a 10% performance advantage on the Light workload and extends this advantage to 12%, 18%, and on to 26% as the IO requirements are incremented from 50MB/sec up to 150MB/sec. AMD Virtualization Competitive Performance Study 5
AMD/Intel Relative Benchmark Performance on VMware ESX 3.0 (12 VMs: 2 SMP DBH, 2 UP DBH, 4 WB, 4 NB) AMD Opteron 2218 (2.6GHz) vs Intel 5150 (2.66GHz Woodcrest) Relative Percentage (AMD/Intel). 160% 140% 120% 100% 80% 60% DBH BatchReq/Sec WebBench Req/Sec NetBench Mb/Sec Geomean 110% 112% 118% 126% 112% 113% 106% 107% 129% 102% 107% 141% 108% 110% 151% 119% 40% 20% 0% Light Medium1 Medium2 Heavy Figure 3: Overall Relative Benchmark Performance AMD Virtualization Competitive Performance Study 6
October 2006 www.veritest.com info@veritest.com This table contains all of the benchmark scores and IO traffic for each of the 3 benchmarks on each of the 4 workloads. The network and disk traffic as well as % CPU and total system power are included in the table as well. From the table, it shows that the AMD Opteron processor-based platform maintained a slight power advantage across the entire range of testing, giving the AMD Opteron platform up to a 26% performance/watt advantage and well as an overall 26% performance advantage. AMD Opteron 2218 (2.6GHz) processor vs. Intel 5150 (2.66GHz Woodcrest) on VMWare ESX 3.0 Light Medium1 Medium2 Heavy Intel AMD Rel % Intel AMD Rel % Intel AMD Rel % Intel AMD Rel % DBH BatchReq/Sec 2,516 2,815 112% 2,494 2,677 107% 2,478 2,662 107% 2,400 2,645 110% DBH BatchReq/Sec WebBench Req/Sec 1,330 1,506 113% 1,080 1,389 129% 903 1,275 141% 481 728 151% WebBench Req/Sec NetBench Mb/Sec 118 126 106% 298 303 102% 425 461 108% 472 564 119% NetBench Mb/Sec Geomean 110% 112% 118% 126% Geomean Network total Mb/s 258 266 103% 499 531 106% 645 738 114% 739 945 128% Net Transmit Mb/s Disk Total MB/s 15 18 120% 17 19 112% 19 21 111% 42 48 114% Total MB/s CPU % 99.25 98.75 99.5% 100.00 98.25 98.3% 100.00 99.50 99.5% 100.00 99.00 99.0% CPU % Power (W) 320.2 316.6 98.9% 321.3 317.6 98.9% 321.1 319.4 99.5% 321.7 320.8 99.7% Power (W)
October 2006 www.veritest.com info@veritest.com Conclusion Commissioned by AMD, VeriTest performed performance testing to evaluate AMD and Intel-based platforms using VMware ESX Server 3.0 virtualization software and to gather real-world measurements. The goal of this testing was to provide an independent, third-party analysis of virtual machines with four distinct workloads, and to use benchmark scoring as a basis for comparing the overall performance of AMD versus Intel processors, in a heterogeneous virtualized environment. The results of this study showed that the AMD Opteron 2218 processor-based Tier 1 platform exhibited a 10% advantage over the Intel 5150 Woodcrest Tier 1 platform for the light IO workload and increased this advantage as the IO workload increased. Additionally, the AMD Opteron 2218 processor-based Tier 1 platform exhibited a 26% advantage over the Intel 5150 Woodcrest Tier 1 platform for the heavy IO workload. Finally, the AMD Opteron 2218 processor-based Tier 1 platform maintained a slight edge in power consumption for all workloads, giving the AMD Opteron platform as much as a 26% performance/watt advantage.
Appendix A: Test Bed Details Workload Total VMs 2 2P VMs of 64bit SQL2005 2 1P VMs of 32bit SQL2005 4 1P VMs of 64bit WebBench (64bit Linux/Apache) 4 1P VMs of 64bit NetBench (64bit Linux/Samba) 12 Workloads o 2P SMP SQL 64-bit Microsoft SQL Server 2005 DBHammer, driven from external clients with 90 instances and 100ms Single network card connected using e1000 driver C: drive located on OS LUN D: drive located on SQL LUN for database and log o 1P UP SQL 32-bit Microsoft SQL Server 2005 DBHammer, driven from external clients with 50 instances and 100ms Single network card connected using flexible VMware driver C: drive located on OS LUN D: drive located on SQL LUN for database and log o WebBench WebBench version 5.0 Apache version 2-2.0.49-27.38 Static workload only with Scaling file sizes. 1 client / 2 engine from 1 physical client per WebBench VM / drive located on OS LUN (file set on OS LUN also) Single NIC per VM, connected to private subnet o NetBench NetBench version 7.0.3 / drive located on OS LUN Samba share drive located on NB LUN for database 1 client / 10 engines from 1 physical client per NetBench VM Dm.scr workload with Scaling file sizes/operations. Single NIC per VM, connected to private subnet. Metrics o System CPU Utilization from ESX_TOP with 2 second sample rate. Avg Disk Rd/sec from each LUN OS, SQL, NB (fm ESX_TOP) Avg Disk Wr/sec from each LUN OS, SQL, NB (fm ESX_TOP) Collect performance data for 25 minutes (covers entire NB and WB runs) Power consumption collected for 25minutes (covers all runs) o SMP SQL SQL Batch Requests/Sec from each SQL VM (from PerfMon) Run SQL and collect performance data for 25 minutes o UP SQL SQL Batch Requests/Sec from each SQL VM (from PerfMon) Run SQL and collect performance data for 25 minutes o WebBench Requests/Sec and Response Time from each WebBench VM AMD Virtualization Competitive Performance Study 2
o Run 4 standard 300sec mixes, reporting score from 3rd iteration NetBench Throughput and Average Response Time from each NetBench VM Run 2 standard 660sec iterations, reporting score from 2 nd iteration. Tests All tests are run with the 12 VMs described above. The only change is the file sizes of the WebBench and NetBench workloads, increasing each file in the respective workload by a specified percentage. Every WebBench and NetBench file is increased by the same percentage to preserve the relative file size distribution. This allows us to scale the amount of IO the workload will generate as it is executed. Test Run Description 1 Light Workload (Unmodified workload) 2 Medium1 Workload (approximately 1.5x the total IO of the unmodified) 3 Medium2 Workload (approximately 2x the total IO of the unmodified) 4 Heavy Workload (approximately 3x the total IO of the unmodified) Software ESX o ESX 3.0.0 o Primary Partitions /boot = 100MB Swap = 1600MB / = 6000MB o Extended Partitions /var = 8000MB (should be plenty for logs) /vmimages = 16,000MB /home = 12,000MB /vmfs = ~50GB o System memory allocation 272MB - service console (default) 2GB for SMP SQL VMs (1GB per VM) 1GB for UP SQL VMs (512 MB per VM) 2GB for WebBench VMs (512 MB per VM) 2GB for NetBench VMs (512 MB per VM) ~8GB free o No affinity set for any VMs Guest Operating Systems o 64-bit Windows 2003 Server Enterprise Edition for SMP SQL SQL 2005 standard install as service o 32-bit Windows 2003 Server Enterprise Edition SP1 for UP SQL VMs SQL 2005 standard install as service o 64-bit SLES9.3 SP3 for WebBench and NetBench Apache version 2-2.0.49-27.38 standard install Samba standard install for sharing AMD Virtualization Competitive Performance Study 3
Hardware Intel o Undisclosed Tier 1 OEM model o 2 dual-core Intel Xeon WoodCrest 5150 MP Processors 2.66GHz/1333MHz with 4MB L2 o 16GB of PC5300 FBDDR2 (2GB DIMMs) REG ECC o 2 Intel PRO/1000 PT Dual Port Server Adapters o Fibre Channel QLogic QLE2462 o 63GB 15k rpm SAS Drive with ESX 3.0 o 1 Power Supply AMD o Undisclosed Tier 1 OEM model o 2 AMD Opteron dual-core processors, model 2218, 95W (2.6GHz/1GHz HT with 2x1MB L2 per core) o 16GB of PC2-4200 DDR2 (2GB DIMMs) REG ECC o 2 Intel PRO/1000 PT Dual Port Server Adapters o Fibre Channel QLogic QLE2462 o 63GB 15k rpm SAS Drive with ESX 3.0 o 1 Power Supply SAN o Each System under Test has two SANs consisting of: HP Modular Storage Array (MSA1500) 2 MSA30 drive enclosures, each with 14 15k rpm 36GB Ultra320 SCSI drives for a total of 28 drives (1,008GB) 1 2GB fibre connection to SAN 1 st MSA Array configured with 2 LUNs: OS LUN: 200GB RAID5 for Guest VMs o 8GB per Guest OS SQL LUN: 160GB RAID10 for SQL Databases o 4 5GB databases for SMP SQL o 4 5GB databases for UP SQL 2nd MSA Array configured with 1 LUNs: NB LUN: 120GB RAID10 for NetBench Databases o 4 5GB databases Clients 36 Physical Clients all consisting of o 2P - Sun Microsystems Sun Fire V20z (2.2GHz) o 2GB DDR1 o Onboard Broadcom NetXtreme Gigabit Ethernet (1 port used) o 69 GB Fujitsu SCSI drive NTFS formatted o Windows Server 2003 Enterprise Edition SP1 Control PC o 2P AMD (3.0GHz) o 16GB DDR1 o Onboard Broadcom NetXtreme Gigabit Ethernet (2 ports used) o 2 Intel PRO1000 MT Dual Port NIC o Fibre Channel Qlogic QLA2342 o 18 GB Seagate SCSI drive for OS o 73 GB Seagate SCSI drive for Images and Isos o Windows Server 2003 Enterprise Edition SP1 AMD Virtualization Competitive Performance Study 4
Network Figure 4: Test Bed Configuration AMD Virtualization Competitive Performance Study 5
Storage Figure 5: Storage Configuration AMD Virtualization Competitive Performance Study 6
Appendix B: Timing for Benchmarks and Analysis Time: 0 Start of all Benchmarks and Sampling Time: 13 minutes Begin sampling Window End Power, ESXtop, & DBHammer End NetBench 12 12 Sample Mix 6 6 Sample Mix 6 6 25 0 15 30 Minutes Legend: Power, ESXtop, SQL-DBHammer NetBench WebBench Time: 21minutes End sampling Window End WebBench Figure 6: Timing AMD Virtualization Competitive Performance Study 7
October 2006 www.veritest.com info@veritest.com AMD, the AMD Arrow Logo (AMD Opteron) and combinations thereof are trademarks of Advanced Micro Devices, Inc. Windows is a registered trademark of Microsoft Corporation. VeriTest (www.veritest.com), the testing division of Lionbridge Technologies, Inc., provides outsourced testing solutions that maximize revenue and reduce costs for our clients. For companies who use high-tech products as well as those who produce them, smoothly functioning technology is essential to business success. VeriTest helps our clients identify and correct technology problems in their products and in their line of business applications by providing the widest range of testing services available. VeriTest created the suite of industry-standard benchmark software that includes WebBench, NetBench, Winstone, and WinBench. We've distributed over 20 million copies of these tools, which are in use at every one of the 2001 Fortune 100 companies. Our Internet BenchMark service provides the definitive ratings for Internet Service Providers in the US, Canada, and the UK. Under our former names of ZD Labs and etesting Labs, and as part of VeriTest since July of 2002, we have delivered rigorous, objective, independent testing and analysis for over a decade. With the most knowledgeable staff in the business, testing facilities around the world, and almost 1,600 dedicated network PCs, VeriTest offers our clients the expertise and equipment necessary to meet all their testing needs. For more information email us at info@veritest.com or call us at 919-380-2800. Disclaimer of Warranties; Limitation of Liability: VERITEST HAS MADE REASONABLE EFFORTS TO ENSURE THE ACCURACY AND VALIDITY OF ITS TESTING, HOWEVER, VERITEST SPECIFICALLY DISCLAIMS ANY WARRANTY, EXPRESSED OR IMPLIED, RELATING TO THE TEST RESULTS AND ANALYSIS, THEIR ACCURACY, COMPLETENESS OR QUALITY, INCLUDING ANY IMPLIED WARRANTY OF FITNESS FOR ANY PARTICULAR PURPOSE. ALL PERSONS OR ENTITIES RELYING ON THE RESULTS OF ANY TESTING DO SO AT THEIR OWN RISK, AND AGREE THAT VERITEST, ITS EMPLOYEES AND ITS SUBCONTRACTORS SHALL HAVE NO LIABILITY WHATSOEVER FROM ANY CLAIM OF LOSS OR DAMAGE ON ACCOUNT OF ANY ALLEGED ERROR OR DEFECT IN ANY TESTING PROCEDURE OR RESULT. IN NO EVENT SHALL VERITEST BE LIABLE FOR INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES IN CONNECTION WITH ITS TESTING, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. IN NO EVENT SHALL VERITEST'S LIABILITY, INCLUDING FOR DIRECT DAMAGES, EXCEED THE AMOUNTS PAID IN CONNECTION WITH VERITEST'S TESTING. CUSTOMER S SOLE AND EXCLUSIVE REMEDIES ARE AS SET FORTH HEREIN.