January 2003 www.veritest.com info@veritest.com VERITAS Software - Storage Foundation for Windows Dynamic Multi-Pathing Performance Testing Test report prepared under contract from VERITAS Software Corporation Executive summary VERITAS Software Corporation ( VERITAS ) commissioned VeriTest, a division of Lionbridge Technologies Inc., to conduct a series of performance tests between VERITAS Dynamic Multi-Pathing and EMC PowerPath in a Microsoft Cluster Server environment with an EMC Clariion CX600 storage array. The VERITAS Dynamic Multi-Pathing is part of VERITAS Storage Foundation for Windows software, previously VERITAS Volume Manager, and provides transparent, redundant paths to disk drives from the server. PowerPath is a stand-alone product from EMC that provides transparent redundant paths to disk drives from the server. Key findings In our test configurations, we found that VERITAS Dynamic Multi-Pathing generated better Tpm scores compared to the EMC PowerPath product in both the out-of-box and vendor recommended HBA configurations. In our test configurations, we found that VERITAS Dynamic Multi-Pathing generated better TpmC scores compared to the EMC PowerPath product in both the out-of-box and vendor recommended HBA configurations. During configuration and testing, we liked the fact that we were able to perform all operations using the GUI interface provided with VERITAS Storage Foundation for Windows. Installing the EMC PowerPath product required that we make use of the command line interface. For these tests, we configured identical testbeds for both products. East testbed consisted of a Dell PowerEdge 4400 configured with dual 1GHz Pentium III Xeon processors, 2GB of RAM and running Windows Server 2003 Enterprise Edition with all latest updates. We installed a Qlogic QLA-2342 Host Bus Adapter ( HBA ) in the PowerEdge 4400 and connected it to the EMC CX600 using a Brocade 3800 2Gb 16-port switch. Additionally, we set up four identical Dell OptiPlex GX115 client systems configured with a single 866Mhz Pentium III processor and 256MB of RAM for use as load generation and monitoring systems. Finally, we used Microsoft Cluster Server Software to create a highly-available server environment. To measure the performance using the products above, we executed a series of Microsoft Benchcraft TPC-C tests. The Benchcraft software is provided as part of the Microsoft SQL Server TPC-C Benchmark Kit. We conducted these tests using an out-of-box configuration as well as a vendor-recommended configuration that consisted primarily of modifying the default HBA settings to those recommended by VERITAS and EMC for their respective products. The Benchcraft TPC-C software generates two different metrics, namely Tpm and TpmC. Tpm is defined as transactions per minute and is the number of total transactions, new orders, payments, delivery, stock level and order status that the system is able to process in one minute. TpmC is defined as transactions per minute order entry. This is the numbers of new orders the system is able to process in one minute. Please refer to the Test Methodology section of this report for complete details on how we conducted these tests.
Figure 1 below compares the results of the TpmC testing for both the VERITAS Dynamic Multi-Pathing and EMC PowerPath products using out-of-box and vendor-recommended HBA configurations. Testing with our configurations showed that VERITAS Dynamic Multi-Pathing outperformed EMC PowerPath in the out-of-box HBA configurations as well as after tuning the HBA using the vendor-recommended guidelines. Additionally, we found that using the EMC vendor-recommended HBA tunings had no impact on the performance generated using the VERITAS Dynamic Multi-Pathing product. TpmC Scores Out-of-Box HBA Configuration Vendor-Recommended HBA Configuration VERITAS Dynamic Multi- Pathing 3661 3661 EMC PowerPath 3240 3328 Figure 1. TpmC Scores Out-of-Box and Vendor-Recommended HBA Configurations Figure 2 below compares the results of the Tpm testing for both the VERITAS Dynamic Multi-Pathing and EMC PowerPath products using out-of-box and vendor-recommended HBA configurations. Testing with our configurations showed that VERITAS Dynamic Multi-Pathing outperformed EMC PowerPath in the out-of-box HBA configurations as well as after tuning the HBA using the vendor-recommended guidelines. Additionally, we found that using the EMC vendor-recommended HBA tunings had no significant impact on the performance generated using VERITAS Dynamic Multi-Pathing. Tpm Scores Out-of-Box HBA Configuration Vendor-Recommended HBA Configuration VERITAS Dynamic Multi- Pathing 8557 8555 EMC PowerPath 8508 7825 Figure 2. Tpm Scores Out-of-Box and Vendor-Recommended HBA Configurations VERITAS Software - SFW Dynamic Multi-Pathing Performance Testing 2
Testing methodology VERITAS Software Corporation ( VERITAS ) commissioned VeriTest, a division of Lionbridge Technologies Inc., to conduct a series of performance tests between VERITAS Dynamic Multi-Pathing and EMC PowerPath in a Microsoft Cluster Server environment with an EMC Clariion CX600 storage array. The VERITAS Dynamic Multi-Pathing is part of the VERITAS Storage Foundation for Windows software and provides transparent, redundant paths to disk drives from the server. EMC PowerPath is a stand-alone product from EMC that provides transparent, redundant paths to disk drives from the server. The testing consisted of installing both products under test on identical testbeds consisting of server-class systems, four load-generating clients and an EMC CX600 system used as an SQL database repository. We then used Microsoft s Benchcraft TPC-C software to measure the performance of both products under test. The Benchcraft software is provided as part of the Microsoft SQL Server TPC-C Benchmark Kit. There are significant parameters that EMC recommends be applied to the Qlogic HBA when used with PowerPath that are not required by VERITAS when using VERITAS Dynamic Multi-Pathing. Therefore, the decision was made to run two different tests. The first with the Qlogic HBA in an out-of-box configuration, or the way the HBA comes configured from the factory. Running two separate tests enables us to see what effect, if any, the parameters make to either environment. Figure 3 below shows both the default Qlogic settings, as well as those recommended by EMC. These parameters were taken directly from the EMC Fibre Channel with Qlogic Host Bus Adapters in the Windows environment guide, P/N 300-000-164 - REV A03. Parameter QLogic Default Setting Recommended EMC Setting Data Rate 0 (1 Gb/s) 2 (Auto Select) Execution Throttle 16 256 Connection options (topology) 2 (Loop preferred, otherwise point-to-point) 1 (point to point only) Enable LIP Full Login Yes Yes Enable Target Reset No Yes Port Down Retry Count 8 45 Luns Per Target 8 256 Adapter Hard Loop ID Enabled Disabled Hard Loop ID 125 0 Descending Search LoopID 0 1 Figure 3. Default and EMC-Recommended Qlogic HBA Settings Testbed Configuration Details This section describes the details of the testbeds used in these tests for both products under test. We configured identical testbeds for both products. Please refer to Appendix A for a graphic depiction of the testbeds we used. Each testbed consisted of a Dell PowerEdge 4400 configured with dual 1GHz Pentium III Xeon processors, 2GB of RAM and running Windows Server 2003 Enterprise Edition with all latest updates. Additionally, we set up four identical Dell OptiPlex GX115 client systems configured with a single 866Mhz Pentium III processor, 256MB of RAM and running Windows Server 2000 and Service Pack 4 for use as loadgeneration and monitoring systems. We connected the server and client systems using 100 Mbps full duplex network connections. We installed a Qlogic QLA-2342 Host Bus Adapter in the PowerEdge 4400 and connected it to the EMC CX600 using a Brocade 3800 2Gb 16-port switch. Finally, we used Microsoft Cluster Server Software to create a highly-available server environment. VERITAS Software - SFW Dynamic Multi-Pathing Performance Testing 3
When testing with VERITAS Dynamic Multi-Pathing, we connected the PowerEdge 4400 server running Windows 2003 Enterprise Server using the VERITAS-recommended topology as detailed in the VERITAS Storage Foundation for Windows Implementations Guide. We installed Windows 2003 Enterprise Server with VERITAS Storage Foundation for Windows as detailed in the VERITAS Storage Foundation for Windows Administration Guide (VERITAS part number N102768) and applied the most current patch level as of January 19, 2004. MSCS was installed and configured to work in conjunction with VERITAS Storage Foundation for Windows. MS SQL Server was installed with the Latin1_General_Binary collation option as a virtual server in MSCS and patched with SP3a. When testing with EMC PowerPath, we connected the PowerEdge 4400 server running Windows 2003 Enterprise Server using the EMC-recommended topology as detailed in the EMC PowerPath Version 3.0 Installation and Administration Guide for Windows P/N 300-000-512 REV A-03. We installed Windows 2003 Enterprise Server with EMC PowerPath and applied the most current patch level as of January 19, 2004. MSCS was installed and configured to work in conjunction with EMC PowerPath. MS SQL Server was installed with the Latin1_General_Binary collation option as a virtual server in MSCS and patched with SP3a. After setting up the testbeds for each product as shown above, we created and configured a set of LUNs on the CX600 for use during the testing. This configuration was identical for each of the products under test and is shown in figure 4 below. All LUNs created on the CX600 were multi-pathed. Testing using this configuration allows the SQL log LUN to have a dedicated channel to the array during testing. The transaction log is critical to database performance. As only cold backups were performed during testing, the SQL log LUNs were not in use at the same time as the backup LUNs. Finally, the EMC PowerPath connections to the CX600 were identical to those used for the VERITAS Dynamic Multi-Pathing configuration with the exception that both service processor ports 0 and 1 are connected to the fabric in accordance with EMC s documentation. Volume ID Volume Size Usage LUN # Storage Processor / Port Disk Configuration Q 500MB Quorum 1 A / 0 Disk Group1 - Single Disk J 5GB SQL Core 2 A / 0 Disk Group1 - Single Disk K 40GB SQL Data 3 A / 0 Disk Group2-6 disk RAID 0 L 20GB SQL Logs 4 B / 0 Disk Group3-6 disk RAID 1 Y 150GB Backup 1 5 B / 0 Disk Group4-4+1 RAID 5 Z 150GB Backup 2 6 B / 0 Disk Group4-4+1 RAID 5 Figure 4. CX600 LUN Configuration for All Testing Configuring and Running Benchcraft TPC-C To measure the performance using the products above, we executed a series of Microsoft Benchcraft TPC-C tests. The Benchcraft software is provided as part of the Microsoft SQL Server TPC-C Benchmark Kit. We conducted these tests using an out-of-box configuration as well as a vendor-recommended configuration that consisted primarily of modifying the default HBA settings to those recommended by VERITAS and EMC for their respective products. The Benchcraft TPC-C software generates two different metrics, namely Tpm and TpmC. Tpm is defined as Transactions per minute and is the number of total transactions, new orders, payments, delivery, stock level and order status that the system is able to process in one minute. TpmC is defined as Transactions per minute order entry. This is the numbers of new orders the system is able to process in one minute. For more information or additional detail on TPC-C benchmarking tests please visit the Transaction Processing Performance Council s website at http://www.tpc.org. We modified the Benchcraft SQL scripts to conform to the current environment. This primarily entailed changing pathnames in the 800.war setup files to match the current environment. The TPCC SQL database was created using Benchcraft s supplied GUI, with the supplied default operations. In addition to the default VERITAS Software - SFW Dynamic Multi-Pathing Performance Testing 4
operations, we selected the configure SQL Server option. This allows Benchcraft to set up SQL Server as recommended. After configuring the Benchcraft software for use with SQL Server, we configured the set of four Dell Optiplex GX-115 clients for use in the testing as follows: Performed a complete installation of the Benchcraft software on Client 1. This included the Master, Slave and Processor options. All test configuration and execution was performed from this system. Performed a slave-only Benchcraft installation on the remaining three clients. Two of these were actively used during the testing and the third was redundant. We created two driver engines for each client specifying a unique log file name, default parameter set, a Connect Rate of zero and a Start Rate of zero. Additionally, we configured each driver engine using the following parameters as shown in figure 5 below Client Emulator Connect Method Warehouse Users Connect Rate/Min Start Rate/Min client2 dvra ODBC 1-50 500 500 after start 500 after Connect dvrb ODBC 51-100 500 500 after start 500 after Connect client3 dvra ODBC 101-150 500 500 after start 500 after Connect dvrb ODBC 151-200 500 500 after start 500 after Connect client4 dvra ODBC 201-250 500 500 after start 500 after Connect dvrb ODBC 251-300 500 500 after start 500 after Connect Figure 5. Benchcraft Load Client Emulator Parameters After completing the test configurations and before conducting the final tests, we performed basic interoperability testing including basic I/O testing utilizing the Benchcraft product in order to verify the functionality and stability of all related software and test tools. The test was executed with no special settings or configuration parameters. The servers used were configured as stated above. Additionally, we conducted a Benchcraft test with no multi-pathing software installed and confirmed there were no errors. After verifying that the test configuration was operating correctly we conducted the tests using Benchcraft for each product using the following steps: Double-checked the Rates to verify both connect and start rates were set to zero. Selected all drivers. o We activated all of the drivers and waited until they were fully active. (The drivers start in a paused state.) o We changed the connect rate to 500 connections per minute and we allowed all simulated users to connect to the database. o After all users were connected, we changed the start rate to 500 users per minute. o Finally, after all simulated users had started transactions, we verified the number of connections and the number of active users to be 3000. Once all users were active, we monitored the test using the Benchcraft monitoring facility. After the test ran for the allotted time, we stopped the test. We then gathered and analyzed the test results. Two tests were executed with the out-of-box HBA configuration. The best result of the two tests was kept and the other discarded. The same test was performed after vendor-recommended HBA settings were applied. The tests were allowed to execute for enough time to generate stable results. The initial and final 10 minutes of each test s results were discarded during reporting. This further eliminated test startup and shutdown anomalies and provided more accurate results. Benchcraft s reporting tools facilitated the generation of the results detailed in this report. VERITAS Software - SFW Dynamic Multi-Pathing Performance Testing 5
Test results This section provides the complete set of test results for both the out-of-box and vendor-recommended testing. Please refer to the Test Methodology section for compete details on how we conducted these tests. Out-of-Box HBA Configuration Test Results Figure 6 below compares the results of the TpmC testing for both VERITAS Dynamic Multi-Pathing and EMC PowerPath using out-of-box HBA configurations. Testing with our configurations showed that VERITAS Dynamic Multi-Pathing outperformed EMC PowerPath in the out of the box HBA configurations. TpmC Performance - Out-of-Box HBA Configuration TpmC Score 4000 3500 3000 2500 2000 1500 1000 500 0 3661 3240 TpmC Veritas Dynamic Multi-Pathing EMC PowerPath Product Under Test Figure 6. TpmC Performance Out-of-Box HBA Configuration VERITAS Software - SFW Dynamic Multi-Pathing Performance Testing 6
Figure 7 below compares the results of the Tpm testing for both VERITAS Dynamic Multi-Pathing and EMC PowerPath using out-of-box HBA configurations. Testing with our configurations showed that VERITAS Dynamic Multi-Pathing outperformed EMC PowerPath in the out-of-box HBA configurations. Tpm Performance - Out-of-Box HBA Configuration Tpm Score 9000 8000 7000 6000 5000 4000 3000 2000 1000 0 8557 8508 Veritas Dynamic Multi-Pathing EMC PowerPath Product Under Test Tpm Figure 7. Tpm Performance Out-of-Box HBA Configuration VERITAS Software - SFW Dynamic Multi-Pathing Performance Testing 7
Vendor-Recommended HBA Configuration Test Results Figure 8 below compares the results of the TpmC testing for both VERITAS Dynamic Multi-Pathing and EMC PowerPath using vendor-recommended HBA configurations. Testing with our configurations showed that VERITAS Dynamic Multi-Pathing outperformed EMC PowerPath in vendor-recommended HBA configurations. TpmC Performance - Vendor-Recommended HBA Configuration TpmC Score 4000 3500 3000 2500 2000 1500 1000 500 0 3661 3328 Veritas Dynamic Multi-Pathing EMC PowerPath Product Under Test TpmC Figure 8. TpmC Performance Vendor-Recommended HBA Configuration VERITAS Software - SFW Dynamic Multi-Pathing Performance Testing 8
Figure 9 below compares the results of the Tpm testing for both VERITAS Dynamic Multi-Pathing and EMC PowerPath using vendor-recommended HBA configurations. Testing with our configurations showed that VERITAS Dynamic Multi-Pathing outperformed EMC PowerPath in the vendor-recommended HBA configurations. Tpm Performance - Vendor-Recommended HBA Configuration Tpm Score 9000 8000 7000 6000 5000 4000 3000 2000 1000 0 8555 7825 Veritas Dynamic Multi-Pathing EMC PowerPath Product Under Test Tpm Figure 9. Tpm Performance Vendor-Recommended HBA Configuration VERITAS Software - SFW Dynamic Multi-Pathing Performance Testing 9
Appendix A. Testbed Topology Details Figure 10. VERITAS Dynamic Multi-Pathing Testbed Configuration Figure 11. EMC PowerPath Testbed Configuration VERITAS Software - SFW Dynamic Multi-Pathing Performance Testing 10
Appendix B. Testbed System Disclosure SQL Server System Machine Type Dell PowerEdge 4400 BIOS A11 Processors 2 1Ghz Intel Pentium III Xeon Expansion Bus 64bit/66Mhz PCI Memory 2.0GB Disk(s) Clariion CX600 Network Adapter(s) Intel PRO/100 OS Windows 2003 Enterprise Figure 12. Benchcraft Client Systems Machine Type Dell OptiPlex GX-115 BIOS A03 Processors 1 866Mhz Intel Pentium III Expansion Bus 32-bit PCI Memory 256MB Disk(s) IDE Network Adapter(s) 3Com 3C920 10/100 OS Windows 2000 Server Figure 13. Core Software Versions System O/S Patch Level Hot Fixes HBA HBA/FW HBA Driver Server1 W2K3 WU Current QLA-2342 1.34 8.2.3.21 Server2 W2K3 WU Current *Dell QLA-2342 1.34 8.2.3.21 R66424 Client1-4 W2K SP4-2195 N/A *Dell R66424 required by EMC PowerPath Figure 14. Application Software Versions VERITAS Storage EMC System Foundation PowerPath for Windows SQLServer SQLServer Patch Level Server1 5.40.130.9 SQL 2K SP3a SP3a Benchcraft Server2 3.0.5 SQL 2K SP3a SP3a Client1-4 ODBC 2K-SP3a 4.11 Figure 15. VERITAS Software - SFW Dynamic Multi-Pathing Performance Testing 11
VeriTest (www.veritest.com), the testing division of Lionbridge Technologies, Inc., provides outsourced testing solutions that maximize revenue and reduce costs for our clients. For companies who use high-tech products as well as those who produce them, smoothly functioning technology is essential to business success. VeriTest helps our clients identify and correct technology problems in their products and in their line of business applications by providing the widest range of testing services available. VeriTest created the suite of industry-standard benchmark software that includes WebBench, NetBench, Winstone, and WinBench. We've distributed over 20 million copies of these tools, which are in use at every one of the 2001 Fortune 100 companies. Our Internet BenchMark service provides the definitive ratings for Internet Service Providers in the US, Canada, and the UK. Under our former names of ZD Labs and etesting Labs, and as part of VeriTest since July of 2002, we have delivered rigorous, objective, independent testing and analysis for over a decade. With the most knowledgeable staff in the business, testing facilities around the world, and almost 1,600 dedicated network PCs, VeriTest offers our clients the expertise and equipment necessary to meet all their testing needs. For more information email us at info@veritest.com or call us at 919-380-2800. Disclaimer of Warranties; Limitation of Liability: VERITEST HAS MADE REASONABLE EFFORTS TO ENSURE THE ACCURACY AND VALIDITY OF ITS TESTING, HOWEVER, VERITEST SPECIFICALLY DISCLAIMS ANY WARRANTY, EXPRESSED OR IMPLIED, RELATING TO THE TEST RESULTS AND ANALYSIS, THEIR ACCURACY, COMPLETENESS OR QUALITY, INCLUDING ANY IMPLIED WARRANTY OF FITNESS FOR ANY PARTICULAR PURPOSE. ALL PERSONS OR ENTITIES RELYING ON THE RESULTS OF ANY TESTING DO SO AT THEIR OWN RISK, AND AGREE THAT VERITEST, ITS EMPLOYEES AND ITS SUBCONTRACTORS SHALL HAVE NO LIABILITY WHATSOEVER FROM ANY CLAIM OF LOSS OR DAMAGE ON ACCOUNT OF ANY ALLEGED ERROR OR DEFECT IN ANY TESTING PROCEDURE OR RESULT. IN NO EVENT SHALL VERITEST BE LIABLE FOR INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES IN CONNECTION WITH ITS TESTING, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. IN NO EVENT SHALL VERITEST'S LIABILITY, INCLUDING FOR DIRECT DAMAGES, EXCEED THE AMOUNTS PAID IN CONNECTION WITH VERITEST'S TESTING. CUSTOMER S SOLE AND EXCLUSIVE REMEDIES ARE AS SET FORTH HEREIN. VERITAS Software - SFW Dynamic Multi-Pathing Performance Testing 12