Performance benefit of MAX5 for databases The MAX5 Advantage: Clients Benefit running Microsoft SQL Server Data Warehouse (Workloads) on IBM BladeCenter HX5 with IBM MAX5 Vinay Kulkarni Kent Swalin IBM System x June 2012
Contents Introduction 3 Decision Support System (DSS/DW) Workload... 3 IBM BladeCenter HX5... 3 IBM MAX5 for BladeCenter... 4 Microsoft SQL Server 2012... 6 Test Environment 7 Analysis 7 Test Results... 7 Sequential Query Execution... 7 Conclusion 10 Appendix A 11 BladeCenter HX5 with MAX5 Parts List... 11 References 11 Page 2 Copyright IBM Corporation 2012
Introduction This paper describes the series of tests performed by the IBM System x Performance Lab in Kirkland, Washington that demonstrate the benefits of using IBM BladeCenter HX5 with an IBM MAX5 memory expansion blade for a large-scale Decision Support System workload. IBM developed the BladeCenter MAX5 product after conducting a research study on customer pain points and finding out that memory became a bottleneck long before processors and I/O. When an HX5 server maxes out the memory at 256GB you have the option to attach a second HX5 server to the first, doubling the processors, memory, and other components. This creates a much more powerful double-wide 4-socket server, ideal for certain workloads. However, this is a fairly expensive upgrade if all that is needed is more memory. With IBM ex5 technology, you now have the option of adding a MAX5 expansion unit, which adds scalable memory capacity of up to 640GB in a double-wide blade at a relatively low cost. Adding a MAX5 with additional memory in the configuration reduces the load on I/O subsystem and reduces the amount of storage infrastructure needed. MAX5 is a great option to reduce the I/O footprint of a data warehouse configuration in your datacenter. By reducing the number of disks in the configuration, additional savings, such as space, power and cooling apply also making it even more attractive in terms of price/performance. For this series of tests, a 300GB data warehouse under the control of Microsoft SQL Server 2012 was used on an IBM BladeCenter HX5 server configured with two 10-core Intel Xeon E7 processors and IBM MAX5 for BladeCenter. Tests were conducted both with and without the MAX5 expansion unit to measure the performance benefit of the MAX5 with additional memory. The IBM HX5 blade server with MAX5 on average performed the queries 6 times faster than the BladeCenter HX5 alone, and increased the processor utilization to 76% from 18%, due to reduced I/O. Decision Support System (DSS/DW) Workload A data warehouse (DW) is a type of database that is designed for archiving data for the purposes of reporting and analysis. Data warehouses are typically used as the back-end data storage for Decision Support Systems (DSS). A data warehouse environment is different from an Online Transaction Processing (OLTP) database in that it is often subjected to ad hoc queries rather than the predefined queries that an OLTP database is designed to accommodate. Queries run against data warehouse databases often scan millions of rows as opposed to typical OLTP queries, which generally scan a handful of records. The I/O pattern is mostly large sequential reads compared to small random reads with an OLTP workload. IBM BladeCenter HX5 The BladeCenter HX5 is built with the fifth generation of IBM X-Architecture chipset technology (ex5). This new system incorporates the latest in hardware acceleration and scalability, providing resource configurations that push the boundaries of x86/x64-based systems while providing industry-leading flexibility. Figure 1 shows the BladeCenter HX5 server. Page 3 Copyright IBM Corporation 2012
Figure 1. IBM BladeCenter HX5 Using multi-core processors from Intel the IBM BladeCenter HX5 offers up to 2 processors per single-wide HX5 and 4 processors in a double-wide HX5 blade configuration. The HX5 memory capacity is up to 256GB via 16 very low profile (VLP) DIMM slots, running PC3-10600 DDR3 DIMMs, per single-wide HX5. Using a MAX5 expansion blade, the HX5 is scalable to 640GB at speeds up to 1067 MHz, via 40 DIMM slots. 1 Using 16GB 4Rx4 VLP memory modules resulted in a cost-effective, high-performance solution for this data warehouse application. We used VLP quad-rank RDIMMs to maximize performance with the Xeon processors in the HX5. The BladeCenter chassis has high-availability features, such as redundant fans, hot-swap fully redundant power supplies with 220VAC input, and hot-swap solid-state drives that deliver high levels of reliability and availability to maximize up-time. These kinds of features provide the reliability and high availability needed for the most demanding database workloads. This paper will demonstrate how leveraging the memory capabilities of the BladeCenter HX5 can maximize the performance, efficiency and reliability of your Data Warehouse environment. IBM MAX5 for BladeCenter The BladeCenter HX5 server can be expanded with the MAX5 memory expansion unit to scale the memory by an additional 24 DIMMs. Using 16GB DIMMs, a 2-processor BladeCenter HX5 with MAX5 supports a total of 640GB memory. Figure 2 shows the BladeCenter HX5 and MAX5 together. 1 Due to the design of the Intel processor, in an environment where only one processor is used, memory inside the HX5 is limited to only 128GB, However, MAX5 can still be used, offering up to 512GB even with only one processor. This creates an extremely cost-effective server when maximum memory is more important than processor cycles Page 4 Copyright IBM Corporation 2012
Figure 2. MAX5 is an industry-first technology from IBM that decouples memory from the processor, eliminating the need to buy another server to support memory-intensive workloads and dramatically changing the economics of the x86 server market. The MAX5 connects to the BladeCenter HX5 with two QPI links, one from each of the two sockets. Figure 3 illustrates the logical connections between the MAX5 and BladeCenter HX5. Page 5 Copyright IBM Corporation 2012
Figure 3. IBM BladeCenter HX5 connectivity with the MAX5 MAX5 might not be beneficial in all scenarios. If you have less than 256GB memory in your BladeCenter HX5 and need to upgrade memory to no more than 256GB, then buying highercapacity DIMMs is more economical and a better performing option. If you have maxed out the memory on BladeCenter HX5 at 256GB and you still have an I/O bottleneck that is slowing down your data warehouse queries, then MAX5 is an ideal fit for your configuration. Adding a MAX5 into that configuration is a better option than trying to add more storage into the configuration. The MAX5 will give you better latencies than even SSD storage. If you cache your database entirely in memory, you will see additional performance gains as listed in this paper. Page 6 Copyright IBM Corporation 2012
Microsoft SQL Server 2012 Microsoft SQL Server 2012 running on Microsoft Windows Server 2008 R2 enables organizations to build comprehensive, enterprise-scale analytic solutions that deliver intelligence where clients want it. Microsoft s SQL Server 2012 makes a major advance in data warehouse scalability. The database engine contains numerous enhancements including a new data warehouse query acceleration feature based upon a new type of index called the ColumnStore Index. This new index, combined with enhanced query processing features, improves data warehouse query performance, and can routinely give a tenfold speedup for a broad range of decision support queries. Test Environment The test environment consisted of a single BladeCenter HX5 server with two Intel Xeon E7-4870 10-core processors. Decision support workloads are frequently very I/O intensive. Two memory configurations were evaluated: 256GB of memory installed exclusively in the BladeCenter HX5 (no MAX5) 640GB of memory 256GB in the BladeCenter HX5 server and 384GB installed in the MAX5 attached to the BladeCenter HX5. Data warehouse workloads are host-based environments. There were no clients involved to generate the workload. The server ran Microsoft Windows Server 2008 R2 x64 Enterprise Edition. Database Layout: The SQL database was spread out over 8 LUNS. The database log was configured on a RAID-10 LUN using 6 SAS drives. The Windows Server 2008 R2 OS was configured on a 2-disk RAID-1 LUN on internal solid-state drives. SQL TempDB was also spread out on the 8 LUNS. Analysis The graphs below show the many benefits of MAX5 and additional memory. You will see that when more memory is used, thereby enabling the database to be cached in memory, the query response times are greatly improved. This is a result of reduced I/O and increased CPU utilization. As a result of using large amounts of memory, the processor utilization is maximized to make the best use of the server. Test Results Sequential Query Execution Sample queries were run sequentially to generate the results. The queries were a mix of CPUintensive and I/O-intensive queries. Some queries touched only a few rows in the database and others quite a few. Some queries did a full scan through a table, others were more computeintensive. This resulted in some queries executing in just a few seconds and others taking several minutes to run. The same set of queries was executed in the same order on both configurations. Page 7 Copyright IBM Corporation 2012
The graph in Figure 4 shows the percentage improvement of query response times for each of the queries using 640GB with the MAX5 as compared to 256GB of server memory. Other than the MAX5 and additional DIMMs, the two configurations were identical. Figure 4. Query response time improvements when using additional memory in the MAX5 The graph in Figure 5 shows the comparison of I/O operations per second during the sequential execution of the queries with 256GB and 640GB of server memory. With more memory, the database is cached, thereby significantly reducing I/O accesses to storage. Page 8 Copyright IBM Corporation 2012
8000 7000 6000 5000 4000 3000 2000 1000 0 HX5 With MAX5 I/O HX5 Without MAX5 I/O Figure 5. Physical storage I/O per second The graph in Figure 6 shows the comparison of processor utilization percentage during the sequential execution of the queries with 256GB and 640GB of server memory. Because I/O is reduced with MAX5 processor utilization increases, resulting in higher query throughput. 80% 70% 60% 50% 40% 30% 20% 10% 0% HX5 With MAX5 Processor Time HX5 Without MAX5 Processor Time Figure 6. Processor utilization percentages The graph in Figure 7 shows the comparison of disk queue lengths during the sequential execution of the queries with 256GB and 640GB of server memory. With more memory, the database is cached, thereby reducing the I/O bottleneck. Page 9 Copyright IBM Corporation 2012
1600 1400 1200 1000 800 600 400 200 0 HX5 With MAX5 Disk Queue Length HX5 Without MAX5 Disk Queue Length Figure 7. Disk queue lengths Conclusion The I/O subsystem is the slowest component in a data warehouse configuration. When you run a query against a database, the server needs to process data. Data resides in the storage, and read requests are issued to gather the dataset. On a system with inadequate amounts of system memory, the data that resides in memory must be evicted to make room for new data. On a system with a large amount of memory, more of the dataset can remain in memory, which enables future references to the dataset to be satisfied at very low latencies, cutting down the query response times significantly. If achieving faster query response times is your goal and I/O is the bottleneck, then the I/O bandwidth can be increased by either adding more storage or more memory. Adding memory will let you reduce the amount of storage needed, which in turn offsets the cost of adding that memory by reducing the space, power and cooling needs of storage in the datacenter. IBM has an effective solution: the 2-socket BladeCenter HX5, which supports up to 256GB system memory and up to 640GB of total memory when the MAX5 memory blade is attached. The results presented in this paper show that the IBM HX5 blade server with MAX5 on average performed the queries 6 times faster than the BladeCenter HX5 alone, and increased the processor utilization to 76% from 18%, by removing I/O bottleneck. 2 2 Just as MAX5 for BladeCenter offers cost-effective, high-performance memory expansion for BladeCenter HX5 servers, there is also a 1U MAX5 expansion unit for IBM System x3690 X5, x3850 X5, and x3950 X5 rack-optimized servers. This MAX5 unit supports up to 512GB of memory. One MAX5 can be attached to the x3690 X5 and x3850 X5, and up to two MAX5 units can be attached to an x3950 X5. Page 10 Copyright IBM Corporation 2012
Appendix A IBM BladeCenter HX5 and MAX5 parts list: Part Number Description Quantity 7873AC1 IBM BladeCenter HX5 1 88Y6160 Intel Xeon Processor E7-2870 10C 2.40GHz 30MB Cache 130W 1 90Y3221 16GB (1x16GB,4Rx4,1.35V) PC3L-8500 CL7 DDR3-1066 VLP RDIMM 16 46C0599 16GB (1x16GB, 2Rx4, 1.35V) PC3L-10600 CL9 ECC DDR3 1333MHz VLP RDIMM 24 59Y5877 IBM HX5 MAX5 1-node scalability kit 1 88Y6128 IBM MAX5 V2 for BladeCenter 1 46M6908 SSD Expansion Card for IBM BladeCenter HX5 1 43W7726 IBM 50GB SATA 1.8" MLC SSD 2 46M6140 Emulex 8Gb Fibre Channel Expansion Card (CIOv) for IBM BladeCenter 1 References IBM BladeCenter HX5: IBM Redbooks Product Guide http://www.redbooks.ibm.com/redbooks.nsf/redbookabstracts/tips0824.html#contents IBM BladeCenter HX5 Overview http://ibm.com/systems/bladecenter/hardware/servers/hx5/index.html Netlist Memory http://www.netlist.com/products/vlp/briefs/16gb_vlp_for_hx5_brief_final.pdf Microsoft Windows Server 2008 R2 http://www.microsoft.com/en-us/server-cloud/windowsserver/default.aspx Microsoft SQL Server 2012 http://www.microsoft.com/sqlserver/en/us/default.aspx Page 11 Copyright IBM Corporation 2012
The MAX5 Advantage: Client Benefits running Microsoft SQL Server Data Warehouse (Workloads) on Copyright 2012 by International Business Machines Corporation. IBM Systems and Technology Group Dept. U2SA 3039 Cornwallis Road Research Triangle Park, NC 27709 Produced in the USA June 2012 Warranty Information: For a copy of applicable product warranties, write to: Warranty Information, P.O. Box 12195, RTP, NC 27709, Attn: Dept. JDJA/B203. IBM makes no representation or warranty regarding third-party products or services including those designated as ServerProven or ClusterProven. Telephone support may be subject to additional charges. For onsite labor, IBM will attempt to diagnose and resolve the problem remotely before sending a technician. IBM, the IBM logo, System x, and X-Architecture are trademarks or registered trademarks of IBM Corporation in the United States and/or other countries. For a list of additional IBM trademarks, please see http://ibm.com/legal/copytrade.shtml. Intel and Xeon are registered trademarks of Intel Corporation. Microsoft, Windows Server, and SQL Server are registered trademarks of Microsoft Corporation in the United States and/or other countries. All other company/product names and service marks may be trademarks or registered trademarks of their respective companies. This document could include technical inaccuracies or typographical errors. IBM may make changes, improvements or alterations to the products, programs and services described in this document, including termination of such products, programs and services, at any time and without notice. Any statements regarding IBM s future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. The information contained in this document is current as of the initial date of publication only, and IBM shall have no responsibility to update such information. Performance data for IBM and non-ibm products and services contained in this document was derived under specific operating and environmental conditions. The actual results obtained by any party implementing and such products or services will depend on a large number of factors specific to such party s operating environment and may vary significantly. IBM makes no representation that these results can be expected or obtained in any implementation of any such products or services. THE INFORMATION IN THIS DOCUMENT IS PROVIDED AS-IS WITHOUT ANY WARRANTY, EITHER EXPRESS OR IMPLIED. IBM EXPRESSLY DISCLAIMS ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR INFRINGEMENT. References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business. Any reference to an IBM program or product in this document is not intended to state or imply that only that program or product may be used. Any functionally equivalent program or product, that does not infringe IBM s intellectually property rights, may be used instead. It is the user s responsibility to evaluate and verify the operation of any non-ibm product, program or service. Information in this presentation concerning non-ibm products was obtained from the suppliers of these products, published announcement material or other publicly available sources. IBM has not tested these products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. The provision of the information contained herein is not intended to, and does not grant any right or license under any IBM patents or copyrights. Inquiries regarding patent or copyright licenses should be made, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive Armonk, NY 10504-1785 U.S.A. Page 12 Copyright IBM Corporation 2012 BLW03027-USEN-01