EMC INFRASTRUCTURE FOR HIGH PERFORMANCE MICROSOFT AND ORACLE DATABASE SYSTEMS

Transcription

1 White Paper EMC INFRASTRUCTURE FOR HIGH PERFORMANCE MICROSOFT AND ORACLE DATABASE SYSTEMS EMC Symmetrix VMAX 40K, EMC XtremSF, EMC XtremCache, NEC Express5800/A1080a-E, Simplified storage management with FAST VP Accelerated performance with XtremCache EMC Solutions Abstract This white paper describes an automated storage tiering solution for multiple mission-critical applications virtualized with VMware vsphere on the EMC Symmetrix VMAX 40K storage platform. With EMC XtremCache enabled on the host, read I/O is cached and offloaded from the VMAX storage virtual pool. December 2013

2 Copyright 2013 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All trademarks used herein are the property of their respective owners. Part Number H

3 Table of contents Executive summary... 6 Business case... 6 Solution overview... 6 Key results... 6 Introduction... 7 Purpose... 7 Scope... 7 Audience... 7 Terminology... 7 Key technology components... 9 Overview... 9 EMC XtremSF and XtremCache... 9 Server-side flash caching for maximum speed... 9 Write-through caching to the array for total protection... 9 Application agnostic Integration with vsphere Minimum impact on system resources XtremCache active/passive clustering support EMC Symmetrix VMAX 40K EMC Virtual Provisioning EMC FAST VP NEC Express5800/A1080a-E VMware vsphere 5 components VMware vsphere VMware vcenter Server EMC PowerPath/VE Oracle Database 11g R Oracle Automatic Storage Management Oracle Grid Infrastructure Microsoft SQL Server SQL Server Failover Clustering Solution architecture and design Overview Physical architecture Hardware resources Software resources Storage connectivity

4 Storage Virtual Provisioning design FAST VP configuration Storage design considerations Oracle database and workload profile Oracle database schema Oracle database services Oracle LUN configuration Microsoft SQL Server workload type SQL Server 2012 DSS workload and profile SQL Server 2012 DSS LUN configuration SQL Server 2012 OLTP workload and profile SQL Server 2012 OLTP LUN configuration SQL Server 2012 and Windows 2008 R2 settings for DSS and OLTP workloads Operating system and SQL Server instance settings Database settings VMware vsphere configuration VMware virtual machine configuration SQL Server 2012 clustering on VMware EMC Virtual Storage Integrator XtremCache configuration with VMware Performance testing processes Overview Validation Application workloads Test procedure Test scenarios Three-tier FAST VP without and with XtremCache Objective Test scenarios Test result summary Three-tier FAST VP performance results Two-tier FAST VP without and with XtremCache Overview Test scenarios Two-tier FAST VP OLTP performance results Two-tier XtremCache and FAST VP OLTP performance results Two-tier FAST VP performance results breakdown XtremCache impact on FAST VP

5 Overview Three-tier FAST VP behavior with XtremCache XtremSF and XtremCache with DSS workload Overview Caching Local storage XtremCache with SQL Server failover cluster instance Overview Microsoft failover clustering (active/passive) support Validation Conclusion Summary Findings References White papers Product documentation Other documentation

6 Executive summary Business case Solution overview As enterprises move their databases and applications to the private cloud, their IT organizations must strive for more efficiency and improved quality of service, including: Extending the high performing flash technology from storage to host to support mixed workloads. Moving the workload from the storage array to host-based flash so that the array can serve more I/Os for other applications. Reducing capital expenditures and ongoing costs. Maintaining high performance levels and providing predictable performance to deliver the quality of service required in these environments. It is essential that infrastructure and tools simplify storage management processes and improve performance with a minimum of manual tasks. EMC Symmetrix VMAX 40K, and associated management tools, have been developed to be the foundation of this infrastructure and to meet real business needs: Performance optimization Optimizing and prioritizing business applications, allowing customers to dynamically allocate resources within a single array. Ease of management Elimination of manually tiering applications when performance objectives change over time. Host-side storage acceleration Accelerating application performance to extreme levels and placing hot data closest to server memory through consolidation with EMC XtremSF and EMC XtremCache. NEC Express5800/A1080a-E is the base server platform of this solution. Representing the fifth generation of enterprise server architecture from NEC, this line of server maintains the NEC legacy for developing scalable enterprise servers that offer exceptional configuration flexibility, capacity, reliability and availability features. Pairing the NEC Express5800/A1080a-E with VMware vsphere 5, the platform creates an outstanding solution for enterprise virtualization needs. Key results Our testing shows that this solution, based on Symmetrix VMAX with Enginuity 5876, EMC Fully Automated Storage Tiering for Virtual Pools (FAST VP ), XtremSF, and XtremCache, provides the following performance results: XtremSF and XtremCache improve OLTP performance by offloading much of the read I/O traffic from the storage array for other applications. XtremSF and XtremCache can solidly support an OLTP workload with SAN-based central storage, and can improve the application performance on two-tier FAST VP. Active/passive-hypervisor and physical-cluster support means that XtremSF and XtremCache can ensure data integrity while accelerating application performance in a highly available environment. 6

7 Introduction Purpose This white paper describes the design, testing, and validation of an enterprise VMware infrastructure using the Symmetrix VMAX 40K storage platform with Enginuity 5876, XtremSF, and XtremCache as its foundation. This solution demonstrates how XtremSF and XtremCache complement FAST VP in providing performance, scalability, and application-specific functionality to the solution using representative application environments, including Microsoft SQL Server and Oracle. Specifically, this solution: Validates that XtremSF and XtremCache can be shared to and serve multiple applications in a VMware virtualized environment. Validates that XtremCache can be consolidated with FAST VP enabled on Symmetrix VMAX, and also that recently accessed data within workloads can be effectively offloaded from SAN-based central storage to XtremSF. Validates that XtremSF and XtremCache can improve performance to provide excellent response times for the read-intensive workloads. Demonstrates that data integrity can be guaranteed by adding XtremCache to a clustered SQL Server instance. Scope Audience This white paper discusses multiple EMC products as well as those from other vendors. Some general configuration and operational procedures are outlined. However, for detailed product installation information, refer to the user documentation provided with those products. This white paper is intended for EMC employees, partners, and customers, including IT planners, virtualization architects and administrators, and any other IT professionals involved in evaluating, acquiring, managing, operating, or designing infrastructure that leverages EMC technologies. Throughout this white paper, we assume that you have some familiarity with the concepts and operations related to enterprise storage and virtualization technologies and their use in information infrastructures. Terminology Table 1 lists several terms used in this paper. Table 1. Terminology Term ASM DSS FAST VP FC FCI Definition Oracle Automatic Storage Management Decision Support System (that is, data warehouse) Fully Automated Storage Tiering for Virtual Pools Fibre Channel Failover Cluster Instance 7

8 Term HBA HS IOPS LUN NIC OLTP prdm RAID SAN SAS SATA SCSI SLC TDev TPS VSI Definition Host bus adapters Hot swap I/Os per second Logical unit number Network interface controller Online transaction processing physical Raw Device Mapping Redundant array of independent disks Storage area network Serial Attached SCSI Serial Advanced Technology Attachment Small Computer System Interface Single-Level Cell Thin device Transactions per second Virtual Storage Integrator 8

9 Key technology components Overview This solution used the following key hardware and software components: EMC XtremSF EMC XtremCache Note: Any mention of VFCache or XtremSW Cache in this white paper refers to XtremCache. EMC Symmetrix VMAX 40K EMC Virtual Provisioning EMC FAST VP NEC Express5800/A1080a-E VMware vsphere Oracle Database 11g R2 Enterprise Edition Microsoft SQL Server 2012 Enterprise Edition EMC XtremSF and EMC XtremCache EMC XtremSF is PCIe flash hardware deployed in the server to dramatically improve application performance by reducing latency and accelerating throughput. XtremSF can be used as a local storage device to accelerate read and write performance. It can also be used in conjunction with server flash caching software EMC XtremCache for accelerated read performance with data protection. XtremCache is intelligent caching software that leverages server-based flash technology to reduce latency and accelerate throughput for dramatic application performance improvement. A number of XtremCache features are highlighted below. For more information refer to the XtremCache Installation and Administration Guide. Server-side flash caching for maximum speed XtremCache software caches the most frequently referenced data on the flash in the server (XtremSF or other), which puts the data closer to the application. XtremCache automatically adapts to changing workloads by determining which data is most frequently referenced and promoting it to the server flash. This means that the hottest (that is, the most active) data automatically resides on the PCIe card in the server for faster access. XtremCache offloads the read traffic from the storage array, which allows it to allocate greater processing power to other applications. While one application is accelerated with XtremCache, the array s performance for other applications is maintained or even slightly enhanced. Write-through caching to the array for total protection XtremCache accelerates reads and protects data by using a write-through cache to the storage to deliver persistent high availability, integrity, and disaster recovery. 9

10 Application agnostic XtremCache is transparent to applications, therefore no rewriting, retesting, or recertification is required to deploy XtremCache in the environment. Integration with vsphere Integration of the VSI plug-in with VMware vsphere vcenter simplifies the management and monitoring of XtremCache. Minimum impact on system resources XtremCache does not require a significant amount of memory or CPU cycles, as a majority of flash management is done on XtremSF. Unlike other server flash solutions, there is no significant overhead from using XtremSF and XtremCache on server resources. XtremCache active/passive clustering support XtremCache clustering support ensures data integrity of an active/passive clustered application. The XtremCache-enabled cluster also accelerates application performance. EMC Symmetrix VMAX 40K EMC Symmetrix VMAX 40K with Enginuity 5876 provided the tiered storage configuration used in the test environment. Built on the strategy of powerful, trusted, smart storage, this solution incorporated a highly scalable Virtual Matrix Architecture that enables Symmetrix VMAX arrays to grow seamlessly and cost-effectively. Symmetrix VMAX supports flash drives, Fibre Channel (FC) drives, and SATA drives within a single array, as well as an extensive range of RAID types. The EMC Enginuity operating environment controls all components in the Symmetrix VMAX array. Enginuity 5876 for Symmetrix VMAX offers: More efficiency: New zero downtime technology for migrations (technology refreshes) and lower costs with automated tiering. More scalability: Up to two times more performance, with the ability to manage up to 10 times more capacity per storage administrator. More security: Built-in encryption, RSA-integrated key management, increased value for virtual server and mainframe environments, replication enhancements, and a new electronic licensing model. EMC Virtual Provisioning EMC Virtual Provisioning is the EMC implementation of thin provisioning. It is designed to simplify storage management, improve capacity utilization, and enhance performance. Virtual Provisioning provides for the separation of physical storage devices from the storage devices as perceived by host systems. This enables nondisruptive provisioning and more efficient storage use. This solution uses virtually provisioned storage for all deployed applications. For detailed information on Virtual Provisioning, refer to the EMC Solutions Enabler Symmetrix Array Controls CLI v7.4 Product Guide. 10

11 EMC FAST VP EMC FAST VP is a feature of Enginuity 5875 and higher, that provides automatic storage tiering at the sub-lun level. Virtual pools are Virtual Provisioning thin pools. FAST VP provides support for sub-lun data movement in thin-provisioned environments. It combines the advantages of Virtual Provisioning with automatic storage tiering at the sub-lun level to optimize performance and cost, while simplifying storage management and increasing storage efficiency. FAST VP data movement between tiers is based on performance measurement and user-defined policies, and is executed automatically and non-disruptively. NEC Express5800/A10 80a-E VMware vsphere The NEC Express5800/A1080a has many key design features that are ideal for mixedworkload large-scale virtualization. With a maximum memory configuration of 2 TB, eight CPU sockets (160 threads) and 14 PCI Express 2.0 slots, consolidating entire database, application, and Web infrastructures onto a single NEC Express5800/A1080a is the preferred solution to growing IT needs. For this solution, the Microsoft SQL Server and Oracle application servers are fully virtualized using VMware vsphere 5. This section describes the virtualization infrastructure, which uses the following components and options: VMware vsphere VMware vcenter Server EMC PowerPath /VE for VMware vsphere Version 5.7 VMware vsphere 5 VMware vsphere 5 is a complete, scalable, and powerful virtualization platform, with infrastructure services that transform IT hardware into a high-performance shared computing platform, and application services that help IT organizations deliver the highest levels of availability, security, and scalability. VMware vcenter Server VMware vcenter is the centralized management platform for vsphere environments, enabling control and visibility at every level of the virtual infrastructure. EMC PowerPath/VE EMC PowerPath/VE for VMware vsphere delivers PowerPath multipathing features to optimize VMware vsphere virtual environments. PowerPath/VE installs as a kernel module on the VMware ESXi host and works as a multipathing plug-in (MPP) that provides enhanced path management capabilities to ESXi hosts. Oracle Database 11g R2 Oracle Database 11g Release 2 Enterprise Edition delivers performance, scalability, security, and reliability on a choice of clustered or single servers running Windows, Linux, or UNIX. It provides comprehensive features for transaction processing, business intelligence, and content management applications. Oracle Automatic Storage Management Oracle Automatic Storage Management (ASM) is a volume manager and a file system for Oracle database. ASM is Oracle's recommended storage management solution 11

12 that provides an alternative to conventional volume managers, file systems, and raw devices. ASM uses disk groups to store data files. An ASM disk group is a collection of disks that ASM manages as a unit. Within a disk group, ASM exposes a file system interface for Oracle database files. The content of files stored in a disk group is evenly distributed, or striped, to eliminate hot spots and to provide uniform performance across the disks. The performance is comparable to the performance of raw devices. Oracle Grid Infrastructure For this solution, Oracle Grid Infrastructure was installed with the Standalone Server option. The Oracle Grid Infrastructure for a standalone server provides system support for an Oracle database including volume management, file system, and automatic restart capabilities. If you plan to use Oracle Restart or Oracle ASM, then you must install Oracle Grid Infrastructure before you install and create the database. Oracle Grid Infrastructure for a standalone server combines Oracle Restart and Oracle ASM into a single set of binaries that is installed in the Oracle Grid Infrastructure home. Microsoft SQL Server 2012 Microsoft SQL Server 2012 is Microsoft s database management and analysis system for e-commerce, line-of-business, and data warehousing solutions. By enabling a modern data platform with SQL Server 2012, users can get built-in, mission-critical capabilities and enable breakthrough insights across the organization with familiar analytics tools and enterprise-ready Big Data solutions. SQL Server Failover Clustering In SQL Server failover clustering, the operating system and SQL Server work together to provide availability in case of an application failure, hardware failure, or operating system error. Failover clustering provides hardware redundancy through a configuration in which vital, shared resources are automatically transferred from a failing computer to an equally configured server. SQL Server failover clustering in the active/passive mode is for one instance of a set of databases. 12

13 Solution architecture and design Overview Physical architecture EMC solutions are validated architectures that are designed to reflect real-world deployments. This section describes the key components, resources, and overall architecture that make up this solution and its environment. Figure 1 depicts the physical architecture for this solution. Figure 1. Physical architecture diagram This solution is built on an EMC Symmetrix VMAX 40K array running Enginuity The array provides a mix of flash, FC, and SATA drives. FAST VP continually monitors and tunes performance by relocating data across storage tiers, based on access patterns and predefined FAST policies. 13

14 We provisioned Microsoft SQL Server 2012 (two OLTP workloads and one Decision Support System (DSS)) and Oracle 11g R2 (OLTP). We also built Microsoft failover clustering on the virtualized environment to validate XtremCache and Microsoft Cluster Service (MSCS) consolidation. These applications ran on virtual machines in a VMware vsphere 5 cluster environment on EMC VMAX 40K storage. Load generation tools drove these applications simultaneously to validate the infrastructure and acceleration function from XtremCache. Failover was performed for SQL Server failover clustering to verify XtremCache and Windows Server Failover Clustering (WSFC) integration. The effects of applying the FAST policy are documented in Performance testing processes. Hardware resources Table 2 lists the hardware resources used in this solution environment. Table 2. Hardware resources Equipment Quantity Configuration EMC Symmetrix VMAX 40K Enginuity Three-engine, 128 GB cache per engine 33 flash 200 GB (including 1 HS) GB 10k FC drives (including 6 HS) 70 2 TB 7.2k SATA drives (including 3 HS) 64 x 450 GB 15k FC drives (including 2 HS) NEC Express5800/A1080a-E 2 8-socket (10 C/2.40 GHZ/30 MB cache) 1 TB RAM 4 GbE IP ports GB 2.5-in. 15k SAS disks 1 internal RAID controller 12 8 PCIe slots/2 16 PCIe slots 2 dual-port 8 Gb/s HBAs (4 FC) 1 quad-port GbE NIC SAN 1 8 Gb enterprise-class FC switch EMC XtremSF GB SLC PCIe cards 14

15 Software resources Table 3 lists the software resources used in this solution environment. Table 3. Software resources Software Version EMC Symmetrix VMAX Enginuity code 5876 EMC XtremCache 1.5 EMC PowerPath/VE for VMware 5.7 EMC Unisphere for VMAX 1 EMC Solutions Enabler 7.4 VMware vsphere 5 (Enterprise Plus) Oracle ASMlib Oracle Database 11g R Microsoft Windows Server 2008 R2 Microsoft SQL Server 2012 SP1 RTM Microsoft TPC-E toolkit Quest Benchmark Factory Red Hat Enterprise Linux Server 5.7 Swingbench 2.3 Storage connectivity The application workloads were logically separated using masking views within the VMAX 40K and HBAs. Figure 2 shows the front-end port use for each application. Oracle and SQL Server OLTP workloads were on separate ESXi hosts but shared the same eight front-end ports on the VMAX. The SQL Server OLTP and DSS workloads were on the same ESXi host but the front-end ports on the VMAX were separate. The purpose was to separate the application by the I/O size. The OLTP application is typically 8k 64k I/O in size and is measured in IOPS. For DSS with large I/O sizes ranging from 8k to 256k, disk performance is typically measured by throughput (in megabytes per second). 15

16 Figure 2. Logical grouping of ports to applications Storage Virtual Provisioning design EMC Virtual Provisioning greatly simplifies storage design. We created four thin pools on the array, based on the drive types available. Table 4 shows the thin pool definitions. Table 4. Thin pool configuration Thin pool name Drive size/ technology/ RPM RAID protection No. of drives Data device size No. of data devices Pool capacity FAST VP usage FLASH_3RAID5 200 GB flash RAID GB TB Oracle/SQL Server OLTP FC10K_RAID1 600 GB FC 10k RAID GB TB Oracle/SQL Server OLTP FC15K_RAID1 450 GB FC 15k RAID GB TB DSS SATA_6RAID6 2 TB SATA 7.2k RAID GB TB Oracle/SQL Server OLTP/DSS For this solution, the Oracle and Microsoft OLTP applications were bound to the FC10K_RAID1 pool. The Microsoft DSS application was bound to the FC 15K_RAID1 pool, which was backed by a smaller number of drives. 16

17 FAST VP configuration VMAX administrators can set high-performance policies that use more flash drive capacity for critical applications, and cost-optimized policies that use more SATA drive capacity for less critical applications. The ideal FAST VP policy is to specify a 100-percent allocation for each of the tiers included. Such a policy provides the greatest amount of flexibility to an associated storage group, as it allows 100 percent of the storage group s capacity to be promoted or demoted to any tier within the policy. However, data warehouse applications tend to issue scan-intensive operations that access large portions of the data at a time and also commonly perform bulk loading operations. These operations result in larger I/O sizes than OLTP workloads, and they require a storage subsystem that can provide the required throughput. This makes the throughput or megabytes per second (MB/s) the critical metric. Although flash disk storage can provide more than a 100 MB/s throughput, generally it is best suited to serving a small portion of the database s hot data. Therefore, in this solution, we used a two-tier policy consisting of FC and SATA storage to provide a cost-efficient mix of storage to satisfy the needs of DSS workloads. Table 5 shows the FAST VP policies used for the application workloads in this solution for Oracle, SQL Server OLTP, and SQL Server DSS. Table 5. FAST VP policy for Oracle, SQL Server OLTP, and SQL Server DSS Storage group FAST policy name Flash FC SATA MSSQL_OLTP MSSQL_OLTP 100 percent 100 percent 100 percent MSSQL_DSS MSSQL_DSS 0 percent 100 percent 100 percent Oracle Oracle 100 percent 100 percent 100 percent Storage design considerations The design incorporates the following recommended practices for mission-critical database applications with FAST VP: Use separate storage volumes for data files and log files. Use separate file groups for large databases. For ASM, EMC recommends separate ASM disk groups for DATA, REDO, FRA, and TEMP. Bind all thin devices to the FC tier. Pin log devices and temp files to the FC tier. Figure 3 shows an overview of how each critical application is configured for FAST VP. In this implementation, only data LUNs are managed by FAST VP. LUNs for OS, temp, and LOG are pinned to the FC tier, excluding them from FAST VP decisions and movement. Note: The DSS SQL Server instance tempdb is moved out after moving the DSS tempdb to XtremCache. 17

18 Figure 3. General view of FAST VP configuration for mission-critical applications Oracle database and workload profile The Swingbench Order Entry PL/SQL Server (SOE) schema was used to deliver the OLTP workloads required by this solution. Swingbench consists of a load generator, a coordinator, and a cluster overview. The software enables a load to be generated and the transactions and response times to be charted. Table 6 details the Oracle database and workload profile for this solution. Table 6. Oracle database and workload profile Profile characteristic Database size Database version Storage type Oracle system global area (SGA) Workload type Workload profile Database metric Details 2 TB Oracle Database 11g R2 single instance Oracle ASM 24 GB OLTP Swingbench Order Entry (TPC-C-like) workload Transactions per second (TPS) Workload read/write ratio 80/20 Swingbench sessions 1,800 18

19 Oracle database schema Two identical schemas were used to deliver the OLTP workloads required by this solution: Schemas SOE1 and SOE2. A Swingbench Order Entry workload was generated and run against schema SOE1. Since the I/O distribution across the database was completely even and random, this reduced sub-lun skewing (since the entire database was highly active), therefore the second schema SOE2 remained idle to simulate a more normal environment where some objects are not highly accessed. Table 7 lists the tables and indexes for the SOE schema used in this solution (SOE1). Table 7. SOE schema Table name CUSTOMERS INVENTORIES ORDERS ORDER_ITEMS PRODUCT_DESCRIPTIONS PRODUCT_INFORMATION WAREHOUSES Index CUSTOMERS_PK (UNIQUE), CUST_ACCOUNT_MANAGER_IX, CUST_ _IX, CUST_LNAME_IX, CUST_UPPER_NAME_IX INVENTORY_PK (UNIQUE), INV_PRODUCT_IX, INV_WAREHOUSE_IX ORDER_PK (UNIQUE), ORD_CUSTOMER_IX, ORD_ORDER_DATE_IX, ORD_SALES_REP_IX, ORD_STATUS_IX ORDER_ITEMS_PK (UNIQUE), ITEM_ORDER_IX, ITEM_PRODUCT_IX PRD_DESC_PK (UNIQUE), PROD_NAME_IX PRODUCT_INFORMATION_PK (UNIQUE), PROD_SUPPLIER_IX WAREHOUSES_PK (UNIQUE) LOGON Oracle database services Database services are entry points to an Oracle database that enable the management of workloads across the cluster. For this solution, each of the test schemas had a corresponding database service mapped to it, as shown in Table 8. This enabled monitoring of the Oracle I/O to be mapped to each individual service and hence schema. Table 8. Oracle database services Schema Service Instance Swingbench sessions SOE1 SOE1.oracledb.ie orafast 1800 SOE2 SOE2.oracledb.ie orafast 0 19

20 Oracle LUN configuration Table 9 lists the LUN configuration for the Oracle application. Table 9. Oracle LUN configuration ASM disk group TDev hyper size No. of TDevs Capacity (GB) +DATA 64 GB SOE1 64 GB SOE2 64 GB FRA 64 GB TEMP 64 GB REDO 64 GB 1 64 Total (GB) 2304 Microsoft SQL Server workload type SQL Server 2012 DSS workload and profile In the test environment, the following two applications generated the different workload patterns running on the Microsoft SQL Server 2012 enterprise class platform: A TPC-E-like application, acting as a typical OLTP application A TPC-H-like application, acting as a typical DSS application The test workload for the SQL Server 2012 DSS application was based on a TPC-H-like workload. The TPC-H-like application models the analysis part of the business environment where trends are computed and refined data is produced to support the making of sound business decisions. In a TPC-H-like application, periodic refresh functions are performed against a DSS database whose content is queried on behalf of, or by, various decision makers. Table 10 details the SQL Server DSS database and workload profile for this solution. Table 10. SQL Server DSS database and application profile Profile characteristic Total SQL Server database capacity Details 2 TB Number of SQL Server instances 1 Concurrent users 2 Read/write ratio (user databases) 100:0 (typical) 20

21 SQL Server 2012 DSS LUN configuration Table 11 shows the LUN utilization for Microsoft SQL Server DSS LUNs, and how they were used. Table 11. Table LUN use for Microsoft SQL Server DSS Purpose LUN size No. of TDevs Capacity (GB) SQL Server DSS virtual machine data store File group Log tempdb log tempdb Total (GB) 4,448 SQL Server 2012 OLTP workload and profile The test workload for the SQL Server 2012 OLTP application instances was based on a TPC-E-like workload. It was composed of a set of transactional operations that simulate an online stock trading floor, which is latency-sensitive and combines multiple query types. These include insert and updates per application transaction. The OLTP applications are primarily heavily indexed to support low latency retrieval of small numbers of rows from data sets that often have little historical data volume. These types of database operations induce significant disk head movement and generate classic random I/O scan patterns. Table 12 details the SQL Server OLTP database and workload profile for this solution. Table 12. SQL Server OLTP database and application profile Profile characteristic Total SQL Server database capacity Details 1 TB Number of SQL Server instances 2 Number of user databases for each virtual machine Concurrent users 1 (400 GB, 600 GB) Mixed workloads to simulate both a hot and a warm application environment. Read/write ratio 90:10 21

22 SQL Server 2012 OLTP LUN configuration Table 13 and Table 14 show the LUN use for Microsoft SQL Server LUNs. Table 13. Microsoft SQL Server LUN use SQL Server OLTP instance #1 Purpose LUN size No. of TDevs Capacity (GB) SQL Server OLTP virtual machine data store SQL1 tpce root SQL1 file group SQL1_Log SQL1 tempdb log SQL1 tempdb Total (TB) 1,792 Table 14. Microsoft SQL Server LUN use SQL Server OLTP instance #2 Purpose LUN size No. of TDevs Capacity (GB) SQL Server OLTP virtual machine data store SQL2 tpce root SQL2 file group SQL2_Log SQL2 tempdb log SQL2 tempdb Total (TB) 1,120 SQL Server 2012 and Windows 2008 R2 settings for DSS and OLTP workloads Operating system and SQL Server instance settings For the SQL Server 2012 tests we used Windows 2008 R2 as the operating system. Our settings for DSS and OLTP workloads were: Large-page memory support was enabled for the SQL Server instance by enabling the 834 startup parameter. The Lock pages in memory option was used for the SQL Server instances. All data and log LUNs were formatted using a 64 KB allocation unit size. Database settings For user databases, we used these settings: For DSS user database multiple data files 16 data files on 16 thin devices (TDevs). For OLTP user database multiple data files eight data files on eight TDevs. Disabled the autogrow option for data files and manually grew all data files. 22

23 For tempdb, we used these settings: Pre-allocated space and added a single data file per LUN. We ensured all files were the same size. Assigned temp log files to one of the LUNs dedicated to log files. Enabled autogrow In general, the use of a large growth increment is appropriate for data warehouse workloads. A value equivalent to 10 percent of the initial file size is a reasonable starting point. We followed standard SQL Server best practices for database and tempdb sizing considerations. For more information, see Capacity Planning for tempdb in SQL Server Books Online. For the transaction log we used this configuration: Created a single transaction log file per database on one of the LUNs assigned to the transaction log space. Spread log files for different databases across available LUNs or use multiple log files for log growth, as required. Enabled the autogrow option for log files. VMware vsphere configuration VMware vcenter Server provided a scalable and extensible platform to centrally manage vsphere environments, providing control and visibility at every level of the virtual infrastructure. We connected two ESXi5 hosts to the VMAX 40K array. Host A ran the virtual machine for Oracle, and also ran the FCI standby SQL Server virtual machine. Host B ran the virtual machines for the SQL Server OLTP, the DSS, and ran the FCI active SQL Server virtual machine. VMware virtual machine configuration The clustered SQL Server data LUNs used physical Raw Device Mapping (prdm) disks, besides these shared disks, all virtual machines in this configuration used virtual machine disks (VMDK) from VMware Virtual Machine File System (VMFS) data store volumes, including the OS and boot LUNs. Each VMFS data store hosts a single VMDK disk, ensuring high performance and zero contention. This practice also ensures you have the ability to restore at an application level with EMC TimeFinder Clone and Snap on the VMAX 40K array. 23

24 Table 15 shows the virtual machines CPU and memory allocation for each application virtual machine. Table 15. Virtual machine CPU and memory allocation Application Virtual machine name CPU count Memory size (MB) Oracle ORACLEDB 32 54,272 Microsoft SQL Server DSS SQLTPCH ,072 Microsoft SQL Server OLTP SQLTPCE ,000 SQLTPCE ,000 Domain controller 4 4,096 SQL Server 2012 clustering on VMware In this solution, a clustered SQL Server instance was built across two ESXi hosts for the XtremCache function test. The cluster requires specific hardware and software. The ESXi hosts had the following configuration: One physical network adapter dedicated to the VMkernel. Shared storage must be on an FC SAN. In this solution, the two shared disks were from VMAX 40K. RDM in physical compatibility (pass-through) mode. Table 16. Microsoft SQL Server Failover Cluster Instance (FCI) LUN use Purpose Quantity of LUNs Capacity (GB) SQL Server FCI boot LUN data store Microsoft Distributed Transaction Coordinator (MSDTC) data store Microsoft SQL Server user database store Total (TB)

25 EMC Virtual Storage Integrator EMC Virtual Storage Integrator (VSI) provides enhanced visibility into Symmetrix VMAX 40K directly from the vcenter GUI. Figure 4 shows the data store and storage pool information, which provides information about virtual pool usage for the Oracle_SOE1_1 data store. Figure 4. Data store and storage pool information viewed from VSI VMAX 40K volumes hosted the VMFS data stores and prdm disks for this solution. Figure 4 shows the ESXi server and the Symmetrix VMAX 40K storage mapping with details about VMFS data stores and the LUNs. The VSI Storage Viewer feature identifies details about VMFS data stores such as the VMAX storage volumes hosting the data store, the paths to the physical storage, pool usage information, and data store performance statistics. Figure 5 shows the LUN view from VSI. From here, administrators can identify the Symmetrix device ID for LUNs and data stores, if user-defined labels are set on VMAX LUNs. Administrators can export these listings to CSV files for manipulation with VMware PowerCLI scripts for the rapid provisioning of data stores to the ESXi hosts. Figure 5. EMC Virtual Storage Integrator LUN view 25

26 XtremCache configuration with VMware In a VMware environment, the XtremCache card resides on the ESXi server, while XtremCache software is installed on each of the virtual machines that are accelerated by XtremCache. The XtremCache VSI plug-in, which resides on the vcenter client, is used to manage XtremCache. XtremCache can accelerate performance for either RDM or VMFS LUNs in a VMware environment. The XtremCache installation is distributed over various vsphere system components. Figure 6 illustrates the location of these installed components. XtremCache software is installed on the guest machines and the XtremCache VSI plug-in is installed on a vsphere client. Figure 6. XtremCache in VMware environment 26

27 Multiple virtual machines on the same ESXi server can share the performance advantages of a single XtremSF card. As shown in Figure 7, the flash device (VMFS) is carved into virtual disks and presented to the virtual machines. Refer to XtremCache Installation Guide for VMware 1.5 for the detailed XtremCache VMware configuration. Figure 7. XtremCache in VMware environment XtremCache is integrated with VSI plug-ins to simplify XtremCache management and monitoring. Figure 8 shows how VSI is used to manage XtremCache in the VMware environment. Figure 8. VSI XtremCache management 27

28 We monitored and observed how many I/Os were offloaded by the XtremSF card, as shown in Figure 9. Figure 9. VSI XtremSF monitor 28

29 Performance testing processes Overview Notes This section describes how we tested the applications in the solution environment. Each test is described in more detail in later sections. Benchmark results are highly dependent on workload, specific application requirements, and system design and implementation. Relative system performance will vary as a result of these and other factors. Therefore, this workload should not be used as a substitute for a specific customer application benchmark when critical capacity planning or product evaluation decisions are contemplated. The environment was rigorously controlled; results obtained in other operating environments may vary significantly. EMC Corporation does not warrant that a user can or will achieve performances similar to these. Validation To validate the environment, we deployed all applications and populated them with test data. Each of the applications (Oracle, SQL Server OLTP, and SQL Server DSS) was deployed at the production location, and workloads were driven against each application running simultaneously on the VMAX 40K storage array. We used Unisphere s Performance Analyzer module on VMAX to monitor and gather storage performance data in addition to application performance monitoring tools. Application workloads For each application, we used load generation tools to simulate realworld user interactions. The details are as follows: We used a Microsoft TPC-E toolkit on the client virtual machines to generate TPC-E-like loads simultaneously for SQL Server OLTP databases. This emulated warm and hot workloads. The SQL Server OLTP application I/O pattern is typically 8 KB read/write, with a read/write ratio of 90:10 percent, respectively. We used Quest Benchmark Factory to generate a TPC-H-like load for the SQL Server DSS database. The DSS application I/O pattern is typically 64 KB, with 100 percent read ratio on the data LUNs. We generated a Swingbench TPC-C-like order entry workload with 1,800 users and ran it against the Oracle database. The Oracle I/O pattern is 8 KB read and 8 KB write, with a read/write ratio of 80/20 percent, respectively. 29

30 Test procedure The test procedure was as follows: 1. Baseline test The baseline performance metrics were measured when the application workloads (Oracle, SQL Server OLTP, and SQL Server DSS) were run together and stabilized within three hours. We measured each application s performance to ensure it was within predefined KPIs and that all workloads co-existed without a negative impact on each other. 2. Enable XtremCache After the application workloads stabilized, we enabled XtremCache on the OLTP workload and measured the performance acceleration and workload offloading from three-tiered FAST VP storage to XtremSF. We enabled a card on each of the SQL Server OLTP and Oracle virtual machines, and configured as much space as possible for all three workloads, according to demand. Enabling XtremCache on a DSS workload was verified as cache and as a local disk store for tempdb. The minimum space requirement for the XtremCache is 25 GB. The two 700 GB XtremSF cards (651 GB usable space) were divided by capacity and allocated to the four virtual machines on the two ESXi servers. The detailed allocation is listed in Table 17. Table 17. XtremSF allocation on virtual machines XtremSF allocation per application/virtual machine ESXi 01 (allocation unit: GB) Oracle OLTP 600 N/A SQL Server DSS N/A 200 SQL Server OLTP 01 N/A 200 SQL Server OLTP 02 N/A 200 Total ESXi 02 (allocation unit: GB) Test scenarios The performance tests contained the following scenarios: Three-tier FAST VP without and with XtremCache Two-tier FAST VP without and with XtremCache XtremCache impact on FAST VP XtremCache with DSS workload XtremCache with SQL Server failover cluster instance 30

31 Three-tier FAST VP without and with XtremCache Objective The objective of this test was to validate the solution build under normal operating conditions for a normal work day, with FAST VP storage tiering enabled. Tests were run without and with XtremCache enabled to evaluate the heavy workloads offloading from three-tiered FAST VP storage to XtremSF. We evaluated all aspects of this solution, including the VMware vsphere server and virtual machine performance, Oracle, SQL Server OLTP, and SQL Server DSS server and client experiences. Test scenarios Test result summary The XtremCache offloading test had two scenarios: Before enabling XtremCache: All OLTP workloads were on VMAX with three-tier FAST VP enabled. The storage can support the workload with an excellent application response time; however the workload was heavy on the array. After enabling XtremCache on the virtual machines: The read I/O can be offloaded to XtremCache. The array still has three-tier FAST VP enabled and can handle other I/O requests. The test result summary is as follows: Without XtremCache enabled the array received more than 40,000 IOPS from the host side. With XtremCache enabled this number fell to only 12,000 IOPS (approximately) from the host side. XtremCache significantly reduced the IOPS and back-end adapter utilization of the storage array in the three-tier FAST VP configuration, while there was no change in the OLTP TPS performance. The array was then freed up for other I/O requests. The SQL Server OLTP, Oracle Database, and LUN response times decreased after XtremCache was enabled. 31

32 As shown in Figure 10, the IOPS received by the VMAX 40K array front-end adapter fell from approximately 40,000 to 12,000, while the back-end adapter busy percentage decreased from 67 percent to 33 percent. This means that approximately 28,000 IOPS were offloaded by XtremCache. Figure 10. Array workload before and after XtremCache was enabled The test procedure was carried out with the three application (Oracle, SQL Server OLTP, and SQL Server DSS) workloads running together. Figure 11 shows XtremCache reduced storage device IOPS for Oracle and SQL Server OLTP data LUNs respectively. Figure 11. Storage device IOPS before and after XtremCache is enabled 32

33 Three-tier FAST VP performance results Table 18 shows detailed results of running workloads on the source array for Oracle and SQL Server OLTP applications before and after enabling XtremCache on the threetier FAST VP. With flash, FC, and SATA tiers in the FAST VP pool, after enabling XtremCache: The storage IOPS reduced significantly from more than 40,000 to approximately 12,000, because the XtremCache offloaded the IOPS and the SAN-based storage could serve I/Os from other applications. As the result of the decrease in back-end storage utilization, the virtual machine and ESXi CPU utilization increased. There was no obvious increase in OLTP TPS because the three-tier FAST VP could serve the application with excellent performance. Response times decreased, because the XtremCache solution cached the most frequently referenced data on the server-based PCIe card, thereby putting the data closer to the application. Table 18. Detailed results before and after enabling XtremCache Components Performance Three tiers configured without XtremCache Three tiers configured with XtremCache VMAX VMAX OLTP IOPS (total) 40,605 12,275 ESXi 01 Average CPU utilization 65 percent 77 percent ESXi 02 Average CPU utilization 42 percent 43 percent Swingbench TPS 8,535 8,537 Oracle OLTP Average Oracle database response time (ms) 4 3 vcpu utilization 84.4 percent 91.4 percent SQL01 latency (ms) (read/write/transfer) 7/8/7 3/5/3 SQL Server SQL02 latency (ms) (read/write/transfer) 3/4/4 2/4/2 SQL01 vcpu utilization 30 percent 73 percent SQL02 vcpu utilization 66 percent 81 percent Transaction/sec 5,725 5,846 33

34 Two-tier FAST VP without and with XtremCache Overview Test scenarios Two-tier FAST VP OLTP performance results With only FC and SATA tiers in the FAST VP pool, central storage may be a performance bottleneck because the limited spindle numbers may not provide excellent response times for heavy workloads. However, with XtremCache enabled on the virtual machines, the performance bottleneck can be overcome and the application continues to experience excellent levels of storage latency. The XtremCache offloading test had two scenarios: Before enabling XtremCache: All OLTP workloads were on VMAX with two-tier FAST VP enabled. The two-tier storage has limited spindles to provide excellent response times for read I/O intensive applications. After enabling XtremCache on the virtual machines: XtremCache can significantly improve application performance. After disabling the flash tier in FAST VP, most of the workload was served by the FC tier. The FC tier disk utilization was very high and the SAN-based central storage (FC and SATA) could not support enough IOPS or provide acceptable response times for the I/O-intensive OLTP applications. The test results were as follows: The average FC disk IOPS was 142, which in theory is the maximum value for 10K FC disks. The maximum FC disk utilization was almost at 100 percent. The total OLTP IOPS was approximately 10,000 on average; the storage could not serve more IOPS. For SQL Server OLTP the average disk response time was more than 20 ms, and for Oracle, the application response time was more than 35 ms. These times exceed vendor-recommended limits. CPU utilization for the ESXi and the virtual machine was low: ESXi CPU utilization was less than 20 percent. Virtual machine CPU utilization was less than 5 percent. 34

35 Figure 12 shows the FC-tier disk heat map. The red color means the disk workload was very high (hot). Utilization reached 100 percent of the total IOPS capacity. Figure 12. FC-tier disk heat map without XtremCache Two-tier XtremCache and FAST VP OLTP performance results Performance improved after enabling the XtremCache on the Oracle and SQL Server OLTP virtual machines. The test results were as follows: The average FC disk IOPS was 43, which was approximately 30 percent of the maximum capacity for 10K FC disk IOPS. The FC disk maximum utilization was approximately 65 percent. For SQL Server OLTP, the average disk response time was no more than 3 ms, and for Oracle, the application response time was no more than 3 ms. ESXi and virtual machine CPU utilization increased greatly when compared with the results before XtremCache was enabled: On ESXi-1 CPU utilization increased from 16 percent to 21 percent, and on ESXi-2 increased from 2 percent to 40 percent. The SQL Server OLTP database CPU utilization increased from less than 2 percent to percent; for the Oracle database it increased from 48 percent to 89 percent. XtremCache can provide excellent response times for a read I/O intensive OLTP workload, and reduce the storage workload from the system. As a result, the system was able to handle more OLTP TPS. ESXi and virtual machine CPU usage increased because of the increased SQL Server and Oracle utilization with transactional processing. 35

36 Figure 13 shows the FC-tier disk heat map. The yellow color means the disk workload was normal compared with those in Figure 12. The previous heavy workload on this tier was removed by XtremCache. Figure 13. FC tiers disk heat map with XtremCache Figure 14 shows that high Oracle latency and high SQL Server data LUN latency on the two-tiered SAN-based storage can be effectively eliminated by adding XtremCache to the virtual machine hosting the applications. The average response times fell by approximately six to ten times for SQL Server OLTP and Oracle, respectively. Figure 14. Oracle and SQL Server latency before and after XtremCache 36

37 Two-tier FAST VP performance results breakdown Table 19 shows the detailed performance metrics for storage, ESXi, and virtual machines of the running workloads on the source array for Oracle and SQL Server OLTP applications before and after enabling XtremCache on the two-tier FAST VP. With FC and SATA tiers in the FAST VP pool, after enabling XtremCache: OLTP application TPS and response times significantly improved because XtremCache can offload read I/O processing from the storage array, while reducing disk latencies, thus enabling higher transactional throughput. It can address hot-spots in the data center and alleviate high utilization of a twotier FAST VP storage environment. CPU usage increased because of the increased SQL Server and Oracle utilization with transactional processing. With XtremCache enabled the system was able to handle more SQL Server and Oracle TPS. Table 19. Performance metrics for storage, ESXi, and virtual machines Components Performance Two tiers configured without XtremCache Two tiers configured with XtremCache VMAX VMAX IOPS 23,514 13,798 ESXi 01 Average CPU utilization 16 percent 21 percent ESXi 02 Average CPU utilization 2 percent 40 percent Swingbench TPS 6,653 8,590 Oracle OLTP Average Oracle response (ms) 35 3 vcpu utilization 47.6 percent 89 percent SQL01 latency (ms) (read/write/transfer) 22/4/21 3/2/3 SQL Server SQL02 latency (ms) (read/write/transfer) 21/4/21 2/3/2 SQL01 vcpu utilization 1.20 percent percent SQL02 vcpu utilization 1.46 percent percent Transaction/sec 2,073 6,054 37

38 XtremCache impact on FAST VP Overview In this solution, we evaluated the impact of XtremCache on FAST VP to measure if XtremCache offloading caused an impact on the existing three-tiered FAST VP. FAST VP moves data between tiers. The ingress and egress track records the data moving in or out from each tier. If there are many ingress tracks and egress tracks per second, the back-end performance is impacted when the application is running on the corresponding tiers. Generally, FAST VP has a quality of service (QoS) setting to control the reallocation rate of the data movement. The minimum value of the setting is 10; this means 1 GB/sec is the maximum moving rate. If the ingress/egress of each tier is far less than this value, the impact to the back-end is minimal. In the solution, the ingress/egress track of each tier (FLASH, FC, and SATA) was monitored when we enabled the XtremCache on the virtual machines. The purpose was to evaluate if the XtremCache causes lots of data movement between the FAST VP controlled storage pools, and if the FLASH tier can be released automatically when we enable the XtremCache. Three-tier FAST VP behavior with XtremCache In the three-tier FAST VP implementation, after enabling XtremCache, the flash tier ingress/egress tracks per second were less than 200 tracks per second, as shown in Figure 15, Figure 16, and Table 20. These charts show that XtremCache started taking the load off the flash tier of the three-tier FAST VP to the FC and SATA tier, freeing up these resources for other I/O intensive applications. This movement was controlled by FAST VP. Figure 15. Track ingress of three tiers after enabling XtremCache on three-tiered FAST VP 38

39 Figure 16. Table 20. Track egress of three tiers after enabling XtremCache on three-tiered FAST VP Average track ingress/egress per second FC tier Flash tier SATA tier FAST ingress track per second FAST egress track per second The maximum ingress or egress data between tiers was approximately 200 x 64 KB = 13 MB/sec. The moving rate has a minimum impact to the back-end disk workload. Since FAST VP has a long demotion period (demoting the cold data from the flash or FC tier to the lower tier), the running workload with XtremCache does not actively demote the data. This means that although the previously frequently accessed data can be served by XtremCache and the workload is freed from the flash tier, the flash capacity can still be occupied. The flash tier in the FAST VP can be proactively freed to handle more IOPS. While XtremCache can accelerate the performance for the SQL Server or Oracle OLTP workloads, FAST VP with flash tier is still complementary to XtremCache. The central storage can be a cost-effective solution with XtremCache. When the read-intensive I/Os are served by XtremCache, FAST VP with flash tier enabled can handle other I/O requests. 39

40 XtremSF and XtremCache with DSS workload Overview Caching XtremCache has a split-card functionality in which a part of the XtremSF card can be used as a cache and the other part can be used as local storage. We tested XtremCache in split-card mode with a DSS workload as the cache and the SQL Server instance tempdb as the direct-attached storage (DAS). We carved a 200 GB LUN from the 700 GB available capacity and tested this as a cache to accelerate the TPC-H-like database data LUNs, as shown in Figure 17. The Max IO parameter is the maximum cached I/O size for XtremCache, which means that users can adjust the maximum cached I/O size for different application I/O patterns. In this solution, it was set to 128 KB for DSS workload. This means that any I/O size less than 128 KB is cached; whereas all other I/O greater than 128 KB is bypassed by XtremCache. Figure 17. LUN creation Figure 18 shows the space allocation for DSS from the 700 GB available capacity. Figure 18. XtremCache allocation for DSS 40