Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
Table of Contents Overview... 1 Goal... 1 Audience... 1 Objectives... 1 Authors... 1 Companies... 1 Summary... 2 Infrastructure Components... 3 UCS... 3 Applications... 5 Cisco UCS Servers Change the Economics of the Data Center... 5 Fusion-io... 7 Consistent, Low-Latency Performance... 7 Industry-leading Capacity... 7 Lowest CPU Utilization per Unit of Application Work... 8 Unrivaled Endurance... 8 Enterprise Reliability via Self-Healing, Wear Management, and Predictive Monitoring. 8 Extensive OS Support and Flexibility... 8 Fusion ioturbine... 9 ioturbine, Intelligent Application-level Caching Software... 9 ioturbine Benefits... 10 ioturbine Features... 10 Intelligent, Efficient Usage of Virtualization and Infrastructure... 11 ioturbine Deployment... 11 Technical Strengths for ioturbine... 12 2
Product Features... 13 System Requirements... 13 Architecture Design of Microsoft SQL on Cisco UCS, Leveraging Fusion iomemory and ioturbine... 13 Design... 13 Write-through Cache... 14 Logical Design... 14 Physical Design... 15 UCS Configuration... 17 Service Profile... 32 Installing the UCS Fusion-io Mezz Card... 37 ESXi Image Configuration... 40 Obtaining Offline Driver Bundles... 40 Adding Software Depots... 41 Building and Export the Image... 42 ioturbine Configuration... 43 Virtual Management Appliance VME2 Install... 43 Installing the Host... 47 Installing the Guest Package... 50 VMware Cluster Considerations... 53 Dynamic Resource Scheduler Configuration... 53 VM Configuration... 59 Performance Validation... 60 Iometer Configuration... 61 iometer Results... 64 Performance Charts: IOPS... 65
Performance Chart: CPU Utilization... 68 Performance Chart: IOPs... 69 Performance Chart: Average Read Response Time... 70 TPC-E Test Results... 71 Comparison Summary... 71 TPS vs. Userload... 72 Response Time vs. Userload... 72 SQLIO Results... 73 ESXTOP... 75 Best Practices... 76 ioturbine MS SQL Best Practices... 76 Available CPU... 76 Available Memory... 76 Database Components to Cache... 76 Measuring Success... 76 Additional Recommendations... 76 References... 77 APPENDIX... 77 Solution Validation... 77 Test Setup and Configuration... 77 IOMeter Tests... 77 Installing, Configuring, and Uninstalling ioturbine via the CLI... 80 Setup/Prerequisites... 80 Procedure... 80 Troubleshooting Commands... 84 Support... 84 4
System... 84 Account... 84 List... 85
Overview Goal The goal of this document is to provide an architecture reference to enhance workload performance in a virtual environment. Results and design will be based on the concept of caching high I/O closer to the CPU, through the use of ioturbine on a Cisco UCS platform. Testing results and best practices will also be provided. The test results in this document are intended to show caching functionality, not to act as performance benchmarks. Your own test results may vary from those shown here. Audience The document is intended to aid architects in their efforts to design highperformance workloads within a virtual platform. It is assumed that there is a basic understanding of UCS, virtualization, caching concepts and storage design. Objectives Provide a comprehensive document that outlines reference architecture for high performance SQL workloads. Authors Kyle Wallace Senior Solutions Engineer, Datalink Victor Backman Systems Engineer, Fusion-io William Sahm Systems Engineer, Fusion-io Companies Cisco is the worldwide leader in networking that transforms how people connect, communicate and collaborate. To learn more, call 800.553.6387 or visit www.cisco.com. Fusion-io delivers the world s data faster. Our Fusion iomemory platform accelerates databases, virtualization, cloud computing, big data, and the applications that drive our economy and our daily lives. For additional information, call 800.578.6007 or visit www.fusionio.com. Datalink is a complete data center solutions and services provider for Fortune 500 and mid-tier enterprises, transforming data centers so they become more efficient, manageable and responsive to changing business Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 1
needs. Datalink helps leverage and protect storage, server, and network investments with a focus on long-term value, offering a full lifecycle of services, from consulting and design to implementation, management and support. For more information, call 800.448.6314 or visit www.datalink.com. Intel is a world leader in computing innovation. The company designs and builds the essential technologies that serve as the foundation for the world s computing devices. Additional information about Intel is available at newsroom.intel.com and blogs.intel.com. Summary Virtualization allows organizations to use and manage infrastructure much more efficiently. However, virtualization places extremely high demand on storage systems by making all access random even sequential access. Because random access is precisely the pattern that most challenges traditional storage, much of virtualization s cost savings are offset by the need to purchase and maintain expensive storage systems just for performance purposes. Fusion-io's industryleading, low-latency iomemory eliminates the I/O bottleneck to improve virtual machine workload capabilities in all servers, allowing organizations to unleash virtualization s true potential. This makes it easy for organizations to size by capacity instead of by performance. With an iomemory solution from Fusion-io, you can Increase the performance of virtual machines. o Run more virtual machines per server. o Improve the performance of each virtual machine. o Virtualize even the most data-intensive workloads, including databases. o Reduce I/O latency to drive higher throughput of virtual machines. Simplify virtual environments. o Use fewer servers to support more virtual machines. o Consolidate and simplify virtual environments. o Accelerate existing storage infrastructure. o Utilize disk storage for capacity, not performance. Fully leverage your hardware investment. 2 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
o Replace large amounts of costly RAM with a Fusion-io memory tier. o Achieve better CPU utilization. o Reduce reliance on costly high-speed network storage. o Continue to leverage vmotion (with ioturbine caching). o Dynamically reallocate flash capacity as VMs come and go, to assure maximum efficiency of flash resources. o Fully utilize your hardware resources and stop overprovisioning them. Infrastructure Components The infrastructure components for the virtual performance cluster described in this document are as follows: Cisco UCS (Unified Computing System) Fusion iomemory and VSL (Virtual Storage Layer) Fusion ioturbine caching software Microsoft SQL Server Each of these components is described in this section. UCS The Cisco Unified Computing System (Cisco UCS ) combines Cisco UCS B-Series Blade Servers and C-Series Rack Servers with networking and storage access in a single converged system. This simplifies management and delivers greater cost efficiency and agility, with increased visibility and control. The latest server expansion of the Cisco UCS portfolio includes the Cisco UCS B200 M3, B22 M3, and B420 M3 Blade Servers, along with the C22 M3, C24 M3, C220 M3, C240 M3 and C420 M3 Rack Servers. These servers increase compute density through more cores and cache, balanced with more memory capacity and drives, and with faster I/O. Together, these server improvements and complementary Cisco UCS system advancements deliver the best combination of features and cost efficiency to support IT s diverse needs. The Cisco UCS B200 M3 Blade Server delivers performance, versatility, and density without compromise. It addresses the broadest set of workloads, from IT and web infrastructure, through a distributed database. Building on the success of the Cisco Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 3
UCS B200 M2 Blade Server, the enterprise-class Cisco UCS B200 M3 Blade Server further extends the capabilities of the Cisco UCS portfolio in a half-width blade form factor. The Cisco UCS B200 M3 harnesses the power of the latest Intel Xeon processor E5-2600 product family, with up to 768 GB of RAM (using 32-GB DIMMs), two disk drives, and up to dual 4x 10 Gigabit Ethernet throughput. In addition, Cisco UCS has the architectural advantage of not having to power and cool excess switches in each blade chassis. With a larger power budget per blade server, Cisco can design uncompromised expandability and capabilities in its blade servers, as evidenced by the new Cisco UCS B200 M3, with its leading memory slot and drive capacity. The Cisco UCS B200 M3 provides: Two multi-core, Intel Xeon processor E5-2600 product family CPUs, for up to 16 processing cores 24 DIMM slots for industry-standard double-data-rate 3 (DDR3) memory running up to 1600 MHz, and up to 768 GB of total memory (using 32-GB DIMMs) Two optional, hot-pluggable SAS or SATA hard disk drives (HDDs) or solid state drives (SSDs) Remote management through a Cisco Integrated Management Controller (CIMC) that implements policy established in Cisco UCS Manager Out-of-band access by remote keyboard, video, and mouse (KVM) device, Secure Shell (SSH) Protocol, and virtual media (vmedia), as well as the Intelligent Platform Management Interface (IPMI) In addition, the Cisco UCS B200 M3 is a half-width blade (Figure 1). Up to eight of these high-density, two-socket blade servers can reside in the 6RU Cisco UCS 5108 Blade Server Chassis, offering one of the highest densities of servers per rack unit in the industry. Another Cisco innovation, the Cisco UCS Virtual Interface Card (VIC) 1240 is a 4-port 10 Gigabit Ethernet, Fibre Channel over Ethernet (FCoE)-capable modular LAN on motherboard (LOM), designed exclusively for the M3 generation of Cisco UCS B- Series Blade Servers. When used in combination with an optional I/O expander, the Cisco UCS VIC 1240 capabilities can be expanded up to eight ports of 10 Gigabit Ethernet. The Cisco UCS VIC 1240 enables a policy-based, stateless, agile server infrastructure that can present up to 256 PCI Express (PCIe) standards-compliant interfaces to the host that can be dynamically configured as either network interface cards (NICs) or host bus adapters (HBAs). In addition, the Cisco UCS VIC 1240 supports Cisco Data Center Virtual Machine Fabric Extender (VM-FEX) technology, which extends the Cisco UCS fabric interconnect ports to virtual machines, simplifying server virtualization deployment. 4 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
Figure 1. Cisco UCS B200 M3 Blade Server The Cisco UCS B200 M3 Blade Server continues Cisco s commitment to delivering uniquely differentiated value, fabric integration, and ease of management that is exceptional in the marketplace. The Cisco UCS B200 M3 further extends the capabilities of Cisco UCS by delivering new levels of manageability, performance, energy efficiency, reliability, security, and I/O bandwidth for enterprise-class applications: Applications The Cisco UCS B200 M3 is suited for a broad range of IT workloads: Virtualized workloads IT and web infrastructure Virtual desktops Databases Middleware Enterprise resource planning (ERP) and customer relationship management (CRM) applications Cisco UCS Servers Change the Economics of the Data Center IT infrastructure matters now more than ever, as organizations seek to achieve the full potential of infrastructure as a service (IaaS), bare metal, virtualized servers, and cloud computing. Cisco continues to lead in Data Center innovation with the introduction of new building blocks for the Cisco Unified Computing System that extend its exceptional simplicity, agility, and efficiency (Figure 2). Innovations such as the Cisco UCS B200 M3 Blade Server bring even better industry-leading performance from IT infrastructure to enterprise applications. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 5
Figure 2. Cisco UCS Components Cisco innovations, such as the Cisco UCS Manager software, allow administrators to create a software model of a desired server (using Cisco service profiles and templates) and then instantiate that server and its I/O connectivity by associating a model with physical resources. This approach contrasts with the traditional approach of configuring each system resource manually, one at a time, through individual element managers. Unlike legacy vendors of traditional systems, Cisco uses a unified management model with service profiles that can be moved easily between any Cisco UCS servers whether blade server or rack server in a Cisco UCS Manager domain. In other words, Cisco UCS service profiles are form-factoragnostic. Other Cisco UCS building blocks include new Cisco UCS VIC options, expanded Cisco UCS fabric interconnects, and additional fabric extender options that expand throughput and management simplicity for both blade and rack servers across baremetal, virtualized, and cloud environments. Cisco UCS also provides investment protection. For example, fabric extenders can be upgraded in the same Cisco UCS 5108 Server Chassis using the same fabric interconnects. Fabric interconnects can be upgraded independently of fabric extenders and blade servers within the chassis. The Cisco UCS 5108 Chassis high-performance mid-plane provides 8 blades with 1.2 6 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
terabits (Tb) of available Ethernet throughput for future blade and I/O requirements. In addition, Cisco continues to innovate in all these areas, helping ensure that newer, more powerful blade servers have matching I/O bandwidth and computing power through continuous innovation across the Cisco UCS environment. The Cisco UCS B200 M3 is also part of a large family of blade servers: the Cisco B- Series Blade Servers. Designed to operate as part of Cisco UCS, the Cisco UCS B- Series servers employ many innovative Cisco technologies to help customers handle the most challenging workloads. Cisco UCS B-Series servers operating in a Cisco UCS management framework incorporate a standards-based unified network fabric, Cisco VM-FEX virtualization support, Cisco UCS Manager software, Cisco fabric extender architecture, and Cisco Extended Memory Technology. Again, Cisco is innovating across all these technologies. Together, these Cisco UCS architectural advantages and Cisco s software advances, Cisco s continuous innovation, and unique blade server and chassis designs combine to make Cisco UCS the first truly unified data center platform. In addition, Cisco UCS can transform IT organizations through policy-based automation and deep integration with familiar systems management and orchestration tools. Fusion-io Fusion iomemory accelerates businesses from the smallest e-tailers to the world s largest data centers, social media leaders, and Fortune Global 500 businesses. Fusion-io combines its VSL (Virtual Storage Layer) and iomemory modules into an iomemory platform that takes enterprise applications and databases to the next level. This platform leverages flash memory to significantly increase datacenter efficiency, with enterprise grade performance, reliability, availability, and manageability. Consistent, Low-Latency Performance The iomemory platform provides consistent, low-latency access for mixed workloads. It features 15-microsecond access latency, 3GB/s bandwidth, over 700,000 read IOPS, and over 1,100,000 write IOPS. The sophisticated iomemory architecture allows for nearly symmetrical read and write performance, with bestin-class low queue depth performance. This makes the iodrive2 and iodrive2 Duo devices, powered by Fusion iomemory, an ideal solution across a wide variety of real world, high-performance enterprise environments. Industry-leading Capacity Get the maximum capacity in the smallest footprint to cache more process-critical data close to application CPUs where it s needed most. The Fusion iodrive2 is available in 365GB, 785GB, and 1,205GB capacities, while the iodrive2 Duo features 2.4TB of capacity to fuel CPU efficiency even in the most data-intensive environments. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 7
Lowest CPU Utilization per Unit of Application Work The iomemory platform integrates with host system CPUs as a memory tier to give multiple (and mostly idle) processor cores, direct and parallel access to the flash. The platform s cut-through architecture gives systems more work per unit of processing, and it continues to deliver performance increases as CPU power increases. Unrivaled Endurance The iomemory platform offers best-in-class endurance in all capacities, which is crucial for caching and write-heavy databases and applications. Enterprise Reliability via Self-Healing, Wear Management, and Predictive Monitoring The iomemory platform includes powerful features to eliminate concerns about reliability, including NAND failures and excessive wear. The all-new intelligent, selfhealing feature called Adaptive Flashback provides complete chip-level fault tolerance, which enables an iomemory product to repair itself after a single-chip or multi-chip failure, without interrupting business continuity. Extensive OS Support and Flexibility The platform supports all major operating systems, including Windows, Linux, OSX, Solaris x86 and ESXi 5.0. Field programmability allows IT professionals maximum flexibility in customizing their iomemory platform to meet the unique needs of their enterprise. Figure 3. Cisco Fusion iodrive2 The 785GB PCI Express Flash model has the following performance characteristics: 1.5GB/s bandwidth 535,000 IOPS 15-microsecond latency 8 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
Fusion ioturbine ioturbine, intelligent application-level caching software from Fusion-io, transforms the iodrive into a transparent acceleration device for virtualized workloads to cache hot data off any storage system, whether SAN- or NFS-based. ioturbine caches data within hosts, close to application workloads, delivering unparalleled low latency. With ioturbine, Fusion-io customers can deploy terabytes of flash-based cache to significantly accelerate performance of virtualized workloads, eliminate I/O bottlenecks, increase virtual machine density per physical host, and minimize incremental investments in external storage infrastructure. ioturbine, Intelligent Application-level Caching Software The diagram below shows the role of ioturbine in providing a VMware caching solution when combined with iodrives and vmotion. Figure 4. ioturbine in a virtualization solution ioturbine caches frequently used data on iodrives. This enables your system to Virtualize without compromise. Maximize performance for I/O intensive applications. Seamlessly integrate with existing storage. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 9
Take full advantage of vmotion. Example applications include Any Windows application SQL Server Microsoft Exchange Sharepoint Active Directory IIS ioturbine Benefits ioturbine enables you to Transparently cache hot data off any existing SAN- or NFS-based storage volumes to eliminate I/O bottlenecks and significantly accelerate virtualized workloads. Increase the VM density per physical host and minimize incremental server investments. Minimize expensive investments in external storage infrastructures. Minimize overprovisioning of LUNs and maximize storage efficiency. Minimize investment in storage networking and maximize available network bandwidth. Virtualize I/O-intensive critical applications, as NAND flash memory in the host can be hundreds of times faster than disk-based solutions. Leverage policy-based management by treating the iodrive as an elastic cache resource when VMs are created, destroyed or migrated. Build a scalable, performance-oriented foundation for cloud computing. ioturbine Features ioturbine enables simple administration and optimum control of resources. Guest VMs continue to see SAN- or NFS-based ESX data stores as-is, without requiring any configuration change. 10 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
iodrives can be used transparently as cache Seamless live migration of VMs with VMware vmotion from one physical host to another Centralize VM cache policy management across the data center Integration with VMware vcenter to provide VM-to-physical host relationship Intelligent, Efficient Usage of Virtualization and Infrastructure Virtualization becomes more efficient and effective with ioturbine. Here are some key virtualization and performance advantages that ioturbine offers: Support for VMware vsphere Graphical reports to monitor cache hits and performance statistics Fine-grain control for advanced users to cache at the disk, volume, or individual file level Policy-based management on a per-vm basis. This allows the iodrive cache size to be dynamically adjusted when VMs are created, destroyed, or migrated between hosts. ioturbine Deployment The diagram below shows ioturbine deployed on a Hypervisor host and multiple guest VMs, using the Fusion iosphere VM Edition. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 11
Figure 5. ioturbine deployment example with VMware ESX Technical Strengths for ioturbine Here are some of the ways ioturbine provides technical advantages in your virtualized environment: Accelerates any OS-visible storage volume, disk or file as a write-through cache Runs as two components: in the hypervisor, and in the guest operating system Is tightly coupled with guest OS I/O routines and transparently redirects read requests to iomemory for significantly faster random read access Features strong per-vm performance statistics and rich graphing of hits, reads, and writes, with percentages and IOPS Supports logs and ftp to Fusion-io Support for diagnostics 12 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
Product Features Cache type Cache mode Additional CPU and RAM overhead Management Power loss protection Fully associative cache Write-through No noticeable increase over the iomemory VSL requirements GUI, CLI Yes System Requirements Compatible cache devices Operating systems Hypervisor iodrive, iodrive2, iodrive Duo, iodrive2 Duo, and iodrive Octal Microsoft Windows Server 2008 and 2008 R2 (x64-bit only) Windows Server 2003 (32- and 64-bit), Windows Server 2003 R2 (32- and 64-bit), Windows Server 2003 SP2 (32- and 64-bit) Red Hat Enterprise Linux (RHEL) 5.6, 5.7, 5.8 (64-bit) SUSE Linux Enterprise Server (SLES) 11 SP1 (64-bit) VMware ESX 4.0 Update 2 and Update 3, ESX 4.1 Update 1 and Update 2, ESXi 4.1 Update 1 and Update 2, ESXi 5.0, ESXi 5.1 Architecture Design of Microsoft SQL on Cisco UCS, Leveraging Fusion iomemory and ioturbine Design Maximizing SQL performance in a shared infrastructure can be challenging. However, a solution that combines the strengths of the Cisco UCS with iomemory and ioturbine can offer compelling advantages. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 13
In an effort to minimize bottlenecks from back-end, SAN-based architectures, this design is focused on keeping read requests local to the ESXi host on the PCIe bus (served from a NAND flash medium). This will enable the array to serve writes more efficiently, by not having high random demand from reads. When read requests are processed through the VM and Hypervisor stack through to the array, the reads become highly randomized (known as the blender effect). For this design, the database is installed on a separate drive or VMDK from the log and TempDB data. Only the database files will be enabled with cache, which optimizes cache usage. Write-through Cache Write-through caches tend to be the most common, because it provides a good balance of performance and reliability. A block of data can get into the cache when an application either reads or writes data. Once data is read or written, any subsequent read of the block is quickly returned to the application from the cache instead of from slower primary storage. The diagram below shows a typical flow for write-through caching. 1. READ (not yet in cache) 2. READ (already in cache) 3. WRITE Figure 6. Write-through caching Logical Design The diagram below shows a design for deploying ioturbine in a cluster, on Cisco M3 Blade servers running VMware ESXi Server. 14 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
Figure 7. Deploying ioturbine in a clustered configuration Physical Design The diagram below shows a sample physical design to deploy a UCS network system. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 15
Key: 10GbE Twinax (copper w/pre-terminated SFP) 1000BASE-T Copper Figure 8. Sample Physical Design for deployment of a UCS-based system for ioturbine and iomemory Assuming the physical installation is completed for the components described above, the following tasks need to be completed: Configure the UCS Create a Service Profile Install and configure the Cisco/Fusion-io Mezz cards Each of these tasks is described below. 16 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
UCS Configuration To configure the UCS, follow the steps below (adapted from the Cisco UCS Manager Configuration Guide): 1. Install the cables for LAN/SAN/Server. 2. Connect a console cable to Fabric Interconnect A and complete the initial cluster configuration wizard. 3. Connect a console cable to Fabric Interconnect B and complete the initial configuration wizard. 4. Access the UCS via the VIP (Virtual IP) in a web browser. 5. In the UCS Manager, configure the DNS and NTP settings in the Admin tab. A sample screen is shown below. 6. In the Equipment tab, choose the Policies tab in the right-hand pane. 7. Set the policies for power and FEX discovery. An example is shown below. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 17
8. Enable the server ports in the Equipment tab, as shown below. 9. In the Equipment tab, choose the General tab in the right-hand pane. 10. Click Acknowledge Chassis in the Actions section. 18 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
11. In the Equipment tab, click the Firmware tab in the right-hand pane. 12. In the sub-tabs that appear, choose Installed Firmware. 13. Verify that the current Cisco UCS Manager firmware version is 2.1 or later (required for Fusion-io cards). For example: Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 19
If the displayed firmware version is older than 2.1, upgrade the firmware according to the Cisco UCS B-Series GUI Firmware Management Guide, Release 2.1. 14. In the LAN tab, click the General tab in the right-hand pane. 15. Create KVM IPs by adding a block of IP Addresses to the ext-mgmt pool. An example is shown below. 16. Click OK to continue. Next, you need to create LAN uplinks and port channels. 17. In the LAN tab, open the Fabric A entry inside LAN Cloud. 20 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
18. Click Create Port Channel (circled above) to open the Create Port Channel dialog. 19. Use the dialog to designate ports in the port channel. To do this, select an entry from the Ports list and click the right-arrow button to transfer it to the Ports in the port channel list. 20. Click Finish to close the dialog. 21. In the LAN tab, open the Fabric B entry inside LAN Cloud and complete the previous three steps to create a port channel on the B fabric. 22. Click the VLANs entry to open the Create VLANs dialog. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 21
23. In the Create VLANs dialog, specify the VLAN Name/Prefix. 24. Enter the VLAN IDs as necessary. 25. Click OK to save changes. 26. If necessary, create SAN Uplinks and port channels, using the previous steps as a guide. 27. Click the VSANs entry to open the Create VSANs dialog. 22 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
28. Enter the VSAN name, VSAN ID, and FCoE VLAN ID. 29. Click OK to save changes. 30. In the LAN tab, select QoS System Class in the list. 31. In the General Tab in the right-hand pane, specify QoS settings as appropriate for the environment. 32. Click OK to save the changes. 33. In the LAN tab, open Policies > Root > QoS Policies in the list. The Create Qos Policy dialog opens. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 23
34. Specify the settings you want for this QoS policy. If more policies are needed, specify the settings for each policy. 35. Click OK. MAC Pool Creation 36. In the LAN tab, open Pools > Root > MAC Pools. 37. Right-click MAC Pools and select Create MAC Pool. A wizard opens for MAC pool creation. 38. Follow the steps in the wizard to create the MAC pool for Fabric A. 24 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
39. When the wizard is finished, repeat the previous two steps to create a similar pool for Fabric B. The completed MAC pools are displayed in the right-hand pane and in the entry list. WWNN Pool Creation 40. In the SAN tab, open Pools > Root > WWNN Pools. 41. Right-click WWNN Pools and select Create WWNN Pool. A wizard opens for WWNN pool creation. 42. Follow the steps in the wizard to create the WWNN pool. The completed WWNN pool is displayed in the right-hand pane and in the entry list. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 25
WWPN Pool Creation 43. Right-click WWPN Pools and select Create WWPN Pool. A wizard opens for WWPN pool creation. 44. Follow the steps in the wizard to create the WWPN pool for Fabric A. 45. When the wizard is finished, repeat the previous two steps to create a similar pool for Fabric B. The completed WWPN pools are displayed in the right-hand pane and in the entry list. UUID Suffix Pool Creation 46. In the Server tab, open Pools > Root > UUID Suffix Pools. 47. Right-click UUID Suffix Pools and select Create UUID Suffix Pool. 26 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
A wizard opens for UUID suffix pool creation. 48. Follow the steps in the wizard to create the UUID suffix pool. The completed UUID suffix pools are displayed in the right-hand pane and in the entry list. vnic Template Creation 49. In the LAN tab, open Policies > Root > vnic Templates. 50. Right-click vnic Templates and select Create vnic Template. A wizard opens for vnic template creation. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 27
51. Follow the steps in the wizard to create the vnic template for Fabric A. 52. When the wizard is finished, repeat the previous two steps to create a similar template for Fabric B. The completed vnic templates are displayed in the right-hand pane and in the entry list. 53. In the SAN tab, open Policies > Root > vhba Templates. 54. Right-click vhba Templates and select Create vhba Template. The Create vhba Template wizard appears. 28 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
55. Fill in the necessary fields and click OK to save changes. 56. In the Servers tab, open Policies > Root > Boot Policies. The Create Boot Policy dialog appears. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 29
57. Fill in the dialog fields. 58. In the bottom half of the dialog, select devices from the left pane and add them to the right pane, to create the boot order you prefer. 59. Click OK to continue. 60. In the Servers tab, open Policies > Root > Host Firmware Packages in the list. The Create Host Firmware Package dialog appears. 61. Fill in the dialog fields and click OK to continue. 62. In the Servers tab, open Policies > Root > Local Disk Configuration Policies in the list. The Create Local Disk Configuration Policy dialog appears. 30 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
63. Fill in the dialog fields and click OK to continue. 64. In the Servers tab, open Policies > Root > Maintenance Policies in the list. The Create Maintenance Policy dialog appears. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 31
65. Fill in the dialog fields to create a User_Acknowledge policy and click OK. Service Profile The next task is to create a Service Profile Template using the pools, policies, and templates created earlier. 1. In the Server tab, right-click Service Profile Templates and choose Create Service Profile Template. 32 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
The Create Service Profile Template wizard begins. 2. Specify the name and the UUID for the service profile template. 3. Click Next to continue. The Networking screen appears. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 33
4. Choose the Expert option for configuring LAN connectivity. 5. Add vnics the server, selecting the vnic Templates that were previously created. 6. Click Next to continue. The Storage screen appears. 7. Choose the local storage policy you created earlier (RAID 1 in this case). 34 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
8. Choose the Expert option for configuring SAN connectivity. 9. Specify the WWNN (World Wide Node Name) pool previously created. 10. Add vhbas the server, selecting the vhba Templates that were previously created 11. Click Next to continue. The Server Boot Order screen appears. 12. Choose the Boot Policy you created earlier (Local-Disk in this case). The details of the Boot Order you specified earlier will appear in the Boot Order area. 13. Click Next to continue. The Maintenance Policy screen appears. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 35
14. Choose the Maintenance Policy you created earlier (User_Acknowledge in this case). 15. Click Next to continue. The Server Assignment screen appears. 36 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
16. Select the Host Firmware package, then click Finish Installing the UCS Fusion-io Mezz Card To install the UCS Fusion-io Mezz Card in the UCS, follow the steps below. 1. Power down and remove the blade and blade cover. The installation slot is highlighted in red in the photo below. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 37
2. Locate the installation receptacle on the bottom side of the UCS Fusion-io Mezz card (highlighted in red below). 3. Turn the card over as shown below, noting the installation area ( Press Here to Install ). 38 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
4. Install the card and tighten the screws. For assistance with installation and safety guidelines, contact Cisco Customer Support. 5. Replace the blade cover and insert the blade. 6. When the blade boots, the Fusion-io card should be discovered in the blade s inventory. To check this, Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 39
a) In the Equipment tab, open the view for the server where the UCS Fusionio Mezz card is installed, such as Chassis > Chassis 1> Servers > Server3. b) Click the Inventory tab, then the Storage tab in the right-pane and check the PID and Serial Number fields. As of the UCS Del Mar release (firmware version 2.1(1a)), firmware management for the UCS Fusion-io Mezz cards is not available. This feature will be added in a later UCS release. ESXi Image Configuration The following procedure slipstreams the Fusion-io and Cisco VIC drivers into a customized ESXi.iso file used to build the hosts. The drivers can also be installed after the host is built. To create the customized ESXi image, these tasks need to be completed using VMware vsphere PowerCLI: Obtain offline driver bundles. Add software depots to be included in the ESXi profile. Build and export the image. These tasks are described below. Obtaining Offline Driver Bundles Refer to the Cisco UCS Hardware and Software Interoperability Matrix to identify the appropriate enic/fnic drivers. http://www.cisco.com/en/us/products/ps10477/prod_technical_reference_list.ht ml 40 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
To get the UCS VMware drivers for enic/fnic, 1. Browse to http://downloads.vmware.com. 2. Click VMware vsphere. 3. Click the Drivers and Tools tab. 4. Expand the Driver CDs section and download both the enic and fnic drivers. To download the Fusion-io VSL for the Cisco UCS Fusion iodrive2 card, navigate to http://support.cisco.com and download the UCS B-Series drivers bundle: Downloads Home > Products > Servers - Unified Computing > Cisco UCS B-Series Blade Server Software > Unified Computing System (UCS) Drivers. Adding Software Depots 1. To add the base ESXi software depot, run this command at the PowerCLI command prompt: Add-ESXSoftwareDepot https://hostupdate.vmware.com/software/vum/production/main/vmwdepot-index.xml 2. Add the Cisco enic drivers software depot by running this command: Add-EsxSoftwareDepot C:\ESXi\enic_driver_2.1.2.22-offline_bundle- 564611.zip 3. Add the Cisco fnic driver offline bundle by running this command: Add-EsxSoftwareDepot C:\ESXi\fnic_driver_1.5.0.8-offline_bundle- 758653.zip Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 41
4. Add the Fusion-io drivers software depot by running this command: Add-EsxSoftwareDepot C:\ESXi\iomemory-vsl-5X-3.2.2.869- offline_bundle-895853.zip Building and Export the Image 1. Determine the name of the Image Profile to clone: Get-EsxImageProfile Sort-Object "ModifiedTime" -Descending format-table -property Name,CreationTime,Description 2. Build a new image by cloning the desired VMware image: New-EsxImageProfile -CloneProfile ESXi-5.0.0-20120904001-standard -Name ESXi5.0U1_Cisco_FIO Vendor Datalink 3. Add drivers to the image by running the commands described below: Add-EsxSoftwarePackage -ImageProfile ESXi5.0U1_Cisco_FIO - SoftwarePackage scsi-fnic 42 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
Add-EsxSoftwarePackage -ImageProfile ESXi5.0U1_Cisco_FIO - SoftwarePackage net-enic Add-EsxSoftwarePackage -ImageProfile ESXi5.0U1_Cisco_FIO - SoftwarePackage block-iomemory-vsl 4. Compare Image Profiles: Compare-EsxImageProfile -ComparisonProfile ESXi5.0U1_Cisco_FIO - ReferenceProfile ESXi-5.0.0-20120904001-standard 5. Export the Image Profile: Export-EsxImageProfile ESXi5.0U1_Cisco_FIO -ExportToIso -FilePath C:\ESXi\ESXi5.0U1_Cisco_FIO.iso ioturbine Configuration Virtual Management Appliance VME2 Install The ioturbine software can be obtained from support.fusionio.com. There should be three files in the ioturbine software download folder: ioturbine-disk1.vmdk.gz ioturbine.mf ioturbine.ovf Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 43
To install VME2, 1. In the vsphere Client, choose File > Deploy OVF Template. 2. Open the Source screen. 3. Click Browse to navigate to the location of the unzipped files, and select ioturbine.ovf 4. Click Next. 5. In the OVF Template Details screen, click Next. 6. In the End User License Agreement screen, click Accept > Next. 7. Open the Name and Location screen. 8. Accept the default name or change it if desired. 9. Select the location for the VM. 10. Click Next. The Deploy OVF Template screen appears. 44 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
11. Enter the fully qualified name in the Hostname field, and networking information in the Networking Properties fields. 12. Once the VME2 is successfully deployed, connect via web browser to https://<ipaddress> where <ipaddress> is the Network 1 IP Address you specified in the Deploy OVF Template screen. 13. From this address, enter the name or address of the vcenter server and login credentials. This will install the Fusion-io plugin. For example: Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 45
14. In the Plug-in Manager (vcenter plugin tab), verify the plugin is installed and enabled. 15. Select an ESX host. 16. From the Fusion-io tab, click the Admin tab. 17. Choose Licensing > Upload License File. 46 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
18. Accept the end-user agreement. 19. Click Upload and apply license. Installing the Host This process will reboot the ESXi Host. 1. Put ESXi in maintenance mode. To do this, right-click the ESXi host and select Enter Maintenance Mode. 2. Enable the http ESXi firewall httpclient as shown in the example below. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 47
3. Click OK. 4. Right-click the host and select Fusion-io > Host Package Management. 5. Select the host to deploy the package and version. 48 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
6. Click Install. 7. When the host package install is finished, click the Configuration tab under Fusion-io. The Caching Device Selection area of the screen is where you will add the Fusion iodrive as a caching device 8. Select the Fusion-io device as shown in the example below. This will format the selected device, destroying any user data on the device. 9. Click the Performance tab. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 49
10. Click Enable to enable performance statistics. 11. Click OK to finish. Installing the Guest Package This process will reboot the VM guest. 1. Right-click the host in the tree view and select Fusion-io Guest Package Management. 2. Select the VM(s) to install the package, and then select the package version. 3. Click Install. 50 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
4. To verify the installation of the Guest Package Management, click Refresh. 5. In the Caching Shares area select a weighting value and click Apply. 6. To configure volume-based caching, add the drive letters (in this case, e: ) in the Volume Filter area and click Start. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 51
7. To verify the volume on the VM Guest, run the following command from the the ioturbine Shell (under Start > Programs): list --volumestatus 8. Click Refresh in the GUI. A Volume caching is working message should appear in the Volume Filter area of the screen. 52 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
VMware Cluster Considerations When designing a VMware cluster with read-only local cache, it is best to keep the workload local to that host whenever possible. In this case you may want to disable Dynamic Resource Scheduling (DRS) so that cache warming does not take place unnecessarily. Other workloads on the VMware cluster should be allowed to utilize DRS during contention. Disabling DRS from workloads utilizing ioturbine cache will not preclude the operation of vmotion or HA. It is not a necessary to disable DRS; however, doing so avoids performance degradation caused by unnecessary vmotion of cached workloads. Dynamic Resource Scheduler Configuration 1. Open the Host Settings dialog. 2. Right-click the name of the host to be configured, and select Edit Settings. 3. To disable DRS for a virtual machine, select the VM name and click the dropdown menu to select Disabled, as in the example below. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 53
4. Repeat the previous step to disable DRS for as many VMs as necessary. 5. Click OK. If a subset of hosts in a cluster is enabled with read-only cache, it is also recommended to redirect workloads to them, so the workloads do not end up on a non-caching ESX host. In the configuration used in this example, a total of 8 SQL servers can be divided equally between the two nodes in the cluster. This will ensure that 4 SQL servers, when powered on, will be assigned to each node. In the event of resource contention, the VMs will not migrate to other nodes in the cluster but will restart on another node in the event of a host failure. 6. Select DRS Groups Manager in the vsphere DRS list, as shown below. 54 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
Next, you need to assign the VMs that are to be associated with each node to a DRS group. 7. Select a group in the Virtual Machines DRS Groups list (such as ioturbinevms_groupa) and click Edit. The Edit DRS Group dialog appears. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 55
8. Select a VM from the left-hand list and add it to the DRS group by clicking the double-arrow button. The added VM then appears in the right-hand list. 9. Repeat the previous step for each VM to be added to the DRS group. 10. When you are finished adding VMs to the group, click OK. Then you need to add the ESXi host to the individual DRS group. 11. In the Host DRS Groups area, select a group and click Edit. 56 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
The Edit DRS Group dialog appears. 12. Select a Host from the left-hand list and add it to the DRS group by clicking the double-arrow button. The added Host then appears in the right-hand list. 13. Repeat the previous step for each Host to be added to the DRS group. 14. When you are finished adding Hosts to the group, click OK. The added VMs and Hosts now appear in the DRS Groups: Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 57
The next task is to create rules for the VMs in the cluster. 15. Click Rules under vsphere DRS. 58 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
16. In the right-hand window, click Add to create a new rule, selecting the groups created earlier. 17. Click OK to finish. VM Configuration The baseline VM configuration used for testing included the following components: 4vCPU 2GB RAM 20GB database on D:\ The screens below illustrate these components in more detail. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 59
Performance Validation The test results in this section are intended to show caching functionality, not to act as performance benchmarks. Your own test results may vary from those shown here. To ensure the functionality of the ioturbine caching software in the high IOPs cluster, three methods of simulated tests were run: Iometer test to simulate a 100%-random-read and 70%-read, 30%-write workload. TPC-E benchmark SQLIO simulated workload with various block sizes 60 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
These were all tested without cache enabled, and again with cache warmed to 80% cache-hit ratio. Performance results varied based on the underlying infrastructure. Although this document is not intended to be a benchmark, it shows the validation of cache working from the results of the IO characteristics under simulated loads. Iometer Configuration The following Iometer configuration was used for the performance tests: 4 workers 4KB, 8KB, and 64KB workloads 16 I/Os per worker The Edit Access Specification screen below shows the settings being used for the SQL Load Test. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 61
1. To verify that the CACHE setting is off, run the following commands from the ioturbine Shell: provision --StopVolumeCache list -- volumestatus 62 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
2. In the GUI, go to the Configuration tab under the Fusion-io tab. 3. Verify that the message Volume caching is not working appears. 4. Click the Results Display tab in Iometer to view the real-time performance results for the test. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 63
iometer Results The table below shows iometer results for a single VM, with caching alternating OFF and then ON. Block sizes are 4KB, 8KB, and 64KB; mixes are 70%-read and 100%- read. Average Response Time Average Read Response Time Average Write Response Time Average Transaction Time Access Spec. Name IOPS Read IOPS Write IOPS MB/s Read MB/s Write MB/s Transactions per Second % CPU Utilization 1VM ON 4K; 70% Read; 100% random 10988.357 7695.149 3293.209 42.923 30.059 12.864 10988.357 5.823 7.641 1.575 5.823 13.303 1VM OFF 4K; 70% Read; 100% random 6846.220 4789.795 2056.425 26.743 18.710 8.033 6846.220 9.347 12.927 1.009 9.347 3.818 1VM ON 4K; 100% Read; 100% random 22616.769 22616.769 0.000 88.347 88.347 0.000 22616.769 2.829 2.829 0.000 2.829 24.341 1VM OFF 4K; 100% Read; 100% random 6870.964 6870.964 0.000 26.840 26.840 0.000 6870.964 9.314 9.314 0.000 9.314 3.496 1VM ON 8K; 70% Read; 100% random 6906.486 4838.765 2067.721 53.957 37.803 16.154 6906.486 9.265 12.710 1.203 9.265 9.788 1VM OFF 8K; 70% Read; 100% random 5457.166 3820.670 1636.496 42.634 29.849 12.785 5457.166 11.727 16.239 1.191 11.727 3.124 1VM ON 8K; 100% Read; 100% random 47042.723 47042.723 0.000 367.521 367.521 0.000 47042.723 1.360 1.360 0.000 1.360 43.131 1VM OFF 8K; 100% Read; 100% random 5654.282 5654.282 0.000 44.174 44.174 0.000 5654.282 11.318 11.318 0.000 11.318 3.249 1VM ON 64K; 70% Read; 100% random 7310.720 5114.695 2196.026 57.115 39.959 17.156 7310.720 8.753 12.011 1.164 8.753 9.752 1VM OFF 64K; 70% Read; 100% random 5502.202 3851.451 1650.751 42.986 30.089 12.896 5502.202 11.631 16.108 1.185 11.631 3.137 1VM ON 64K; 100% Read; 100% random 10931.665 10931.665 0.000 683.229 683.229 0.000 10931.665 5.853 5.853 0.000 5.853 26.319 1VM OFF 64K; 100% Read; 100% random 1404.454 1404.454 0.000 87.778 87.778 0.000 1404.454 45.565 45.565 0.000 45.565 1.423 The charts in the next sections show the above results in a graphical view. Each chart contains results for caching ON (blue bars) and caching OFF (red bars). 64 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
Performance Charts: IOPS 12000 10000 8000 6000 4000 1VM ON 4K; 70% Read; 100% random 1VM OFF 4K; 70% Read; 100% random 2000 0 IOps Read IOps Write IOps 4KB block size, 70% read mix 25000 20000 15000 10000 1VM ON 4K; 100% Read; 100% random 1VM OFF 4K; 100% Read; 100% random 5000 0 IOps Read IOps Write IOps 4KB block size, 100% read mix Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 65
8000 7000 6000 5000 4000 3000 2000 1VM ON 8K; 70% Read; 100% random 1VM OFF 8K; 70% Read; 100% random 1000 0 IOps Read IOps Write IOps 8KB block size, 70% read mix 50000 45000 40000 35000 30000 25000 20000 15000 10000 5000 0 IOps Read IOps Write IOps 1VM ON 8K; 100% Read; 100% random 1VM OFF 8K; 100% Read; 100% random 8KB block size, 100% read mix 66 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
8000 7000 6000 5000 4000 3000 2000 1VM ON 64K; 70% Read; 100% random 1VM OFF 64K; 70% Read; 100% random 1000 0 IOps Read IOps Write IOps 64KB block size, 70% read mix 12000 10000 8000 6000 4000 1VM ON 64K; 100% Read; 100% random 1VM OFF 64K; 100% Read; 100% random 2000 0 IOps Read IOps Write IOps 64KB block size, 100% read mix Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 67
Performance Chart: CPU Utilization The chart below shows the percentage of CPU utilization by each of the VM test workloads. 100 90 80 70 60 50 40 30 20 10 0 1VM ON 4K; 70% Read; 100% random 1VM OFF 4K; 70% Read; 100% random 1VM ON 4K; 100% Read; 100% random 1VM OFF 4K; 100% Read; 100% random % CPU Utilization 1VM ON 8K; 70% Read; 100% random 1VM OFF 8K; 70% Read; 100% random 1VM ON 8K; 100% Read; 100% random 1VM OFF 8K; 100% Read; 100% random 1VM ON 64K; 70% Read; 100% random 1VM OFF 64K; 70% Read; 100% random 1VM ON 64K; 100% Read; 100% random 1VM OFF 64K; 100% Read; 100% random 68 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
Performance Chart: IOPs The chart below shows the I/O operations per second produced by each of the VM test workloads. 50000 IOPS 45000 40000 35000 30000 25000 20000 15000 10000 5000 0 1VM ON 4K; 70% Read; 100% random 1VM OFF 4K; 70% Read; 100% random 1VM ON 4K; 100% Read; 100% random 1VM OFF 4K; 100% Read; 100% random 1VM ON 8K; 70% Read; 100% random 1VM OFF 8K; 70% Read; 100% random 1VM ON 8K; 100% Read; 100% random 1VM OFF 8K; 100% Read; 100% random 1VM ON 64K; 70% Read; 100% random 1VM OFF 64K; 70% Read; 100% random 1VM ON 64K; 100% Read; 100% random 1VM OFF 64K; 100% Read; 100% random Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 69
Performance Chart: Average Read Response Time The chart below shows the average read response times for each of the VM test workloads. 50 Average Read Response Time - ms 45 40 35 30 25 20 15 10 5 0 1VM ON 4K; 70% Read; 100% random 1VM OFF 4K; 70% Read; 100% random 1VM ON 4K; 100% Read; 100% random 1VM OFF 4K; 100% Read; 100% random 1VM ON 8K; 70% Read; 100% random 1VM OFF 8K; 70% Read; 100% random 1VM ON 8K; 100% Read; 100% random 1VM OFF 8K; 100% Read; 100% random 1VM ON 64K; 70% Read; 100% random 1VM OFF 64K; 70% Read; 100% random 1VM ON 64K; 100% Read; 100% random 1VM OFF 64K; 100% Read; 100% random 70 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
TPC-E Test Results Comparison Summary Test Run with Cache ON performed 268% better than Test Run with Cache OFF, comparing the Avg. Response times of two runs. Test Name Run ID Workload Summary Database Under Test TPC-E Transaction Mix Cache ON Mix test with 10 transactions and userload(s) of 4,8,10. Connected to Microsoft SQL Server database using ODBC DSN TPC-E as administrator TPC-E Transaction Mix Cache OFF Mix test with 10 transactions and userload(s) of 4,8,10. Connected to Microsoft SQL Server database using ODBC DSN TPC-E as administrator User Load Run TPS Avg. Response Time (sec) Total Executions Total Rows Total Errors 4 Cache OFF 0.018 164.546 22 2392 0 4 Cache ON 0.047 55.825 19 1250 0 8 Cache OFF 0.166 16.456 46 3046 0 8 Cache ON 0.420 4.798 96 8047 0 10 Cache OFF 0.017 63.530 46 3051 0 10 Cache ON 0.592 5.873 115 7115 0 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 71
Response Time (ms) Transaction / Second TPS vs. Userload 0.7 0.6 0.5 0.4 0.3 0.2 Cache OFF Cache ON 0.1 0 4 8 10 User Load Response Time vs. Userload 180 160 140 120 100 80 60 Cache OFF Cache ON 40 20 0 4 8 10 Userload 72 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
SQLIO Results The test results for SQLIO are all based on a 100%-read workload. The table below shows SQLIO results for one to four VM nodes, with caching alternating OFF and then ON. Block sizes are 4KB, 8KB, and 64KB. Name IOPS Throughput MB/s Latency ms 1node_OFF_4KB 13382.12 52.27 37 1node_OFF_8KB 14770.63 115.39 34 1node_OFF_64KB 5636.17 352.26 90 1node_ON_4KB 110961.98 433.44 4 1node_ON_8KB 99021.35 773.6 4 1node_ON_64KB 15985.21 999.07 31 2node_OFF_4KB 21079.38 82.33 47.5 2node_OFF_8KB 15541.91 121.41 65 2node_OFF_64KB 6452.4 403.27 157.5 2node_ON_4KB 113193.58 442.15 8 2node_ON_8KB 102850.65 803.51 9 2node_ON_64KB 15136.33 946.01 67.5 3node_OFF_4KB 23879.17 93.26 63.33333333 3node_OFF_8KB 20634.03 161.19 73.33333333 3node_OFF_64KB 3111.35 194.44 492.3333333 3node_ON_4KB 114656.53 447.86 12.33333333 3node_ON_8KB 103715.31 810.26 14 3node_ON_64KB 14856.89 928.55 103.3333333 4node_OFF_4KB 14245.47 55.62 142.75 4node_OFF_8KB 11937.02 93.23 170.5 4node_OFF_64KB 2924.5 182.76 697.75 4node_ON_4KB 116818.61 456.31 16.5 4node_ON_8KB 109428.4 854.89 17.5 4node_ON_64KB 15075.07 942.17 136.5 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 73
1node_OFF_4 1node_OFF_8 1node_OFF_6 1node_ON_4KB 1node_ON_8KB 1node_ON_64 2node_OFF_4 2node_OFF_8 2node_OFF_6 2node_ON_4KB 2node_ON_8KB 2node_ON_64 3node_OFF_4 3node_OFF_8 3node_OFF_6 3node_ON_4KB 3node_ON_8KB 3node_ON_64 4node_OFF_4 4node_OFF_8 4node_OFF_6 4node_ON_4KB 4node_ON_8KB 4node_ON_64 1node_OFF_4KB 1node_OFF_8KB 1node_OFF_64 1node_ON_4KB 1node_ON_8KB 1node_ON_64KB 2node_OFF_4KB 2node_OFF_8KB 2node_OFF_64 2node_ON_4KB 2node_ON_8KB 2node_ON_64KB 3node_OFF_4KB 3node_OFF_8KB 3node_OFF_64 3node_ON_4KB 3node_ON_8KB 3node_ON_64KB 4node_OFF_4KB 4node_OFF_8KB 4node_OFF_64 4node_ON_4KB 4node_ON_8KB 4node_ON_64KB The charts below show a graphical view of the IOPS, throughput, and latency results for the SQLIO test runs. IOPS 140000 120000 100000 80000 60000 40000 20000 0 1200 1000 800 600 400 200 0 Throughput MB/s 74 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
1node_OFF_4KB 1node_OFF_8KB 1node_OFF_64KB 1node_ON_4KB 1node_ON_8KB 1node_ON_64KB 2node_OFF_4KB 2node_OFF_8KB 2node_OFF_64KB 2node_ON_4KB 2node_ON_8KB 2node_ON_64KB 3node_OFF_4KB 3node_OFF_8KB 3node_OFF_64KB 3node_ON_4KB 3node_ON_8KB 3node_ON_64KB 4node_OFF_4KB 4node_OFF_8KB 4node_OFF_64KB 4node_ON_4KB 4node_ON_8KB 4node_ON_64KB Latency ms 800 700 600 500 400 300 200 100 0 ESXTOP esxtop is a VMware ESXi monitoring tool you can use to see how well VMs are performing. 1. To access esxtop, SSH into the host and run esxtop from the command line. 2. Enter the v command (VM disk) to display performance statistics for each VM associated with the host. Sample output is displayed below. Performance categories include commands/sec, read IOPS, write IOPS, MB/s read, MB/s written, read latency (ms), and write latency (ms). Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 75
Best Practices ioturbine MS SQL Best Practices Available CPU ioturbine improves the performance of a system by improving the I/O horsepower at the guest VM level. This results in an increased usage of an under-utilized CPU. However, if the server is already constrained for CPU resources (say, the average CPU utilization is already above 90%), then performance gains will most likely not be optimal as the bottleneck will shift from I/Os to the processor. It is advisable that the current average CPU usage be not more than 60% for optimal performance gains. Adding vcpus may be required for some workloads. Available Memory MS SQL Server uses the Buffer Cache to store most recently queried data. So if the memory available to MS SQL Server is more than the total database size, or if the memory is more than the size of the frequently accessed data-set, then the performance gains with ioturbine caching will be marginal. This is because most of the MS SQL Server query requests will be served from the Buffer Cache (which resides in RAM). It is advisable that the RAM available to MS SQL Server be no more than 25% of the total database size. Database Components to Cache In the current version, ioturbine caches the read requests and passes the write requests through. So to effectively make use of ioturbine, ensure that the data files (*.mdf) are on separate logical volume from the transaction log (*.ldf) files. The data files are more read-heavy, whereas the log files are more write-heavy. One exception is tempdb. Because tempdb is frequently read from, it is a strong candidate for placement on volumes that are cached with ioturbine. Measuring Success If possible, measure the average duration of a SQL statement in pre-ioturbine and post-ioturbine scenarios. This can be done using the SQL profiler. This metric shows a much clearer improvement for IOPS, transactions per second, etc. Additional Recommendations Use volume caching with SQL Server; file caching is not fully functional with SQL. Cache the (*.mdf) files (data files). Don t cache the (*.ldf) files (log files). 76 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
References ioturbine_2.0.1.0_administrator s Guide.pdf ioturbine_2.0.1.0_installation_guide.pdf ioturbine_2.0.1.0_release_notes.pdf To access ioturbine documents, visit http://support.fusionio.com VMware Image Builder: http://pubs.vmware.com/vsphere- 50/index.jsp?topic=%2Fcom.vmware.vsphere.install.doc_50%2FGUID- C84C5113-3111-4A27-9096-D61EED29EF45.html Cisco UCS configuration guides: http://www.cisco.com/en/us/products/ps10281/products_installation_and _configuration_guides_list.html APPENDIX Solution Validation Test Setup and Configuration IOMeter Tests TestBatch: 1 SQL_Test: Arc1_SQL_Virtualized_Cache 1. Capture 5-minute output for each Iometer run per VM: TEST#_VMs#_CacheOFF_%hostname% 2. Capture ESXTOP data for the host: esxtop -b -a -d 2 -n 150 > TEST#_VMs#_CacheOFF.cvs Iterations: Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 77
TEST1_VMs1_CacheOFF TEST2_VMs2_CacheOFF TEST3_VMs3_CacheOFF TEST4_VMs4_CacheOFF TEST9_VMs1_CacheON TEST10_VMs2_CacheON TEST11_VMs3_CacheON TEST12_VMs4_CacheON Cache OFF testing for Mspfiov01: 1. Launch the ioturbine Shell. 2. Disable caching by running this command: # ioturbine: provision -stopvolumecache 3. Verify that caching is not started by running this command: # ioturbine: list -volumestatus 4. Ensure that the Iometer test file has been deleted. 5. Connect to the ESX console and prep the esxtop tool: # esxtop -b -a -d 2 -n 150 > TEST1_VMs1_CacheOFF.cvs 6. Load Iometer and launch SQL_Test from the following URL: \\TestShare\TestResults\SQL_Test 7. Launch Iometer and save the file to this URL: \\TestShare\TestResults\TEST1_VMs1_CacheOFF_%computername%.csv 8. As soon as drive is finished prepping, execute the esxtop command. It will run for 5minutes (the same duration as for the Iometer job). # esxtop -b -a -d 2 -n 150 > TEST1_VMs1_CacheOFF.cvs Cache OFF testing for Mspfiov02: 1. Launch the ioturbine Shell. 78 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
2. Disable the cache by running this command: # ioturbine: provision stopvolumecache 3. Verify that caching has not started, by running this command: #ioturbine: list -volumestatus 4. Ensure that the Iometer test file has been deleted. 5. Connect to the ESX console and prep the esxtop tool: # esxtop -b -a -d 2 -n 150 > TEST1_VMs1_CacheOFF.cvs 6. Load Iometer and launch SQL_Test from the following URL: \\TestShare\TestResults\SQL_Test 7. Launch Iometer and save the file to this URL: \\TestShare\TestResults\TEST1_VMs2_CacheOFF_%computername%.csv 8. As soon as drive is finished prepping, execute the esxtop command. It will run for 5minutes (the same duration as for the Iometer job). 9. Start job again on mspfio01: \\TestShare\TestResults\TEST1_VMs2_CacheOFF_%computername%.csv ioturbine Test: Test 1 2 VMs, CACHE OFF esxtop commands: In this example we are gathering esxtop in batch mode (-b) gathering all stats (-a), in 2sec intervals (-d 2) for 300 seconds (-n 150). Saving the file in.csv format will allow you to analyze the data in esxplot. http://labs.vmware.com/flings/esxplot ~ # vim-cmd vmsvc/getallvms Vmid Name File Guest OS Version Annotation 10 mspfiov02 [nfs01] mspfiov02/mspfiov02.vmx windows7server64guest vmx-08 13 mspfiov01 [nfs01] mspfiov01/mspfiov01.vmx windows7server64guest vmx-08 15 mspfiov03 [nfs01] mspfiov03/mspfiov03.vmx windows7server64guest vmx-08 8 mspfiov04 [nfs01] mspfiov04/mspfiov04.vmx windows7server64guest vmx-08 ~ # esxtop -b -a -d 2 -n 150 > /vmfs/volumes/iso\:01b\:iso/fusionio/testresults/test2_vms4_cacheon.cvs Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 79
~ # esxtop -b -a -d 2 -n 150 > /vmfs/volumes/iso\:01b\:iso/fusionio/testresults/test2_vms3_cacheon.cvs ~ # esxtop -b -a -d 2 -n 150 > /vmfs/volumes/iso\:01b\:iso/fusionio/testresults/test2_vms2_cacheon.cvs ~ # esxtop -b -a -d 2 -n 150 > /vmfs/volumes/iso\:01b\:iso/fusionio/testresults/test2_vms1_cacheon.cvs ~ # esxtop -b -a -d 2 -n 150 > /vmfs/volumes/iso\:01b\:iso/fusionio/testresults/test2_vms1_cacheoff.cvs ~ # esxtop -b -a -d 2 -n 150 > /vmfs/volumes/iso\:01b\:iso/fusionio/testresults/test2_vms2_cacheoff.cvs ~ # esxtop -b -a -d 2 -n 150 > /vmfs/volumes/iso\:01b\:iso/fusionio/testresults/test2_vms3_cacheoff.cvs ~ # esxtop -b -a -d 2 -n 150 > /vmfs/volumes/iso\:01b\:iso/fusionio/testresults/test2_vms4_cacheoff.cvs ~ # Installing, Configuring, and Uninstalling ioturbine via the CLI Setup/Prerequisites To install, configure, or uninstall ioturbine with the Command-Line Interface (CLI), make sure the following things are set up: Vcenter Server ESXi 5.0 Host with an IOTurbine-supported guest OS platform An ioturbine Linux MS VM appliance, which should be in a different ESX host Procedure The steps below help you install, configure, and uninstall ioturbine using the CLI. 1. Establish an MS connection to vcenter: # iot vmp --register --vmpaddress <vcenter address> --vmpuser <vcenter user> --vmppassword <vcenter password> --vmplogin If Step 1 fails with a vcenter already registered message, log in directly using the following command: # iot vmp --login --vmpaddress <vcenter address> --vmpuser <vcenter user> --vmppassword <vcenter password> 2. To install host bits, power off all the VMs and put the host in maintenance mode, using the Vsphere CLI or ESXi vim-cmds. 80 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
# iot package -y --install --vmhost < host address > --version <version number> 3. List all the LUNS present in the host. # iot list --vmhost < host address > --listluns 4. Copy the license from/var/lib/vme2/licenses to the MS, using the <vcenter user> user. # lmreread all 5. Restart the vme2 service. 6. Assign the caching device to the host from the info got from Step 3. # iot provision -y --vmhost < host address > --assigndevice <SSD CanonicalName > 7. Take the host out of maintenance mode. 8. Power on the guest VM using the vsphere CLI or ESXi vim-cmd commands. 9. When the VMs are up and the vcenter password tools are in a running state, install the guest package. # iot package -y --install --vmguest < guest address > -- guestuser <guest user> --guestpassword <guest password> --version <version number> 10. Set or assign caching shares to the guest VM. Caching shares are automatically set in the guest install step above, but if specific caching shares are needed, use this command: # iot provision -y --vmguest < guest address > --guestuser <guest user> --guestpassword <guest password> --assigncachingshares <caching share value> For Linux VMs there is no need set filter level shares, as there is only one filter level. For Windows VM, use the command below to set filter-level shares for file, volume, and disk. # iot provision --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --setallshares <file shares> <vol shares> <disk shares> 11. Determine all the volumes present on the guest VM. # iot list --vmguest <guest address> --guestuser <guest user> -- guestpassword <guest password> --allvolumes Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 81
12. Add the above volume for caching. # iot provision -y --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --addvolume <volume> 13. Similarly, determine all the disks present on the guest VM. # iot list --vmguest <guest address> --guestuser <guest user> -- guestpassword <guest password> --alldisks 14. Add the above disk for caching. # iot provision -y --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --adddisk <disk> 15. For Windows VMs, determine all files for caching. # iot list --vmguest <guest address> --guestuser <guest user> -- guestpassword <guest password> --allfiles 16. Add the above file or any other file to the VM guest. # iot provision -y--vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --addfile <file name> 17. Start all three types of caching on each VM. For Linux, volume and disk are available caching methods, but file is not. # iot provision --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --startvolumecache # iot provision --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --startdiskcache # iot provision --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --startfilecache 18. Start I/O on the caching volume/disk and file using any I/O tool for each VM. IOmeter/IOZone/FIO 19. Check the status of each filter. # iot list --vmguest <guest address> --guestuser <guest user> -- guestpassword <guest password> --diskstatus # iot list --vmguest <guest address> --guestuser <guest user> -- guestpassword <guest password> --filestatus # iot list --vmguest <guest address> --guestuser <guest user> -- guestpassword <guest password> --volumestatus 82 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
20. Stop caching on all the filters. File/Volume/Disk for each VM. For Linux only, Disk/Volume as File is not supported. # iot provision --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --stopvolumecache # iot provision --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --stopdiskcache # iot provision --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --stopfilecache 21. Delete all the filters (caching volume/disk and file for each VM). For Linux only, Disk/volume as File is not supported. # iot provision --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --deletevolume <volume name> # iot provision --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --deletedisk <disk name> # iot provision --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --deletefile <file name> 22. Un-assign caching shares that were assigned in Step 7 for each guest VM. # iot provision -y --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --unassigncachingshares 23. Check the File/Disk/Volume status for each VM, as was done in step 19 above. 24. Uninstall the guest package for each VM. # iot package -y --uninstall --vmguest < guest address > -- guestuser <guest user> --guestpassword <guest password> 25. List and un-assign the SSD assigned in Step 5. # iot list --vmhost < host address > --listssds # iot provision -y --vmhost < host address > --unassigndevice <SSD CanonicalName > 26. Uninstall the host bits. # iot package -y --uninstall --vmhost <host name > 27. Unregister the vcenter. Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 83
# iot vmp -unregister -vmpaddress <vcenter address> --vmpuser <vcenter user> --vmppassword <vcenter password> Troubleshooting Commands There are several troubleshooting commands that can be used in error scenarios. These are described below, along with their corresponding options. For more information and additional options, use the corresponding Help commands (# iot <command> --help). Support The support command generates guest, host, and MS support logs (SUB) at any point in time. It can also enable or disable ASUB. Examples: # iot support --ms # iot support --vmguest <guest name> --guestuser <user name> --guestpassword <guest password> # iot support --vmhost <host name> # iot support statusasub # iot support disableasub/--enableasub System The system command is used to upgrade, get the hostname, set the time, execute the db command, etc. Examples: # iot system gethostname # iot system upgrade <url file> # iot system doctor <vcenter/host> # iot system gettimezone # iot system settimezone <TZ/Location> # iot system db <cmd> Account All the above commands on the VM guest can be done by creating, saving, and mapping an account to the respective guest. So instead of using the guest 84 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster
credentials, you can use an account that is mapped to the respective guest credentials. Here are two commands that accomplish this: # iot account --save <account name> --guestuser <guest user> -- guestpassword <guest password> # iot provision -y --vmguest <guest address> --account <account name> --unassigncachingshares In the second command, an account name is used instead of guest credentials, as the account is mapped to the same guest credentials in the first command. Similarly, you can replace the guest credentials with an account previously saved in every guestrelated command. You can also delete or list the accounts present or already mapped. Below are the commands to do this: # iot account --delete <account name> # iot account --list List The list command displays the values for particular variables at any point in time. Examples: # iot list --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --diskstatus # iot list --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --filestatus # iot list --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --volumestatus # iot list license # iot list --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --allvolumes # iot list --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --alldisks # iot list --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --allshares # iot list vmp # iot list --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --configureddisks Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster 85
# iot list --vmguest <guest address> --guestuser <guest user> --guestpassword <guest password> --configuredfiles 86 Cisco UCS Fusion-io MS SQL in a Virtual Performance Cluster