Using Iometer to Show Acceleration Benefits for VMware vsphere 5.5 with FlashSoft Software 3.7 WHITE PAPER Western Digital Technologies, Inc. 951 SanDisk Drive, Milpitas, CA 95035 www.sandisk.com
Table of Contents Executive Summary...3 Audience...3 FlashSoft and Production Workloads...3 Notes about Baseline Testing and Reporting...3 Tools for Generating Simulated Test Workloads...3 Flash Device Discovery and Configuration...4 Installing FlashSoft...4 Installing Iometer...4 Testing Procedure Guidelines...5 Example: Configuring Iometer to Generate Maximum Read IOPS...5 Editing the Disk Targets Tab...6 Using the Access Specifications Tab...7 Editing the Access Specification Dialog...8 Editing the Test Setup Tab...9 Editing the Results Display Tab... 10 Figure 5 Results Display tab... 10 Running the Iometer Test without FlashSoft Acceleration... 10 Starting FlashSoft Caching and Re-running the Iometer Test... 10 Creating the Baseline for the Iometer Test Environment... 11 Starting I/O Acceleration and Warming the Cache... 11 Additional Testing Suggestions... 12 Viewing FlashSoft Host Performance... 12 Viewing FlashSoft VM Performance... 13 Tools for Monitoring Performance... 14 Viewing Performance Results in Windows Perfmon... 14 Conclusion... 15 References... 15 2
Executive Summary This paper describes how to use the Iometer workload tool to generate a read-intensive I/O workload for demonstrating the capabilities of the FlashSoft software, version 3.7 in a Windows Virtual Machine with VMware vsphere 5.5. Audience This white paper is intended for VMware vsphere or Storage Administrators who want to demonstrate the benefits of I/O acceleration with FlashSoft software using the Iometer I/O workload tool. FlashSoft and Production Workloads The FlashSoft software from SanDisk can be used to increase the read performance of a production workload, such as a database application, running in a virtual machine (VM). It is not always practical or possible to test FlashSoft with actual production workloads to demonstrate the benefits of caching shared storage. For a variety of reasons, simulating a production workload in a test environment is a good method to achieve those results. Simulated workloads can quickly demonstrate the benefits of FlashSoft and immediately show that the solution is working, before applying it to a production workload. Notes About Baseline Testing and Reporting Baseline tests form a very important part of performance testing, as they create the starting point for comparison. Baseline testing shows the solution performance before changing the environment. When baseline tests use a simulated workload, there is greater control and inherent repeatability. Repeatable baseline performance tests ensure that the performance results are accurate and valid before any of the variables are changed. It is important that the baseline performance test is conducted on the same hardware and software configuration that will be used for the FlashSoft performance test. This allows accurate results to be recorded before and after caching is enabled, for any of the I/O workloads under test. It s recommended that baseline tests be run for 30 minutes or more so the sample time is long enough to gather realistic, steady-state statistics. As with all performance testing, proper monitoring should be done while the baseline tests execute, and results should be reviewed after the tests complete. Once a baseline has been established and the results have been recorded, the next step is to test the same workloads on virtual machines running on a host that is accelerated through use of the FlashSoft software to compare the performance with and without caching. If there is sufficient time, consider using more than one test tool in order to get useful steady-state averages. It s generally not a good idea to rely on just one tool to provide accurate performance measurements, due to differences and overall test results. Tools for Generating Simulated Test Workloads For testing performance, it is suggested that you use FlashSoft with default settings. If the performance is not satisfactory, then make configuration changes. For those environments where production applications cannot be loaded onto test systems, the following applications can be used to simulate a production workload: Iometer, the focus of this paper, is a GUI tool that generates a workload on the I/O subsystem. It is used as a general workload generation benchmark as well as a troubleshooting tool. Instructions for downloading and using the product are available at http://www.iometer.org 3
Diskspd is a workload tool from Microsoft that combines robust and granular I/O workload definition with flexible runtime and output options, creating a tool for synthetic storage subsystem testing and validation. As of December 2015, DiskSpd.exe supersedes the SQLIO test tool. For more information, see http://aka.ms/diskspd SQLIOSim is a tool that determines the I/O capacity of a hardware configuration prior to deployment of SQL Server. For more information on how to use and configure this tool, see http://support.microsoft.com/kb/231619 Jetstress is used to verify the performance of a disk subsystem by simulating an Exchange disk I/O workload. Specifically, Jetstress simulates the Exchange database and log file workloads. Jetstress is available for download at http://www.microsoft.com/download/en/details.aspx?id=4167 fio stands for flexible I/O tester. It is a freeware command-line tool that can generate a variety of artificial workloads. It works on many of the common OS platforms, has many options, and is designed to be scriptable. For more information and to download see http://freecode.com/projects/fio Flash Device Discovery and Configuration If you are using Fusion iomemory flash, install the VSL Software for ESXi first. The VSL driver presents a raw device to the ESXi host for use by FlashSoft as the cache device. After installing the Fusion iomemory device into the appropriate PCIe slot, install the VSL driver with esxcli software vib install with the ESXi host command line tool as shown in the example output below. # esxcli software vib install -v /vmfs/volumes/bcdemo1-local3/fusion-io_bootbank_block-i omemory-vsl_3.11.5.126-1oem.500.0.0.472560.vib The above is a brief description of the driver installation process, and there may be other required steps depending on environmental factors. See the VSL Software Guide for ESXi for detailed installation instructions. SanDisk SAS and SATA SSDs are discovered automatically by ESXi, so no additional driver installation is required. Once the local flash devices are discovered in ESXi, they are ready to use for caching. Flash devices from other vendors, including NVMe PCIe and AHCI PCIe devices may also be used to evaluate FlashSoft performance. Installing FlashSoft To perform testing with VMware vsphere 5.5, install FlashSoft version 3.7 for VMware vsphere. The FlashSoft software has two components that require installation: a client plugin that is installed on the system with the vsphere client, and an ESXi host driver. For performance testing, it s recommended that you install FlashSoft with default settings. This will cache and accelerate all VMware virtual disk files (VMDKs) that are Windows volumes for the VM. See the FlashSoft Administrator Guide for installation details. Installing Iometer The Windows version of the Iometer tool is available for download from http://www.iometer.org. In addition to the installer and prebuilt binaries, the site also includes documentation on using the tool. Download the Iometer product to your Windows VM and double-click the package to install. 4
Testing Procedure Guidelines The following example demonstrates how to configure Iometer and FlashSoft to show the maximum number of read I/Os per second (IOPS). With Windows Server, it s best to avoid using the boot volume (typically C:) as the volume under test. Instead, use a data volume such as J:. For best results, configure the VM with four vcpus. This will allow each of the four I/O threads in the example to own a vcpu. After the environment is setup, the first step should be to create a baseline for the environment, by running the Iometer I/O workload tool on worker threads to get a baseline result on a non-accelerated VM. After repeatable baseline tests are completed, use the FlashSoft software to begin acceleration and conduct the same target workloads to observe the benefits. The example will walk you through these steps. Example: Configuring Iometer to Generate Maximum Read IOPS Demonstrating maximum read IOPS with the Iometer tool is a matter of correctly configuring the Iometer tool. This is done by setting parameters in three Iometer tabs; Disk Targets, Access Specification, and Results Display. The following sections provide details on configuring the Iometer tool. 5
Editing the Disk Targets Tab Set the following values in the Disk Targets tab. 1. The Topology box displays Workers. Reduce the number of Workers to the number of vcpus configured in the VM. 2. To determine the number of vcpus, open Windows Task Manager and click the Performance tab. In the example there is one vcpu, so one worker is configured. 3. Set the number of Outstanding I/Os to 32. 4. Set Maximum Disk Size to 2,048,000 sectors to create a 1GB target file for the test. 5. Select one Disk Target per worker. In the example, the J:\ volume is selected as the target. Repeat and select the same target volume for each worker. Figure 1 Disk Targets tab in Iometer 6
Using the Access Specifications Tab Create a new Global Access Specification called 100% Read 100% Random. This will open the Edit Access Specification dialog. The values for this Access Specification are detailed in the next section. Figure 2 Access Specifications Tab in Iometer 7
Editing the Access Specification Dialog Set the following values in the Edit Access Specification tab: 1. Set Transfer Request Size to 4KB. 2. Set Percent of Access Specification to 100%. 3. Set Percent Read/Write Distribution to 100% Read. 4. Set Percent Random/Sequential Distribution to 100% Random. 5. Keep the Burstiness Transfer Delay to 0 ms and keep the Burst Length at 1 I/O. 6. Set Align I/Os on to 4KB. 7. Keep Reply Size to No Reply. Figure 3 Access Specifications tab 8
Editing the Test Setup Tab Leave the default values in the Test Setup tab. Figure 4 Test Setup tab 9
Editing the Results Display Tab Set the following values in the Results Display tab to view real time statistics during the test. 1. Set Update Frequency to 2 or 3 seconds. 2. Set Results Since to Last Update. Figure 5 Results Display tab Running the Iometer Test Without FlashSoft Acceleration Perform the first test runs with the FlashSoft acceleration turned off in order to create baseline test results for your system. FlashSoft acceleration is started and stopped in the FlashSoft Configuration Tab located in the vcenter client. If you have already enabled the cache for acceleration, return to the Configuration Tab and click Stop Acceleration. See Figure 6 below for details. Start the Iometer test by clicking the Green Flag icon in the top of the results tab screen in the Iometer GUI. Before the tool generates a workload, the tool will first build a target file on the drive letter you selected in the Disk Targets tab. When this task has completed, the worker(s) will start and run the workload against the target file. Starting FlashSoft Caching and Re-running the Iometer Test Start FlashSoft Volume caching for the VM and re-run the Iometer test. This will demonstrate the increased performance that the FlashSoft caching product provides. Notice that the cache takes time to warm up, so your 10
test results will increase over time. Be sure to run the test for 30 minutes or even longer. We have observed caching performance results that continue to increase even after several hours of testing. Creating the Baseline for the Iometer Test Environment To create baseline test results for your system, you need to perform the first test runs with the FlashSoft cache stopped. 1. Go to the FlashSoft tab in the vsphere client to start or stop caching. 2. If you have already enabled the cache, return to the Configuration Tab in vcenter and click Stop Acceleration (see the diagram below). Figure 6 FlashSoft Tab in vsphere C# Client 3. For the maximum number of IOPS with caching turned off, use the settings as discussed in Example: Configuring Iometer to Generate Maximum Read IOPS. Starting I/O Acceleration and Warming the Cache 1. Start FlashSoft acceleration. By default, all volumes in the Windows VM will be accelerated. The J: volume will be accelerated automatically. 2. Re-run the Iometer test above. This will demonstrate the increased performance that the FlashSoft software provides. Note that the cache takes time to warm up, so your test results will increase over time. 3. FlashSoft acceleration is started and stopped in the FlashSoft Configuration tab in the vsphere GUI. With Iometer, it typically takes 30 to 40 minutes of I/O runtime to warm up the cache. 4. For the maximum number of IOPS, use the following Iometer configuration settings as discussed in Configure Iometer to Generate Maximum Read IOPS. 11
5. Run the I/O load for at least 30 minutes to ensure the cache is warm. While the I/O is running, FlashSoft will collect counter data statistics. 6. While the workload is running, use the FlashSoft vcenter plugin to watch the real time statistics and see the cache warm up in real time. For this workload, the cache is warmed up once the hit ratio is 95% or so. Ideally, the cache hit ratio will be near 100%. Additional Testing Suggestions Here are some suggestions for additional testing. All these variations can be done easily using the Iometer GUI. 1. Add more vcpus to the VM. Increase the number of Outstanding I/Os to match the number of vcpus in the VM you are testing. 2. Increase the Maximum Disk Size of the test file to 10GB or larger. This will create a larger file and thus increase the working-set-size for Iometer to use for testing. 3. Change the %Read/Write ratio to less than 100%. This will add write I/Os and provide a more realistic environment. Typical Read/Write ratios for real-world applications are 70% Read/ 30% Write. Viewing FlashSoft Host Performance The FlashSoft client plugin includes graphs and data that can be used to view cache performance for each ESXi host. Below is a sample screenshot and a brief explanation. Figure 7 FlashSoft host performance in vsphere client Cache Status shows status of the cache, the license status, and the cache mode. Cache Size shows the total size and used space. 12
Cache Utilization shows Read %, Read Hit %, and Read Miss %. The cache is typically warm with Read Hit % > 90%. Cache shows the device ID and the total size. Performance data can also be observed at the ESXi host command line, using the following command: # esxcli fsc cache stats grep metric_name readtotal Total number of read requests for all accelerated VMs on the host writetotal Total number of write requests for all accelerated VMs on the host readhit Number of read requests satisfied from the cache for all accelerated VMs on the host readmiss Number of read requests satisfied from back end storage for all accelerated VMs on the host readhitsec Cumulative size of all requests satisfied from the cache for all accelerated VMs on the host. This value is expressed in the number of 512 byte sectors. For more details, read the Monitoring Performance section in the FlashSoft Administrator Guide. Viewing FlashSoft VM Performance The Utilization section of the screen provides information about the efficiency of FlashSoft acceleration for the selected VM. The statistics display VM I/Os since the VM was last started or accelerated, whichever is latest. It shows cumulative stats for all VDISKs currently associated with the VM (K indicates thousands, M indicates millions). To update the display, click Refresh at the top-right corner of the panel. Figure 8 FlashSoft VM performance in vsphere client FlashSoft performance data can be also obtained per VM by running the following command on the host: esxcli fsc vm vmstats -v vmx_file_path_name grep metric_name The vmx_file_path_name is the full path name of the VM configuration file (.vmx) on the host, and the metric_name indicates the following: totalreads Total number of read requests for the accelerated VM totalwrites Total number of write requests for the accelerated VM 13
totalreadhits Number of read requests satisfied from SSD cache for the accelerated VM totalreadmiss Number of read requests satisfied from back end storage for the accelerated VM For more details, read the Monitoring Performance section in the FlashSoft Administrator Guide. Tools for Monitoring Performance There are a number of tools and applications available to monitor performance results, beyond what the performance test tool provides. SanDisk does not support these monitoring tools, so the tester should be wellversed in the tools listed here. Windows Performance Monitor (PerfMon) is a simple yet powerful visualization tool for viewing performance data, both in real time and from log files. This tool is typically included with Windows so there is no need to install it on the VM. With it, you can examine performance data in a graph, histogram, or report. Information for using PerfMon in Windows can found here: http://technet.microsoft.com/en-us/library/cc749249.aspx VMware vcenter Client/Server includes a system for gathering extensive metrics on performance, resource utilization, and basic statistics of all parts of the environment, including virtual machine, ESX Server hosts, resource pools, and host clusters. All measurements may be obtained in real-time or from a historical record. More information on vsphere 5 monitoring and performance can be found at http://pubs.vmware.com/vsphere-50/topic/com.vmware.icbase/pdf/vsphere-esxi-vcenter-server-50- monitoring-performance-guide.pdf VMware ESXi includes esxtop, a tool for gathering detailed statistics at the ESXi command prompt. It is typically accessed via an SSH session. More information can be found at http://kb.vmware.com/selfservice/microsites/search.do?language=en_us&cmd=displaykc&externalid= 1008205 Orion Network Performance Monitor by SolarWinds is powerful performance management software that makes it easy to detect, diagnose, and resolve issues. A trial version is available at http://www.solarwinds.com/products/network-management/network-performance-monitor.aspx Viewing Performance Results in Windows Perfmon Disk Performance Counters can be viewed and graphed in the Windows perfmon.exe tool. Perfmon is highly customizable, with many different categories of counters. For disks, there are three general areas to measure: latency, IOPS and throughput. It is best to open Perfmon in the MMC.exe framework, as this allows you to save the Perfmon configuration data and settings. By itself, Perfmon does not allow you to save configuration data and settings. For details, see Microsoft documentation. A good place to start is the Getting Started Guide at https://technet.microsoft.com/enus/library/dd744567%28v=ws.10%29.aspx?f=255&mspperror=-2147217396 14
Conclusion When used properly, Iometer can be used to demonstrate FlashSoft I/O acceleration benefits. This paper describes the techniques and usage of Iometer workloads to showcase FlashSoft acceleration benefits, using a practical example. References SanDisk FlashSoft: https://www.sandisk.com/flashsoft VMware vsphere: https://www.vmware.com/products/vsphere Iometer Information: http://www.iometer.org/ Iometer Latest release downloads: http://sourceforge.net/projects/iometer/ Specifications are subject to change. 2016 Western Digital Corporation or its affiliates. All rights reserved. SanDisk and the SanDisk logo are trademarks of Western Digital Corporation or its affiliates, registered in the U.S. and other countries. FlashSoft, Fusion iomemory, and VSL are trademarks of Western Digital Corporation or its affiliates. Other brand names mentioned herein are for identification purposes only and may be the trademarks of their respective holder(s). Western Digital Technologies, Inc. is the seller of record and licensee in the Americas of SanDisk products. 5061EN 20160614 Rev. 1 15