Using VMWare VAAI for storage integration with Infortrend EonStor DS G7i Application Note Abstract: This document describes how VMware s vsphere Storage APIs (VAAI) can be integrated and used for accelerating the virtual machine operations and offloading the overhead of virtualization hosts. Copyright 2013 Infortrend Technology, Inc. All rights reserved. Infortrend, ESVA, EonStor, EonNAS and EonPath are trademarks or registered trademarks of Infortrend. All other marks and names mentioned herein may be trademarks of their respective owners. The information contained herein is subject to change without notice. The content provided as is, without express or implied warranties of any kind.
Table of Content Products Covered in This Document... 3 Overview... 4 What is VAAI... 5 VAAI Benefits... 6 VAAI Primitives... 7 Full Copy... 7 Block Zeroing... 8 Hardware-Assisted Locking (ATS; Atomic Test & Set)... 8 How VAAI is Supported in the Storage System... 10 Check VAAI support via the hardware acceleration status... 10 Toggle the options for VAAI primitives... 10 Test Cases for VAAI integrated with EonStor DS G7i... 12 Test environment configuration... 12 Storage capacity configuration... 13 Test case 1 Full Copy... 13 Test case 2 - Block Zeroing... 16 Test case 3 - Hardware-Assisted Locking... 17 Conclusion... 18 Copyright 2013 Infortrend Technology, Inc. All rights reserved. Page 2 / 18
Products Covered in This Document This application note applies to the following product models: EonStor DS G7i Fibre Channel (FC)-host systems For more related information regarding subsystem configuration, please check the User Manual included with the storage system, or visit the official web site of Infortrend for more product solutions: http://www.infortrend.com For further information of the virtualized environment mentioned in this document, you can also contact VMWare via their web site: http://www.vmware.com/ Copyright 2013 Infortrend Technology, Inc. All rights reserved. Page 3 / 18
Overview Administrators always look for ways to improve performance & efficiency and to reduce system overhead & costs -especially virtual infrastructure administrators and storage administrators who are responsible for deploying VMware vsphere on external storage systems. They require the integration between VMware and hardware storage array to reduce the various virtual machine overhead on physical servers from the mission critical applications. Now we have a way to do this in VMware environments, through the VAAI, or VMware vstorage APIs for Array Integration. By integrating with storage system, such as the Infortrend EonStor DS G7i subsystem, it can enable significant improvements in virtual machine operations in VMware vsphere environments. Copyright 2013 Infortrend Technology, Inc. All rights reserved. Page 4 / 18
What is VAAI VAAI is a set of APIs (application programming interface), or primitives, that allow the virtualization hypervisor to offload I/O processing for certain data-related services to VAAI-supported storage systems, such as the Infortrend EonStor DS family storage systems. It enables VMware ESX hosts to convey virtual machine operations to storage hardware at a meta-level instead of at the traditional data level and reduces operational latency and traffic on a Fibre Channel storage network. Some of primitives enable the storage hardware to participate in data block allocation functions for virtual machines. These primitives, also known as hardware offloading functions, are typically implemented in-band from VMware vsphere environment to a storage array. Virtual Machines VMware vsphere VAAI; vstorage APIs for Array Integration Storage Array Figure 1: The topology of VAAI between host and storage array Copyright 2013 Infortrend Technology, Inc. All rights reserved. Page 5 / 18
VAAI Benefits To meet all various requirements from hosts and applications, enhanced functionalities are required by the SCSI layer stack so that SCSI compatible storage can meet the demands of emerging virtualized infrastructures. There are currently a number of issues that can be solved via these enhancements: Speed up the virtual machine clone and migration tasks Reduce the host overhead associated with initialization job result from space allocation in virtual machine creation Solve the virtual machine locking issue by competing for the same storage resources Infortrend storage system works seamlessly with VMware s ESX/ESXi virtualization hypervisor to improve the efficiency of tasks such as Storage vmotion, cloning and provisioning new virtual machines, all while reducing SCSI reservation locks and increasing the storage scalability by adding new related SCSI commands to the standard SCSI command set. And all enhancements mentioned above have related primitives and SCSI commands for storage system integration. Copyright 2013 Infortrend Technology, Inc. All rights reserved. Page 6 / 18
VAAI Primitives The VAAI storage array offload capability supports three primitives: Full Copy, Block Zeroing, and Hardware-Assisted Locking (ATS; Atomic Test & Set): Full Copy The primitive is also called hardware-accelerated copy, or copy offload, and it leverages the standard SCSI XCOPY command. In real case, it always takes lots of time when the user deploys, clones or migrates an existing virtual machine with application data. The primitive lets those processes be entirely executed on storage array and offload the hypervisor server, and resolves the host overhead associated with VMware initiating and managing cloning or migrating operations. It can reduce overall traffic on host and network significantly. The full copy primitive is helpful for the tasks such as virtual machine data cloning, or VMDK file migrating between datastores within a storage system using VMware Storage vmotion function. The storage system will help to reduce the number of disk I/O requests from the host - saving the host-side I/O bandwidth while doing the copy related operations. Virtual Machines Virtual Machines VMware ESX VMware ESX VAA I Source Target Source Target Data Volumes Data Volumes Figure 2: The difference of data copying or migration between without or with VAAI support. It can significant reduce the I/O requests from host to storage array Copyright 2013 Infortrend Technology, Inc. All rights reserved. Page 7 / 18
Block Zeroing The primitive is also called hardware-accelerated zero, or zero offload, and it leverages the standard SCSI WRITE-SAME command. When user tries to create a new virtual machine or new virtual disks, a format called eager zeroed thick could be used for space pre-allocated and pre-zeroed. The zeroed task is to write data with zero to clean and initialize space for the new space allocation. It means it will take much longer time to provision such virtual disks because it s a slow and inefficient process to zero out each disk space. The virtualization server must complete all the zero writes of the entire virtual disk before it reports back that the disk zeroing is complete, and it s especially time-consuming for large disks. With the block zeroing primitive, these zeroing operations also could be offloaded to the storage system without issuing multiple zero writing commands. Virtual Machines Virtual Machines VMware ESX VMware ESX VAAI 0 0 0 0 0 0 0 0 New Data Volume New Data Volume Figure 3: The difference of block zeroing between without or with VAAI support. VAAI helps to reduce lots of zeroing write commands issued by server. Hardware-Assisted Locking (ATS; Atomic Test & Set) If the storage system was used for sharing to multiple virtualization servers, VMware s file system needs to coordinate all the accesses from multiple servers. It allocates part of the storage available to it for data describing virtual machines and their configurations, as well as the virtual disks that Copyright 2013 Infortrend Technology, Inc. All rights reserved. Page 8 / 18
they try to access. The implemented locking mechanism within the datastores that are used to prevent any virtual machine from being run on, or modified by, more than one host at a time. Rather than locking the entire LUN, Hardware-Assisted Locking (ATS) provides a granular LUN locking method and only locks the blocks on the LUN that are allocated to specific VMDK used by a specific virtual machine. The initial implementation of mutual exclusion for updates to these locking structures was built using SCSI RESERVE and RELEASE commands. Under the protection of those locking methods, one host could update its data records on the device without the risk of interference from other hosts attempting to update the same records. Multiple hosts still can access its LUNs concurrently and also reduces the number of commands required to successfully acquire on-disk locks. Host 1 Host 2 Host 1 Host 2 Virtual Virtual Virtual Virtual Machines Machines Machines Machines VMware ESX VMware ESX VMware ESX VMware ESX Accessing Blocked Accessing VAAI Accessing Virtual Disks Virtual Disks Data Volume Data Volume Figure 4: The difference of disk locking between without and with VAAI support. Instead of locking the entire LUN, VAAI supported granular LUN locking for a specific virtual disk used by a specific virtual machine. Copyright 2013 Infortrend Technology, Inc. All rights reserved. Page 9 / 18
How VAAI is supported in the Storage System Check VAAI support via the hardware acceleration status First of all, we can check if the storage system supports the VAAI related features. In the graphic user interface of VMware vsphere client, there is a basic view for VAAI support: Figure 5: Check the VAAI support in VMware vsphere client interface You can check the Configuration tab under the Hardware -> Storage page, in the list of datastores, there is a column present the Hardware Acceleration status of every storage device. This attribute indicates the support status of the VAAI primitives for the device of a specific datastore. There are three possible values that can populate this column as Supported, Not supported, or Unknown. As the screenshot above shows, there are three datastores and one from the local disk, and two partitions from the EonStor DS G7i storage system. Since the EonStor DS subsystem has already integrated with VMware and applied those VAAI primitives, it shows the storage devices support the hardware acceleration capabilities via VAAI. Toggle the options for VAAI primitives For the VMware vsphere software, all VAAI primitives are enabled in the default configuration. Each primitive also had its related switch for user toggling manually. Before starting the test, we can also check those options first and make sure all primitives are already enabled. For Full Copy and Block Zeroing Using the vsphere Client, Full Copy and Block Zeroing features can be disabled or enabled independently by altering the respective settings in the Configuration tab under Software > Copyright 2013 Infortrend Technology, Inc. All rights reserved. Page 10 / 18
Advanced Settings. You can check the option of DataMover, and there are two sub items as DataMover.HardwareAcceleratedMove and DataMover.HardwareAcceleratedInit. HardwareAcceleratedMove capability is equal to the Full Copy primitive and HardwareAcceleratedInit is the Block Zeroing feature. Using 1 as the value of those settings means to enable the feature, otherwise, 0 uses to disable the option if desired. Block Zeroing Full Copy Figure 6: Check the Hardware-Accelerated Full Copy or Block Zeroing options For Hardware-Assisted Locking The setting of Hardware-Assisted Locking option is also located in the Configuration tab under Software > Advanced Settings. You can check the option for VMFS3, and disabled or enabled by changing the value of VMFS3.HarwareAcceleratedLocking item by user s requirement. Hardware-Assisted Locking Figure 7: Check the Hardware-Assisted Locking option in vsphere client Copyright 2013 Infortrend Technology, Inc. All rights reserved. Page 11 / 18
Test Cases for VAAI integrated with EonStor DS G7i Test environment configuration Here we describe some test cases to prove the advantages of integrating the storage array with VMware via VAAI support. First, here is the test environment configuration for the next cases: Management Server vcenter Server + vsphere client Virtualization Hypervisor VMware ESX server Ethernet switch HBA Virtual Machines Fibre Channel switch EonStor DS G7i with VAAI support Storage expansion (JBOD) Figure 8: Test environment configuration We use an EonStor DS G7i storage system with another expansion enclosure as the main storage device to serve the virtualization hypervisor server, which installed the VMware ESX server and hosts several virtual machines. And leverage the Fibre Channel as the host channel connection with a FC SAN topology via Fibre Channel switch. For the hypervisor server management, we prepared another server as the management host and installed the vcenter server to control the ESX server, and also installed the management GUI vsphere client to connect the local vcenter server for all the configuration and test operations. The management network topology leveraged the common Ethernet switch within the local area network (LAN). Copyright 2013 Infortrend Technology, Inc. All rights reserved. Page 12 / 18
Storage capacity configuration The Infortrend EonStor DS G7i series is a high-performance data storage designed for mid-range enterprise usage. With storage virtualization technology, the capacity and computing power of multiple EonStor DS G7i systems can be consolidated into a single storage pool called a Logical Volume. In the test environment, we consolidated all the capacity from two enclosures as one large storage pool for capacity management. Since the VMware hypervisor server needs physical disks for the creation of datastore, where the virtual disks of virtual machines located, we allocated desired space in logical volume, creating an EonStor DS G7i partition as the host LUN and map to virtualization server. This partition serves as the physical storage device in VMware configuration. Virtual Machines VMware ESX Datastores Storage Array Logical Volume Partition Partition Figure 9: Storage configuration for the test environment Test case 1 Full Copy For the Full Copy test, we planned to clone a 150GB virtual machine from one datastore to another first, which resided in different partitions in the Logical Volume of the storage system. Then we can measure the consumption of time, host computing utilization and network traffic, to tell the difference of VAAI and non-vaai supported storage. Copyright 2013 Infortrend Technology, Inc. All rights reserved. Page 13 / 18
With one virtual machine clone operation, we got the result as below: Time (Sec) 18.75% improved VAAI Disable VAAI Enable Figure 10: Relative v time difference between cloning a virtual machine without or with Full Copy primitive supported using the same storage system. It s almost 20% (18.75%) faster with VAAI enhancement. CPU % 91.16% improved VAAI Disable VAAI Enable Figure 11: Relative host (VMware ESX server) CPU utilization difference. The CPU utilization decreased over 90% (91.16%). Copyright 2013 Infortrend Technology, Inc. All rights reserved. Page 14 / 18
Commands/Sec 97.38% improved VAAI Disable VAAI Enable Figure 12: Relative network traffic difference. The network traffic is result from the I/O commands between host and storage system. With VAAI support, the decrease of network traffic was reduced over 97% (97.38%)! Since the EonStor DS G7i system can serve most of mission critical applications in enterprise environment, we can simulate the environment with more virtual machines to see the performance scalability on storage. We came out another test with multiple virtual machine operations simultaneously, for example, to clone four to ten virtual machines at the same time. More virtual machines may bring more significant time saving enhancement by customer s environment. Time (Sec) Time (Sec) 22% 40.24% VAAI Disable VAAI Enable VAAI Disable VAAI Enable Figure 12: With more virtual machine cloning simultaneously, we got more significant time saving via the Full Copy primitive with cloning 4 virtual machines (22%) to 10 virtual machines (40.24%). Copyright 2013 Infortrend Technology, Inc. All rights reserved. Page 15 / 18
Test case 2 - Block Zeroing Block Zeroing provides benefits and performance increases in a variety of use cases. We planned the test to cover the most commonly encountered scenario with deploying fully allocated virtual machine, which was created using the eagerzeroedthick virtual disk format. The format using the allocation mechanism to fully zero out whole the disk at initial creation, and it can take a great deal of time to write these zeros, especially in virtual machines with large virtual disks. We decided to create a 100GB virtual machine resided in specific partition from EonStor DS G7i storage system, with the eagerzeroedthick format, and measure the indexes to see the difference. With one virtual machine creation operation, we got the result as below: Time (Sec) 20.75% improved VAAI Disable VAAI Enable Figure 13: With VAAI Block Zeroing primitive support, the virtual machine creation time can be reduced over 20% (20.75%). Writes / Sec 85.59% improved VAAI Disable VAAI Enable Figure 14: Since the creation job will issue lots of I/O commands to write zero to storage, the difference of write command per second can tell the reason why saving the traffic between host and storage. With the Block Zeroing support, the decrease of write operation (related to network traffic) was 85.59 % reduced. Copyright 2013 Infortrend Technology, Inc. All rights reserved. Page 16 / 18
Test case 3 - Hardware-Assisted Locking To find out the benefit with Hardware-Assisted Locking, we planned to simulate the multiple virtual machine resource competition via lots of virtual machine booting simultaneously. While powering-on lots of virtual machines with a single shared datastore, which was mapping from a partition of EonStor DS G7i system, the number of SCSI locking conflicts should be also increased and slow down the performance of virtual machine deploying. We can see the effect of Hardware-Assisted Locking primitive and how it helps to reduce the bottleneck caused by LUN locking. In this test case, we prepared four ESX servers to deploy 100 virtual machines for testing, and each server hosted 25 virtual machines. Those servers all shared the storage resource from one EonStor DS G7i system. Time (Sec) 10% improved VAAI Disable VAAI Enable Figure 15: Relative 100 virtual machines power-on time difference between non-vaai and VAAI supported storage. With Hardware-Assisted Locking, the power-on time was 10% faster for all virtual machine booting accomplished. Copyright 2013 Infortrend Technology, Inc. All rights reserved. Page 17 / 18
Conclusion Infortrend EonStor DS G7i family storage system, when integrated with VMware s VAAI primitives, can provide customers with the combination to benefit overall system performance and efficiency in virtual environments. These enhanced primitives, including Full Copy, Block Zeroing, and Hardware-Assisted Locking, are enabled by default and will be utilized on EonStor DS G7i systems without any user configuration or intervention. With a redundant and robust EonStor DS G7i system, users can create a reliable and highly available infrastructure to support the high density virtual machine workloads, as reported in this document. As we described, while the use of VAAI has many benefits by offloading functions from the virtualization server to storage system, leveraging VAAI in a virtual environment makes the choice of storage systems even more critical. It is important to choose a highly available, scalable, high performance storage system like the Infortrend EonStor DS G7i. It not only meets the mission-critical storage demands with advanced hardware design & comprehensive data services but with the innovative architecture, various features including storage virtualization, thin provisioning, distributed load balancing, automatic data migration, prioritized volume accessibility, and array-based snapshot and replication, are consolidated to realize optimal business benefits. Making EonStor DS G7i virtualized storage complement VMware s virtual infrastructure, users can enjoy the enhanced benefits of optimized returns of investment, simplified storage infrastructure & management, and maximized storage-related application productivity. Copyright 2013 Infortrend Technology, Inc. All rights reserved. Page 18 / 18