How To Create A Server Virtualization Solution For A Large-Scale Data Center

Transcription

1 SERVER VIRTUALIZATION WITH EMC XTREMIO ALL-FLASH ARRAY AND VMWARE VSPHERE 5.5 EMC Solutions Abstract This white paper highlights the performance and operational advantages of server virtualization based on EMC XtremIO all-flash array technology. This document describes the reference architecture of the EMC Proven Infrastructure for a validated server virtualization solution enabled by the XtremIO all-flash array and VMware vsphere version 5.5.

2 Copyright 2013 EMC Corporation. All rights reserved. Published in the USA. Published October EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. EMC 2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. Part Number H Server Virtualization with EMC XtremIO all-flash Array and VMware vsphere 5.5

3 Contents Contents Chapter 1 Introduction 7 Executive summary... 8 Target audience... 8 Document purpose... 8 Business requirements... 9 Chapter 2 Solution Functionality and Features 11 Introduction Server virtualization Data center demands Performance Workload portability Scalability Virtual machine provisioning Deduplication Thin provisioning Data protection VAAI integration Summary Chapter 3 Solution Technology 18 Overview Key components Virtualization Overview VMware vsphere VMware vsphere HA VMware vcenter EMC XtremIO VAAI support EMC XtremIO storage Cluster design Deduplication capacity savings Thin provisioning Fault protection Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5 3

4 Contents Scalability In-memory metadata operations Chapter 4 Solution Architecture 24 Overview Reference architecture Overview Logical architecture Key components Hardware resources Software resources Storage configuration guidelines Overview XtremIO X-Brick scalability XtremIO server virtualization validated maximums External data considerations Sizing guidelines Overview Reference workload Defining the reference workload Applying the reference workload Overview Example 1: Custom-built application Example 2: Point-of-sale system Example 3: Web server Example 4: Decision-support database Summary Use cases for server virtualization with XtremIO Overview Use Case 1: Software build environments Use Case 2: Virtual classrooms Use Case 3: Service provider for cloud deployment XtremIO test results Overview Capacity savings from deduplication Thin provisioning Steady-state response and array utilization at high scale points VAAI Simple provisioning and monitoring Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5

5 Contents Chapter 5 Conclusion 51 Summary Findings Appendix A References 53 References EMC documentation Other documentation Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5 5

6 Contents Figures Figure 1. I/O randomization brought by server virtualization Figure 2. Figure 3. Storage vmotion is highly dependent on array I/O and cloning performance Array-based virtual machine cloning affects storage I/O performance Figure 4. Server virtualization components Figure 5. Logical architecture Figure 6. XtremIO scalability Figure 7. EMC XtremIO volume configuration and mapping Figure 8. Logical architecture with optional VNX array for external data Figure 9. Resource pool flexibility Figure 10. Capacity savings from deduplication Figure 11. Capacity savings from thin provisioning Figure 12. IOPS and array latency scale results Figure 13. CPU utilization scale test results Figure 14. Average I/O throughput of Storage vmotion migrations with VAAIenabled virtual machines Figure 15. XtremIO I/O throughput during virtual machine migration with Storage vmotion Figure 16. Total and average virtual machine deployment duration Figure 17. XtremIO implementation of the VAAI XCOPY Figure 18. Average IOPS per SSD during virtual machine deployment Figure 19. Performance dashboard of the XtremIO array Tables Table 1. Solution hardware Table 2. Solution software Table 3. Validated profile characteristics Table 4. Validated test metrics for 2,800 virtual machines Table 5. Example of a virtualized Exchange 2013 server Table 6. Virtual machine test configuration Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5

7 Chapter 1: Introduction Chapter 1 Introduction This chapter presents the following topics: Executive summary... 8 Target audience... 8 Document purpose... 8 Business requirements... 9 Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5 7

8 Chapter 1: Introduction Executive summary Target audience Document purpose Server virtualization has been a driving force in data center efficiency gains for the past decade. However, mixing multiple virtual machine workloads on a single physical server creates a randomization of input/output (I/O) for the storage array, stalling virtualization of I/O-intensive workloads. EMC XtremIO all-flash arrays not only cost-effectively address this performance challenge, but they also provide new levels of speed and provisioning agility to virtualized environments. This document is a comprehensive guide to the technical aspects of this server virtualization solution. Server capacity is provided in generic terms for required minimums of CPU, memory, and network interfaces. You can select the server and networking hardware that meets or exceeds the tested minimums. This white paper is intended for EMC employees, partners, and customers, including storage and VMware administrators who want to understand how EMC XtremIO storage and VMware vsphere can provide an easy-to-use, high-performance storage solution for server virtualization. We 1 assume that readers are familiar with the following products: EMC XtremIO and EMC storage systems VMware vsphere and VMware vcenter This document describes the reference architecture of the EMC Infrastructure for server virtualization with the EMC XtremIO all-flash array and VMware vsphere 5.5. The EMC Solutions Group tested and validated this solution. This architecture provides a modern system capable of hosting many virtual machines at a consistent performance level. This solution runs on the VMware vsphere virtualization layer backed by the EMC XtremIO all-flash array. The redundant compute and network components, which are chosen by customers, are sufficiently powerful to handle the processing and data needs of the virtual machine environment. Because not every virtual machine has the same requirements, this white paper contains methods and sizing guidance to adjust the system for cost-effective deployment. 1 In this paper, we represents the EMC Solutions team that tested and validated this solution. 8 Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5

9 Chapter 1: Introduction Business requirements Business applications are moving into consolidated compute, network, and storage environments. EMC and VMware server virtualization reduces the complexity of configuring every component of a traditional deployment model. Virtualization reduces the complexity of integration management while maintaining application design and implementation options. Administration is unified, while process separation is adequately controlled and monitored. This solution addresses the complexity of deploying and managing complex, consolidated environments by providing the following: End-to-end virtualization using the capabilities of the unified infrastructure components A server virtualization solution for VMware that efficiently virtualizes thousands of virtual machines for varied customer use cases for a reliable, flexible, and scalable reference design A simplified management interface for a data center environment Better support of service-level agreements and compliance initiatives Lower operational and maintenance costs Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5 9

10 Chapter 1: Introduction 10 Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5

11 Chapter 2: Solution Functionality and Features Chapter 2 Solution Functionality and Features This chapter presents the following topics: Introduction Server virtualization Data center demands Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere

12 Chapter 2: Solution Functionality and Features Introduction This chapter describes the storage needs and challenges of data centers and how the unique benefits of the XtremIO all-flash array and vsphere address these challenges. Server virtualization Server virtualization solves the problems of underutilized storage assets, overburdened power and cooling resources, and high administrator-to-asset ratios. Virtualized server resources are the foundation for delivering cloud infrastructure services. Consolidating underutilized physical server resources into highly available virtual machines can lead to considerable cost savings and improved data center efficiency. And while virtualization has enhanced the efficiency of resources in the data center, it has also magnified the challenges associated with expanding virtualization beyond specific IT zones (or noncritical applications) across the entire data center. The complexity of virtualized infrastructures, including concerns about storage cost, performance, availability, and agility have caused enterprises to delay the roll out of an end-to-end shared, virtualized IT infrastructure. Large-scale deployment of virtualized environments requires increased density of virtual machines for each physical server, the virtualizing of tier-one, I/O-intensive applications, fast-growing datastores, and rapid virtual machine provisioning. In a dynamic large-scale environment, manual storage administration requires time, slows deployments, and takes resources away from proactive business management. If you factor in the effort required to scale the environment when more capacity or performance is necessary, the unique challenges of a virtualized data center become clear. These environments require new solutions to maintain operational efficiency, performance, and cost targets. Data center demands Performance, application provisioning, and data management requirements were easy to meet when discrete applications used physical servers and dedicated storage systems. However, moving those applications into large-scale, agile VMware virtual environments places new demands on the infrastructure. These environments require high performance and support for a high density of virtualized applications with unpredictable workloads, and rapid virtual machine provisioning and cloning. The promise of flash storage arrays meeting large-scale virtualization requirements looms large. However, the reality is that all-flash arrays must have an optimized architecture for both storage input/output (I/O) performance and storage efficiency to effectively address these challenges. Because acquisition and operational costs of storage infrastructure are among the top challenges of cloud-based virtual server environments, storage efficiency has an important role to play. Storage efficiency is a combination of maximizing both available storage capacity and processing resources, which often are competing efforts. Storage efficiency is key to enabling the promise of elastic scalability, 12 Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5

13 Chapter 2: Solution Functionality and Features pay-as-you-grow efficiency, and a predictable cost structure, all while increasing productivity and innovation. Performance CPUs historically have gained power through increases in transistor count and clock speed. More recently, a shift has been made to multicore CPUs and multithreading. These advances combined with server virtualization technology allow massive consolidation of applications onto a single physical server. The result is intensive randomization of the workload for the storage array. Imagine a dual socket server with six cores per socket and two threads per core. With virtualization technology, this server can easily present shared storage with a workload of 24 unique, intermixed data streams. Now imagine numerous servers on a SAN sharing the same storage array. The array workload very quickly becomes an I/O blender of completely random I/O from hundreds or thousands of intermixed sources, as shown in Figure 1. Flash arrays are ideal for handling high volumes of random I/O that have traditionally been too expensive for large-scale virtualization deployments. Figure 1. I/O randomization brought by server virtualization Workload portability Being able to move active virtual machines as quickly and seamlessly as possible from one physical server to another with no service interruption is a key element of a large-scale virtualized infrastructure. VMware vsphere vmotion enables the live migration of virtual machines from one VMware vsphere host to another, with no perceivable impact for users. vmotion is an important enabler for a number of key VMware technologies, including vsphere Distributed Resource Scheduler (DRS) and vsphere Distributed Power Management (DPM). vmotion requires virtual-machine physical memory (as large as 1 TB) to be transferred during a virtual machine migration while leveraging vsphere suspend and resume functionality. This functionality momentarily quiesces the virtual machine on the source vsphere host, then copies the last set of memory changes to the target vsphere host, and then resumes the virtual machine on the target. The suspend and resume phase is the most likely phase to affect guest performance, during which an Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere

14 Chapter 2: Solution Functionality and Features abrupt, temporary increase of latency can occur. The impact depends on a variety of factors including the performance of the storage I/O. Large-scale virtual environments commonly use VMware Storage vmotion for live, nondisruptive migration of virtual machine files within and across storage arrays for performing proactive storage migrations, improving virtual machine performance, and optimizing storage utilization. You can use the vstorage APIs for Array Integration (VAAI) Extended Copy (XCOPY) to accelerate Storage vmotion with compliant storage arrays, which enable the host to offload specific virtual machine and storage management operations to the storage array. The host issues the XCOPY command to the array from the source LUN to the destination LUN, or to the same source LUN, if required. The choice depends on the configuration of the virtual machine file system (VMFS) datastores on the relevant LUNs. The array uses internal mechanisms to complete the cloning operation and, depending on the efficiency of the array to implement the VAAI XCOPY support, can accelerate the performance of Storage vmotion. Figure 2 shows how array-enabled vmotion and Storage vmotion operations are managed. vmotion vsphere vsphere Array VAAI LUN01 LUN02 Figure 2. Storage vmotion is highly dependent on array I/O and cloning performance Scalability An agile, virtualized infrastructure must also scale in the multiple dimensions of performance, capacity, and operations. It must have the ability to scale efficiently, without sacrificing performance and resiliency, and without scaling the number of people that manage the environment. However, deploying traditional discrete dualcontroller flash appliances to address scalability challenges can lead to system 14 Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5

15 Chapter 2: Solution Functionality and Features sprawl, performance bottlenecks, and suboptimal availability, which increases storage administration time. Virtual machine provisioning The promise of increased agility is a major reason why organizations choose to virtualize their infrastructures. However, IT responsiveness often exponentially slows as virtual environments grow. Resources typically are unable to be provisioned quickly enough to meet rapidly changing business requirements. Bottlenecks occur because organizations do not have the proper tools to quickly determine the capacity and health of the physical and virtual resources. Standard virtual machine provisioning or cloning methods, commonly implemented in flash arrays, can be expensive, because full copies of virtual machines can require 50 GB or more storage for each copy. In a large-scale cloud data center, when shared storage is cloning up to hundreds of virtual machines each hour while concurrently delivering I/O to active virtual machines, cloning can become a major bottleneck for optimal data center performance and operational efficiency, as shown in Figure 3. Figure 3. Array-based virtual machine cloning affects storage I/O performance Deduplication Storage arrays can accumulate duplicate data over time, which increases costs and management overhead. In particular, large-scale virtual server environments create large amounts of duplicate data when virtual machines are deployed by cloning existing virtual machines or have the same operating system and applications installed. Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere

16 Chapter 2: Solution Functionality and Features Traditionally, deduplication eliminates duplicate data by replacing it with a pointer to a unique data block. This post-processing operation writes incoming data to disk and then the array deduplicates the data, both of which affect array performance. The XtremIO all-flash array uses inline deduplication, which eliminates performance hits incurred by traditional deduplication mechanisms. Thin provisioning Thin provisioning is a popular technique that improves array utilization. The storage capacity is consumed only when data is written instead of when storage volumes are provisioned. For administrators of large-scale virtualized environments, thin provisioning removes the need for overprovisioning storage up front to meet anticipated future capacity demands and, instead, allows virtual machine storage to be allocated on-demand from an available storage pool. Most storage arrays are designed to be statically installed and run, yet virtualized application environments are naturally dynamic and variable. Change and growth in virtualized workloads causes organizations to actively redistribute workloads across storage array resources (or use other features such as VMware DRS) for load balancing to avoid running out of space or reducing performance. Unfortunately, this ongoing load balancing is a manual, iterative task that is often costly and timeconsuming. As a result, storage arrays that support large-scale virtualization environments require optimal and inherent data placement to ensure maximum utilization of both capacity and performance without any planning demands. XtremIO s thin provisioning support can result in significant savings by reducing unused storage. Data protection While storage arrays have traditionally supported several RAID data-protection levels, the arrays required storage administrators to choose between data protection and performance for specific workloads. The challenge for large-scale virtual environments is the shared storage system that stores data for hundreds or thousands of virtual machines with different workloads. Some storage systems allow live migrations between RAID levels, requiring repeatedly proactive administration as workloads evolve. Optimal data protection for virtualized environments requires that arrays support data-protection schemes, which combine the best attributes of existing RAID levels while avoiding the drawbacks. Because flash endurance is a special consideration in an all-flash array, the scheme maximizes the service life of the array s solid-state drives (SSDs) while complementing the high I/O performance of flash media. XtremIO provides a robust toolset of data-protection and management methods. Tight integration with VMware extends benefits into the virtualized environment. VAAI integration In contrast to a custom integration of virtualized environments and storage arrays, VAAI is a set of APIs that enables VMware hosts to offload common storage operations to the array. VAAI reduces resource overhead on VMware hosts and can significantly improve performance for storage-intensive operations such as storage cloning for virtual machine provisioning. While VAAI removes the involvement of vsphere hosts in storage-intensive operations, the actual performance benefits of VAAI-enabled flash arrays are highly 16 Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5

17 Chapter 2: Solution Functionality and Features dependent on the array architecture. For example, the performance of VAAI-enabled XCOPY for copying virtual disk files (up to hundreds of GBs) for cloning or storage vmotion is highly dependent on the efficiency of deduplication and metadata models supported by the array. If the XCOPY operation requires read and write of data blocks to and from the SSDs as compared to only creating metadata pointers to deduplicated data blocks on SSDs, the performance can vary widely for both the copy operation and I/O of the live virtual machines. Summary To meet the multiple demands arising from a large-scale virtualization data center, you need a storage array that can provide the following: Superb performance and capacity scale-out for infrastructure growth Built-in data deduplication Thin provisioning for capacity efficiency and cost mitigation Flash-optimized data protection techniques Near-instantaneous virtual machine provisioning and cloning Inherent load balancing Automated virtual machine disk (VMDK) provisioning The EMC XtremIO all-flash array is built to unlock the full performance potential of flash storage and to deliver array-based data management capabilities that make it an optimal storage solution for large-scale virtualization. The next chapter includes more details about how to apply XtremIO features for optimal performance. Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere

18 Chapter 3: Solution Technology Chapter 3 Solution Technology This chapter presents the following topics: Overview Key components Virtualization EMC XtremIO storage Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5

19 Chapter 3: Solution Technology Overview This solution uses the EMC XtremIO all-flash array and VMware vsphere 5.5 to provide storage and server hardware consolidation for a private cloud or server virtualization. The new virtualized infrastructure is centrally managed to provide efficient deployment and management of a scalable number of virtual machines and associated shared storage. Figure 4 depicts the solution components. Figure 4. Server virtualization components Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere

20 Chapter 3: Solution Technology Key components Virtualization Key components of this solution are as follows: Virtualization The virtualization layer decouples the physical implementation of resources from the applications that use them, so the application view of the available resources is not directly tied to the hardware. Compute The compute layer provides memory and processing resources for the virtualization layer software and for the applications running in the virtual servers. This solution defines the minimum amount of required compute layer resources and enables you to implement the solution by using any server hardware that meets these requirements. Network The network layer connects the users of the virtual servers to the resources, and the storage layer to the compute layer. This solution defines the minimum number of required network ports, provides general guidance on network architecture, and enables you to implement the solution by using any network hardware that meets these requirements. Sto rage The storage layer is critical for the implementation of the server virtualization. The EMC XtremIO all-flash array used in this solution provides extremely high performance and supports a number of capacity-efficient and data-service capabilities. Reference architecture on page 25 provides details about the components in the reference architecture. Overview VMware vsphere 5.5 The virtualization layer is a key component of any server virtualization or private cloud solution. It decouples the application resource requirements from the underlying physical resources. This enables greater flexibility in the application layer by eliminating hardware downtime for maintenance and changes to the physical system without affecting the hosted applications. In a server virtualization or private cloud use case, this layer enables multiple independent virtual machines to share the same physical hardware, instead of being directly implemented with dedicated hardware. VMware vsphere transforms the physical resources of a computer by virtualizing the CPU, RAM, hard disk, and network controller. This transformation creates fully functional virtual machines that run isolated and encapsulated operating systems and applications like physical computers. The High Availability (HA) features of VMware vsphere such as vmotion and Storage vmotion enable seamless migration of virtual machines and stored files from one vsphere server to another, or from one data storage area to another, with minimal or no performance impact. Coupled with vsphere DRS and Storage DRS, virtual machines have access to the appropriate resources at any point in time through load balancing of compute and storage resources. 20 Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5

21 Chapter 3: Solution Technology VMware vsphere HA VMware vsphere HA enables the virtual layer to automatically restart virtual machines in various failure conditions. If an operating system error occurs on the virtual machine, the virtual machine automatically restarts on the same hardware. If an error occurs on the physical hardware, the impacted virtual machines can automatically restart on other servers in the cluster. Note: To restart virtual machines on other servers in the cluster, the servers must have available resources. With VMware vsphere HA, you can configure policies to determine which machines automatically restart and under what conditions to try these operations. VMware vcenter VMware vcenter is a centralized management system for a VMware virtual infrastructure. This system provides you with a single interface that you can access from multiple devices for all aspects of monitoring, managing, and maintaining a virtual infrastructure. VMware vcenter also manages some advanced features of a VMware virtual infrastructure, such as VMware vsphere HA and DRS, vmotion, and Update Manager. EMC XtremIO VAAI support XtremIO is fully integrated with VMware vsphere through VAAI for virtual machine provisioning and cloning, VMDK provisioning, and overall seamless deployment of large-scale virtualization. It delivers high-performance, low-latency response times, and low provisioning times, for all storage provisioning choices at the VMDK level. XtremIO supports the VAAI block zero primitive and has a unique way to write zero blocks that removes the performance drawbacks of provisioning eager zero thick (EZT) volumes for virtual disks. EMC XtremIO storage Cluster design XtremIO includes a scale-out cluster design that adds capacity and performance in balance together to meet any storage requirement. Each cluster building block has high-availability, fully active/active storage servers with no single point of failure. As you expand a cluster, XtremIO automatically balances the workloads of all hosts and clusters to maintain performance. The XtremIO operating system (XIOS) manages storage clusters and provides the following functionality: Ensures that all SSDs in the cluster are evenly loaded to provide the highest possible performance and endurance required for high-demand workloads throughout the life of the array. Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere

22 Chapter 3: Solution Technology Eliminates the need to perform complex configuration steps for traditional arrays. It negates the need to set RAID levels, determine drive-group sizes, set stripe widths, set caching policies, build aggregates, or do any other manual configuration. Automatically and optimally configures volumes and ensures that I/O performance on existing volumes and data sets automatically increases when a cluster scales out. Manages the process of expanding clusters and ensures that data remains balanced across new X-Bricks (the fundamental building block of XtremIO clusters). It also ensures that I/O performance of existing volumes and data sets automatically increases when the cluster scales out. It eliminates the need to restripe data if application requirements change. Every volume receives the full performance potential of the entire XtremIO cluster. Deduplication capacity savings The XtremIO all-flash array performs in-line data deduplication based on an algorithm that checks to ensure that duplicate data blocks are not stored on SSDs. The result is that every storage I/O is deduplicated in real-time on ingest with only unique blocks ever being written to the flash storage. Moreover, deduplication on XtremIO aids performance, as all metadata is in memory. The better the data deduplicates, the better XtremIO performs. This ensures the maximum host I/O performance. The XtremIO array also provides deduplication-aware caching where blocks that are held in cache can be served for any logical reference to those blocks. Deduplicationaware caching combined with inline deduplication dramatically decreases latencies for handling challenging situations, such as multiple concurrent virtual machine boots, providing consistent sub-millisecond data access. Thin provisioning Fault protection Along with delivering high performance, the XtremIO array natively offers thin provisioning capabilities to allocate on-demand capacity as applications need it without any impact on array-storage I/O performance. XtremIO thin provisioning is also granular in that capacity is allocated in 4 KB blocks to ensure thrifty use of flash capacity, which is consistent with how VMware vsphere uses I/O block sizes. The EMC XtremIO array delivers the utmost in reliability and availability with completely redundant components and the ability to tolerate any component failure without loss of service. XtremIO includes the following fault protection: Dual power supplies in controllers and disk array enclosures (DAEs) to support loss of a power supply while keeping the controller/dae in service Redundant active/active controllers to support controller failures Redundant Serial Attached SCSI (SAS) interconnect modules in the DAEs Redundant inter-controller communication links Multiple host connections with multipath capabilities to survive path failures XtremIO Data Protection (XDP) to tolerate SSD failures Multiple techniques to ensure initial and ongoing data integrity 22 Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5

23 Chapter 3: Solution Technology Scalability In-memory metadata operations XtremIO storage clusters support a fully distributed, scale-out design that allows linear increases in both capacity and performance for infrastructure agility. XtremIO uses a building-block approach in which the array is scaled with additional X-Bricks. It provides host access using N-way active/active controllers for linear scaling of performance and capacity for simplified support of growing virtualized environments. As a result, as capacity in the array grows, performance grows in lockstep with the addition of storage controllers. The XtremIO cluster distributes metadata evenly across all storage controllers, maintaining metadata in memory during runtime. Metadata is hardened to SSD to allow the array to tolerate failures and power loss, but during normal operations all metadata lookups are memory-based. This is possible only by segmenting the metadata tables and spreading them evenly across all storage controllers. In contrast, a dual-controller design could not contain enough RAM to store all metadata in memory and would require de-staging large amounts of metadata to flash, with several associated performance drawbacks. The XtremIO in-memory metadata and unique in-line data reduction model combine to deliver new, unprecedented capabilities in virtualized data centers. Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere

24 Chapter 4: Solution Architecture Chapter 4 Solution Architecture This chapter presents the following topics: Overview Reference architecture Storage configuration guidelines Sizing guidelines Applying the reference workload Use cases for server virtualization with XtremIO XtremIO test results Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5

25 Chapter 4: Solution Architecture Overview This chapter is a comprehensive guide to the major aspects of this solution. It presents server capacity for the required minimum CPU, memory, and network resources. Users can select the server and networking hardware that meets or exceeds the stated minimums. This EMC Proven Infrastructure validated the specified storage architecture for providing high levels of performance while delivering a highly available architecture for your private cloud deployment. Each Proven Infrastructure balances the storage, network, and compute resources needed for a set number of virtual machines validated by EMC. In practice, each virtual machine has its own set of requirements that rarely fits a predefined idea of a virtual machine. In any discussion about virtual infrastructures, defining a reference workload first is important. Not all servers perform the same tasks, and building a reference that takes into account every possible combination of workload characteristics is impractical. Reference architecture Overview This section provides a summary and characterization of the tests performed to validate the EMC Proven Infrastructure for Server Virtualization enabled by the EMC XtremIO all-flash array and VMware vsphere 5.5. It involves building a 2,800-virtualmachine environment on XtremIO and integrating the new features of this platform to provide a high-performance, compelling, and cost-effective server virtualization platform. The defined configuration forms the basis for creating a custom solution. Note: This solution uses the concept of a reference workload to describe and define a virtual machine. One physical or virtual server in an existing environment might not be equal to one virtual machine in this solution. Evaluate your workload for an appropriate point of scale. Applying the reference workload on page 37 describes the process. Logical architecture Figure 5 shows the logical architecture of the solution and characterizes the validated infrastructure, where an 8 Gb Fibre Channel (FC) or 10 Gb iscsi SAN carries storage traffic and 10 GbE carries management and application traffic. Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere

26 Chapter 4: Solution Architecture Figure 5. Logical architecture Key components This architecture includes the following key components: VMware vsphere Provides a common virtualization layer to host a server environment. vsphere provides the highly available infrastructure through features such as the following: vmotion Provides live migration of virtual machines within a virtual infrastructure cluster, with no virtual machine downtime or service disruption Sto rage vmotion Provides live migration of VMDK files within and across storage arrays with no virtual machine downtime or service disruption vsphere HA Detects and provides rapid recovery for a failed virtual machine in a cluster Distributed Resource Scheduler (DRS) Provides load balancing of computing capacity in a cluster Storage Distributed Resource Scheduler (SDRS) Provides load balancing across multiple datastores based on space usage and I/O latency VMware vcenter Server Provides a scalable and extensible system that forms the foundation for virtualization management for the VMware vsphere cluster. vcenter manages all vsphere hosts and their virtual machines. SQL Server VMware vcenter Server requires a database service to store configuration and monitoring details. This solution uses Microsoft SQL Server 2008 R2. 26 Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5

27 Chapter 4: Solution Architecture DNS server Uses DNS services for name resolution of various solution components. This solution uses the Microsoft DNS Service running on a Windows 2012 server. Active Directory (AD) server Uses AD services for various solution components to function correctly. Microsoft AD runs on Windows Server IP network Performs as a standard Ethernet network with redundant cabling and switching. It is a shared IP network that carries and manages all of the data traffic. Storage network The storage network is isolated to provide host access to the array with the following options: 8 Gb FC Performs high-speed serial data transfer with a set of standard protocols. FC provides a standard data transport frame for servers and shared storage devices. 10 Gb Ethernet (iscsi) Enables the transport of SCSI blocks over a TCP/IP network. iscsi works by encapsulating SCSI commands into TCP packets and sending the packets over the IP network. XtremIO all-flash array The XtremIO all-flash array includes the following components: X-Brick Represents a building block that contains two active storage controllers, as the fundamental scaling unit of the array, and a shelf with 25 enterprise multi-level cell (emlc) SSDs. When the XtremIO cluster scales, the array clusters together multiple X-Bricks with redundant Infiniband back-end switches. Sto rage controller Represents a physical computer (1U in size) that acts as storage controllers in the cluster, providing block data that supports FC and iscsi protocols. Storage controllers can access all SSDs in the same X-Brick over the SAS connection to the X-Brick DAE. Or they can access SSDs in remote X-Bricks, leveraging remote direct memory access (RDMA) transfer over the Infiniband inter-controller network. Battery backup unit (BBU) Provides enough power to each storage controller to ensure that any data in flight de-stages to SSD in the event of a power failure. The first X-Brick has two BBUs for redundancy. Each additional 1U X- Brick requires only a single BBU. DAE Houses the flash drives that the array uses and is 2U in size. I n finiband switch Connects multiple X-Bricks together and is 1U in size. The array uses two separate switches so that even the fabric that ties the controllers together is high availability. Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere

28 Chapter 4: Solution Architecture Hardware resources Table 1 lists the hardware used in this solution. Table 1. Solution hardware Component VMware vsphere servers CPU Memory Network Configuration 1 vcpu per virtual machine 4 vcpus per physical core For 3,600 virtual machines: 3,600 vcpus Minimum of 900 physical CPUs 2 GB RAM per virtual machine 2 GB RAM reservation per VMware vsphere host For 3,600 virtual machines: Minimum of 7200 GB RAM Add 2 GB for each physical server 2 x 10 GbE NICs per server 2 HBAs per server Note: You must add to the infrastructure at least one additional server beyond the minimum requirements to implement VMware vsphere HA functionality and to meet the listed minimums. Network infrastructure Minimum switching capacity 2 physical switches 2 x 10 GbE ports per VMware vsphere server for management 2 ports per VMware vsphere server for the storage network (FC or iscsi) 2 ports per storage controller for storage data (FC or iscsi) EMC XtremIO all-flash array Shared infrastructure One X-Brick with 25 x 400 GB emlc SSD drives In most cases, a customer environment already has infrastructure services such as AD and DNS configured. The setup of these services is beyond the scope of this document. If you implement the solution without the existing infrastructure, the minimum requirements are as follows: 2 physical servers 16 GB RAM per server 4 processor cores per server 2 x 1 GbE ports per server Note: You can migrate the services into this solution post-deployment. However, the services must exist before the solution is deployed. 28 Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5

29 Chapter 4: Solution Architecture Software resources Table 2 lists the software used in this solution. Table 2. Solution software Software Configuration VMware vsphere vsphere Server Enterprise Edition 5.5 vcenter Server Enterprise Edition 5.5 Operating system for vcenter Server Note: You can use any operating system that is supported for vcenter. Microsoft SQL Server Note: You can use any database that is supported for vcenter. Microsoft Windows Server 2008 R2 SP1 Standard Edition Version 2008 R2 Standard Edition E MC PowerPath EMC PowerPath/VE Release 5.9 EMC XtremIO (for vsphere datastores) Virtual machines (used for validation, but not required for deployment) Base operating system Microsoft Windows Server 2012 Datacenter VDBench (workload generator) Release Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere

30 Chapter 4: Solution Architecture Storage configuration guidelines Overview This section provides guidelines for setting up the storage layer to provide high availability and the expected level of performance. VMware vsphere allows more than one method of storage when hosting virtual machines. This tested solution uses the FC and iscsi block protocols, and the storage layout adheres to all current best practices. You can modify this solution, if necessary, based on your system usage and load requirements. XtremIO X-Brick scalability XtremIO storage clusters support a fully distributed, scale-out design that allows linear increases in both capacity and performance to provide infrastructure agility. XtremIO uses a building-block approach in which the array can be scaled by adding X-Bricks. With clusters of two or more X-Bricks, XtremIO uses a redundant 40 Gb/s quad data rate (QDR) Infiniband network for back-end connectivity among the storage controllers. This ensures a highly available, ultra-low latency network. N-way active controllers provide host access for linear scaling of performance and capacity for simplified support of growing virtual environments. As a result, as capacity in the array grows, performance also grows with the addition of more storage controllers. Figure 6 shows what the different cluster configurations look like as you scale up. You can start from one single X-Brick (a 6U system). As you scale, you can add a second X-Brick, and then a third and fourth X-Brick. You will see the performance linearly scaling as additional X-Bricks are added. 2 Note: In Figure 6, IOPS (mix) is measured with 4 KB fully random storage, 50 percent writes and 50 percent reads, while IOPS (read) is measured with 4 KB and 100 percent reads. 2 Online cluster expansion will be supported in a post-ga release. 30 Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5

31 Chapter 4: Solution Architecture Figure 6. XtremIO scalability XtremIO server virtualization validated maximums This solution uses a single X-Brick that is validated with the environment profile described in Table 3. Table 3. Profile characteristic Validated profile characteristics Maximum number of virtual machines Virtual machine OS Number of processors per virtual machine 1 Number of virtual processors per physical CPU core 4 Value 2,800 per X-Brick Windows Server 2012 Datacenter RAM per virtual machine 2 GB Average storage available per virtual machine 100 GB I/O per second (IOPS) per virtual machine 25 I/O pattern Random I/O read/write ratio 2:1 Number of datastores to store virtual machine disks 28 Number of virtual machines per datastore 100 Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere

32 Chapter 4: Solution Architecture Profile characteristic Disk and RAID type for datastores Value 400 GB emlc SSD drives XDP (XtremIO proprietary data protection that delivers the equivalent of RAID 6 data protection and better performance than RAID 10) Note: We tested and validated this solution with Windows Server 2012 for vsphere virtual machines. However, the solution also works with Windows Server 2008 on VSphere with the same configuration and sizing. The EMC XtremIO array configuration included the following: Twenty-eight 6 TB volumes for virtual machines. Each volume stored 100 virtual machines. XtremIO supports the VAAI ATS primitive, thereby enhancing virtual server performance. One initiator group using the vsphere host FC World Wide Names (WWNs) from the hosts in the indicated vsphere cluster. Figure 7 shows the simple volume configuration in the EMC XtremIO console and the volume mapping for one of the two initiator groups. Figure 7. EMC XtremIO volume configuration and mapping 32 Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5

33 Chapter 4: Solution Architecture Table 4 provides the validated test metrics for this configuration. Table 4. Validated test metrics for 2,800 virtual machines Performance metrics Value Number of virtual machines 2,800 per X-Brick Number of volumes 28 Volume size TB Address space TB SSD space in use 3.34 TB SSD space 7.48 TB Overall efficiency 50.61% Deduplication ratio 9.13 Thin provisioning savings 81.95% Front-end IOPS Front-end bandwidth Front-end latency MB/s 1.76 ms Bandwidth on XtremIO array MB/s IOPS on XtremIO array Average IOPS per SSD 3137 Average bandwidth per SSD MB/s CPU utilization for storage controllers 61 to 75% The test metrics in Table 4 show an I/O latency of less than 2 milliseconds (ms) for 2,800 concurrently running virtual machines, with approximately 70 percent CPU utilization on the array, which confirms room to scale. We measured front-end latency inside the virtual machines, which is the round trip latency from the guest OS through the hypervisor, over the SAN to the XtremIO array, and back again. On the array we observed latency of less than 1 ms. This test confirms that a single X-Brick supports up to thousands of virtual machines with the ability to scale to multiples of thousands in a four X-Brick cluster. Steadystate response and array utilization at high scale points on page 43 provides more details about the scale-out results and projections. Note: The number of virtual machines in practice is determined by I/O performance and capacity utilization. You must monitor capacity utilization for capacity-intensive applications such as Exchange and SharePoint. Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere

34 Chapter 4: Solution Architecture External data considerations XtremIO provides high-performance, high-efficiency storage when deploying virtual servers. Applications such as Microsoft Exchange and SharePoint require high capacity instead of I/O performance. Table 5 provides an example of a configuration for a medium-sized Exchange 2013 server. Table 5. Example of a virtualized Exchange 2013 server Configuration Value Number of mailboxes 2,500 Maximum mailbox size Mailbox IOPS profile (messages sent and received per mailbox per day) 1.5 GB IOPS (150 messages) DAG copies (including active copies) 2 Deleted items retention (DIR) Backup and truncation failure tolerance Years of growth 14 days 3 days 1 year Annual growth rate of mailbox number 11% To size the Exchange storage IOPS, use the following block calculation to get the total database IOPS required to support the specified number of mailbox users: Total transactional IOPS = IOPS per mailbox x mailboxes per server x (1 + I/O overhead factor) In the example, the calculation is: x 2500 x (1 + 20% + 20%) = 354 IOPS To calculate the Exchange storage capacity requirement, use the following steps: 1. To determine the mailbox size on disk, use the following formula: Mailbox size on disk = Maximum mailbox size + White space + Dumpster Mailbox space White space Dumpster Test formula messages sent and received for each user, each day multiplied by the average message size divided by 1024 Example: A mailbox in IOPS profile sends and receives a total of 150 messages per day on average, so the white space is 150 x 75 / 1024 = 11 MB. messages sent and received for each user, each day multiplied by the average message size multiplied by the deleted item retention window added to the maximum mailbox size x plus maximum mailbox size times 0.03: Example: (150 x 75 x 14 / 1024) + (1536 x 0.012) + (1536 x 0.03) = 218 MB. 34 Server Virtualization with EMC XtremIO All-Flash Array and VMware vsphere 5.5