SAN VIRTUOSITY Series WHITE PAPER High Availability with VMware vsphere and Emulex/Cisco SANs Subscribe to the SAN Virtuosity Series at www.sanvirtuosity.com
Table of Contents Introduction...1 VMware High Availability...2 VMware Fault Tolerance...2 VMware Distributed Resource Scheduler...3 Multiple Emulex Adapter Ports and Multipathing...3 Dual-Fabric SANs...4 Storage Redundancy...5 VMware vcenter Site Recovery Manager...6 Conclusion...8 For More Information...8
Introduction Server virtualization has moved from development and test labs into the heart of the data center. When failure or unplanned downtime of a physical server affects multiple virtual machines (VMs) running business-critical applications, high availability (HA) becomes a high-priority requirement. This second installment in the SAN Virtuosity series co-authored by Emulex, Cisco and VMware will present key high availability technologies for VMware vsphere 4.1 and Emulex/Cisco Storage Area Networks (SANs). The first sections will cover capabilities in VMware vsphere 4.1, which ensure that VMs are always available and server resources are optimized. The focus will then shift to the storage network infrastructure. When used together, the result is end-to-end high availability and uninterrupted service for critical users and applications. Table 1 shows different points of failure in a data center and the appropriate solution for mitigating against that failure. Point of Failure ESX/ESXi, server hardware Server resource availability Adapter, fabric cable, switch, array controller Disk drive Array or drive availability, data corruption Site-wide catastrophic failure Table 1 Point of Failure / Solution Map Solution VMware High Availability (HA), VMware Fault Tolerance (FT) VMware Dynamic Resource Scheduler (DRS) VMware multipathing, multiple HBA ports, dual fabric SAN, director class switch, dual array controller Array-based RAID Array-based mirroring, snapshots VMware Site Recovery Manager 1
VMware High Availability VMware High Availability (HA) delivers the availability needed by many applications running in virtual machines, independent of the operating system and application running in it. VMware HA provides uniform, cost-effective failover protection against hardware and operating system failures within your virtualized IT environment. Monitors virtual machines to detect operating system and hardware failures. Restarts virtual machines on other physical servers in the resource pool without manual intervention when server failure is detected. Protects applications from operating system failures by automatically restarting virtual machines when an operating system failure is detected. VMware HA can be configured with a single click from within the vsphere Client interface to provide failover protection without requiring the complex setup and configuration of solutions tied to operating systems or applications. Because VMware HA is simple to configure and requires minimal resources to provide protection, you can: Provide uniform protection of all of your applications from server and operating system failures, regardless of the server hardware or operating system used by the virtual machine. Establish a consistent first line of defense for your entire IT infrastructure. Protect applications with no other failover options and make high availability possible for software applications that might otherwise be left unprotected. VMware HA can be purchased as a component of all VMware vsphere editions and also in the Essentials Plus kit for the SMB. VMware Fault Tolerance For specific applications requiring even greater uptime than VMware HA can provide with traditional failover, customers also have the option to enable VMware Fault Tolerance. VMware Fault Tolerance provides continuous availability for applications in the event of server failures, by creating a live shadow instance of a virtual machine that is in virtual lockstep with the primary instance. By allowing instantaneous failover between the two instances in the event of hardware failure, VMware Fault Tolerance eliminates even the smallest chance of data loss or disruption. VMware Fault Tolerance can be easily turned on or off for individual virtual machines. Since it leverages existing VMware HA clusters, any number of virtual machines in this cluster can be protected with VMware Fault Tolerance. Applications that require continuous protection during certain critical periods of time such as quarter end processing, can utilize VMware Fault Tolerance for higher assurance of availability during those time periods. VMware Fault Tolerance protected virtual machines are limited to single virtual CPU configurations and requires Intel 31xx, 33xx, 52xx, 54xx, 55xx, 74xx or AMD 13xx,23xx, 83xx series of processors or above. 2
VMware Distributed Resource Scheduler VMware Distributed Resource Scheduler (DRS) continuously balances computing capacity in resource pools to deliver the performance, scalability and availability not possible with physical infrastructure. VMware DRS allows you to: Improve service levels for all applications VMware DRS continuously balances capacity to ensure each virtual machine has access to appropriate resources at any point in time. Easily deploy new capacity VMware DRS seamlessly takes advantage of the additional capacity of new servers added to a resource pool by redistributing virtual machines without system disruption. Automate planned server maintenance VMware DRS can automatically migrate all virtual machines off physical servers to enable scheduled server maintenance with zero downtime. Dramatically increase system administrator productivity Enables system administrators to monitor and effectively manage more IT resources. Automatically defragment cluster capacity When needed by VMware HA to accommodate virtual machine failover needs. Multiple Emulex Adapter Ports and Multipathing In addition to hypervisor-based technologies from VMware, high availability should be supported throughout the SAN infrastructure. The first step in the I/O path is the server and deployment of multiple adapter ports per server, which is a key best practice for high availability. Emulex LightPulse LPe12002 and LPe12004 8Gb/s dual and quad-port Host Bus Adapters (HBAs) provide redundant, high-performance, scalable connectivity to Fibre Channel storage networks with the I/O bandwidth needed to virtualize business-critical applications. The LightPulse series of adapters also delivers industry-leading server performance as measured in I/O transmissions per percent of CPU usage to fully optimize server virtualization ratios. Additionally LightPulse adapters feature leading multi-port scalability, ensuring that all ports are functioning at optimum capacity. Emulex dual-port Converged Network Adapters (CNAs) provide redundant, high-performance connectivity for nextgeneration converged networks that use a common 10Gb/s Ethernet (10GbE) infrastructure, such as the Cisco Nexus 5000 family, for network and storage traffic including Fibre Channel over Ethernet (FCoE), iscsi, NFS and other traffic, providing an ideal solution for scalable, flexible server deployments. Figure 1 Emulex LightPulse LPe12002 8Gb/s Dual-port Fibre Channel Host Bus Adapter For maximum redundancy, multiple singe-port LightPulse LPe12000 8Gb/s adapters can be used instead of dual-port or quad-port adapters Heat issues can also be a concern when multiple adapters are installed in a server. Emulex high-performance 8Gb/s Fibre Channel HBAs and 10Gb/s FCoE CNAs are designed to run with reduced operating temperatures to minimize the risk for failure of server components. 3
Prior to the release of vsphere 4, the ESX hypervisor only supported basic failover with storage adapters. With basic failover, two (or more) HBAs or CNAs are installed on the host server and one of the adapters is used for all I/O. If the storage link goes down, I/O is automatically moved to the standby adapter. The release of vsphere 4 introduced VMware vstorage, which encompasses new capabilities and interfaces, including enhanced multipathing. The ESX 4.0 VMkernel provides a new Pluggable Storage Architecture (PSA) that supports the default ESX Native Multipathing Module from VMware and custom software plug-in modules. Using vstorage multipathing APIs, VMware storage partners are adding new load-balancing capabilities for VMs that are similar to solutions for physical servers. With load-balancing, all of the HBAs or CNAs can be used together to provide maximum total bandwidth. If a storage link goes down, I/O will be directed to the other working adapters with no disruption in service. Emulex adapters and drivers are fully compatible with multipathing failover and load-balancing solutions from VMware and its storage partners. In addition, the Emulex OneCommand Manager application provides an excellent resource for managing high-availability deployments. Emulex s management application enables configuration and management of local and remote adapters (both HBAs and CNAs) throughout the data center from a single management console. Comprehensive tracking and diagnostic capabilities can also help identify potential adapter, fabric and array problems before they result in unplanned server downtime. Dual-Fabric SANs Within each ESX server, having a minimum of two HBA or CNA ports will enhance the availability of the VMs. To take full advantage of redundant adapter ports and the failover capabilities of the ESX multipathing, the common best practice requires the deployment of two parallel, fully independent SANs. The Cisco MDS 9000 Family consists of multilayer director and fabric switches that incorporate industry-leading features such as virtual SANs, quality of service (QoS), stateful application restart and multi-protocol support into the system design. Given that high availability has become the highest priority for customers, fault tolerance and avoidance are needed to maintain continuous access. The Cisco MDS 9000 Family of switches is based upon a high-availability architecture offering intelligent features across both MDS 9500 and MDS 9200/9100 series products. Cisco MDS 9500 series provides a director class switch with full hardware redundancy and stateful failover mechanisms. Like the MDS 9500 series, the Cisco MDS 9200 and MDS 9100 fabric switches also utilize the highly available NX-OS operating system benefiting from NX-OS features such as stateful application restart to minimize disruptions from a failure and In Service Software Upgrade (ISSU). One means of delivering high availability at the network level is the aggregation of multiple physical Inter-Switch Links (ISLs) into a single logical interface. This aggregation allows fabric administrators to provide link redundancy, greater aggregated bandwidth and load-balancing. Cisco calls this technology PortChannel. Each ISL in an MDS PortChannel is treated in the same way to enhance high availability and robustness: a failure of any link will not affect the overall PortChannel stability. An important advantage of Cisco PortChannel technology is the ability for the bundled physical links to be located on any port on any switching module in the switch. Because the physical links are spread across multiple switching modules, protection is provided not only from link failures, such as cable breaks and faulty optics, but also from a switching module failure. The MDS PortChannel solution does not show any performance degradation over long distances, nor does it have any specific cabling requirements. MDS PortChannel uses flow-based load balancing and can deliver predictable and robust performance independent of covered distances. This means PortChannels can be used to improve availability over links such as CWDM (Course Wave Division Multiplexing) and DWDM (Dense Wave Division Multiplexing), where path lengths may differ substantially. Since it does not require a special license, the PortChannel can be deployed immediately to enhance network availability and robustness. 4
Figure 2 SAN Architecture using a Dual Fabric solution The design of the network or fabric that provides the connectivity between hosts and storage is also an important component of the overall high-availability solution. Generally each device is connected to the same physical infrastructure in the same Fibre Channel fabric. This exposes the SAN to fabric-level events that could disrupt all devices on the network. Changes such as adding switches or changing zoning configurations could ripple through the entire connected fabric. Therefore, designing with separate connected fabrics helps to isolate the scope of any such events. While deploying two independent fabrics is considered the best practice, the MDS 9000 Family Virtual SAN (VSAN) capability offers a way to carve out multiple isolated environments in the same physical infrastructure. Storage Redundancy Dual controllers on storage arrays provide additional protection against fabric and cable incidents, as well as against array controller congestion or failure. High availability can also be supported within the storage array using Redundant Array of Inexpensive Disks (RAID) technologies. The simplest approach is RAID 1, or mirroring, which is writing the same data to multiple drives in the array, or its derivative, RAID 10. If a drive fails, a mirror copy is used with no loss of data. Most arrays also support hot swapping, which allows a failed drive to be replaced without powering down the array. Other RAID levels support writing data across multiple disks (disk striping) using parity data, which is calculated to ensure that failure of one or more disks will not result in loss of data. RAID 5 uses three or more disks and prevents a loss of data if one drive fails. RAID 6 uses additional drives and prevents a loss of data if two drives fail. RAID levels are set when logic unit numbers (LUNs) are created using the management application for the storage array. Additional redundancy options include snapshot and mirroring options offered by the storage array vendors. 5
VMware vcenter Site Recovery Manager To address the challenges of a manual disaster recovery process in server virtualization environments, VMware introduced VMware Site Recovery Manager (SRM). VMware SRM combines VMware vsphere and VMware vcenter Server with storage replication technology to automate disaster recovery. VMware SRM automates the execution of the disaster-recovery processes, such as the setup, testing and actual failover, and it plugs into VMware vcenter Server, where recovery plans can be centrally managed. With VMware SRM, organizations can automate and manage failover between active-passive sites production data center (protection site) and disaster recovery (recovery site) location or active-active sites two sites that have active workloads and serve as recovery sites for each other. SRM recovery process requires a mirror copy of primary storage data in the remote secondary data center. Replication can be synchronous, leading to a mirror copy exactly synchronized with the original copy, or asynchronous, leading to a mirror copy that may be late by some data. Data replication allows little or no downtime or data loss, even with an entire site outage, protecting physical and virtual machines. The storage arrays and the SAN include complementary technologies to effectively replicate application data across long distances. For a virtual machine deployment, the guest OS images, applications and data can be replicated using the native replication software provided by the storage vendor, such as EMC CLARiiON MirrorView, RecoverPoint, Symmetrix Remote Data Facility (SRDF) and NetApp SnapMirror. The interconnection between locations can be achieved with Cisco SAN extension technologies such as native Fibre Channel transport and Fibre Channel over Internet Protocol (FCIP). Figure 3 Implementation of Site Recovery Manager using Storage Replication An organization with direct access to dark fiber has several options for extending SAN connectivity. For the simplest and easiest to manage approach, Fibre Channel can be transmitted directly over the dark fiber infrastructure, eliminating the need for additional media and transmission conversion equipment, leading to a solution that is fast to deploy, with low capital and operating costs. Using Small Form Factor Pluggable (SFP) transceivers, existing dark fiber equipment can carry Coarse Wavelength Division Multiplexing (CWDM) signals to a distance between network endpoints on a single-mode fiber connection up to 56 miles (90 km). 6
A CWDM connection allows up to eight wavelengths to share the same fiber, increasing the available bandwidth, and can host mixed SAN and Ethernet traffic, allowing an organization to optimize use of the dark fiber infrastructure. However, support for multiple wavelengths requires filters to be inserted at either end of the network, and the attenuation reduces the distance between endpoints to 41 miles (66 km). Like CWDM, Dense Wavelength Division Multiplexing (DWDM) uses different optical frequencies to allow multiple channels of communication to share a dark fiber infrastructure. However, with much narrower filters, DWDM can provide 32 channels of communication across a dark fiber network, delivering 320 Gbps of aggregated bandwidth. This high-speed, low-latency connectivity is ideal for extending mission-critical storage networks. To enable longer distances and for additional flexibility, FCIP communications travel across TCP/IP networks, supporting almost unlimited distance between the source and target of a replication configuration. With any long-distance communication, the further apart the endpoints in the network causes greater the potential for network latency to affect performance. Nonetheless FCIP has been successfully deployed for distances exceeding 10,000 miles. Figure 4 Fibre Channel over IP allows long distance SAN extension Cisco IP services supports the extension of VSANs across long-distance IP networks to effectively segregate multiple streams of traffic. Although VSANs enforce segregation, Inter-VSAN Routing (IVR) provides a mechanism for traffic to transit VSAN boundaries. This feature adds flexibility without compromising the stability, availability and management isolation of each VSAN. The Cisco MDS 9000 Family of fabric switches and directors provides native Fibre Channel interfaces with a large number of buffer credits for long-distance connection, and integrates multiprotocol support for FCIP, performing local fabric switching and SAN extension services on a single platform. Support for network QoS and VSANs allows storage replication traffic to be segregated effectively and independently managed over a shared network infrastructure. The availability of the MDS 9000 SAN extension links is improved by aggregating links in Port Channels. To preserve application performances, the IOA (I/O Acceleration) services significantly improve replication performance and overall network throughput when traffic travels over extended distances. FCIP compression optimizes bandwidth, allowing data replication across low and intermediate bandwidth long-distance networks. MDS9000 hardware guarantees the data confidentiality and integrity providing native Fibre Channel encryption and native IPsec support for IPFC. The Cisco MDS 9000 NX-OS Software includes tools for monitoring and optimizing the performance of an FCIP connection. 7
Conclusion High availability becomes a high-priority requirement when failure or unplanned downtime of a physical server affects multiple VMs running business-critical applications. This paper described some of the components and concepts involved in designing a highly available deployment of VMware vsphere on a SAN, in conjunction with the Cisco MDS 9000 and Emulex LightPulse 8Gb/s Host Bus Adapters. High availability is achieved by a combination of traditional design choices, like adopting a dual-fabric configuration, and the most advanced functionalities provided by VMware vsphere. Each component contributes to the end-to-end availability with the adequate level of resilience at any given level. Emulex dual-port and quadport HBAs and CNAs enable reliable multipathing, the fabric switches are protected by a highly available operating system and the directors resiliency is assured by the same highly available operating system, on top of a fully redundant architecture. The overall SAN design can benefit from improved isolation by deploying VSAN, and can include the highly available PortChannels to assure inter-switch link availability. The full BC/DR architecture is based on VMware Site Recovery Manger in combination with data replication over fabric extension technologies. This document is the second in an ongoing series that will be jointly published by Cisco, Emulex and VMware. For More Information More references to be added. VMware High Availability: Concepts, Implementation, and Best Practices: http://www.vmware.com/files/pdf/vmwareha_twp.pdf Protecting Mission-Critical Workloads with VMware Fault Tolerance: http://www.vmware.com/files/pdf/resources/ft_virtualization_wp.pdf Virtual Machine Mobility with VMware VMotion and Cisco Data Center Interconnect Technologies: http://www.cisco.com/en/us/solutions/collateral/ns340/ns517/ns224/ns836/white_paper_c11-557822.pdf 8
Cisco Systems, Inc. 170 West Tasman Drive, San Jose, CA 95134-1706 USA Tel 408-526-4000 www.cisco.com 2011 Cisco Systems, Inc. All rights reserved. CCVP, the Cisco logo, and the Cisco Square Bridge logo are trademarks of Cisco Systems, Inc. Emulex Corporation. 3333 Susan Street, Costa Mesa, CA 92626 USA Tel 714-662-5600 www.emulex.com Copyright 2011 Emulex. All rights reserved worldwide. No part of this document may be reproduced by any means or translated to any electronic medium without the prior written consent of Emulex. Information furnished by Emulex is believed to be accurate and reliable. However, no responsibility is assumed by Emulex for its use; or for any infringements of patents or other rights of third parties which may result from its use. No license is granted by implication or otherwise under any patent, copyright or related rights of Emulex. Emulex, the Emulex logo, LightPulse and SLI are trademarks of Emulex. VMware, Inc. 3401 Hillview Ave. Palo Alto CA 94304 USA Tel 650-427-5000 www.vmware.com 2011 VMware, Inc. All rights reserved. VMware, the VMware boxes logo and design, vsphere, Virtual SMP and vmotion are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies. 11-0704 1/11