SN A. Reference Guide Efficient Data Center Virtualization with QLogic 10GbE Solutions from HP

Transcription

1 SN A Reference Guide Efficient Data Center Virtualization with QLogic 10GbE Solutions from HP

2 Reference Guide Efficient Data Center Virtualization with QLogic 10GbE Solutions from HP Information furnished in this manual is believed to be accurate and reliable. However, QLogic Corporation assumes no responsibility for its use, nor for any infringements of patents or other rights of third parties which may result from its use. QLogic Corporation reserves the right to change product specifications at any time without notice. Applications described in this document for any of these products are for illustrative purposes only. QLogic Corporation makes no representation nor warranty that such applications are suitable for the specified use without further testing or modification. QLogic Corporation assumes no responsibility for any errors that may appear in this document. Revision A, December 15, 2011 Document Revision History Changes Sections Affected Initial release All ii SN A

3 Table of Contents Preface Overview Key Benefits Reference Architecture Model System Architecture Network Topology Typical Legacy 1GbE Topology Efficient 10GbE Topology Reference Component List 3 Optimizing the Virtualized Data Center The HP-QLogic 10GbE Solution HP-QLogic NC523SFP Dual Port 10GbE Server Adapter The HP 5800 Switch Series HP ProLiant DL380 G7 Server Building an Efficient Network Design 5 Conclusion A Design Considerations Networking Requirements for the Virtual Infrastructure Network Design with 10GbE Increased Performance Simplified Management Increased Scalability Implementation Resources User Guide A-1 HP-QLogic Product Matrix A-1 Other Resources A-1 v v SN A iii

4 Reference Guide Efficient Data Center Virtualization with QLogic 10GbE Solutions from HP List of Figures Figure Page 1-1 Comparison of a Legacy Deployment Versus an Efficient Deployment Typical Legacy 1GbE Topology Efficient 10GbE Topology List of Tables Table Page 1-1 1GbE and 10GbE Components for Reference Network Topology Network Bandwidth Sizing for Legacy 1GbE Reference Architecture Network Bandwidth Sizing for the Efficient 10GbE Reference Architecture Hardware Components Software Components Management Components iv SN A

5 Preface Overview Key Benefits Designing an efficient data center to support high performance and scalable virtualized server deployment creates a number of challenges when using a legacy 1-gigabit Ethernet (GbE) network I/O infrastructure. This document identifies the challenges of the 1GbE legacy network infrastructures for virtualized data centers and presents a reference architecture solution using QLogic s 10Gb Intelligent Ethernet Adapter with HP infrastructure. The reference architecture discussed in this document uses proven HP and QLogic 10GbE networking technology solutions currently available in the market. System architects can leverage the reference architecture to plan, migrate, and optimize their designs, maximizing data center operational efficiency and savings. As data centers continue to increase the number of virtualized hosts on a physical server, the demands for higher efficiency, performance, and consolidation creates critical challenges with 1GbE legacy networks: Low network bandwidth caps the scaling of virtualized hosts and services. Inefficient usage of server PCI slots limits I/O consolidation. Increased cabling, switch ports, and adapters add management complexity and costs. Peak network performance is reduced for on-demand applications and services. The HP-QLogic 10GbE network solution eliminates the I/O bottlenecks, delivering significant operational savings, high performance, reduced management complexity, and improved scalability for virtualized data center deployments. Efficient data center virtualization has the following key benefits: Reduced management complexity Improved operational efficiency High-performance 10GbE network Scalable virtual hosts and services SN A v

6 Preface Key Benefits vi SN A

7 1 Reference Architecture Model System Architecture Figure 1-1 compares a legacy deployment with 1GbE versus the efficient deployment with 10GbE. Figure 1-1. Comparison of a Legacy Deployment Versus an Efficient Deployment SN A 1-1

8 1 Reference Architecture Model Network Topology Table 1-1 compares the components used for the legacy deployment with the components used for the efficient deployment. Table GbE and 10GbE Components for Reference Network Topology Component Legacy Deployment with 1GbE Efficient Deployment with 10GbE Model Units Model Units Server HP DL380 G7 1 HP DL380 G7 1 NICs Legacy 1 GbE NIC ports 14 HP NC523 SFP (QLogic) ports 2 LAN switch Legacy 1GbE Ethernet switch 2 switches; 14 switch ports HP A5820X-24XG-SFP+ 10GbE 2 switches; 2switch ports Network Topology There are two network topologies described in this document: Typical Legacy 1GbE Topology on page 1-3 Efficient 10GbE Topology on page SN A

9 1 Reference Architecture Model Network Topology Typical Legacy 1GbE Topology The 1GbE network topology shown in Figure 1-2 uses the legacy 1GbE network interfaces with VMware ESX virtualization environment and multiple 1GbE links. An HP DL380 G7 server with four 1GbE LAN-on-motherboard (LOM) ports and five dual-port 1GbE NICs uses five of the available six PCIe slots on the server. Figure 1-2. Typical Legacy 1GbE Topology Twelve virtual machines (VMs) or hosts running various application services (such as a Web server, a file server, and database applications) were considered for this reference architecture model (see Figure 1-2). ESX back-end services (such as NFS, IP storage, and VMotion ) provide access to the storage and high availability required for enterprise deployments. SN A 1-3

10 1 Reference Architecture Model Network Topology The ESX virtual standard switch is configured with multiple port groups (1 6) to support the various networking functions and application groupings. Each port group is supplied with one or more physical 1GbE connections. Virtual LAN (VLAN) tags (A F) are implemented on the port groups for segmenting network traffic. Each VM connects to the data center network through one or more ESX virtual NICs (VNICs), and each VNIC connects to one or more virtual switch (vswitch) port groups. Network bandwidth and high-availability requirements of the various traffic types (see Table 1-2) on the 1GbE physical uplinks are either configured for failover for redundancy or static link aggregation control protocol (LACP) for higher bandwidth. Table 1-2. Network Bandwidth Sizing for Legacy 1GbE Reference Architecture Traffic Types for VMware ESX Provisioned 1GbE Network Ports Available Bandwidth Target Network Bandwidth Management traffic 2 (with failover) 1Gbps 1Gps VMotion traffic 1 1Gbps 4Gbps IP storage traffic 1 1Gbps 2Gbps NFS traffic 1 1Gbps 2Gbps Fault tolerance (FT) traffic 1 1Gbps 2Gbps VM traffic (for 12 VMs) 8 (with failover) 6Gbps 8Gbps Total 14 11Gbps 19Gbps The management network is served by the ESX VMkernel network interface connected to a pair of 1GbE uplinks connected by port group 6 on the ESX virtual standard switch. The management traffic is carried over VLAN F through a redundant pair of 1GbE links (failover configuration) connected to a redundant set of 1GbE switches. While the solid green line is the primary I/O path, the dashed green line depicts a failover path for the network traffic (see Figure 1-2). Two legacy 1GbE switches interconnected to each other create a high-availability configuration that prevents network traffic disruption if one of the 1GbE switches fail. The 1GbE architecture has limited network bandwidth; it is unable to scale to meet the peak network bandwidth requirements for on-demand applications. The reason for this is that most PCIe slots are already occupied, and the network cannot scale with more VMs and application services, which demand additional network bandwidth that cannot increase because of the limitation of additional PCIe slots. 1-4 SN A

11 1 Reference Architecture Model Network Topology There are 14 cables originating from one server using seven switch ports per switch, increasing operational cost and management complexity. Efficient 10GbE Topology The 10GbE network topology with ESX deployment shown in Figure 1-3 uses two 10GbE uplinks. The HP DL380 G7 server with one dual-port 10GbE NIC uses only one of the available six PCIe slots on the server instead of the five slots used in the legacy network. Figure 1-3. Efficient 10GbE Topology The legacy 1GbE topology and efficient 10GbE topology both use the same virtualized model with 12 virtual machines and associated ESX back-end services, vswitch, port groups, and VLAN. The 10GbE network model yields an efficient and scalable network for a virtualized data center. SN A 1-5

12 1 Reference Architecture Model Network Topology The architecture uses the two 10GbE uplinks aggregated (static LACP) to provide a single consolidated 20GbE uplink for the ESX standard virtual switch. The network bandwidth sizing table (see Table 1-3) categorizes various traffic types and the allocation of the 20Gb network bandwidth. Table 1-3. Network Bandwidth Sizing for the Efficient 10GbE Reference Architecture Traffic Types for VMware ESX Management traffic VMotion traffic IP storage traffic NFS traffic FT traffic Virtual machine traffic (for 12 VMs) Provisioned 10GbE Network Ports 2 Available Bandwidth 1Gps 4Gps 2Gps 2Gps 2Gps 8Gps Total 2 19Gps VMware ESX traffic types (such as VMotion) that were limited to 1-2Gbps in the legacy 1GbE deployment can now leverage the increased available bandwidth, resulting in a more efficient VMotion for live migration. IP storage, NFS traffic, and the storage access needs of the virtual machines are no longer limited to the number of 1GbE adapters dedicated to the traffic type. Two HP 5800 series 10GbE network switches replace the legacy 1GbE switches and provide high-speed and high-availability network traffic configuration (see Figure 1-3). The obvious advantage of the 10GbE solution is that it reduces the overall physical connections, simplifying infrastructure management. Fewer physical ports reduce the wiring complexity and improve infrastructure reliability. With just one PCIe slot occupied and significantly higher bandwidth than the 1GbE legacy solution, the 10GbE architecture can scale to accommodate more VMs and higher network bandwidth requirements for peak demands. This consolidation improves the efficiency of the architecture. 1-6 SN A

13 2 Reference Component List Table 2-1, Table 2-2, and Table 2-3 provide lists of hardware, software, and management components required for this reference architecture. See Appendix A for relevant references. Table 2-1. Hardware Components Component HP Part Number Description Server HP DL380 G B21 Base configuration model; various models are available with a different selection of system configurations. Intelligent Ethernet Adapter HP NC523SFP B21 HP ProLiant Ethernet Server adapter Switch HP G JC100A Fixed four RJ-45 10/100/1000 ports and four SFP+ 10GbE ports Table 2-2. Software Components Component Description Version Server VMware ESX Operating system 4.1 Intelligent Ethernet Adapter Driver-network Switch Platform firmware HP NC-Series QLogic P3P Multifunction Driver for VMware ESX HP ProCurve switch firmware and boot code A5800_5.20.R1211 SN A 2-1

14 2 Reference Component List Table 2-3. Management Components Component Description Version Server HP CIM Provider HP Common Information Model 1.1(a) (1 Mar 2011) VMware vcenter TM VMware ESX Management Suite Build Intelligent Ethernet Adapter QLogic vcenter Plugin QLogic CIM Provider HP NC-Series QLogic P3P Flash Update Kit Switch Microsoft Windows -based network management vcenter Plugin QConvergeConsole Management CIM Provider for HP-branded QLogic-based Intelligent Ethernet Adapters A bootable FreeDOS TM operating system along with utilities required to Flash HP P3+-based adapters HP Intelligent Management Center (IMC) platform for HP network switches TBD (Yet to be posted) TBD (Yet to be posted) SN A

15 3 Optimizing the Virtualized Data Center The HP-QLogic 10GbE Solution The HP 10GbE solution comprises the HP-QLogic NC523SFP Dual Port 10GbE Server Adapter hosted in a HP ProLiant DL380 G7 enterprise class server connected to a set of HP A5820 switch series. HP-QLogic NC523SFP Dual Port 10GbE Server Adapter The NC523SFP is an eight-lane (x8) PCI Express Generation 2 (PCIe Gen2) 10Gb network solution offering superior bandwidth in a ProLiant Ethernet adapter. This dual-port PCI Express Gen2 adapter supports SFP+ connectors, requiring either direct attach cable (DAC) for copper environments or fiber transceivers and fiber cables for fiber optic environments. The NC523SFP ships with advanced server features such as support for TCP checksum and large segment offload (LSO) capability, VLAN tagging, jumbo frames, IPv6, and more. The two ports on the NC523SFP have a theoretical maximum of 40Gbps full-duplex, bidirectional throughput. Two ports save valuable I/O slots for other cards and reduce cabling compared to the 1GbE legacy architecture. Ports can be configured for failover, load balancing, or connecting to a separate network. The NC523SFP is ideal for enterprise customers in a VMware ESX environment because it provides the following advantages: The best combination of high speed and low latency for virtualization makes it the ideal choice when migrating from a 1GbE solution to a 10GbE solution. It reduces the number of cables and switch ports needed by using a larger 10GbE pipe in the infrastructure. SN A 3-1

16 3 Optimizing the Virtualized Data Center The HP-QLogic 10GbE Solution The ability to provide a higher virtual machine density by meeting and exceeding the network bandwidth demands of an increasing number of virtual machines. For more information, visit the following Web sites: The HP 5800 Switch Series HP 5800 switches offer a combination of 1Gb and 10GbE port density, high-availability architecture, and full Layer 2 and Layer 3 dual-stack IPv4 and IPv6 capabilities. Besides wire-speed line-rate performance on all ports, the switches include patented intelligent resilient framework (IRF) technology and rapid ring protection protocol (RRPP), which allow local or geographically distributed 5800 switches to be interconnected for higher resiliency and performance. Available in power over Ethernet (PoE) and non-poe models and 1 RU and 2 RU flex chassis configurations, 5800 switches are built on open standards and include an open application architecture (OAA) module slot that enables flexible deployment options for new services. These versatile switches are ideal for use in the network core for a building or department or as a high-performance switch in the convergence layer or network edge of enterprise campus networks. For more information, visit the following Web site: dex.aspx HP ProLiant DL380 G7 Server The HP ProLiant DL380 G7 Server provides up to two-socket Intel Xeon performance in a 2U density. With up to six available PCI Express Gen2 expansion slots (including two x8 slots), it provides for the full use of the high-performance capabilities of the HP-QLogic NC523SFP Dual Port 10GbE Server Adapter. For more information, visit the following Web site: SN A

17 4 Building an Efficient Network Design Design Considerations This reference architecture is built according to the following design goals: Performance The design should provide sufficient network bandwidth to virtual machines and back-end services and allow for growth of virtual machines without bandwidth bottlenecks. Availability The design should be capable of recovery from any single points of failure in the network outside of the VMware ESX server. Traffic should continue to flow if a single switch, cable, or network adapter port fails. Isolation Each traffic type should be logically isolated from every other traffic type. Consolidation The design should reduce the number of server PCIe slots occupied, network cables, and switch ports. Scalability The design allows the seamless growth of the number of virtual machines and the demand for network bandwidth. Networking Requirements for the Virtual Infrastructure The key to developing and implementing the 10GbE reference architecture is to understand how the various networking components of VMware vsphere ESX function and what their requirements are for network performance, isolation, and high availability. SN A 4-1

18 4 Building an Efficient Network Design Design Considerations VMware vsphere ESX offers multiple components that relate to Ethernet networking: Hypervisor The hypervisor runs on physical servers and abstracts processor, memory, storage, and networking resources. HP-QLogic NC523SFP 10GbE Server Adapter drivers run in the context of the hypervisor. Virtual standard switch The virtual standard switch is a software component running under the control of the hypervisor. It provides the necessary network connectivity for virtual machines (through the VNIC interface) and back-end services (through the VMkernel network interface). Virtual center management server The virtual center management server allows for configuring, provisioning, and managing virtualized IT infrastructure. VMware back-end services These services include VMware VMotion (which enables virtual machine migration), VMware FT (which provides a higher level of availability), and VMware IP storage and NFS (which provide access to storage and NFS servers). A VMware vsphere ESX environment involves the following traffic types: Management Management traffic goes through the VMkernel management interface on VMware ESXi. This port is used for all management and configuration and is the port by which VMware ESX communicates with the VMware vcenter server. This port generally has very low network usage, but it should always be available and isolated from other traffic types through a management VLAN. VMware VMotion The VMkernel port is used for migrating a running virtual machine from one VMware ESX host to another. Using a 10GbE connection, VMotion under vsphere 4.1 can use up to 8Gbps of aggregate bandwidth with multiple virtual machines being migrated simultaneously. This traffic is typically implemented on a separate VLAN specific to VMware VMotion, with no outside communication required. 4-2 SN A

19 4 Building an Efficient Network Design Network Design with 10GbE Fault-tolerant logging The VMkernel port for fault-tolerant logging transfers the input network I/O for the fault-tolerant virtual machine plus the read disk traffic to the secondary fault-tolerant virtual machine. Traffic varies according to the network and storage behavior of the application. End-to-end latency between the fault-tolerant virtual machines should be less than 1ms, and the dedicated 2Gbps of network bandwidth is considered sufficient. This traffic is typically implemented on a separate VLAN specific to fault-tolerant logging, with no outside communication required. IP Storage The VMkernel port provides access to a disk subsystem over the IP network. In VMware ESXi 4.0, two VMkernel ports can be bonded to allow IP storage traffic over both physical network interfaces. Traffic varies according to I/O; typically 2Gbps of network bandwidth is considered sufficient. This traffic is typically implemented on a storage-specific VLAN common to initiators and targets, although targets may reside on another VLAN accessible through a Layer 3 gateway. Network file system The VMkernel port communicates with NFS files in VMware ESX. Traffic varies according to I/O, and 1 2Gbps of network bandwidth is considered sufficient. This traffic is typically implemented on an NFS-specific VLAN, although files may reside on another VLAN accessible through a Layer 3 gateway. Virtual machines Virtual machines vary in number, may be distributed over more than one VLAN, and may be subject to different policies defined in port profiles and virtual port groups. The reference architecture design assumes an aggregate virtual machines network bandwidth demand to be 6 8Gbps. Network Design with 10GbE When migrating from a 1GbE network to a 10GbE network, the same or enhanced level of network isolation and high availability requirements must be maintained across the legacy 1GbE and efficient 10GbE solutions. SN A 4-3

20 4 Building an Efficient Network Design Network Design with 10GbE With the HP-QLogic 10Gb network infrastructure solution, the transition from the legacy design is very simple and straightforward. After the VMware ESX driver for the NC523SFP has been loaded, the only task is to map the new 10GbE uplinks to the VMware vswitch and create a static LACP (802.3ad) of the two ports of the NC523SFP. Each of the two ports of the NC523SFP are physically connected to two independent HP 5800 switches; this architecture provides for resilience to any single point of failure. While the use of two 10GbE ports helps to reduce the solution cost, some organizations may prefer the use of four 10GbE ports to provide additional bandwidth, additional redundancy, or simply to interface with existing network infrastructure. NOTE: HP supports up to a maximum of two physical NC523SFP adapters in a supported server. The availability of an aggregate bandwidth of 40Gbps bidirectional across the two ports of the NC523SFP provides for all the network bandwidth requirements of the virtual machines and back-end services, as described in Table 1-3. If planned appropriately, the migration from 1GbE to 10GbE in a VMware ESX environment provides the benefits described in the following sections. Increased Performance 10GbE delivers 10 times the bandwidth of 1GbE in a single pipe. Now, rather than using 8 to 10 1GbE ports in addition to on-board LOMs in each server, organizations can deploy just two 10GbE NICs and meet the network bandwidth demands of the virtual infrastructure. Increased bandwidth per virtual machine 10GbE allows each virtual machine to use bandwidth of more than 1Gbps (and its increments), helping to increase performance and eliminate the network as a bottleneck. Faster VMotion Using a 10GbE connection, VMotion under vsphere 4.1 can use up to 8Gbps of aggregate bandwidth, which enables virtual machines to move to new hosts, quickly reducing downtime. Faster access to IP storage Using a faster 10GbE link, access speeds to IP storage and NFS servers are enhanced, improving overall virtual machine performance. 4-4 SN A

21 4 Building an Efficient Network Design Network Design with 10GbE Simplified Management Adopting 10GbE in virtualized environments reduces the number of Ethernet adapter ports to be managed, components to troubleshoot, and components that can fail. Ease of management Consolidating a number of gigabit Ethernet links onto a single 10GbE connection dramatically decreases the number of management points, including NICs and their firmware/driver, cables, and switch ports. NIC management can be executed in any of three ways: through QConvergeConsole, through the command line interface, or through the Windows network management utility. Simplified troubleshooting Decreasing the number of devices reduces the number of failure points and devices to troubleshoot, contributing to higher availability. Increased Scalability Moving to 10gbE enables the data center s virtual infrastructure to scale and accommodate a virtual machine sprawl and associated I/O infrastructure demands. Higher virtual machine density 10GbE provides increased headroom that gives applications running in virtual machines room to scale and the ability to host more virtual machines per ESX host without network bottlenecks. The increased bandwidth of 10GbE improves scalability for both back-end traffic and production traffic to virtual machines. Free PCIe slots Deploying 10GbE frees up server PCIe slots that were occupied by multiple 1GbE adapters, proving room for expansion. SN A 4-5

22 4 Building an Efficient Network Design Network Design with 10GbE 4-6 SN A

23 5 Conclusion Deploying HP-QLogic 10GbE technology in virtualized environments based on VMware vsphere allows enterprises to reduce the complexity and increase the scalability of the network infrastructure. The 10GbE products from HP and QLogic enable enterprise customers to meet the performance requirements of virtual machines without the large numbers of physical server connections required in 1GbE networks. Increased bandwidth; fewer cables, ports, and network adapters to manage; fewer PCIe slots used; and the capability to consolidate network I/O on a single wire complete the argument for deploying the HP-QLogic 10GbE technology in an efficient data center. SN A 5-1

24 5 Conclusion 5-2 SN A

25 A Implementation Resources User Guide Use the following references for more information about the HP server, Intelligent Ethernet Adapter, and switch. HP NC523SFP 10Gb 2-port Ethernet Server Adapter User Guide: pdf HP-QLogic Product Matrix tm Other Resources QLogic Converged Network Adapter User's Guide (8200 and 3200 Series): _Network_Adaper_8200_and_3200_Series_A.pdf Server QuickSpecs for HP ProLiant DL380 G7 Server: Intelligent Ethernet Adapter: QuickSpecs for HP NC522SFP Dual Port 10GbE Server Adapter: v.pdf HP ProLiant Ethernet Adapters Overview and Features: aspx Switch QuickSpecs for HP 5820 Switch Series: ML SN A A-1

26 A Implementation Resources Other Resources A-2 SN A

27

28 Corporate Headquarters QLogic Corporation Aliso Viejo Parkway Aliso Viejo, CA International Offices UK Ireland Germany France India Japan China Hong Kong Singapore Taiwan 2011 QLogic Corporation. Specifications are subject to change without notice. All rights reserved worldwide. QLogic, the QLogic logo, and QConvergeConsole are registered trademarks of QLogic Corporation. ESX, vcenter, VMotion, vsphere, and VMware are trademarks or registered trademarks of VMware, Inc. FreeDOS is a trademark of Jim Hall. HP, ProLiant, ProCurve, and StorageWorks are registered trademarks of Hewlett-Packard Company. Intel and Xeon are registered trademarks of Intel Corporation. Microsoft and Windows are registered trademarks of Microsoft Corporation. PCI Express and PCIe are registered trademarks of PCI-SIG Corporation. All other brand and product names are trademarks or registered trademarks of their respective owners. Information supplied by QLogic Corporation is believed to be accurate and reliable. QLogic Corporation assumes no responsibility for any errors in this brochure. QLogic Corporation reserves the right, without notice, to make changes in product design or specifications.