Business-Ready Configurations for VMware vsphere 4 with PowerEdge R810 Servers and EqualLogic PS6010 Storage Arrays Utilizing iscsi Boot Technology

Transcription

1 Business-Ready Configurations for VMware vsphere 4 with PowerEdge R810 Servers and EqualLogic PS6010 Storage Arrays Utilizing iscsi Boot Technology A Solution Guide Vishvesh Sahasrabudhe Dell Virtualization Solutions Engineering Feedback: [email protected]

2 THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. Copyright 2010 Dell Inc. All rights reserved. Dell, the DELL logo, PowerEdge,, PowerConnect, and EqualLogic are trademarks of Dell Inc. Microsoft is a registered trademark of Microsoft Corporation. Intel and Xeon are registered trademarks of Intel Corporation in the U.S. and other countries. VMware, vcenter, vmotion, and vsphere are registered trademarks or trademarks of VMWare, Inc. in the United States or other countries. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell disclaims proprietary interest in the marks and names of others. November 2010 ii

3 Contents Introduction... 3 Audience and Scope... 3 Overview... 3 VMware vsphere Dell PowerEdge R810 Servers... 5 EqualLogic PS6010 Series iscsi Arrays... 5 Specifications... 7 Reference Architecture... 8 Network Architecture... 9 NetIOC in vnetwork Distributed Switches Traffic Isolation using VLANs Uplinks and Inter Switch Links Storage Architecture Storage Array Connectivity iscsi network configuration at the server iscsi boot configuration iscsi network configuration EqualLogic Multipathing Extension Module Volume Size Considerations Array RAID Configuration Scalability Management Virtualization Management using VMware vcenter Server Systems Management Storage Management Switch Management Deploying and Configuring the Solution Rack, Stack, and Cable Configuring the Servers Configuring Networking Configuring Storage Page 1

4 ESXi installation and configuration Configuring vcenter References VMware Documentation PowerEdge R810 Server EqualLogic Documentation Feedback Figures Figure 1: Overview of Dell R810 Server, Dell EqualLogic SAN, and VMware vsphere... 4 Figure 2: Reference architecture for the solution... 9 Figure 3: vds configuration with network I/O control Figure 4: vds configuration of a server Figure 5: Storage Connectivity Figure 6: Scalability configuration with more than two switches Page 2

5 Introduction Business-Ready Configurations for Virtualization are pre-configured, easy-to-order, bundled solutions that include Dell PowerEdge servers, storage and networking combined with industry leading hypervisors. This white paper provides detailed reference architectures for VMware vsphere 4.1 ESXi with PowerEdge servers and Dell EqualLogic PS6010 storage arrays. Based on extensive engineering work in architectural design and certification, customers can quickly and confidently deploy these engineering-proven architectures into production environments, thereby helping to eliminate much of the costly and time consuming trial and error often encountered during complex infrastructure design and implementation. The configurations in this paper make use of boot from iscsi SAN technology to boot ESXi hypervisors from boot volumes on EqualLogic arrays onto PowerEdge R810 servers. There are several benefits to using hypervisors from boot volumes instead of local drives. Server deployment and replacement is made easier as the hypervisor image is not tied to the physical server and can be redeployed to a new server seamlessly. Diskless servers reduce deployment and maintenance cost over the life of the configuration. SAN-based backup and restore procedures make it easier to back up hypervisor images and also improve disaster recovery scenarios over localized images. Since all images are hosted on the same SAN, their creation and management is also made easier. Finally, the high-availability features of a SAN improve hypervisor availability by removing a single point of failure at the local disk of a server. Audience and Scope The intended audience for this white paper is IT administrators, IT managers, and channel partners who plan to deploy VMware virtualization using PowerEdge R810 servers and EqualLogic PS6010 series storage arrays. This white paper provides an overview of recommended servers, storage, networking, software, and services. It can be used to plan, scope, and procure the required components to set up a virtualization infrastructure. Based on customer requirements, further customization of the recommended architecture may be required. Overview Error! Reference source not found. provides a high-level product overview of VMware vsphere, PowerEdge R810 servers, and EqualLogic PS6010 arrays. VMware vsphere 4.1 VMware vsphere 4 includes the hypervisor as well as an optional VMware vcenter server that is responsible for configuring and managing VMware hosts. Capabilities for the ESX/ESXi Enterprise Plus license level include: VMware vmotion : vmotion technology provides real-time migration of a running virtual machine (VM) from one server to another with no disruption or downtime. vmotion can be used to achieve manual load balancing and near-zero downtime maintenance. VMware Distributed Resource Scheduler (DRS): VMware DRS technology enables vmotion to automatically achieve load balancing according to resource requirements. VMware Distributed Power Management (DPM): When VMs in a DRS cluster need fewer resources, such as during nights and weekends, DPM consolidates workloads onto fewer servers and powers off the rest to reduce power consumption. When VM resource requirements increase, such as when users log into applications in the morning, DPM brings powered-down hosts back online to ensure service levels are met. VMware High Availability (HA): VMware HA provides high availability at the VM level. Upon server failure, VMware HA automatically restarts VMs on other physical servers. Page 3

6 VMware Fault Tolerance (FT): VMware Fault Tolerance provides continuous availability for applications in the event of server failure by creating a live shadow instance of a VM that is in virtual lockstep with the primary instance. VMware vnetwork Distributed Switch: VMware s vnetwork Distributed Switch spans many ESX/ESXi hosts and aggregates networking to a centralized cluster level. The vnetwork Distributed Switch abstracts configuration of individual virtual switches and enables centralized provisioning, administration, and monitoring of virtual networks through vcenter. VMware Update Manager: VMware Update Manager automates patch management by enforcing compliance to patch standards for VMware ESX/ESXi hosts, as well as select Microsoft Windows VMs. Secure patching of offline VMs, as well as automated snapshots and rollback capabilities, helps to reduce risk. Integration with DRS allows for nondisruptive VMware ESX/ESXi patching with minimal administrative overhead. VMware Storage vmotion: VMware Storage vmotion enables real-time migration of running VM disks from one storage array to another with no disruption or downtime. It minimizes service disruptions due to planned storage downtime previously incurred for rebalancing or retiring storage arrays. Host Profiles: Host Profiles standardize and simplify the deployment and management of VMware ESX host configurations. They capture and store validated configurations, including networking, storage, and security settings, and make it easy to monitor host compliance. Network and Storage I/O Control: These features provide limits and shares for accessing storage and network resources from VMs and VMware networks. I/O control helps improve resource utilization similar to CPU and memory resource allocations available in previous versions. Figure 1: Overview of Dell R810 Server, Dell EqualLogic SAN, and VMware vsphere VMware vsphere 4.1 vmotion VMware HA, DRS and DPM, FT CPU, memory, network, and storage resource pools Dell PowerEdge R810 Servers Efficient Infrastructure Intel Xeon 7500 series processors Up to 512 GB DDR3 memory per server FlexMem Bridge technology Increased memory capacity Dell EqualLogic PS6010 Series iscsi arrays Intelligent and automated management tools Self-managing array Seamless scalability EqualLogic Multipathing Extension Module for multipathing management For more information on VMware vsphere, see Page 4

7 Dell PowerEdge R810 Servers The 11 th generation (11G) PowerEdge R810 server offers performance and reliability with up to four Intel Xeon 7500 series processors in a 2U form factor and with the following key features: Hypervisor Redundancy: The R810 server features internal dual-sd modules to provide failover at the hypervisor layer. It also uses Intel Advanced RAS Technology that can automatically monitor, report, and recover from hardware errors to maintain data integrity and keep mission-critical services online. FlexMem Bridge Technology: Allows scaling to full memory capacity with only two populated processors. The FlexMem Bridge essentially mimics and extends existing connections from the two populated processors to the DIMM channels that would normally only be accessible if all four processors were populated. Hence, using FlexMem Bridge Technology, all 32 DIMMs can be accessed with just two processors. Efficient Infrastructure: Each server supports up to 512GB of DDR3 memory and up to 32 cores with four eight-core Intel Xeon 7500 series processors to help consolidate inefficient workloads into a single server. The servers also feature six PCI Express Gen2 slots to achieve comparable network performance and match memory and CPU performance. Energy-efficient system design built with Energy Smart technologies includes power management features enabling power capping, power inventory, and power budgeting within your specific environment. Logical component layout of the internal components aids with airflow direction, helping to keep the server cool. Embedded Management with Dell s Lifecycle Controller: The Lifecycle Controller is an embedded management solution delivered as part of idrac Enterprise in PowerEdge servers. It includes one GB of managed and persistent storage that embeds systems management features directly on the server. It can be accessed on the server through the Unified Server Configurator (USC), and also provides APIs for remote access. Some of the features available include operating system installation, hardware configuration, and system updates. EqualLogic PS6010 Series iscsi Arrays The Dell EqualLogic PS6010 array features an active-passive controller pairing with two 10 Gigabit Ethernet iscsi network ports per controller, two GB of cache per controller, support for RAID 6, a monitoring application, and SAN Headquarters (SAN HQ) at no additional cost. In addition, Dell EqualLogic SAN devices provide the following capabilities: Reliability: EqualLogic PS6000 Series arrays have hot-swappable redundant components, a choice of RAID types, and hot-spare disks. They also include the Auto- Stat Disk Monitoring System (ADMS), which proactively scans disk drives in the background to help detect and correct media anomalies. Scalability: As each array is added to the storage group, the storage capacity and performance increase in terms of both bandwidth and IOPS. This increased capacity can be utilized without downtime. Thin-provisioning permits predefinition of a group of Virtual Machine File System (VMFS) volumes that are larger than the physical space, allowing for physical space to be added and utilized without downtime when necessary. Self-Managing Arrays: The arrays offer many self-managing features, such as automatic load balancing and storage tiering. A single storage pool can have different models that offer a range of capacity and performance parameters. In addition, different arrays in a storage pool can be configured with different RAID levels, and volumes will automatically be migrated between RAID levels based on performance data and usage patterns. All data and volume movement can be performed online with zero downtime. Page 5

8 Top-Tier Data Protection Software: Advanced data protection features such as Auto Replication and Auto-Snapshot Manager are included in the standard configuration. Auto-Snapshot Manager integrates with VMware vcenter and VMware s native snapshot technology to provide intelligent and consistent snapshots. VMware Site Recovery Manager (SRM) is tightly integrated with the EqualLogic PS Series to offer centralized disaster recovery management, automation, and testing for the entire virtualized datacenter using a native auto-replication feature integrated directly into Site Recovery Manager using specialized storage adapter software. SAN Headquarters (HQ): SAN HQ provides centralized access to detailed performance and event data, reporting, and intelligent alerts from EqualLogic storage array groups. SAN HQ uses Simple Network Management Protocol (SNMP) to collect performance, alarm, and health status data from dozens of array groups that could be spread across multiple locations around the world. Support for vstorage APIs: EqualLogic uses the vstorage API for Array Integration (VAAI) for improved storage performance through storage-based hardware acceleration. This also reduces the load on servers by reducing storage traffic. The EqualLogic Multipathing Extension Module uses the vstorage API for Multipathing to provide improved path selection and management. For more information on Dell EqualLogic storage, see Page 6

9 Specifications Table 1 provides the specifications for three pre-configured solutions. Not that the specifications for each configuration might change based on component availability. Table 1: Solution specifications Entry Configuration Medium Configuration Large Configuration Solution Summary Solution ID ESX Server (3) x R810 with 256GB memory in each server (6) x R810 with 256GB memory in each server (10) x R810 with 256GB memory in each server Storage Device (1) EQL PS6010X with GB 10K drives (1) EQL PS6010X with GB 10K drives (2) EQL PS6010X with GB 10K drives Switch (2) x PowerConnect 8024F (2) x PowerConnect 8024F (2) x PowerConnect 8024F Management Server (1) x R610 with VMware vcenter Server (1) x R610 with VMware vcenter Server (1) x R610 with VMware vcenter Server ESX Server Configuration Server Model R810 R810 R810 Processor (2) x Intel Xeon E7540, 2.00Ghz, 18M Cache, 6.4 GT/s QPI (2) x Intel Xeon E7540, 2.00Ghz, 18M Cache, 6.4 GT/s QPI (2) x Intel Xeon E7540, 2.00Ghz, 18M Cache, 6.4 GT/s QPI Memory 256 GB (32 x 8 GB, DDR3) 256 GB (32 x 8 GB, DDR3) 256 GB (32 x 8 GB, DDR3) Add-in Controllers (1) Broadcom Dual Port 10 GbE DA NIC (1) Broadcom Dual Port 10 GbE DA NIC (1) Broadcom Dual Port 10 GbE DA NIC Hypervisor VMware ESXi Enterprise Plus VMware ESXi Enterprise Plus VMware ESXi Enterprise Plus Storage Configuration Storage Device (1) EqualLogic PS6010X (1) EqualLogic PS6010X (2) EqualLogic PS6010X Drives (16) 600GB 10K SAS drives (16) 600GB 10K SAS drives (32) 600GB 10K SAS drives Storage Capacity 9.6 TB Raw Capacity 9.6 TB Raw Capacity 19.2 TB Raw Capacity Management Server Server Model (1) R610 (1) R610 (1) R610 Processor (1) x Intel Xeon E5620, 2.4Ghz, 12M Cache (1) x Intel Xeon E5620, 2.4Ghz, 12M Cache (1) x Intel Xeon E5620, 2.4Ghz, 12M Cache Memory 8 GB (2 x 4 GB) 8 GB (2 x 4 GB) 8 GB (2 x 4 GB) Network Controller (1) Broadcom Dual Port 10 GbE DA NIC (1) Broadcom Dual Port 10 GbE DA NIC (1) Broadcom Dual Port 10 GbE DA NIC Virtualization Management Services VMware vcenter 4.1, 3Yr Upgrade Subscription VMware vcenter 4.1, 3Yr Upgrade Subscription Software and Services 3 Year ProSupport for IT 3 Year ProSupport for IT and Mission Critical 4HR and Mission Critical 4HR 7x24 Onsite Pack 7x24 Onsite Pack VMware vcenter 4.1, 3Yr Upgrade Subscription 3 Year ProSupport for IT and Mission Critical 4HR 7x24 Onsite Pack Page 7

10 How to order: To order one of the solutions discussed in this paper, contact your Dell Sales Representative with the Solution ID provided in the table above. The configurations can be customized to fit specific needs, but may invalidate some of the best practices discussed in this paper. These solutions can also be directly ordered online: Reference Architecture This section describes the high-level reference architecture for the solution. Design Principles The following principles were used to design this architecture. 1. Optimal hardware configuration for virtualized workloads: Each solution is designed with an optimal hardware configuration to support virtualized workloads: Servers are configured with Intel Xeon E-series processors that offer optimal processing power without sacrificing energy efficiency. If a higher processing power is desired, X- series processors can be chosen. Energy efficiency can be improved further by choosing L-series processors. The PS6010X array features SAS 10K HDDs for high I/O throughput. Alternately, the PS6010XV (with 15K HDDs), PS6010S (with solid-state drives), or the PS6010XVS (mix of SSD and 15K SAS drives) can be used for improved throughput based on the I/O profile of the workload. Use of boot-from-san technology provides all of the benefits enumerated above, including easier server replacement and improved operating system management 2. Redundancy with no single point-of-failure: Redundancy is incorporated in every aspect of the solution: A pair of network adapters in each server provides redundancy at the server. Other features like redundant power supplies are also taken into consideration. A redundant pair of switches provides availability in case of switch failure. The Network Architecture section that starts on page 9 provides recommendations for how to connect both switches to the core network to provide uninterrupted connectivity. EqualLogic arrays use a pair of active-passive controllers for redundancy. Connectivity to each controller is provided from both switches to ensure uninterrupted access to the storage array. 3. Scalability: Additional server and storage scalability is considered in design of the architecture. Scalability within a server can be achieved by choosing the optimal processor/memory combination as explained above. EqualLogic storage arrays can scale up to 16 arrays in a single group to provide for increased storage capacity based on the I/O requirements of the solution. A modular architecture optimizing the number of switch ports required for the network setup enables addition of increased server and storage capacity based on availability of switch ports. Additional scalability beyond a two switch configuration is also discussed. 4. Unified, isolated, redundant, and high-performance network design: Utilizing VMware network I/O control features, a unified 10Gb network architecture is designed to carry all categories of traffic from a server over redundant pairs of network adapters. Traffic isolation helps segregate different categories of traffic using VLANs and also helps manage network utilization using features of vnetwork Distributed Switches in ESX. Page 8

11 5. Integration into existing infrastructure: It is assumed that the existing infrastructure can support connectivity to the redundant 10Gb network switches carrying network traffic. It is also assumed that an out-of-band management network exists to support connectivity to the servers, storage, and switches. Error! Reference source not found. shows high-level reference architecture for the solution. Entry, Medium, and Large configurations have similar reference architecture with varying number of servers and storage arrays. Figure 2: Reference architecture for the solution Network Architecture vsphere network traffic is typically comprised of different classes: virtual machines, management, iscsi, vmotion, FT logging, etc. Separate port groups with independent network subnets and VLANs can be used for each class of traffic present in the VMware cluster. All classes of traffic use a pair of 10Gb network adapters in each server to connect to separate network switches as shown in Error! Reference source not found.. A vnetwork Distributed Switch (vds) is used to manage and control each class of network traffic. A vds provides a standardized management interface for networking to all servers in a cluster by defining a standard switch configuration. This consists of distributed port groups similar to port groups in a standard vswitch and distributed virtual uplinks that can be used to connect physical network adapters (vmnics) to the vds. Additional features are supported in the vnetwork Distributed Switch: Network vmotion (Port state follows VM) Network I/O Control (NetIOC). This is discussed further in the next section. Private VLAN For more information on the vnetwork Distributed Switch: Page 9

12 Note that the vds capability requires an Enterprise Plus license for vsphere. All three configurations noted above come with this license. If the license is not available, vsphere standard switches (vswitch) can be used. However, these vswitches do not provide the NetIOC and load-based teaming features and might require an isolated iscsi network in order to ensure storage connectivity. A configuration with a separate iscsi network that does not utilize NetIOC features is beyond the scope of this paper. A detailed description of other network features used in the design of the configuration follows. NetIOC in vnetwork Distributed Switches NetIOC is a new feature introduced in vsphere 4.1 for vds. It provides a more streamlined way of traffic management on vds. NetIOC classifies network traffic into six different resource pools similar to CPU and memory resource pools. These six predefined resource pools include vmotion, iscsi, FT logging, Management, NFS, and VM traffic. Network I/O Control uses shares, limits, and load-based teaming to allocate bandwidth efficiently among these traffic classes. Shares Shares are used to specify the relative importance of a given resource pool. The shares are enforced at each physical adapter and divide the bandwidth available between the pools currently using the adapter. The shares range in value from 1 to 100 and can be set either as High (100), Normal (50), Low (25) or Custom. Thus, if two pools with share values of 50 and 25 share a link, then the bandwidth is divided in a 2:1 ratio between these pools on the adapter. Limits Limits can be set for each resource pool defining the absolute highest bandwidth available to the pool across all physical adapters. Thus, the limit defines the available bandwidth for a resource pool on each ESX host using that pool. Limits can be used to define the upper bounds of high bandwidth for sporadic traffic flows like vmotion, which can saturate all available bandwidth otherwise. Load-Based Teaming (LBT) vswitches and vds have various options for network load balancing across a team of adapters. Virtual switch port ID hash, source MAC hash, and IP hash based routing use a hashing algorithm to associate each virtual port to a single physical NIC and only failover if there is network disruption, irrespective of network load. LBT available on vds dynamically manages traffic from different dvports based on network load, and can move ports from one link to another if oversaturation occurs. It switches the traffic flows if the network utilization on a link exceeds 75 percent over a 30-second period. Note: iscsi port groups are configured to use only one uplink per port group. This configuration is required for utilizing Multipath I/O to storage arrays, and is further discussed in the Storage Architecture section that starts on page 13. Page 10

13 Figure 3: vds configuration with network I/O control Note that the above bandwidth allocation techniques apply to ingress traffic coming from a port group into a vds (i.e., traffic traveling out of a physical server). It does not control the egress traffic that each server receives from other servers (i.e., traffic traveling from the uplinks through vds to the port groups). To achieve bandwidth control on the egress traffic, traffic shaping policies at a port group level can be utilized. This provides robust, configurable, and manageable network flows for each server attached to a vds. It is recommended that NetIOC be enabled in the current configuration. iscsi traffic can be given higher shares and unlimited bandwidth to ensure connectivity to storage. vmotion traffic can be assigned a bandwidth limit to make sure that infrequent and jittery vmotion events don t saturate the uplinks. FT traffic should be assigned high shares to make sure it has a high priority in a constrained bandwidth environment. The limits and shares for VM and management traffic would vary based on the customer environment and network load generated by VMs. Further information about NetIOC, its configuration on ESX hosts, case studies for certain resource utilizations, and best practices for the configuration can be found in the VMware whitepaper VMware Network I/O Control: Architecture, Performance and Best Practices Traffic Isolation using VLANs VLANs can be used to isolate traffic from the different port groups mentioned above. Traffic isolation is important for unencrypted traffic flows like vmotion and FT logging. If routing these flows through the core network is necessary, VLAN segregation is recommended. VLANs also enable use of Priority Flow Control at the switches, thus giving priority to certain classes of traffic at the switch. For example, the VLAN associated with iscsi traffic can be given a higher priority to ensure uninterrupted storage access in times of high congestion. Page 11

14 Uplinks and Inter Switch Links There are a number of ports available for switch interconnection as part of a Link Aggregation Group (LAG) and uplinks based on the server and storage configuration chosen. For example with the large configuration, there are 12 ports available for these connections. The specific number of ports chosen for the LAG (the maximum is eight), and uplinks will vary based on the core network configuration and bandwidth requirements. It is recommended to route the vmotion, Fault Tolerance logging, and iscsi traffic through the network switches in the configuration so that it is not exposed to the core network. EqualLogic recommends having connectivity between the first-hop switches connected to the storage array. Also, vmotion and FT logging traffic is typically high bandwidth and bursty, and thus it is recommended that there be minimal hops between servers. To ensure uninterrupted connectivity to the cluster, there should be uplinks to both switches for management and (based on datacenter requirements) VM traffic. This can be accomplished by either using Spanning Tree Protocol (STP) to disable the uplink from one of the switches, or using Multiple Spanning Tree Protocol (MSTP), which configures spanning trees on a per VLAN basis. MSTP can be used to route management and VM traffic from the core switches, while using the LAG between the switches to route other traffic as detailed above. Based on the STP setting chosen and the bandwidth requirement for VM traffic, the appropriate number of uplink ports can be determined. If STP is used to disable uplinks from one of the switches, then a pair of redundant uplinks from each switch might be sufficient. If, however, there is highbandwidth VM traffic and MSTP is chosen to route from both switches, then the number of uplinks can be increased at the expense of the number of switch ports in the LAG. It should be noted that a LAG can have a maximum of eight switch ports. A sample vds configuration with different dvportgroups, their corresponding VLAN ids and vmkernel IP addresses is shown in Error! Reference source not found.. Page 12

15 Figure 4: vds configuration of a server Storage Architecture This section discusses the best practices for configuring storage connections between the servers and EqualLogic storage arrays and configuration changes to the iscsi port groups defined above. Storage Array Connectivity Each controller on a PS6010 array has two ports. The controllers are in an active-standby configuration. The ports from each controller are connected to separate switches to provide redundancy at the switch and controller level. Error! Reference source not found. shows the connectivity between two EqualLogic PS6010 arrays and PowerConnect 8024 switches. For simplicity, only the connections from the active controller are shown. EqualLogic does not support VLAN tagging at the controller. The traffic from the controllers can be tagged at the switch level. Thus, it should be ensured that the ports assigned to server and storage connectivity are separate. The ports connected to the storage should be configured in access mode, with the iscsi VLAN tagged at the port for packets coming from the controller. Page 13

16 Figure 5: Storage Connectivity iscsi network configuration at the server There are two components to the iscsi network configuration in each server: The iscsi boot phase for connecting to the iscsi boot LUN and the iscsi network configuration post boot-up. iscsi boot configuration ESXi hosts can be configured to boot from SAN using software or dependent hardware iscsi adapters. Broadcom iscsi boot firmware supports iscsi Boot Firmware Table (ibft) format that can communicate the iscsi boot parameters from boot device to ESXi hypervisor. The VMkernel that boots off the device will use the ibft parameters to configure networking in the hypervisor. This network configuration can later be changed using vsphere client. Some best practices to follow while configuring for iscsi boot: The initial network configuration of the ESXi host will depend on which adapter is used to initially boot from and install to the SAN boot device. If the first enumerated physical adapter (vmnic0) is used, then the IP address of the iscsi initiator becomes the management network IP address. If, however, any other adapter is used, then ESXi tries to acquire a DHCP address over vmnic0, and if it acquires one, it then uses that address as management IP address. The initial configuration of the network can be changed later through the vsphere client; however, the IP address that ESXi sees on first boot will be the IP address listed on the Direct Console User Interface (DCUI) of the server. Page 14

17 Broadcom iscsi boot firmware does not support VLAN tagging. Thus, it is necessary to allow untagged packets on the switch ports connected to the server and tag them with the iscsi VLAN tag at the switch, if required. ibft iscsi boot does not support failover for the ibft-enabled network adapter; however, after the first boot of the VMkernel (when contents are copied from ibft and network settings subsequently changed through vsphere client), the second NIC can be configured as a secondary device in case the network connection to the first adapter is not available. The IP address of the secondary device can be the same as the first device or any other IP address accessible from the boot LUN. iscsi network configuration As detailed in the Network Architecture section, a vnetwork Distributed Switch is used for network connections from servers. It was recommended in that section to use load-based teaming on port groups for balancing network load. iscsi port groups, however, are configured differently. To provide redundant multiple paths to the storage array, VMware requires the use of two separate VMkernel port groups. Each port group has one active dvuplink, while the other is unused. With this configuration, traffic is routed across both active uplinks to a single target and enables use of VMware Multipathing Plugins. Other considerations for configuring the iscsi port groups include: Spanning Trees Protocol: EqualLogic best practices recommend disabling STP on the ports that connect end nodes. If STP is used for connecting to the core network, then the portfast setting should be enabled on these interfaces that transition a port to the forwarding state on linkup. Jumbo frames: Jumbo frames can be enabled in the vds and individual VMkernels used for iscsi traffic. They also need to be enabled at the switch ports, and are enabled by default on the EqualLogic ports. EqualLogic Multipathing Extension Module VMware introduced Pluggable Storage Architecture (PSA) in vsphere 4, which enabled storage vendors to develop Multipath Extension Modules (MEM) that plug into vsphere. The MEMs can provide advanced multipathing capabilities with path redundancy and improved performance by intelligently routing data. The EqualLogic MEM consists of two components: a Path Selection Plugin (PSP) driver that is used by the VMware Native Multipathing Plugin (NMP) to select the best path for each I/O to the EqualLogic storage devices and an EqualLogic Host Connection Manager (EHCM) that manages iscsi sessions to the EqualLogic storage devices. By utilizing the PSP in the EqualLogic MEM, storage I/O is improved with increased bandwidth and reduced network latency. This is because the PSP can use its knowledge of the exact distribution of volumes in the storage arrays. The EHCM provides automatic connection management by distributing iscsi sessions across all configured adapters and PS Series group Ethernet ports. It also automates load balancing across multiple active paths to the storage array. EqualLogic MEM consists of a setup.pl script for configuring the module and an offline bundle to be installed on the ESXi server. The bundle can be installed through the vcenter Update Manager or from a vsphere Management Assistant server using the setup script. As described above, vsphere Multipathing requires configuring two VMkernels, each with active and unused dvuplinks. The setup script can be used to configure a new vswitch or vds with this setup. However, with an already complete network configuration, the EqualLogic MEM can use that configuration if VMware Round Robin Multipathing is used to access one of the volumes. Page 15

18 Volume Size Considerations Volume sizes depend on the customer environment and the type of workloads. Volumes must be sized to accommodate not only the VM hard drive, but also the size of the virtual memory of the VM with enough space for any snapshots of the VM. Depending on the environment, you may decide to create multiple ~500 GB volumes with multiple VMs. It is important to include space for the guest operating system memory cache, snapshots, and VMware configuration files when sizing these volumes. Additionally, you can configure thin-provisioned volumes to grow on demand only when additional storage is needed for those volumes. Thin provisioning can increase the efficiency of storage utilization. The size of boot volumes can also vary based on the amount of scratch storage and swap partition required. VMware recommends a 6GB volume without a defined VMFS partition or datastore. Array RAID Configuration The storage array RAID configuration is highly dependent on the workload in your virtual environment. The EqualLogic PS series storage arrays support four RAID types: RAID 5, RAID 6, RAID 10, and RAID 50. The RAID configuration will depend on workloads and customer requirements. In general, RAID 10 provides the best performance at the expense of storage capacity. RAID 10 generally provides better performance in random I/O situations, and requires additional overhead in the case of a drive failure scenario. RAID 50 generally provides more usable storage, but has less performance than RAID 10. RAID 5 provides the most storage capacity at the expense of a little lower performance and availability, but is superseded by RAID 6 on performance. For more information on configuring RAID in EqualLogic, refer to the white paper: How to Select the Correct RAID for an EqualLogic SAN Scalability As workloads increase, the reference configuration can be scaled to provide additional resources. In this section, guidelines are provided for adding servers and storage arrays to the reference configuration. The main bottleneck to scalability in this architecture is the number of switch ports available for connections to storage and servers. Each switch needs one port per server and two ports per storage array for fully-redundant connections. Another constraint on the availability of ports will be the size of the LAG and the number of ports from each switch used for uplinks. EqualLogic recommends that for five or fewer arrays connected to switches, there should be 20 Gbps to 30 Gbps bandwidth between the switches. For six or more arrays, 40 Gbps to 60 Gbps bandwidth between the switches is recommended. Note that the LAG requirements may vary depending on the I/O requirements of the applications running in the VMs. The number of uplinks to each switch will vary based on whether STP or MSTP is used and the classes of traffic routed through the LAG and those routed through the core network. The bandwidth requirements of the VMs are also important in determining the number of uplinks. Based on these constraints and the compute and storage requirements for the configuration, the ideal number of servers and storage arrays can be calculated. For example a configuration with ten servers and five storage arrays can be used with a three-port LAG and one port for uplinks from each switch. If the compute requirement is higher, a configuration with twelve servers and four arrays can be used with a two-port LAG and two ports for uplinks. The connections for individual switch ports for each of these configurations are shown in Table 2. Page 16

19 Note that these configurations assume the management server connects to the core network to maximize the ports available for connections to compute servers and storage. If the number of ports available in the two switches is insufficient for connecting all the servers and storage arrays, a tiered configuration can be used. In this configuration, the storage arrays are connected to a pair of switches. These switches will have a LAG for iscsi traffic between the arrays. These switches will in turn be connected pairwise to two more switches, which will connect to the servers. The LAG between the first pair of switches can also be configured to support vmotion and Fault Tolerance logging traffic. Alternately, if MSTP is used, then loops can be avoided by designing separate spanning trees such that iscsi traffic is routed through the LAG between the switches connected to the storage arrays, while other traffic is routed through either the core network or a LAG between the pair of switches connected to the servers. This configuration is shown in Error! Reference source not found.. Table 2: Switch port connections for two sample configurations Configuration Server Ports Storage Ports LAG Ports Uplink Ports 12 servers, 4 arrays, 2 uplinks , 22 23,24 10 servers, 5 arrays, 1 uplink Figure 6: Scalability configuration with more than two switches Page 17

20 Page 18

21 Management This section describes the various components used to manage the proposed solution. Virtualization Management using VMware vcenter Server VMware vcenter Server provides a single control and management center for VMware ESXi Servers. Capabilities include the ability to manage all vsphere hosts and VMs; utilize advanced features such as vmotion, HA, and DRS; monitor performance and resource optimization; and use standard templates to provision new servers in minutes. For more information on vcenter Server, see Best Practices for vcenter Server Configuration Install vcenter Server on the dedicated management server. It is highly recommended to use an external database for vcenter Server. After installation, it can easily be accessed remotely using vsphere Client. For information on Setup and Configuration, see the ESXi Installable and vcenter Server Setup Guide at Systems Management The following tools and software can be used to manage the hardware: Dell OpenManage: Dell OpenManage Server Administrator (OMSA) can be installed on the VMware ESX hypervisor and used to manage the hardware. For more information on Dell OpenManage and its capabilities, see Deployment and change management using Lifecycle Controller: The Lifecycle Controller is the engine for advanced embedded management, and is delivered as part of the optional idrac Express or idrac Enterprise in the PowerEdge R810. The Lifecycle Controller is a chip that is integrated on the server. It helps to simplify administrator tasks by performing a comprehensive set of provisioning functions such as system deployment, system updates, hardware configuration, and diagnostics from a single, intuitive interface called the Unified Server Configurator (USC) in a pre-os environment. Out-of-band idrac: The integrated Dell Remote Access Controller (idrac) provides a single, secure interface to manage inventory, configuration, monitoring, and alerting for server components. It also provides flexibility to remotely manage the server through console redirection and Virtual CD- ROM/DVD/Floppy/Flash capabilities. Dell Management Console: The Dell Management Console (DMC), powered by Symantec, is a systems management solution designed to provide a central console offering features from basic hardware management to advanced enterprise functionality. The DMC creates a foundation for more advanced management functionality, and provides a single view into the deployment, inventory, monitoring, and updating of your IT infrastructure including servers, storage, desktops, notebooks, network devices, printers, and other non-dell devices. Storage Management EqualLogic storage arrays provide a rich set of management features that are available at no additional cost and come with easy-to-use management tools. For more information on features and management capabilities, see Page 19

22 Switch Management Dell OpenManage Network Manager is a comprehensive, centralized network management solution designed to help customers: Minimize the time, complexity, and cost of network management Increase overall network efficiency Effectively manage corporate networks across platforms Maximize their IT investment Deploying and Configuring the Solution This section provides guidelines for the initial deployment and configuration of the solution. Rack, Stack, and Cable 1. Rack the servers and switches 2. Rack the EqualLogic PS6010 storage arrays. 3. Connect power to the servers and storage. 4. Cable the servers and storage. a. Cable the server and storage for Ethernet connectivity as per the architecture. b. Connect the switches with the given configuration. Configuring the Servers 1. Enable VT in the BIOS. 2. Configure idrac with a static IP address (preferable). 3. Configure USC for self-update and firmware repository. 4. Disable local storage media (hard disk drives, SD card etc.) for boot-from-san install. 5. If not already configured, enable iscsi boot through Broadcom adapters using the Broadcom DOS utilities available at Configuring Networking 1. Configure the Dell PowerConnect 8024F switches. The ports should be configured based on whether they will be used to connect the servers or storage as described earlier. It is recommended that a configuration file be used for this. Configuring Storage Configure the EqualLogic PS storage array. For information on configuring storage, see the Dell EqualLogic PS QuickStart Guide at (registration required). 1. Create iscsi boot volumes. 2. Create access lists for these volumes, restricting access for each volume to the corresponding iscsi IP addresses configured at the servers for iscsi boot. 3. Create datastore volumes. 4. Add the VMkernel iscsi IPs to the access list. (Use * to denote the IP address range.) 5. Modify the datastore volume settings to allow simultaneous connections from multiple servers (clusters). ESXi installation and configuration 1. Change the boot order to have iscsi boot first followed by DVD. 2. Configure the Broadcom iscsi boot parameters, either through the boot menu or using USC. Page 20

23 3. Set the Boot to iscsi target parameter in the Broadcom iscsi boot menu to One Time Disabled. 4. Boot to the iscsi volume through the Broadcom adapter. 5. Choose the EqualLogic volume to install the hypervisor during the ESXi DVD installation. 6. After installation and first boot, the hypervisor is assigned the iscsi initiator IP address. To change the management IP address, connect to the server from a vsphere Client, and add a new VMkernel with the Management IP address to the existing vswitch with the iscsi IP address. More detailed instructions about iscsi boot configuration and setup can be found in the VMware iscsi SAN Configuration Guide at under the section Booting from iscsi SAN. Configuring vcenter Installing a SQL database is required before installing VMware vcenter. See the VMware ESXi 4.1 Installable and vcenter Server Setup Guide at to install vcenter on the management server. Deployment Steps 1. Create a cluster in vcenter with DRS and DPM features enabled. 2. Add the ESXi servers to the cluster using the IP address of the Management network configured after first boot of the server 3. Place the servers in maintenance mode. 4. Configure the vnetwork Distributed Switches as described in the Network architecture section above, and add to the server. 5. Attach the vmkernel NICS to the software iscsi initiators. 6. Add an EqualLogic volume to be used for storage as a datastore to each of the servers. 7. Change the path selection policy on the datastore to Round Robin. 8. Install the EqualLogic Multipath Extension Module, using either the update manager or the setup.pl script. 9. Reboot the server and confirm that the path selection policy is DELL_PSP_EQL_ROUTED. The EqualLogic MEM should take over path selection for all subsequent volumes created and added to the servers. Page 21

24 References VMware Documentation ESXi Configuration Guide iscsi SAN Configuration Guide What s new in vsphere Networking VMware Network I/O Control Best Practices PowerEdge R810 Server Server Manual EqualLogic Documentation PS Series Array Network Connection and Performance Guidelines Configuring and Installing the EqualLogic Multipathing Extension Module for VMware vsphere 4.1 and PS Series SANs Feedback We are interested in your feedback regarding this document. If you identify any errors, including typographical, grammatical, or architectural errors, or if you have any questions, please us at [email protected]. Page 22