Dell FlexAddress for PowerEdge M-Series Blades



Similar documents
CUTTING-EDGE SOLUTIONS FOR TODAY AND TOMORROW. Dell PowerEdge M-Series Blade Servers

Dell PowerEdge Blades Outperform Cisco UCS in East-West Network Performance

Manage Dell Hardware in a Virtual Environment Using OpenManage Integration for VMware vcenter

Dell idrac7 with Lifecycle Controller

Direct Attached Storage

Converged Networking Solution for Dell M-Series Blades. Spencer Wheelwright

Building Microsoft Windows Server 2012 Clusters on the Dell PowerEdge VRTX

Enhanced IBM BladeCenter Open Fabric Manager offers an easy to use Web-based interface to easily configure, deploy, and manage BladeCenter products

RACADM Command Line Reference Guide for idrac6 1.95, idrac6 3.5, and CMC 3.2

Compellent Storage Center

Best Practices Guide: Network Convergence with Emulex LP21000 CNA & VMware ESX Server

HBA Virtualization Technologies for Windows OS Environments

Windows Host Utilities Installation and Setup Guide

Feature Comparison: idrac 7 & 8 and idrac8 License Chart

Dell DR4000 Disk Backup System. Introduction to the Dell DR4000 Restore Manager A primer for creating and using a Restore Manager USB flash drive

Dell PowerVault MD3400 and MD3420 Series Storage Arrays Deployment Guide

Reboot the ExtraHop System and Test Hardware with the Rescue USB Flash Drive

SAN Conceptual and Design Basics

Private cloud computing advances

IBM BladeCenter H with Cisco VFrame Software A Comparison with HP Virtual Connect

Management Software. Web Browser User s Guide AT-S106. For the AT-GS950/48 Gigabit Ethernet Smart Switch. Version Rev.

DELL. Unified Server Configurator: IT and Systems Management Overview. A Dell Technical White Paper

What is VLAN Routing?

Using EonStor FC-host Storage Systems in VMware Infrastructure 3 and vsphere 4

Dell Server Management Pack Suite Version 6.0 for Microsoft System Center Operations Manager User's Guide

Enhancements to idrac7 Alert Notification

Proactively Managing Servers with Dell KACE and Open Manage Essentials

Installing idrac Certificate Using RACADM Commands

N_Port ID Virtualization

EMC ViPR Controller. User Interface Virtual Data Center Configuration Guide. Version REV 01

Cascade Profiler Fibre Channel SAN Attached Storage Installation Guide. Version 8.2 March 2009

Emulex 8Gb Fibre Channel Expansion Card (CIOv) for IBM BladeCenter IBM BladeCenter at-a-glance guide

Remote Installation of VMware ESX Server Software Using Dell Remote Access Controller

HP Converged Infrastructure Solutions

Achieve Automated, End-to-End Firmware Management with Cisco UCS Manager

Deploying SAP on Microsoft SQL Server 2008 Environments Using the Hitachi Virtual Storage Platform

Emulex OneConnect NIC Teaming and Multiple VLAN Driver and Application Release Notes

Agent-free Inventory and Monitoring for Storage and Network Devices in Dell PowerEdge 12 th Generation Servers

DVS Enterprise. Reference Architecture. VMware Horizon View Reference

Intel Entry Storage System SS4000-E

Windows Host Utilities 6.0 Installation and Setup Guide

DELL. Unified Server Configurator Security Overview. A Dell Technical White Paper. By Raja Tamilarasan, Wayne Liles, Marshal Savage and Weijia Zhang

SAN Implementation Course SANIW; 3 Days, Instructor-led

Dell High Availability Solutions Guide for Microsoft Hyper-V

Managing Web Server Certificates on idrac

OneCommand Manager Application for Windows Release Notes

How To Design A Data Centre

Brocade Enterprise 20-port, 20-port, and 10-port 8Gb SAN Switch Modules IBM BladeCenter at-a-glance guide

Express5800/320Lb System Release Notes

A Whitepaper on. Building Data Centers with Dell MXL Blade Switch

IP SAN Best Practices

Dell Compellent Storage Center SAN & VMware View 1,000 Desktop Reference Architecture. Dell Compellent Product Specialist Team

Systems Management Tools And Documentation Version 8.1 Installation Guide

Using RAID Admin and Disk Utility

Understanding DRAC/MC Alerts

DELL Remote Access Configuration Tool

VTrak A-Class Release Notes R1.00

Integrated Dell Remote Access Controller 8 (idrac8) Version User's Guide

Introducing logical servers: Making data center infrastructures more adaptive

Fibre Channel HBA and VM Migration

Setup for Microsoft Cluster Service ESX Server and VirtualCenter 2.0.1

FIBRE CHANNEL OVER ETHERNET

Quantum Scalar 50 Tape Drive Installation Instructions

Configuring Dell OpenManage IT Assistant 8.0 to Monitor SNMP Traps Generated by VMware ESX Server

HP Virtual Connect: Common Myths, Misperceptions, and Objections

Dell EqualLogic Red Hat Enterprise Linux 6.2 Boot from SAN

Integrated Dell Remote Access Controller 7 (idrac7) Version User's Guide

Installing an OS on a Server

Quick Start Guide. for Installing vnios Software on. VMware Platforms

EMC Virtual Infrastructure for SAP Enabled by EMC Symmetrix with Auto-provisioning Groups, Symmetrix Management Console, and VMware vcenter Converter

3.5 EXTERNAL NETWORK HDD. User s Manual

Dell Server Management Pack Suite Version For Microsoft System Center Operations Manager And System Center Essentials User s Guide

How to Setup and Configure ESXi 5.0 and ESXi 5.1 for OpenManage Essentials

VMware vsphere 5 On Dell PowerEdge Systems Deployment Guide

3G Converged-NICs A Platform for Server I/O to Converged Networks

The Benefit of Migrating from 4Gb to 8Gb Fibre Channel

Dell PowerVault MD32xx Deployment Guide for VMware ESX4.1 Server

Exploring the Remote Access Configuration Utility

idrac7 Version With Lifecycle Controller 2 Version 1.1 Quick Start Guide

Dell Networking Solutions Guide for Microsoft Hyper-V

High Performance Tier Implementation Guideline

Integration of Microsoft Hyper-V and Coraid Ethernet SAN Storage. White Paper

Quick Start Guide. Cisco Small Business. 200E Series Advanced Smart Switches

How to Set Up Your NSM4000 Appliance

QLogic 3810, QLogic 5000 Series, and QLogic 5800V Series Switch Firmware. Version Table of Contents

Dell OpenManage Mobile Version 1.4 User s Guide (Android)

Using VMWare VAAI for storage integration with Infortrend EonStor DS G7i

QLogic 4Gb Fibre Channel Expansion Card (CIOv) for IBM BladeCenter IBM BladeCenter at-a-glance guide

SAP CRM Benchmark on Dual-Core Dell Hardware

EMC Integrated Infrastructure for VMware

Using Cisco UC320W with Windows Small Business Server

integrated lights-out in the ProLiant BL p-class system

The HBAs tested in this report are the Brocade 825 and the Emulex LPe12002 and LPe12000.

Management of VMware ESXi. on HP ProLiant Servers

PS Series Storage Array - Configuration, Operation and Management

Converting SSG 300M-series and SSG 500M-series Security Devices to J-series Services Routers with a USB Storage Device

Load Balancing for Microsoft Office Communication Server 2007 Release 2

Integrating idrac7 With Microsoft Active Directory

HP OneView Administration H4C04S

AT-S84 Version ( ) Management Software for the AT-9000/24 Gigabit Ethernet Switch Software Release Notes

Transcription:

Dell FlexAddress for PowerEdge M-Series Blades June 16, 2008 Authored By: Rick Ward, Mike J Roberts and Samit Ashdhir Information in this document is subject to change without notice. Copyright 2008 Dell Inc. All rights reserved. Reproduction in any manner whatsoever without the written permission of Dell Inc. is strictly forbidden. THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. Dell, the Dell Logo, PowerEdge and OpenManage are trademarks of Dell Inc. Microsoft is a registered trademark of Microsoft Corporation. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell disclaims proprietary interest in the marks and names of others.

Table of Contents DELL FLEXADDRESS SIMPLIFYING IT 1 THE TECHNOLOGY 2 THE IMPLEMENTATION 2 FLEXADDRESS FUNCTIONAL COMPONENTS 2 SECURE DIGITAL (SD) CARD 3 CHASSIS MANAGEMENT CONTROLLER (CMC) 4 IO CONTROL PANEL 7 INTEGRATED DELL REMOTE ACCESS CONTROLLER (IDRAC) 7 SYSTEM BIOS 7 DEVICES LAN-ON-MOTHERBOARD (LOM) AND MEZZANINES 7 DEPLOYING FLEXADDRESS ON POWEREDGE M1000E 8 SOFTWARE/FIRMWARE REQUIREMENTS 8 FLEXADDRESS ON POWEREDGE M1000E 9 FLEXADDRESS OPERATION AND USAGE 9 ACTIVATION 9 DEACTIVATION 9 CMC FIRMWARE DOWNGRADE 9 RECOVERY 9 ENABLING/DISABLING 10 OPERATION 10 FLEXADDRESS ASSIGNMENT 11 FLEXADDRESS FORMAT FOR FIBRE CHANNEL DEVICES 11 FLEXADDRESS FORMAT FOR ETHERNET DEVICES 11 WAKE-ON-LAN WITH FLEXADDRESS 12 VMWARE ESX CONFIGURATIONS WITH FLEXADDRESS 13 APPENDIX A: FLEXADDRESS ERROR LOGGING 14 NORMAL OPERATION 14 DOWN REV SERVER BIOS 15 DOWN REV IDRAC FIRMWARE 17 DOWN REV MEZZANINE SOFTWARE 19 APPENDIX B: REFERENCES 21 SOFTWARE DOWNLOAD LINKS 21

Dell FlexAddress Simplifying IT Maintaining availability of network and storage resources is one of the biggest challenges faced by server administrators. Availability becomes a key when there are mission critical applications deployed on these resources. Application availability is directly impacted by how the network and storage resources are configured, especially when changes take place. Any downtime associated with the configuration of these resources impacts availability. Modular systems, such as Dell PowerEdge M1000e, provide a great option to address availability requirements at the hardware level. Modular systems have inherent capabilities for hot plugging of modules without the need to bring down an entire chassis. When server modules are swapped however, the networking (Ethernet) and storage (Fibre Channel/iSCSI) devices within them (including their unique hardware identifiers (Ethernet/iSCSI MAC/FC WWN) are changed as well. When this happens, configuration for these devices on external switches, deployment frameworks, DHCP servers, and storage arrays must be updated to account for the new unique identifiers associated with these new controllers. Depending upon the tools and policies used by the network and storage administrators; accounting for these changes can require significant amounts of time and effort and have significant impact to the availability of compute resources. FlexAddress for Dell M-Series products is a patent pending technology that helps solve such resource availability issues associated with changes in system configurations. FlexAddress simplifies management of network and storage resources in the data centers by providing the following key benefits: Swap out or upgrade Dell PowerEdge M-Series server blades with Ethernet, iscsi, & Fibre Channel controllers seamlessly without affecting the LAN/SAN. No reconfiguration of deployment frameworks, DHCP servers, MAC based licensing schemes, or SAN Zoning FlexAddress is administered by the Chassis Management Controller (CMC); therefore it is completely independent of the IO Mezzanine cards on the blade and IO modules in the enclosure. Customers get the benefit of FlexAddress across all Ethernet controllers and Fibre Channel HBAs, as well as Cisco, Brocade, and Dell chassis IO modules (including Pass-Through Modules) FlexAddress is centrally managed, enabled, and configured at the chassis level via web and CLI interfaces, so it is extremely simple to use. In addition, the CMC GUI and CLI present users a consolidated view of all MAC/WWNs within a system. Each FlexAddress enablement card comes provisioned with a unique pool of Ethernet/iSCSI MACs and Fibre Channel WWNs. This ensures that there are never duplicate MAC/WWNs on the network without any additional configuration required by the administrator. FlexAddress is deployed to the server s controllers during the first part of the boot process, so the persistent MACs/WWNs are assigned and used immediately when the server comes up.

The Technology FlexAddress is an optional feature for the Dell PowerEdge M1000e system that, when enabled, substitutes UNIQUE chassis assigned, slot persistent addresses in place of the factory programmed protocol specific addresses on Ethernet and Fibre Channel devices while blades are within a FlexAddress enabled chassis. Both factory and chassis assigned addresses are 48-bit IEEE Standard 802.1A Universal LAN MAC Addresses (ULA) for Ethernet devices and World Wide Names (WWN) for Fibre Channel devices. The Chassis Management Controller (CMC) in Dell PowerEdge M1000e manages assignment of the FlexAddress MAC/WWNs to all supported devices prior to the blades boot up. This feature supports deploying chassis assigned, slot persistent addresses for both the LAN- On-Motherboard (LOM) and optional IO expansion Mezzanine card Ethernet controllers and Fibre Channel HBAs. FlexAddress assigned MAC/WWNs are blade slot specific and globally unique, complying with the IEEE OUI based addressing format for both Ethernet and Fibre Channel devices. Devices with assigned FlexAddress MAC/WWNs will use them for address reporting in management tools and protocol communication. As a result, no changes are required in the management or supporting tools of a device using the FlexAddress feature. All network configuration tools and utilities will report the FlexAddress MAC/WWNs for a device when it has FlexAddress enabled. The Implementation The Chassis Management Controller (CMC) is responsible for managing all system level management functions for the modules within the PowerEdge M1000e enclosure, including the FlexAddress feature. FlexAddress Functional Components The implementation of FlexAddress in the Dell PowerEdge M1000e involves the following functional components: 1. FlexAddress Secure Digital (SD) feature card 2. CMC 3. IO Control Panel 4. idrac 5. Server Blade BIOS 6. Ethernet/Fibre Channel BIOS/firmware Interaction between these components is depicted in Figure 1 below: 2

Figure 1: FlexAddress Solution Stack SD card CMC I2C IO Control Panel IPMI idrac SysBIOS Device (LOM/Mezzanie) Blade Secure Digital (SD) card A Secure Digital (SD) card is used to deliver the unique MACs/WWNs for deploying FlexAddress feature for Dell M-series products. The data is stored in an encrypted form and is intended to be used only by the Dell PowerEdge M1000e CMC. Users should not alter the data on the SD card in any way, as it may inhibit system function and cause a malfunction. The CMC reads the contents of SD card during initial activation of the FlexAddress feature. During the process of feature activation, CMC validates the contents on feature card and binds the card to that chassis. It is critical to note that an activated feature card for a given chassis cannot be installed into another chassis CMC. Though the feature card CAN be removed and installed in a 3

different chassis IF FlexAddress is disabled on the initial chassis first. Data read from the feature card is stored at a common location, the IO Control Panel, in the chassis for future use and reference. Once the FlexAddress feature has been activated, users may choose to leave the SD card plugged into the CMC for long term storage, or it may be removed. Only one FlexAddress feature card is required for a chassis even if the chassis has redundant CMCs. Chassis Management Controller (CMC) The CMC forms the heart of FlexAddress feature in Dell PowerEdge M1000e system. CMC manages the following functions specific to FlexAddress: 1. Provides user interface for enabling or disabling the FlexAddress feature on a per blade slot basis, a per fabric basis or both 2. Identifies and reports device information for each of the supported fabric types LOMs, Ethernet and Fibre Channel mezzanine cards 3. Validates all the components of the FlexAddress feature SD card validation, System BIOS, IO controller firmware, CMC firmware, and, idrac firmware versions. 4. Reports FlexAddress feature status for each of the blade slots 5. Provides information for both server-assigned (Factory programmed) and chassis-assigned (FlexAddress) addresses on each supported device 6. Logs any system level errors that may prevent the FlexAddress feature from being used on the chassis or on a given device. When a user enables the FlexAddress feature for a given blade slot or a given fabric, CMC will assign FlexAddress MAC/WWNs to devices on enabled fabrics in slots with the feature enabled. If a slot or a fabric already contains an operational blade, it would need to be powered off prior to enabling FlexAddress feature on that blade slot or fabric. FlexAddress enabled blades will receive addressing data from the CMC over IPMI that will eventually be communicated to the end device as shown in FlexAddress solution stack diagram. Please refer to Figure2: Deploying FlexAddress on Fabrics and Server slots for details on deploying FlexAddress on Dell PowerEdge M1000e chassis. It is important to note here that if Wake on LAN (WoL) is to be used, because of how FlexAddress works, the server must first be powered up in order for it to get its FlexAddress MAC and then it can be powered down again. If this is not done, the first time WoL is used, the server will only respond to its server assigned or factory programmed MAC address. For details please refer to section Wake-On-LAN with FlexAddress. 4

Figure2 : Deploying FlexAddress on Fabrics and Server slots Users can enable or disable the FlexAddress feature from either a web based GUI or from a Command Line Interface (CLI). Status of the feature is reported by the CMC in both the web based GUI and CLI. Please refer to Figure 3 FlexAddress Summary for details, which shows FlexAddress is enabled for Fabric A while slot 1 is disabled and slot 2 is enabled. Also observe CMC GUI or CLI will show green check or enabled only against Fabrics which have FlexAddress enabled. In the example below, only Fabric A has FlexAddress enabled while both Fabric B and C have FlexAddress disabled. Any errors encountered by the CMC for the FlexAddress feature are logged in the System Event Log (SEL) or the system log, depending on the severity of the error. Please refer to Appendix A: FlexAddress Error Logging for more details. Prior to assigning FlexAddress MAC/WWNs to a server, the CMC validates the minimum required capabilities for the underlying system components idrac firmware, system BIOS and IO controller firmware. If any of the components are found to be below the 5

required level, FlexAddress MAC/WWNs will not be assigned to the blade. Minimum required firmware versions are listed in section Software/Firmware Requirements. Figure 3: FlexAddress Summary 6

IO Control Panel All system configuration and runtime settings are stored in the system s IO control panel. Once CMC validates the FlexAddress feature card for FlexAddress data, it copies all chassis-assigned addresses to the IO control panel. CMC then uses data from IO control panel to actually assign FlexAddress MAC/WWNs to the individual devices. It is important to note that the IO control panel is a field serviceable component that can be replaced without disrupting the operation of the servers themselves. In the unlikely event of a control panel failure, the control panel can be replaced and reprogrammed with the correct chassis service tag. FlexAddress information would be re-sent from the SD card to the new IO control panel. There will be no changes to the servers MAC/WWN addresses in this case. Integrated Dell Remote Access Controller (idrac) idrac is responsible for communicating FlexAddress data to System BIOS along with any FlexAddress status messages from System BIOS to the CMC. FlexAddress data is communicated to System BIOS in SMASH/CLP format. idrac is also responsible for logging any blade level system event logs (SEL) for FlexAddress features. When a system event is logged, an amber LED on the front of the blade will flash, alerting users about the issue. For details refer to section Appendix A: FlexAddress Error Logging and the PEM600/605 Users guide for processing SEL entries in idrac. System BIOS SMASH/CLP commands received from idrac for FlexAddress are either passed directly to the end device (e.g. Fibre Channel supporting PCI 3.0 Firmware) or parsed by the system BIOS prior to being communicated to the device (e.g. Ethernet Devices LOM and Ethernet Mezzanines). System BIOS is also responsible for validating minimum firmware versions required on the end devices to support the FlexAddress solution as shown in figure FlexAddress Solution Stack above. When the system BIOS finds a device to be below the minimum required level it communicates the error to idrac for logging and reports a POST error. Devices LAN-On-Motherboard (LOM) and Mezzanines The FlexAddress feature is supported on both Ethernet and Fibre Channel based devices. FlexAddress capable devices include: 1. Broadcom 5708/5709 based LAN-On-Motherboard (LOM) controllers 2. Broadcom 5708t Ethernet IO Mezzanine Cards 3. Broadcom 5709tis Ethernet IO Mezzanine Cards 4. Qlogic QME2472 Fibre Channel IO Mezzanine Cards 5. Emulex LPe1105-M Fibre Channel IO Mezzanine Cards 6. All future Ethernet and Fibre Channel IO Mezzanine Cards in development 7

Fibre Channel devices use a SMASH/CLP interface to receive FlexAddress data from System BIOS whereas for Ethernet based devices, System BIOS parses the received SMASH/CLP command and provides data to the devices. When enabled, all the devices retain FlexAddress data across blade reboots. Note: Device will revert back to server-assigned address when it is removed from a FlexAddress enabled slot or chassis. Deploying FlexAddress on PowerEdge M1000e Supporting FlexAddress feature on PowerEdge M1000e REQUIRES all the FlexAddress functional components to be properly installed and updated to recommend minimum software/firmware levels as detailed out in following sections. Software/Firmware Requirements Delivering a successful FlexAddress solution on Dell PowerEdge M1000e requires that ALL the functional components support FlexAddress. Table 1 below identifies minimum firmware and BIOS versions required for a complete FlexAddress solution: Server Blades Component PowerEdge M600 BIOS PowerEdge M605 BIOS Broadcom 5708 based LOM idrac Firmware Software versions BIOS 2.04 or later BIOS 2.06 or later Boot code firmware 4.4.1 or later iscsi boot firmware 2.7.11 or later 1.11 or later CMC CMC Firmware 1.10 or later IO Mezzanine Cards Broadcom 5708 Qlogic QME 2472 FC4 Boot code firmware 4.4.1 or later iscsi boot firmware 2.7.11 or later PXE firmware 4.4.3 or later BIOS 2.04 or later BIOS 3.03a3 and firmware 2.72A2 Emulex LPe1105-M4 FC4 or later Table 1: Minimum Firmware/Software Required to Support FlexAddress Any components not listed above already support the FlexAddress feature, and do not require any additional updates. It is important to note that all server blades, IO mezzanine cards, and CMCs shipped after the release date of FlexAddress will come with the correct levels of software/firmware to support this feature. If component updates are needed, they must be performed in the following order: 1) Update IO Mezzanine cards QME2472, LPe1105-M4 or Broadcom 5708 as applicable. 8

2) Update Server BIOS 3) Update server idrac 4) Update CMC 5) Finally plug in the FlexAddress enablement SD card. FlexAddress on PowerEdge M1000e FlexAddress Operation and usage Activation Activation of the FlexAddress feature is automatic when a CMC containing a FlexAddress feature card is inserted and made active in a chassis that does not already have the FlexAddress feature. During FlexAddress activation, unique FlexAddress MAC/WWNs are copied to the IO Control Panel and the FlexAddress feature card is bound to the chassis service tag. Activation of FlexAddress can be verified with the command racadm feature s. The status of the FlexAddress feature card can be viewed with the command racadm featurecard s which will also report if the card is bound to the chassis. Note: FlexAddress feature will be activated at Dell factory for all the Dell PowerEdge M1000e chassis which were ordered with this feature. Any chassis which was ordered without the FlexAddress feature will be activated once user inserts the FlexAddress feature card into the CMC as discussed above. Deactivation FlexAddress can be deactivated on a chassis by issuing the command racadm feature d c flexaddress. The chassis must be powered down prior to issuing the deactivation command. If the FlexAddress feature card that was used to activate FlexAddress is on the active CMC, it will be returned to an unbound state and it can be used to install FlexAddress on another chassis. The unbound feature card should be removed from the CMC after deactivation, since it may automatically reactivate FlexAddress the next time the CMC is reset with feature card still inserted. CMC Firmware Downgrade If a CMC with FlexAddress activated has its firmware updated to a version prior to 1.10, all configuration related to FlexAddress will be lost. Updating the CMC s firmware to a version that supports FlexAddress will not restore the lost configuration data, although FlexAddress can be reactivated and reconfigured. Please refer to CMC 1.10 Users guide for more details. Recovery FlexAddress configuration data is on the IO Control Panel. If the IO Control Panel is replaced during a service call, the chassis service tag will need to be re-stored in the new 9

part in order to reactivate FlexAddress. Once the chassis service tag has been programmed, the FlexAddress feature card can be reinserted and FlexAddress can be reactivated and reconfigured. Enabling/Disabling After FlexAddress has been activated, the chassis must still be configured to assign FlexAddress MAC/WWNs for each of the server slots. FlexAddress can be enabled or disabled on each network fabric. Additionally, FlexAddress can be enabled or disabled on each server slot separately. In order for a network device on a server to obtain a FlexAddress MAC/WWN, its slot and its fabric must be enabled. Fabrics and slots can be enabled for FlexAddress from the CMC GUI (Servers Setup Deploy FlexAddress) as shown in Figure: Deploying FlexAddress on Fabrics and Server slots of section Chassis Management Controller (CMC), or using the following racadm commands. To enable a fabric: racadm setflexaddr f <fabricname> <state> <fabricname> = A, B or C <state> = 0 or 1 (0 = disable, 1 = enable) To enable a server slot: racadm setflexaddr i <slot #> <state> <slot #> = 1 16 <state> = 0 or 1 Prior to enabling FlexAddress on a fabric, all the blades with a device on that fabric must be powered down. Similarly, enabling FlexAddress on a server slot requires the server in the slot to be powered down. Empty server slots can always be enabled. FlexAddress configuration can be viewed with the command racadm getflexaddr. Operation Once FlexAddress is activated and configured, its operation will be transparent in most cases. Certain network configurations may require refreshing of network tables, such as ARP table on IP switches and routers in order for new MAC/WWNs to be logged in by the switch or router. Depending on the network topologies and setup these tables may refresh automatically. Also using Wake-on-LAN with FlexAddress necessitates the server being power cycled before it will respond to its FlexAddress assigned MAC/WWN. This exception is discussed in the section Wake-On-LAN with FlexAddress. The addresses being used by each server can be viewed on the CMC GUI (Servers Properties WWN/MAC) as shown in Figure FlexAddress Summary of section Chassis Management Controller (CMC) or with the following racadm command: racadm getflexaddr i <slot #> 10

<slot #> = 1-16 FlexAddress Assignment FlexAddress data on a PowerEdge M1000e chassis is blade slot specific. CMC maintains a database of pre-allocated globally unique 48-bit IEEE Standard 802.1A Universal LAN MAC Addresses (ULA). Once the FlexAddress feature is enabled on the CMC, these MAC addresses are pre-assigned for each of the 16 blade slots on a given chassis. Depending on the device type Ethernet vs. Fibre Channel installed on a blade, CMC assigns addresses to the device from the block of pre-assigned address. Ethernet device will receive a MAC address, while a Fibre Channel device will receive a WWPN for each of the ports. Format for these addresses will be as described in following sections. FlexAddress Format for Fibre Channel Devices CMC assigns World Wide Port Name (WWPN) for Fibre Channel devices per Fibre Channel Framing and Signaling specifications. WWPN provided by CMC will be in IEEE Extended format with NAA set to 2h. A multi-port device will be assigned WWPN for each of the ports. Devices with FlexAddress assigned WWNs will use this address for all protocol and address reporting functions. Devices will also generate World Wide Node Name (WWNN) based on the assigned WWPN. E.g. If a Fibre Channel device has 2 ports then, CMC may assign following WWPN for each of the ports: Port1 = 0x200100ACDE4B5078 // WWPN for Port 1 Port2 = 0x200200ACDE4B5078 // WWPN for Port 2 Based on above WWPN, Mezzanine HBA will report WWNN as: WWNN = 0x200000ACDE4B5078 // WWNN for device FlexAddress Format for Ethernet Devices CMC assigns Media Access Controller (MAC) addresses for Ethernet devices per IEEE802 standard. MAC address assigned by CMC is a 48-bit IEEE Standard 802.1A Universal LAN MAC Address (ULA). A multi-port Ethernet device will be assigned MAC address for each of the ports. Ethernet device assigned with FlexAddress MACs will use these persistent MAC address for all protocol level communications and address reporting, including device management. Ethernet devices Broadcom 5708 LOM and IO Mezzanine for Dell PowerEdge M1000e systems can operate in following modes: Ethernet Function only Ethernet with iscsi HBA (iscsi offload) function Operating the device with iscsi HBA function may require additional licensing depending on the device. Please refer to Users documentation for more details on iscsi HBA function. 11

Irrespective of the iscsi HBA licensing, the CMC will always assign and report a set of two FlexAddress MACs per port for Ethernet devices (1 Ethernet and 1 iscsi). Devices with iscsi function disabled would only use one of these FlexAddress assigned MACs and ignore other addresses. Please refer to Figure 4 below, which shows ports associated with fabric A1 and A2 which are assigned two FlexAddress MACs each. Note: Second MAC address for a given port will always be for iscsi interface. As shown in Figure 4 below, MAC addresses 00:1C:23:CC:12:45 and 00:1C:23:CC:12:47 are iscsi addresses for fabric A1 and A2. Their corresponding FlexAddress are 00:1D:09:FF:BA:D0 and 00:1D:09:FF:BA:02 respectively. Figure 4: FlexAddress Summary showing Ethernet and iscsi MAC assignment Wake-On-LAN with FlexAddress Ethernet devices with FlexAddress enabled will substitute factory programmed addresses with chassis assigned addresses. All protocol specific communication will utilize FlexAddress instead of the factory programmed address. Note: When the FlexAddress feature is deployed for the first time on a given server, it requires at least one power-down and power-up sequence of the server for FlexAddress 12

to take effect. As discussed above in System BIOS section, FlexAddress for Ethernet devices is programmed by the system BIOS. In order for the system BIOS to program the address, it needs to be operational, which requires server to be powered up. Once the power-down and power-up sequence of the server has been completed, FlexAddress will be available for Wake-On-LAN (WOL) function on the given server. Users may perform a power-cycle sequence on the blade to fully deploy FlexAddress via idrac or CMC interface as detailed in PowerEdge M1000e user documentation. VMware ESX Configurations with FlexAddress In VMware configurations, when the FlexAddress feature is enabled AFTER installing ESX on a blade server, care needs to be taken for refreshing ARP tables on the network switches and routers. Once the FlexAddress feature is enabled on an ESX server, underlying network MAC address will be substituted with a chassis assigned FlexAddress MAC. Depending on network topologies it may take a while for the ARP tables to get refreshed on network switches and routers. During this time, users may experience loss of connectivity to the Virtual Center. Besides letting the ARP table refresh automatically on the switches and routers, users may also force an ARP table refresh by either ping -ing Virtual Center from ESX server or by force-refreshing the table on the switch and router. This procedure is not necessary if FlexAddress is deployed to the server PRIOR to installing VMWare ESX or other applications. 13

Appendix A: FlexAddress Error Logging Normal Operation Depending on the type of error, Dell PowerEdge M1000e system logs error in CMC or in Blade server idrac. In addition to the error logging, system may also blink the LCD panel amber or the blade LED amber. Once all the FlexAddress components have been updated to the required software versions as detailed in Software/Firmware Requirements section, FlexAddress will function normally. The CMC GUI interface will display the FlexAddress state for a given server with all green checks as displayed below in Figure 5. Figure 5: FlexAddress Status Summary 14

Down rev Server BIOS If server BIOS is not updated to the required level as indicated in section Software/Firmware Requirements, CMC would log in a warning and update FlexAddress status accordingly as shown below in Figure 6. Note that since FlexAddress cannot be supported in this case, the green check marks are on the server-assigned MAC/WWNs indicating the FlexAddress chassis-assigned MAC/WWNs are not being used Flexaddress Status Figure 6: FlexAddress Status Page Showing Error due to down-rev BIOS 15

CMC Log In addition to the indications on the FlexAddress status page shown above, CMC will log a warning in CMC log as shown in Figure 7 below: Figure 7: CMC Log Showing Error Due to Down-rev BIOS 16

Down rev idrac firmware If a server s idrac is not updated to the required level, CMC would log in a warning and update FlexAddress status accordingly as indicated in Figure 8 below. Again, note FlexAddress cannot be enabled on this slot due to the idrac firmware and thus the server-assigned MAC/WWNs are being used. FlexAddress Status Figure 8: Server Summary Showing Error Due to Down-rev idrac Firmware 17

CMC Log In addition to the indications on the FlexAddress status page shown above, the CMC will log a warning in CMC log as shown in Figure 9 below: Figure 9: CMC Log Showing Error Due to Down-rev idrac Firmware 18

Down rev Mezzanine Software Enabling FlexAddress feature on a blade with down rev Mezzanine card software will mark the server state Critical and log a idrac System Event Log (SEL) along with a blinking amber LED on both the sever and LCD panel. FlexAddress Status FlexAddress status for a Mezzanine with down rev software version and FlexAddress enabled will have the status as shown in Figure 10 below. Note: Although the status page indicates a green check-mark against Chassisassigned addresses, in reality the down rev device will be using server-assigned addresses because of the down rev software version. Please refer to CMC 1.1 README for more details. Figure 10: Server Summary Showing Error Due Down Rev IO Mezzanine Card Firmware 19

idrac Log idrac will log the event in SEL as shown in Figure 11 below: Figure 11: idrac SEL Log Entry Showing FlexAddress Error Due to Down Rev IO Mezz Card Firmware 20

Appendix B: References Software Download links QME2472 http://driverdownloads.qlogic.com/qlogicdriverdownloads_ui/resourcebyos.aspx?prod uctid=1040&oemid=65&oemcatid=58604 LPe1105-M4 http://www.emulex.com/emc/support/blades/dell_e1105m4.jsp 21