Redpaper. IBM BladeCenter Virtual Fabric Solutions. Front cover. ibm.com/redbooks

Front cover IBM BladeCenter Virtual Fabric Solutions Solutions based on the Emulex Virtual Fabric Adapter and the BNT Virtual Fabric 10Gb Switch Concepts and benefits of virtual NICs (vnics) Step-by-step instructions for how to create various vnic solutions Tom Boucher Scott Lorditch Mark McCool David Watts ibm.com/redbooks Redpaper

International Technical Support Organization IBM BladeCenter Virtual Fabric Solutions July 2010 REDP-4673-00

Note: Before using this information and the product it supports, read the information in Notices on page v. First Edition (July 2010) This edition applies to Emulex Virtual Fabric Adapter (CFFh) for IBM BladeCenter, firmware V2.101.411.7 and BNT Virtual Fabric 10 Gb Switch Module for IBM BladeCenter, firmware V6.3.1.0. Copyright International Business Machines Corporation 2010. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

Contents Notices..................................................................v Trademarks.............................................................. vi Preface................................................................. vii The team who wrote this paper............................................... vii Now you can become a published author, too!.................................. viii Comments welcome........................................................ ix Stay connected to IBM Redbooks............................................. ix Chapter 1. Introduction.................................................... 1 1.1 Virtual NICs overview................................................... 2 1.2 Solution architecture.................................................... 3 1.3 Products that support Virtual Fabric........................................ 5 1.3.1 BladeCenter H.................................................... 6 1.3.2 BNT Virtual Fabric 10Gb Switch Module................................. 8 1.3.3 Emulex Virtual Fabric Adapter....................................... 13 Chapter 2. Configuration basics........................................... 17 2.1 General characteristics................................................. 18 2.2 Basic setup.......................................................... 19 2.3 Creating Virtual NICs.................................................. 19 2.4 vnic group configuration............................................... 22 2.5 Configuring vnic VLANs................................................ 23 2.6 vnic and VMready.................................................... 26 2.7 High availability and vnics.............................................. 27 Chapter 3. Configuring the Virtual Fabric.................................... 31 3.1 Configurations overview................................................ 32 3.2 VFA/VFS vnic correlation.............................................. 34 3.3 Case 1: vnics only, no uplinks........................................... 34 3.4 Case 2: vnics and pnics, no uplinks...................................... 39 3.5 Case 3: vnics only, one uplink........................................... 43 3.6 Case 4: vnics and pnics, one uplink with failover............................ 48 3.7 Case 5: vnics only, one uplink trunk...................................... 52 3.8 Case 6: vnics and pnics, one uplink trunk................................. 58 Chapter 4. Example of use................................................ 65 4.1 Scenario 1........................................................... 66 4.1.1 BNT switch configuration for example 1................................ 67 4.1.2 Upstream router configuration (Cisco)................................. 68 4.1.3 Linux host configuration............................................ 68 4.2 Troubleshooting commands for vnic...................................... 69 Abbreviations and acronyms.............................................. 75 Related publications..................................................... 77 IBM Redbooks........................................................... 77 Other publications........................................................ 77 Online resources......................................................... 77 How to get Redbooks...................................................... 78 Copyright IBM Corp. 2010. All rights reserved. iii

Help from IBM........................................................... 78 iv IBM BladeCenter Virtual Fabric Solutions

Notices This information was developed for products and services offered in the U.S.A. IBM might not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service might be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right might be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-ibm product, program, or service. IBM might have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785 U.S.A. The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Certain states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement might not apply to you. This information can include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM might make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-ibm Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk. IBM might use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you. Information concerning non-ibm products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-ibm products. Questions on the capabilities of non-ibm products should be addressed to the suppliers of those products. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental. COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrate programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. Copyright IBM Corp. 2010. All rights reserved. v

Trademarks IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. These and other IBM trademarked terms are marked on their first occurrence in this information with the appropriate symbol ( or ), indicating US registered or common law trademarks owned by IBM at the time this information was published. Such trademarks might also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at http://www.ibm.com/legal/copytrade.shtml The following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both: AIX BladeCenter IBM Redbooks Redpaper Redbooks (logo) System x Tivoli The following terms are trademarks of other companies: Windows, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both. Intel, Intel logo, Intel Inside logo, and Intel Centrino logo are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Linux is a trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names might be trademarks or service marks of others. vi IBM BladeCenter Virtual Fabric Solutions

Preface The deployment of server virtualization technologies in data centers requires significant efforts in providing sufficient network I/O bandwidth to satisfy the demand of virtualized applications and services. For example, every virtualized system can host several dozens of network applications and services, and each of these services requires certain bandwidth (or speed) to function properly. Furthermore, because of different network traffic patterns relevant to different service types, these traffic flows may interfere with each other, leading to serious network problems including the inability of the service to perform its functions. IBM BladeCenter Virtual Fabric virtual network interface controller (vnic) solution addresses these issues. The solution is based on the IBM BladeCenter H chassis with a 10 Gb Converged Enhanced Ethernet infrastructure built on the BNT Virtual Fabric 10 Gb Switch Modules in the chassis and Emulex Virtual Fabric Adapters in each blade server. In this configuration, each blade server has up to 20 Gb of available network bandwidth with the ability to split it between up to eight vnics starting from 100 Mb to a maximum of 10 Gb per one vnic with 100 Mb increments. This IBM Redpaper publication is for customers who want to learn how to implement an IBM BladeCenter Virtual Fabric vnic solution. It provides step-by-step instruction about how to configure the switch. This paper assumes that you are already familiar with IBM BladeCenter and with the procedures to configure switches such as the BNT Virtual Fabric 10 Gb Switch Module. The team who wrote this paper This paper was produced by a team of specialists from around the world. Tom Boucher is a Senior Systems Engineer with Emulex. Tom has been involved with supporting, designing, and administrating Intel -based computers on numerous operating systems for over 20 years. His primary expertise is high-speed networking, including Fibre Channel, Ethernet, and InfiniBand technologies. In previous roles he was an IBM Product Marketing Manager for IBM System x and BladeCenter servers, and an IBM Field Technical Sales Specialist. Scott Lorditch is a Sales Network Architect for Blade Network Technologies, where he develops customer designs and proposals on Ethernet products for the IBM BladeCenter, including overall network architecture assessments. He has also developed several training and lab sessions for IBM technical and sales personnel and has provided field feedback to the product team. He has over 20 years of experience working in the networking field, including electronic securities transfer projects for a major bank based in New York City. He has worked also as a Senior Network Architect for a multi-national soft drink company, and as a Product Manager for managed hosting services for a large telecommunications provider. He holds a Bachelor of Science degree in Operations Research with a specialization in Computer Science from Cornell University. Mark McCool is an Advisory Engineer at the BladeCenter and Modular Networking Center of Competency at IBM in Research Triangle Park, North Carolina. He has more than 28 years of experience in Communication/Networking ASIC and product development and has been with Copyright IBM Corp. 2010. All rights reserved. vii

IBM for 25 years. His areas of expertise include LAN switching and BladeCenter switch product development. He holds a Bachelor of Science degree in Electrical Engineering Technology from Old Dominion University. David Watts is a Consulting IT Specialist at the IBM ITSO Center in Raleigh. He manages residencies and produces IBM Redbooks publications for hardware and software topics that are related to IBM System x and IBM BladeCenter servers and associated client platforms. He has authored over 80 books, papers, and Web documents. He holds a Bachelor of Engineering degree from the University of Queensland (Australia) and has worked for IBM both in the U.S. and Australia since 1989. David is an IBM Certified IT Specialist and a member of the IT Specialist Certification Review Board. Thanks to the following people for their contributions to this project: Mike Easterly, BNT Kevin Murray, Emulex A portion of this paper comes from the IBM Redbooks publication IBM BladeCenter Products and Technology, SG24-7523, the authors of which are: David Watts Ilia Kroutov Randall Davis Now you can become a published author, too! Here's an opportunity to spotlight your skills, grow your career, and become a published author - all at the same time! Join an ITSO residency project and help write a book in your area of expertise, while honing your experience using leading-edge technologies. Your efforts will help to increase product acceptance and customer satisfaction, as you expand your network of technical contacts and relationships. Residencies run from two to six weeks in length, and you can participate either in person or as a remote resident working from your home base. Find out more about the residency program, browse the residency index, and apply online at: ibm.com/redbooks/residencies.html viii IBM BladeCenter Virtual Fabric Solutions

Comments welcome Your comments are important to us! We want our papers to be as helpful as possible. Send us your comments about this paper or other IBM Redbooks publications in one of the following ways: Use the online Contact us review Redbooks form found at: ibm.com/redbooks Send your comments in an e-mail to: redbooks@us.ibm.com Mail your comments to: IBM Corporation, International Technical Support Organization Dept. HYTD Mail Station P099 2455 South Road Poughkeepsie, NY 12601-5400 Stay connected to IBM Redbooks Find us on Facebook: http://www.facebook.com/ibm-redbooks Follow us on twitter: http://twitter.com/ibmredbooks Look for us on LinkedIn: http://www.linkedin.com/groups?home=&gid=2130806 Explore new Redbooks publications, residencies, and workshops with the IBM Redbooks weekly newsletter: https://www.redbooks.ibm.com/redbooks.nsf/subscribe?openform Stay current on recent Redbooks publications with RSS Feeds: http://www.redbooks.ibm.com/rss.html Preface ix

x IBM BladeCenter Virtual Fabric Solutions

1 Chapter 1. Introduction The deployment of server virtualization technologies in data centers requires significant efforts in providing sufficient network I/O bandwidth to satisfy the demand of virtualized applications and services. For example, every virtualized system can host several dozens of network applications and services, and each of these services requires certain bandwidth (or speed) to function properly. Furthermore, because of different network traffic patterns relevant to different service types, these traffic flows might interfere with each other, leading to serious network problems, including the inability of the service to perform its functions. Providing sufficient bandwidth and isolation to virtualized applications might be particularly challenging for blade-based deployments in which the number of I/O ports per blade is limited. For example, from the maximum of eight ports per blade, at least two ports are used to connect to the external storage. Therefore, only six ports are available, with a total network bandwidth of 6 Gb per blade for Gigabit Ethernet infrastructures, leaving no room for future growth. In addition, the traffic flows are isolated on a physical port basis with a minimum bandwidth of 1 Gb per flow, thus limiting flexibility of bandwidth usage. The IBM BladeCenter Virtual Fabric virtual network interface controller (vnic) solution addresses these issues. The BladeCenter vnic solution is based on the IBM BladeCenter H chassis with a 10 Gb Converged Enhanced Ethernet infrastructure built on the BNT 1 Virtual Fabric 10 Gb Switch Modules in the chassis and Emulex 2 Virtual Fabric Adapters in each blade server. In this configuration, each blade server has up to 20 Gb of available network bandwidth with the ability to split it between up to eight vnics, starting from 100 Mb up to a maximum of 10 Gb per one vnic with 100 Mb increments. In addition, integrated Gigabit Ethernet ports on blade servers can be utilized, and a free CIOv expansion slot provides additional expansion capabilities, for example, to connect the blade server to the external storage. 1 Portions of this chapter are from Blade Network Technologies. Reprinted by Permission. 2 Portions of this chapter are from Emulex. Reprinted by Permission. Copyright IBM Corp. 2010. All rights reserved. 1

1.1 Virtual NICs overview A Network Interface Controller (NIC) is a component within a blade server that allows the server to be connected to a network. The NIC provides the physical point of connection, as well as internal software for encoding and decoding network packets. With the introduction of Virtual Fabric, IBM, Emulex, and BNT established a new feature called virtual NIC. This feature is unrelated to the commonly held concept of virtualization as a hypervisor-based operating system that provides virtual instances. vnic enables virtualizion of the physical interface. Rather than eight physical connections between the adapter and the switch, there are two 10 Gigabit interfaces, and each interface can be presented to the operating system as four physical interfaces. Splitting up a single interface into four is achieved by following the PCIe standard of a PCI DeviceID. Each device can have one ID, and that device can have eight functions. This is nothing new. In fact, many dual or quad port Ethernet or Fibre Channel adapters have a single ASIC chip on them as well, just the physical interfaces are iterated as a function. So in a sense, we have virtualized Layer 1 of the OSI Model, by virtualizing the physical interface. Virtualizing the NIC helps to resolve issues caused by limited NIC slot availability. By virtualizing a 10 Gbps NIC, its resources can be divided into multiple logical instances known as vnics. Each vnic appears as a regular, independent NIC to the server operating system or a hypervisor, with each vnic using a portion of the physical NIC s overall bandwidth. 2 IBM BladeCenter Virtual Fabric Solutions

1.2 Solution architecture Figure 1-1 shows the vnic solution architecture. Different colors represent different traffic flows for different services/applications, and the thickness of the links represent the amount of assigned bandwidth for particular traffic flow. Components and connectivity inside the IBM BladeCenter H chassis I/O Bay 1 I/O Bay 2 I/O Bay 4 I/O Bay 3 Blade Server 14 Blade Server 1 On-Board 1GbE CIOv FC Expansion Card Virtual Fabric Adapter CFFh Legend Standard I/O bays connections vnic virtual connections External physical trunks vswitch virtual trunk connections vnic1 vnic2 vnic3 vnic4 vnic1 vnic2 vnic3 vnic4 BNT Virtual Fabric Switch (Bay 7) INT1.1 EXT1 INT1.2 vswitch1 INT1.3 EXT2 INT1.4 EXT3 vswitch2 EXT4 EXT5 EXT6 vswitch3 EXT7 EXT8 INT14.1 INT14.2 EXT9 INT14.3 vswitch4 INT14.4 EXT10 BNT Virtual Fabric Switch (Bay 9) INT1.1 EXT1 INT1.2 vswitch1 INT1.3 EXT2 INT1.4 EXT3 vswitch2 EXT4 EXT5 EXT6 vswitch3 EXT7 EXT8 INT14.1 INT14.2 EXT9 INT14.3 vswitch4 INT14.4 EXT10 Figure 1-1 BladeCenter Virtual Fabric vnic solution architecture Mid-Plane The Virtual Fabric Adapter (VFA) must be in vnic mode to support the vnic solution (vnic mode is the default mode of operations for the VFA). In such a case, each physical port (pnic) on VFA is split into four virtual vnics that are visible by the operating system as independent NICs. The Virtual Fabric Switch (VFS) recognizes these vnics and manages and isolates network traffic flows between them. Bandwidth allocation and metering is controlled by the VFS, and in cases when incoming or outgoing traffic flow for a particular vnic exceeds established bandwidth limits, the switch begins to drop packets until the traffic falls below configured thresholds. To isolate vnic traffic flows from each other, vnics can be assigned to vnic logical groups (or vswitches) that behave as independent switches inside VFS. That is, vnics in the same group can communicate with each other, whereas vnics in different groups cannot. This is enforced by assigning a unique VLAN ID to every vswitch. In addition to external or internal Chapter 1. Introduction 3

switch ports, external trunks can also be assigned to a particular vnic group to isolate upstream connections. Trunk failover is supported on a per-vnic basis, so you can set up NIC teaming in a failover mode for vnics. In case of external trunk failure, the switch shuts down only those vnics that belong to the same vnic group as the failed external trunk belongs, whereas other vnics on the same pnic port are untouched and continue to perform their functions. vnics are configured on BNT Virtual Fabric Switch by dividing internal VFS ports into sub-interfaces (or logical ports). Each internal interface of VFS can be divided into up to four vnics, and they have strict one-to-one mapping to vnics on VFA. Table 1-1 lists vnic mappings for VFA and VFS as well as vnic identification data to differentiate vnics on the host operating system (vnic PCIe Function ID). Table 1-1 vnic mappings and identification vnic PCIe function ID VFA port number VFS bay number vnic number vnic ID a 0 0 7 1 INTx.1 2 0 7 2 INTx.2 4 0 7 3 INTx.3 6 0 7 4 INTx.4 1 1 9 1 INTx.1 3 1 9 2 INTx.2 5 1 9 3 INTx.3 7 1 9 4 INTx.4 a. x in vnic ID represents the VFS internal port number to which the VFA physical port is connected (x is 1-14). You must take several considerations into account when planning a Virtual Fabric vnic solution: The Virtual Fabric Adapter must operate in vnic mode. Up to four vnics can be configured on each pnic on the VFA (a total of eight vnics on a 2-port VFA). The bandwidth for vnic can be assigned in a range of 100 Mb to 10 Gb with 100 Mb increments (the default is 2.5 Gb). The sum of the bandwidth allocated for all four vnics on the same pnic cannot exceed 10 Gb. Up to 32 vnic groups are supported on VFS. vnics on the same VFA cannot belong to the same vnic group. All vnics on the same VFA physical port must belong to different vnic groups. Only one external port or one trunk made up of external ports is allowed per vnic group. External trunks and ports can belong to at most one vnic group at a time (there is no requirement that trunks and ports belong to a vnic group). Internal ports not connected to a VFA in vnic mode can belong to at most one vnic group at a time. 4 IBM BladeCenter Virtual Fabric Solutions

Virtual machines (VMs) and other virtual entities (VEs) associated with vnics are automatically detected by VFS when VMready in enabled. However, these VEs cannot be assigned to any of VMready s VM groups (because vnic groups used by vnics are fully isolated from other switch entities). Currently, BladeCenter Open Fabric Manager is not supported for vnics (support is planned, however). Tip: All vnic parameters are configured on the BNT Virtual Fabric Switch. 1.3 Products that support Virtual Fabric Table 1-2 lists the components supported by the BladeCenter Virtual Fabric vnic solution. Table 1-2 BladeCenter Virtual Fabric vnic solution support matrix Description Machine type/ part number Feature Quantity Chassis IBM BladeCenter H 8852 None Varies IBM BladeCenter HT 8740/8750 None Varies Blade Servers HS22 7870 None Varies HS22V 7871 None Varies HX5 7872 None Varies Virtual Fabric Adapters Emulex Virtual Fabric Adapter (CFFh) 49Y4235 5755 1 per blade server Virtual Fabric Switches BNT Virtual Fabric 10 Gb Switch Module 46C7191 1639 1 or 2 per chassis External Virtual Fabric Switch connections Copper-based SFP+ Copper Direct Attach Cable - 1 M BN-SP-CBL-1M a 1 per external port SFP+ Copper Direct Attach Cable - 3 M BN-SP-CBL-3M a 1 per external port SFP+ Copper Direct Attach Cable - 7 M BN-SP-CBL-7M a 1 per external port BLADE 1000Base-T SFP BN-CKM-S-T a 1 per external port Fiber Optics-based IBM 10GBase-SR 10 GbE 850 nm Fiber SFP+ Transceiver 44W4408 4942 1 per external port BLADE 1000Base-SX SFP BN-CKM-S-SX a 1 per external port BLADE 10GBase-LR SFP+ BN-CKM-SP-LR a 1 per external port a. This option is ordered from BNT. There is no equivalent IBM Part Number for it. Chapter 1. Introduction 5

1.3.1 BladeCenter H IBM BladeCenter H delivers high performance, extreme reliability, and ultimate flexibility to even the most demanding IT environments. In 9 U of rack space, the BladeCenter H chassis can contain up to 14 blade servers, 10 switch modules, and 4 power supplies to provide the necessary I/O network switching, power, cooling, and control panel information to support the individual servers. The chassis supports up to four traditional fabrics using networking switches, storage switches, or pass-through devices. The chassis also supports up to four high-speed fabrics for support of protocols like 4X InfiniBand or 10 Gigabit Ethernet. The built-in media tray includes light path diagnostics, two front USB inputs, and an optical drive. Figure 1-2 displays the front view of an IBM BladeCenter H. Figure 1-2 BladeCenter H front view 6 IBM BladeCenter Virtual Fabric Solutions

Figure 1-3 displays the rear view of an IBM BladeCenter H. Figure 1-3 BladeCenter H rear view The key features of IBM BladeCenter H chassis are: A rack-optimized, 9 U modular design enclosure for up to 14 hot-swap blades High-availability mid-plane that supports hot-swap of individual blades Two 2,900-watt, hot-swap power modules and support for two optional 2,900-watt power modules, offering redundancy and power for robust configurations Two hot-swap redundant blowers, and six or 12 supplemental fans with power supplies An Advanced Management Module that provides chassis-level solutions, simplifying deployment and management of your installation Support for up to four network or storage switches or pass-through modules Support for up to four bridge modules A light path diagnostic panel, and two USB 2.0 ports Serial port breakout connector Support for UltraSlim Enhanced SATA DVD-ROM and Multi-Burner Drives IBM Director and Tivoli Provisioning Manager for OS Deployments for easy installation and management Energy-efficient design and innovative features to maximize productivity and reduce power usage Density and integration to ease data center space constraints Chapter 1. Introduction 7

Help in protecting your IT investment through IBM BladeCenter family longevity, compatibility, and innovation leadership in blades Support for the latest generation of IBM BladeCenter blades, helping provide investment protection 1.3.2 BNT Virtual Fabric 10Gb Switch Module The BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter offers the most bandwidth of any blade switch and represents the perfect migration platform for clients who are still at 1 Gb outside the chassis by seamlessly integrating in the existing 1 Gb infrastructure. This is the first 10Gb switch for IBM BladeCenter that supports converged networking (that is, is able to transmit Converged Enhanced Ethernet (CEE) to a Fibre Channel over Ethernet (FCoE) capable top-of-rack switch). This new feature is available with firmware Release 6.1. In addition, this switch is a key part of the IBM Virtual Fabric offering, which allows clients to form eight virtual network interface controllers (NICs) from one physical NIC and to manage them in virtual groups. This switch can be managed via a command-line interface (CLI) or a graphical interface of the switch and in the future with BladeCenter Open Fabric Manager, providing all the benefits of I/O Virtualization at 10 Gb speeds. If you have a chassis with multiple servers, several operating at 1 Gb, several at 10 Gb, and several transmitting converged packets, this single switch can handle all these workloads and can connect to a 1Gb infrastructure or a 10 Gb infrastructure or both. With the extreme flexibility of the BNT switch, clients can take advantage of the technologies that they require for multiple environments. For 1 Gb uplinks, they can take advantage of SFP transceivers. For 10 Gb uplinks, they have a choice of either SFP+ transceivers (SR or LR) for longer distances or direct-attached copper (DAC) cables (also known as twin-ax active cables) for shorter distances. DAC cables are more cost-effective and consume less power and can be up to 7 meters in length. They are ideal for connecting chassis together, connecting to a top-of-rack switch, or even connecting to an adjacent rack. Figure 1-4 shows the switch module. Figure 1-4 BNT Virtual Fabric 10Gb Switch Module (BNT, Reprinted by Permission) 8 IBM BladeCenter Virtual Fabric Solutions

Table 1-3 lists the part number to use to order the module. Table 1-3 BNT Virtual Fabric 10Gb Switch Module part number and feature code for ordering Description Part number Feature code BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter 46C7191 1639 The part numbers include the following items: One BNT Virtual Fabric 10Gb Switch Module Three-meter mini-usb-to-db9 serial console cable One filler module BNT Virtual Fabric 10Gb Switch Module Installation Guide BNT user license agreement Important Notices document Documentation CD-ROM Tip: Small form-factor pluggable plus (SFP+) transceivers are not included and must be purchased separately. To communicate outside of the chassis, you must have either SFP+ transceivers or SFP+ direct-attach cables (DAC) connected. Direct-attach cables have SFP+ transceivers on both ends. You have the flexibility to expand your bandwidth as desired, using anywhere from one connection up to 10 connections per switch. Table 1-4 lists the part number to use to order the SFP+ transceivers from IBM. Table 1-4 IBM part numbers for ordering SFP+ transceivers Description Part number Feature code IBM 10GBase-SR 10GbE 850 nm Fiber SFP+ Transceiver 44W4408 4942 Table 1-5 lists additional transceivers and direct-attach cable options that are available directly from BNT. Table 1-5 BNT part numbers for ordering SFP-based transceivers and cables Description BLADE 1000Base-T SFP BLADE 1000Base-SX SFP BLADE 10GBase-LR SFP+ SFP+ Copper Direct Attach Cable - 1 M SFP+ Copper Direct Attach Cable - 3 M SFP+ Copper Direct Attach Cable - 7 M Part number BN-CKM-S-T BN-CKM-S-SX BN-CKM-SP-LR BN-SP-CBL-1M BN-SP-CBL-3M BN-SP-CBL-7M Chapter 1. Introduction 9

The BNT Virtual Fabric 10Gb Switch Module includes the following features and functions: Form-factor Single-height, high-speed switch module Internal ports Fourteen internal auto-negotiating ports: 1 Gb or 10 Gb to the server blades Two internal full-duplex 100 Mbps ports connected to the management module External ports Up to ten 10 Gb SFP+ ports (also designed to support 1 Gb SFP if required, flexibility of mixing 1 Gb/10 Gb) One 10/100/1000 Mb copper RJ-45 used for management or data An RS-232 mini-usb connector for serial port that provides an additional means to install software and configure the switch module Scalability and performance Autosensing 1 Gb/10 Gb internal and external Ethernet ports for bandwidth optimization Non-blocking architecture with wire-speed forwarding of traffic and full line rate performance of 480 Gbps full duplex Media access control (MAC) address learning: automatic update, supports up to 32 Kb MAC addresses Up to 128 IP interfaces per switch Static, EtherChannel, and LACP (IEEE 802.3ad) link aggregation, up to 100 Gb of total bandwidth per switch, up to 18 trunk groups, and up to eight ports per group Support for jumbo frames (up to 12288 bytes) Broadcast/multicast storm control IGMP snooping for limit flooding of IP multicast traffic (IGMP V1, V2, and V3) Configurable traffic distribution schemes over trunk links based on source/destination IP addresses, MAC addresses, or both Fast port forwarding and fast uplink convergence for rapid STP convergence Availability and redundancy VRRP for Layer 3 router redundancy IEEE 802.1D STP for providing Layer 2 redundancy with PVRST+ IEEE 802.1s Multiple STP (MSTP) for topology optimization, up to 128 STP instances supported by single switch IEEE 802.1w Rapid STP (RSTP), providing rapid STP convergence for critical delay-sensitive, traffic-like voice or video Layer 2 Trunk Failover to support active/standby configurations of network adapter teaming on blades Interchassis redundancy (Layer 2 and Layer 3) 10 IBM BladeCenter Virtual Fabric Solutions

VLAN support Up to 1024 VLANs supported per switch, VLAN numbers ranging from 1 to 4095 (4095 used for the management module s connection only) 802.1Q VLAN tagging support on all ports Protocol-based VLANs Security VLAN-based, MAC-based, and IP-based access control lists (ACLs) 802.1X port-based authentication Multiple user IDs and passwords User access control Radius, TACACS+, LDAP Quality of service (QoS) Up to eight queues per port Support for IEEE 802.1p, IP ToS/DSCP, and ACL-based (MAC/IP source and destination addresses, VLANs) traffic classification and processing Traffic shaping and re-marking based on defined policies Eight Weighted Round Robin (WRR) priority queues per port for processing qualified traffic Layer 3 functions IP forwarding IP filtering with ACLs (up to 4096 ACLs supported) VRRP for router redundancy Support for up to 128 static routes Routing protocol support (Router Information Protocol (RIP) V1, RIP V2, OSPF V1, V2, and V3, BGP-4), up to 1024 entries in routing table IPv6 routing including static routes and OSPFv3 (requires firmware V6.3, available as of May 2010) Support for DHCP Relay IPv6 host management IPv6 forwarding based on static routes Manageability Simple Network Management Protocol (SNMP V1, V2, and V3) HTTP/HTTPS browser GUI Industry standard CLI and BLADEOS/AlteonOS CLI Telnet interface for CLI SSH v1/v2 Serial interface for CLI Scriptable CLI Firmware image update (TFTP and FTP) Network Time Protocol (NTP) for switch clock synchronization BNT BLADEHarmony Manager support Chapter 1. Introduction 11

Monitoring Switch LEDs for external port status and switch module status indication Port mirroring for analyzing network traffic passing through switch Change tracking and remote logging with syslog feature POST diagnostics Special functions Serial over LAN (SOL) Virtualization features VMready Virtual Fabric Adapter vnic support Converged Enhanced Ethernet and FCoE features Fibre Channel over Ethernet (FCoE) allows Fibre Channel traffic to be transported over Ethernet links. FCoE Initialization Protocol (FIP) snooping to enforce point-to-point links for FCoE traffic outside the regular Fibre Channel topology. Priority-Based Flow Control (PFC) (IEEE 802.1Qbb) extends 802.3x standard flow control to allow the switch to pause traffic based on the 802.1p priority value in each packet s VLAN tag. Enhanced Transmission Selection (ETS) (IEEE 802.1Qaz) provides a method for allocating link bandwidth based on the 802.1p priority value in each packet s VLAN tag. Data Center Bridging Capability Exchange Protocol (DCBX) (IEEE 802.1AB) allows neighboring network devices to exchange information about their capabilities. Supports the QLogic Virtual Fabric Extension Module for IBM BladeCenter, which provides FCoE gateway functionality inside the BladeCenter Chassis. VMready is a unique solution that enables the network to be virtual machine-aware. The network can be configured and managed for virtual ports (v-ports) rather than just for physical ports. With VMready, as VMs migrate across physical hosts, so do their network attributes. Virtual machines can be added, moved, and removed while retaining the same ACLs, QoS, and VLAN attributes. VMready allows for a define-once-use-many configuration that evolves as the server and network topologies evolve. VMready works with all virtualization products, including VMware, Hyper-V, Xen, and KVM, without modification of virtual machine hypervisors or guest operating systems. It is available as part of the 6.1 software code. The switch module supports the following IEEE standards: IEEE 802.1D STP with PVRST+ IEEE 802.1s MSTP IEEE 802.1w RSTP IEEE 802.1p Tagged Packets IEEE 802.1Q Tagged VLAN (frame tagging on all ports when VLANs are enabled) IEEE 802.1x port-based authentication IEEE 802.2 Logical Link Control IEEE 802.3ad Link Aggregation Control Protocol IEEE 802.3x Full-duplex Flow Control IEEE 802.3ab 1000BASE-T Gigabit Ethernet IEEE 802.3ae 10GBASE-SR 10Gb Ethernet fiber optics short range IEEE 802.3ae 10GBASE-LR 10Gb Ethernet fiber optics long range IEEE 802.3z 1000BASE-SX Gigabit Ethernet 12 IBM BladeCenter Virtual Fabric Solutions

The following network cables are supported for the BNT Virtual Fabric 10Gb Switch Module: 10GBASE-SR for 10Gb ports: 850 Nm wavelength, multimode fiber, 50 µ or 62.5 µ (300 meters maximum), with LC duplex connector 1000BASE-T for RJ-45 port: UTP Category 6 (100 meters maximum) UTP Category 5e (100 meters maximum) UTP Category 5 (100 meters maximum) EIA/TIA-568B 100-ohm STP (100 meters maximum) For more information, see the following documents: BNT Virtual Fabric 10 Gb Switch Module Installation Guide Application Guide Command Reference iscli Reference BBI (Browser-based Interface) Quick Guide Release Notes End User License Agreement These documents are all available in PDF format from the following Web page: http://www.ibm.com/support/docview.wss?uid=psg1migr-5080917 For more information, see the BNT Virtual Fabric 10 Gb Switch Module At-a-Glance Guide at: http://www.redbooks.ibm.com/abstracts/tips0708.html 1.3.3 Emulex Virtual Fabric Adapter The Emulex Virtual Fabric Adapter (CFFh) for IBM BladeCenter is a dual-port 10 Gb Ethernet card that supports 1 Gbps or 10 Gbps traffic, or up to eight virtual NIC devices. The virtual NICs are configured to meet the client's mix of network connectivity and throughput demands for today's complex server application environments. Each physical 10 Gbps port can be divided into four virtual ports with bandwidth allocation in 100 Mbps increments to a maximum of 10 Gbps per physical port. Table 1-6 lists the Emulex Virtual Fabric Adapter (CFFh) ordering part number and feature code. Table 1-6 Ordering part number and feature code Description Part number Feature code Emulex Virtual Fabric Adapter (CFFh) 49Y4235 5755 The Emulex Virtual Fabric Adapter has the following features: Connection to either 1 Gb or 10 Gb data center infrastructure (1 Gb and 10 Gb auto-negotiation) PCI Express 2.0 x8 host interface IBM BladeCenter CFFh form factor, can be combined with a CIOv expansion card on the same server Operates either as an 8-port virtual NIC or as a 2-port 1/10 Gb Ethernet adapter Virtual port bandwidth allocation in 100 Mbps increments Supports up to eight virtual ports Chapter 1. Introduction 13

Wake On LAN support Full-duplex (FDX) capability Bus-mastering support Direct memory access (DMA) support Preboot Execution Environment (PXE) support IPv4/IPv6 TCP, UDP checksum offload Large send offload (LSO) Large receive offload Receive side scaling (RSS) IPV4 TCP Chimney Offload VLAN insertion and extraction Jumbo frames up to 9000 bytes Load balancing and failover support including adapter fault tolerance (AFT), switch fault tolerance (SFT), adaptive load balancing (ALB), teaming support, and IEEE 802.3ad Enhanced Ethernet (draft) Enhanced Transmission Selection (ETS) (P802.1Qaz) Priority-based Flow Control (PFC) (P802.1Qbb) Data Center Bridging Capabilities exchange Protocol, CIN-DCBX and CEE-DCBX (P802.1Qaz) iscsi and FCoE support as a future feature entitlement purchase Configurable from the BNT Virtual Fabric 10Gb Switch Module The expansion card has two modes of operation: Virtual NIC (vnic) mode In vnic mode, each physical port appears to the blade server as four virtual NICs. The default bandwidth for each vnic is 2.5 Gbps. Bandwidth for each vnic can be configured from 100 Mbps to 10 Gbps, up to a maximum of 10 Gb per virtual port. vnics can also be configured to have 0 bandwidth if you must allocate the available bandwidth to fewer than eight vnics. To support vnic mode operation, the expansion card must communicate with the BNT Virtual Fabric 10Gb Switch Module (part number 46C7191), which provides independent control for each vnic. Physical NIC (pnic) mode In pnic mode, the expansion card can operate as a standard 10 Gbps or 1 Gbps 2-port Ethernet expansion card. When in pnic mode, the expansion card functions with any supported 10 Gb switch or 10 Gb pass-thru module installed in I/O module bays 7 and 9. A future entitlement purchase allows for up to two FCoE ports or two iscsi ports. FCoE and iscsi ports can be used in combination with up to six Ethernet ports in vnic mode, up to a maximum of eight total virtual ports. 14 IBM BladeCenter Virtual Fabric Solutions

Figure 1-5 shows the Emulex Virtual Fabric Adapter. Figure 1-5 Emulex Virtual Fabric Adapter (CFFh) (Emulex, Reprinted by Permission) For more information, see the Emulex Virtual Fabric Adapter (CFFh) At-a-Glance Guide at: http://www.redbooks.ibm.com/abstracts/tips0748.html Chapter 1. Introduction 15

16 IBM BladeCenter Virtual Fabric Solutions

2 Chapter 2. Configuration basics This chapter 1, 2 introduces the concepts and steps used to configure a virtual fabric. The topics covered are: 2.1, General characteristics on page 18 2.2, Basic setup on page 19 2.3, Creating Virtual NICs on page 19 2.4, vnic group configuration on page 22 2.5, Configuring vnic VLANs on page 23 2.6, vnic and VMready on page 26 2.7, High availability and vnics on page 27 1 Portions of this chapter are from Blade Network Technologies, Reprinted by Permission. 2 Portions of this chapter are from Emulex, Reprinted by Permission. Copyright IBM Corp. 2010. All rights reserved. 17

2.1 General characteristics General characteristics of this functionality are: Each Server physical NIC (pnic) port is divided into up to four virtual NICs (vnics) Operating system (OS) configurations see eight unique NICs (2 ports x 4 vnics) All vnic parameters are configured from the VFS user interface. The DCBX protocol is used between switch and NIC to convey configuration information The user enables vnics and allocates bandwidth. The allowable vnic bandwidth range is 100 Mbps - 10 Gbps The increments are 100 Mbps. The efault bandwidth setting is 2.5 Gbps. The sum of all four vnics cannot exceed 10 Gbps. VFS-to-server bandwidth metering on a per vnic per port basis Server-to-VFS bandwidth metering on a per-vnic basis The user assigns vnics and, optionally, uplinks to vnic groups Groups serve to isolate virtual NIC traffic flowing on the same physical port Existing VLANs within the customer network are not impacted. No forwarding occurs between uplinks assigned to vnic groups. Up to 32 vnic groups supported per Virtual Fabric Switch (VFS). An uplink (port or trunk) can belong to only one vnic group. A server port (pnic or vnic) can belong to only one vnic group. The failover mechanism is virtual port aware. Key concurrency rules are: vnic and VMready You cannot enable vnic mode on the Emulex Virtual Fabric Adapter (VFA) and implement BNT VMready on that server at the same time. However, certain ancillary functions of VMready that come with vcenter integration work with the VFA in either pnic or vnic mode. Running the VFA in pnic mode and running VMready are supported concurrently. vnic and stacking The BNT VFS does not support stacking and vnic mode at the same time. You can enable stacking when the VFAs in all blade servers are in pnic mode. Stacking is a new feature of the VFS as of firmware Version 6.3. vnic and FCoE You cannot enable vnic mode on the VFA and use Fibre Channel over Ethernet (FCoE) on that same adapter. Note that FCoE is planned to be added as a future feature entitlement purchase, but at the time of writing it was not currently available. vnic and iscsi You cannot enable vnic mode on the VFA and use iscsi on that same adapter. Note that iscsi is planned to be added as a future feature entitlement purchase, but at the time of writing it was not currently available. 18 IBM BladeCenter Virtual Fabric Solutions

2.2 Basic setup The Emulex Virtual Fabric Adapter for IBM BladeCenter (VFA) can operate in 10 Gb dual 4-port Virtual Ethernet mode (vnic) or 10Gb dual Port Ethernet mode (pnic). The VFA default mode is set to vnic. vnic mode is enumerated by the operating system or hypervisor as eight separate Ethernet devices (four per physical port). The pnic mode can be enabled per the instructions in the Installation and User's Guide for Emulex Virtual Fabric Adapter (CFFh). There are two 10 Gb Ethernet switch environments supported, depending on the VFA mode: vnic mode (default): The VFA must be paired with one or two BNT Virtual Fabric 10Gb Switch Modules (VFS) to operate. To enable VFA (virtual) Ethernet connections, the VFS must be configured for vnic mode. The VFS configuration parameters control the speed of the individual vnic links (in increments of 100 Mbps) and assign a collection of vnic links to common communication groups. Unused vnic links can be disabled. Refer to 3.1, Configurations overview on page 32, for more information. pnic mode: The VFA functions as a 10 Gb dual port Ethernet device and can be paired with any high-speed IBM BladeCenter 10Gb Ethernet/CEE (HSSM). This includes VFS and non-virtual fabric 10Gb Ethernet switches or pass-thru modules. No additional HSSM configuration is required when the VFA is set to operate in this mode. Other than mode selection, the VFA has no configuration requirement for virtual NIC operation. The basic steps for permiting vnic operation between the VFA and the VFS are: 1. Enable vnic operation on the applicable blade ports. 2. Optional: Customize the maximum bandwidth for each vnic. 3. Group the vnic links into common communication groups. If an attempt is made to enable vnic operation on a blade port that is not capable, a warning message similar to the following is displayed on the switch command-line interface (CLI) and added to the switch log. Warning: Peer does not support vnic on port INT11 If you are using the switch browser-based user interface (BBI), the pop-up shown in Figure 2-1 is given to alert you of this warning message. Figure 2-1 Error message 2.3 Creating Virtual NICs vnics can be created and configured on the Virtual Fabric switch, which communicates its properties to the Emulex adapter using the DCBX protocol. Note that the switch can be Chapter 2. Configuration basics 19

configured for vnics even if a separate adapter is installed on a blade, but the DCBX exchanges fail and the vnics never become active or pass data. To configure vnics on one blade: 1. Turn vnic functionality on. This uses a master toggle switch, which can be used to temporarily disable vnic without removing all of the configuration details. This command is done once per switch, regardless of how many blades will use vnic. Using BladeOS: /cfg/virt/vnic/on off Using iscli: vnic enable no vnic enable This can also be done via the BBI (Figure 2-2). Figure 2-2 Enabling vnic mode (BNT, Reprinted by Permission) 20 IBM BladeCenter Virtual Fabric Solutions

2. Create up to four vnics on a blade. Note that in most cases this needs to be done twice, once on each of two Virtual Fabric switches. The same commands are used on both switches if the two physical ports are to be configured identically, especially if trunk failover is to be used for high availability. Blade numbers (internal port numbers) are 1-14, and vnic numbers are 1-4. Bandwidth is specified in 100 Mb increments, which also specifies the percentage of 10 Gb, which is allocated to a specific vnic. Using BladeOS: /cfg/virt/vnic/port <blade #>/vnic <vnic #>/ena create the specified vnic /cfg/virt/vnic /port <blade #>/vnic <vnic #>/bw <1-100> - allocate bandwidth Using iscli: vnic port <blade#> index <vnic #> bandwidth <1-100> enable no enable For example, follow these steps: a. Enable vnic #1 on blade 10. b. Allocate 30% of the 10 Gb to this vnic (3 Gb). c. Enable vnic #2 on blade 10. d. Allocate 70% of the 10 Gb to this vnic (7 Gb). The commands to do the above steps with BladeOS are: /cfg/virt/vnic /port 10/vnic 1/ena /cfg/virt/vnic /port 10/vnic 1/bw 30 /cfg/virt/vnic /port 10/vnic 2/ena /cfg/virt/vnic /port 10/vnic 2/bw 70 The commands to do the above steps with iscli are: vnic port 10 index 1 bandwidth 30 enable vnic port 10 index 2 bandwidth 70 enable Chapter 2. Configuration basics 21

This configuration can also be entered via the BBI (Figure 2-3). Figure 2-3 Creating a vnic (BNT, Reprinted by Permission) Consider the following when creating vnics: Up to four vnics can be created on any blade. There is no requirement to create all four. Only 10 Gb of total bandwidth can be allocated to the vnics on one port on a server blade. Oversubscription is not supported. If less than 10 Gb of total bandwidth is allocated to all of the defined vnics on a port, then the unallocated bandwidth is not usable. Bandwidth allocations are policed on both inbound and outbound traffic. Unlike certain configurations that use quality of service, and unlike the virtual NIC functionality built into the AIX operating system, there is no provision to allow a given vnic to exceed its allocated bandwidth at any time. Bandwidth allocations can be changed from the switch command line or browser interface. These changes are propagated to the servers immediately and do not require that the network be brought down or that the server be rebooted. It is possible, for instance, to have one bandwidth allocation scheme in use during business hours and a different one during off hours when batch processing might be in progress. 2.4 vnic group configuration A vnic group consists of one or more vnics, (optionally) one or more uplink (EXTernal) ports, and (optionally) one or more non-vnic server blade-facing (INTernal) ports. For a vnic to communicate, it must be a member of a single vnic group, and only one vnic from a given port can be in any specific group. 22 IBM BladeCenter Virtual Fabric Solutions

To configure a vnic group, do these steps by issuing the following commands: 1. Enable or disable a group. Up to 32 groups can be created. 2. Add a vnic to the group. 3. Add a physical port to the group. 4. Add an external trunk group to the vnic group. 5. Enable or disable trunk failover, as discussed in 2.7, High availability and vnics on page 27. The commands for the above steps are: /cfg/virt/vnic/vnicgrp <number>/{ena dis} /cfg/virt/vnic/vnicgrp <number>/addvnic <blade#.vnic#> /cfg/virt/vnic/vnicgrp <number>/addport <INT or EXT port> /cfg/virt/vnic/vnicgrp <number>/addtrnk <trunk #> /cfg/virt/vnic/vnicgrp <number>/failover {ena dis} The remvnic, remport, and remtrnk commands can be used to remove the corresponding items from the group. The iscli equivalent is: vnic vnicgroup <number> enable member INT<port>.<vnic number 1-4> port INT<x> EXT <x> trunk <number> failover exit The exit command is required to end the configuration of a specific group. Any of the above can be negated by prefixing it with no. At present, only one uplink port or one trunk group of uplink ports aggregated together can be part of a given vnic group. Any uplink port or trunk can only be part of one vnic group at a time, which means that it is not currently possible for multiple groups to share uplinks. At present, only static trunk groups are supported. LACP trunks are planned to be added in an upcoming version of the firmware. Each vnic group is associated with a vnic VLAN. vnic VLANs are discussed in the next section. 2.5 Configuring vnic VLANs Every vnic group is associated with a vnic VLAN. These VLANs differ from normal VLANs configured on the Virtual Fabric switch in significant ways. vnic VLANs are implemented as outer tags on the Ethernet frames that pass through the switch. These outer tags are never forwarded to a blade or to an upstream switch. They serve solely to isolate traffic in a vnic group from other vnic groups and from conventional VLANs. Within a group, conventional VLANs configured by an operating system on a server blade (such as with the vconfig command in Linux ) are implemented as inner tags. These inner tags pass through the group and the Virtual Fabric switch completely unchanged. Chapter 2. Configuration basics 23

The VLAN configuration on upstream switches needs to match that on the blades, but the Virtual Fabric switch simply passes the inner tags through (Figure 2-4). 802.1q "Double Tagging" and the BNT Virtual Fabric Switch Upstream Router BNT Virtual Fabric Switch HS22 Blade Payload Payload Payload Ethernet Header EtherType (of payload).1q VLAN tag VLAN ID "A" EtherType = 0x8100 Tagged traffic for multiple VLANs is sent from the upstream switch. Upon entry to the Virtual Fabric switch, the vnicvlan value is inserted as the second tag (ID "B"). Ethernet Header EtherType (of payload).1q VLAN tag 2 VLAN ID "A" EtherType = 0x8100.1Q VLAN tag 1 VLAN ID "B" EtherType = 0x8100 Traffic flows through the Virtual Fabric switch using the vnicvlan tag and is forwarded to the blade-facing port, where the vnicvlan "outer" tag is removed. Ethernet Header EtherType (of payload).1q VLAN tag VLAN ID "A" EtherType = 0x8100 Ethernet Header Ethernet Header Ethernet Header SA DA SA DA SA DA Figure 2-4 Double tagging (BNT, Reprinted by Permission) The same conventional VLAN can enter the chassis in more than one group, but traffic between groups never occurs within the Virtual Fabric switch. On the Virtual Fabric switch, a given VLAN number can be a conventional VLAN, a vnic group, or a VMready group, but not more than one of these at the same time. Additional notes about vnic VLANs: Routing of traffic between vnic VLANs is not supported, nor is routing between a vnic VLAN and a conventional VLAN. When EXTernal (uplink) ports are added to a vnic group, they are also added to the vnic VLAN, and they are always aggregated together. At present, there is no way to split these ports and aggregate them in more than one trunk. When INTernal (blade facing) ports are added to a vnic group, it is assumed that there will be no vnics created on those blades. (This is with the addport command, not the addvnic command.) Traffic can then flow between vnics and these INTernal ports. It will be forwarded using the vnic VLAN as the outer tag. Any inner tags associated with the non-vnic interface on the ports will be unchanged, just as is the case with vnics. vnic VLANs are configured with the associated vnic groups. 24 IBM BladeCenter Virtual Fabric Solutions

To configure vnic VLANs, the BladeOS command is: /cfg/virt/vnic/vnicgrp <number>/vnicvlan <1-4094> The iscli command is: vnic vnicgroup <number> vlan <1-4094> The selected VLAN number cannot be used within the Virtual Fabric Switch for any other purpose. However, it is of purely local significance in that it is never seen by blades (other than by the vnic driver), and never forwarded out of the Virtual Fabric switch to an upstream switch. Use high numbers that are not used in the customer s network as vnic VLANs (for example, 4000 and up). Chapter 2. Configuration basics 25

Figure 2-5 shows the BBI window to configure a vnic group and the associated VLAN. Figure 2-5 Configuring the vnic group and the associated VLAN (BNT, Reprinted by Permission) 2.6 vnic and VMready At present, a blade-facing port can be in a vnic group or a VMready group, but not both, so there is no way at this time to provide both functions to the same blade. The capability of having a blade-facing port in both a vnic group and a VMready group is planned to be in a forthcoming firmware release. (It is possible today to use vnic to support certain blades and VMready on others using the same switch and within the same chassis.) 26 IBM BladeCenter Virtual Fabric Solutions

2.7 High availability and vnics The trunk failover function, which enables high-availability designs when used with standard ports, is also available when vnics are used. With vnics, if the uplink ports that are members of a group (vnicgrp) fail due to a failure on the upstream device or a cable failing or being erroneously removed, the associated vnic is brought down by the Virtual Fabric switch. In Windows, this appears as a network cable unplugged condition on the server. When triggered, the failover mechanism operates on a per-vnic group basis, thus not affecting vnics/nics, which have no association with the failed uplink. Virtual connections within the group are disabled without bringing down the internal physical port. Refer to 3.6, Case 4: vnics and pnics, one uplink with failover on page 48, for a configuration example. To exploit the trunk failover function to deliver high availability, take the following steps: 1. The other Virtual Fabric switch in the chassis must also be configured to use vnics for the server in question. It is good idea, but is not required, for the same vnic numbers and associated groups and bandwidth allocation to be used on both switches. 2. The corresponding vnic ports as seen by the operating system must be teamed together with the available tools for NIC teaming. In general, the NIC teaming must be in an active/standby mode. Windows servers need to use the available utility from Emulex to implement this. Linux servers have the native bonding driver. ESX has support for multiple NIC teaming modes via the network configuration GUI. Multiple vnics that are part of the same physical 10Gb port can support high availability, but each vnic does so independently. This is different from the HP Flex10 implementation, where if failover occurs it brings down the entire physical port. However, in order to implement this, do the following: 1. Enable failover on the all the vnic groups where the ports from the servers in question are members, on both of the Virtual Fabric switches. 2. Configure NIC teaming (as above) on corresponding pairs of vnics on the server. Make sure that one members of the team is physically connected to one of the Virtual Fabric switches and the other member of the team to the other Virtual Fabric switch. 3. Configure uplink ports or trunks as members of the corresponding vnic groups on the two switches. It is good practice to connect these uplinks to separate upstream physical switches if possible. Chapter 2. Configuration basics 27

Figure 2-6 shows a sample configuration showing only one blade. The same configuration commands are required for each blade used for which failover is desired. Bay 7 10-port switch Switch bay Port INT3 High Availability with vnic vnic 3.1 vnic 3.2 Blade #3 Router vnic 3.1 vnic 3.2 Port INT3 Bay 9 10-port switch Legends: RED GREEN BLACK vnic Group 1, VLAN 2001 (outer tag) vnic 3.1 and EXT10 on each switch vnic Group 2, VLAN 2002 (outer tag) vnic 3.2 and EXT9 on each switch Physical port INT3 on each switch Figure 2-6 High availability with vnic (BNT, Reprinted by Permission) The BladeOS configuration for Figure 2-6 is identical on each of the two switches and includes: /cfg/virt/vnic /port 3/vnic 1/ena /cfg/virt/vnic/port 3/vnic 2/ena /cfg/virt/vnic/vnicgrp 1/ena failover ena vnicvlan 2001 addvnic 3.1 addport EXT10 /cfg/virt/vnic/vnicgrp 2/ena failover ena vnicvlan 2002 addvnic 3.2 addport EXT9 The equivalent iscli configuration is: vnic port INT3 index 1 enable vnic port INT3 index 2 28 IBM BladeCenter Virtual Fabric Solutions

enable vnic vnicgroup 1 failover vlan 2001 member INT3.1 port EXT10 enable exit vnic vnicgroup 2 failover vlan 2002 member INT3.2 port EXT9 enable exit For more information, refer to the BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter, Application Guide. Chapter 2. Configuration basics 29

30 IBM BladeCenter Virtual Fabric Solutions

3 Chapter 3. Configuring the Virtual Fabric In this chapter 1, 2 we describe the steps to create six separate virtual fabric configurations, comprising combinations of: Virtual network interface controllers (vnics) Physical network interface controller (NICs) Virtual NIC (vnic) VLANs Uplink ports Uplink trunks Uplink failover All of the configuration performed here is done using interfaces into the BNT Virtual Fabric Switch. Tip: You also need to ensure that vnic mode is enabled in the Virtual Fabric Adapters. vnic mode is the default. The topics in this chapter are: 3.1, Configurations overview on page 32 3.2, VFA/VFS vnic correlation on page 34 3.3, Case 1: vnics only, no uplinks on page 34 3.4, Case 2: vnics and pnics, no uplinks on page 39 3.5, Case 3: vnics only, one uplink on page 43 3.6, Case 4: vnics and pnics, one uplink with failover on page 48 3.7, Case 5: vnics only, one uplink trunk on page 52 3.8, Case 6: vnics and pnics, one uplink trunk on page 58 1 Portions of this chapter are from Blade Network Technologies, Reprinted by Permission. 2 Portions of this chapter are from Emulex, Reprinted by Permission. Copyright IBM Corp. 2010. All rights reserved. 31

3.1 Configurations overview Table 3-1 Combinations matrix The basic matrix of vnic group possibilities is identified by considering the possible NIC types (physical, virtual) and uplink types (port, trunk). Table 3-1 lists the available configuration options. Case Uplink trunk Single uplink Physical NIC Virtua l NIC Comments 1 No No No Yes vnics only. No uplinks. See section 3.3, Case 1: vnics only, no uplinks on page 34. 2 No No Yes Yes vnics and pnics. No uplinks. See section 3.4, Case 2: vnics and pnics, no uplinks on page 39. 3 No Yes No Yes vnics only. One uplink. See section 3.5, Case 3: vnics only, one uplink on page 43. 4 No Yes Yes Yes vnics and pnics. One uplink with failover. See section 3.6, Case 4: vnics and pnics, one uplink with failover on page 48. 5 Yes No No Yes vnics only. One uplink trunk. See section 3.7, Case 5: vnics only, one uplink trunk on page 52. 6 Yes No Yes Yes vnics and pnics. One uplink trunk. See section 3.8, Case 6: vnics and pnics, one uplink trunk on page 58. 7 a No No Yes No pnics only. No uplinks. 8 a No Yes Yes No pnics only. One uplink. 9 a Yes No Yes No pnics only. One uplink trunk. a. Cases 7-9, though permitted as vnic groups, are typically handled by traditional VLAN configuration and are therefore not covered in this paper. They are included in this table and Figure 3-1 on page 33 for completeness. 32 IBM BladeCenter Virtual Fabric Solutions

Figure 3-1 depicts each configuration possibility. For the purposes of this discussion, we implemented all nine configurations in the one pair of Virtual Fabric Switches in the one chassis. The actions required to implement cases 1-6 are detailed in subsequent sections. Case 1 Case 3 VGrp 1 1.2 2.1 EXT2 1.3 2.2 VFS, I/O Bay 7 Case 1 VGrp 3 Case 3 50% INT1.2 25% INT1.3 25% INT1.4 20% INT2.1 70% INT2.2 10% INT2.4 Fn Fn 0 1 2 4 6 Blade 1 3 5 7 Fn Fn 0 1 2 Blade 3 2 4 5 6 7 INT1.1 30% 30% INT1.3 INT1.4 40% 33% INT2.1 33% INT2.2 INT2.4 34% VFS I/O Bay 9 VGrp 5 1.1 2.2 Case 5 Case 4 VGrp 4 *Failover 1.4 Enabled 2.1 EXT6 3 Case 4 EXT1 EXT3 Case 5 Trnk 1 Case 6 Trnk 5 EXT3 EXT4 VGrp 6 1.4 2.4 4 5 Case 6 INT3 INT4 Fn Fn 0 Blade 1 3 Fn Fn 0 Blade 1 4 INT3 INT4 VGrp 2 1.3 2.4 4 Case 2 Case 2 Case 9 INT5 Fn Fn 0 Blade 1 5 INT5 VLAN 8 5 Case 8 EXT9 Case 8 Trnk 2 EXT7 EXT8 VLAN 9 3 6 Case 9 INT6 Fn Fn 0 Blade 1 6 INT6 VLAN 7 6 7 Case 7 INT7 Fn Fn 0 Blade 1 7 INT7 Case 7 Figure 3-1 Typical vnic configuration scenarios Chapter 3. Configuring the Virtual Fabric 33

3.2 VFA/VFS vnic correlation Setting up virtual connections requires an understanding of how the adapter and switch identify the eight possible vnic devices. Table 3-2 illustrates the designations used. Table 3-2 VFA/VFS vnic correlation Virtual Fabric Adapter (VFA) Virtual Fabric Switch (VFS) PCIe function ID Port I/O bay Port alias (blade slot x) vnic 0 0 7 INTx.1 1 2 0 7 INTx.2 2 4 0 7 INTx.3 3 6 0 7 INTx.4 4 1 1 9 INTx.1 1 3 1 9 INTx.2 2 5 1 9 INTx.3 3 7 1 9 INTx.4 4 Tip: For the Virtual Fabric Adapter (VFA), the PCIe function ID can be determined by running the ethtool -i command in Linux or by viewing the Network Adapter Properties in Windows Device Manager. 3.3 Case 1: vnics only, no uplinks Scenario case 1 is represented as VGRP 1 in Figure 3-1 on page 33. This group consists solely of vnic ports. The configuration is: vnic INT1.2 (blade in slot 1, vnic 2) of the switch in I/O bay 7, set to a maximum bandwidth of 5 Gbps (50% of 10 Gbps) vnic INT2.1 (blade 2, vnic 1) of the switch in I/O bay 7, set to a maximum bandwidth of 2 Gbps (20% of 10 Gbps) One vnic group containing these two vnics (vnic Group VLAN 127) The BladeOS command script in Example 3-1 can be used to implement this configuration. Example 3-1 BladeOS script to create scenario 1 /c/virt/vnic on /c/virt/vnic/port INT1/vnic 2 ena bw 50 /c/virt/vnic/port INT2/vnic 1 ena bw 20 /c/virt/vnic/vnicgrp 1 ena vnicvlan 127 34 IBM BladeCenter Virtual Fabric Solutions

apply addvnic INT1.2 addvnic INT2.1 The iscli command script in Example 3-2 can be used to implement this configuration. Example 3-2 iscli script to create scenario 1 vnic enable vnic port INT1 index 2 enable bandwidth 50 vnic port INT2 index 1 enable bandwidth 20 vnic vnicgroup 1 enable vlan 127 member 1.2 member 2.1 exit Chapter 3. Configuring the Virtual Fabric 35

The Virtual Fabric Switch browser-based interface (BBI) can also be used to implement this configuration as follows: 1. Click the Configure tab of the BBI interface. 2. Enable vnic operation by expanding Virtualization VNIC General, and then select On and click Submit Apply Save. Tip: Click Submit Apply Save on every panel to ensure this setting persists after any switch reboot. Figure 3-2 Case 1 - enable vnic operation (BNT, Reprinted by Permission) 36 IBM BladeCenter Virtual Fabric Solutions

3. Enable and configure each of the vnics. Click Virtualization VNICs (Figure 3-3). Figure 3-3 Case 1 - vnics pane (BNT, Reprinted by Permission) 4. For the first vnic, select Enabled, and then enter in the bandwidth allocation (Figure 3-4). Figure 3-4 Enabling vnic INT1.2 (BNT, Reprinted by Permission) 5. For the second vnic, select Enabled, and then enter in the bandwidth allocation (Figure 3-5). Figure 3-5 Enabling vnic INT2.1 (BNT, Reprinted by Permission) Chapter 3. Configuring the Virtual Fabric 37

6. Enable and configure the vnic group by clicking Virtualization VNIC VNIC Groups (Figure 3-6). Figure 3-6 Case 1 - vnic Groups pane (BNT, Reprinted by Permission) 7. Click vnic Group 1. Figure 3-7 displays. a. Enable the group. b. Set the VLAN to 127. c. Select the two vnics d. Click Add. Figure 3-7 Case 1 - vnic Group Configuration (BNT, Reprinted by Permission) 38 IBM BladeCenter Virtual Fabric Solutions

3.4 Case 2: vnics and pnics, no uplinks Scenario case 2 is represented as VGRP 2 in Figure 3-1 on page 33. This group consists of vnic and pnic ports, but no uplink ports. The configuration is: vnic INT1.3 (blade in slot 1, vnic 3) of the switch in I/O bay 9, set to a maximum bandwidth of 3 Gbps (30% of 10 Gbps) vnic INT2.4 (blade 2, vnic 4) of the switch in I/O bay 9, set to a maximum bandwidth of 3.4 Gbps (34% of 10 Gbps) One 10 Gb physical pnic (port INT4) One vnic group containing these two vnics and the pnic (vnic Group VLAN 310) The vbladeos command script in Example 3-3 can be used to implement this configuration. Example 3-3 BladeOS script for case 2 /c/virt/vnic on /c/virt/vnic/port INT1/vnic 3 ena bw 30 /c/virt/vnic/port INT2/vnic 4 ena bw 34 /c/virt/vnic/vnicgrp 2 ena vnicvlan 310 addvnic INT1.3 addvnic INT2.4 addport INT4 apply The iscli command script in Example 3-4 can be used to implement this configuration. Example 3-4 iscli script for case 2 vnic enable vnic port INT1 index 3 enable bandwidth 30 vnic port INT2 index 4 enable bandwidth 34 vnic vnicgroup 2 enable vlan 310 member 1.3 member 2.4 port INT4 exit Chapter 3. Configuring the Virtual Fabric 39

The Virtual Fabric Switch BBI can also be used to implement this configuration as follows: 1. Click the Configure tab of the BBI interface. 2. Enable vnic operation by expanding Virtualization VNIC General, and then select On and click Submit Apply Save. Tip: Click Submit Apply Save on every panel to ensure that this setting persists after any switch reboot. Figure 3-8 Case 2 - enable vnic operation (BNT, Reprinted by Permission) 40 IBM BladeCenter Virtual Fabric Solutions

3. Enable and configure each vnic. Click Virtualization VNICs (Figure 3-9). Figure 3-9 Case 2 - vnics pane (BNT, Reprinted by Permission) 4. For the first vnic, select Enabled, and then enter the bandwidth allocation (Figure 3-10). Figure 3-10 Case 2, vnic INT1.3 Configuration (BNT, Reprinted by Permission) 5. For the second vnic, select Enabled, and then enter the bandwidth allocation (Figure 3-11). Figure 3-11 Case 2 - vnic INT2.4 Configuration (BNT, Reprinted by Permission) Chapter 3. Configuring the Virtual Fabric 41

6. Enable and configure the vnic group by clicking Virtualization VNIC VNIC Groups (Figure 3-12). Figure 3-12 Case 2 - vnic Groups pane (BNT, Reprinted by Permission) 42 IBM BladeCenter Virtual Fabric Solutions

7. Click vnic Group 2. Figure 3-13 displays. a. Enable the group. b. Set the VLAN to 310. c. Select the vnics. d. Click Add. e. Select the INT4 port. f. Click Add. Figure 3-13 Case 2 - vnic Group Configuration (BNT, Reprinted by Permission) 3.5 Case 3: vnics only, one uplink Scenario case 3 is represented as VGRP 3 in Figure 3-1 on page 33. This group consists of vnic ports and one uplink port. The configuration is: vnic INT1.3 (blade in slot 1, vnic 3) of the switch in I/O bay 7, set to a maximum bandwidth of 2.5 Gbps (25% of 10 Gbps) vnic INT2.2 (blade 2, vnic 2) of the switch in I/O bay 7, set to a maximum bandwidth of 7 Gbps (70% of 10 Gbps) One vnic group containing these vnics (vnic Group VLAN 500) One uplink port, EXT2 Chapter 3. Configuring the Virtual Fabric 43

The BladeOS command script in Example 3-5 can be used to implement this configuration. Example 3-5 BladeOS script for case 3 /c/virt/vnic on /c/virt/vnic/port INT1/vnic 3 ena bw 25 /c/virt/vnic/port INT2/vnic 2 ena bw 70 /c/virt/vnic/vnicgrp 3 ena vnicvlan 500 addvnic INT1.3 addvnic INT2.2 addport EXT2 apply The iscli command script in Example 3-6 can be used to implement this configuration. Example 3-6 iscli script for case 3 vnic enable vnic port INT1 index 3 enable bandwidth 25 (default) vnic port INT2 index 2 enable bandwidth 70 vnic vnicgroup 3 enable vlan 500 member 1.3 member 2.2 port EXT2 exit 44 IBM BladeCenter Virtual Fabric Solutions

The Virtual Fabric Switch BBI can also be used to implement this configuration as follows: 1. Click the Configure tab of the BBI interface. 2. Enable vnic operation by expanding Virtualization VNIC General, and then select On and click Submit Apply Save. Tip: Click Submit Apply Save on every panel to ensure that this setting persists after any switch reboot. Figure 3-14 Case 3 - enable vnic operation (BNT, Reprinted by Permission) Chapter 3. Configuring the Virtual Fabric 45

3. Enable and configure each vnic. Click Virtualization VNICs (Figure 3-15). Figure 3-15 Case 3 - vnics pane (BNT, Reprinted by Permission) 4. For the first vnic, select Enabled, and then enter the bandwidth allocation (Figure 3-16). Figure 3-16 Case 3 - vnic INT1.3 Configuration (BNT, Reprinted by Permission) 5. For the second vnic, select Enabled, and then enter the bandwidth allocation (Figure 3-17). Figure 3-17 Case 3 - vnic INT2.2 Configuration (BNT, Reprinted by Permission) 46 IBM BladeCenter Virtual Fabric Solutions

6. Enable and configure the vnic group by clicking Virtualization VNIC VNIC Groups (Figure 3-18). Figure 3-18 Case 3 - vnic Groups pane (BNT, Reprinted by Permission) 7. Click vnic Group 3. Figure 3-19 displays. a. Enable the group. b. Set the VLAN to 500. c. Select the EXT2 uplink port. d. Select the vnics. e. Click Add. Figure 3-19 Case 3 - vnic Group Configuration (BNT, Reprinted by Permission) Chapter 3. Configuring the Virtual Fabric 47

3.6 Case 4: vnics and pnics, one uplink with failover Scenario case 4 is represented as VGRP 4 in Figure 3-1 on page 33. This group consists of vnic and pnic ports, and one uplink port. The configuration is: vnic INT1.4 (blade in slot 1, vnic 4) of the switch in I/O bay 9, set to a maximum bandwidth of 4 Gbps (40% of 10 Gbps) vnic INT2.1 (blade 2, vnic 1) of the switch in I/O bay 9, set to a maximum bandwidth of 3.3 Gbps (33% of 10 Gbps) One 10 Gb physical pnic (port INT2) One vnic group containing these vnics and the one pnic (vnic Group VLAN 727) One uplink port, EXT6 Failover is enabled for this group, so if the link on EXT6 is disrupted, the VFS disables virtual ports INT1.4 and INT2.1 and physical port INT3. This action facilitates Network Interface Controller (NIC) team failover on the server blade. Tip: After the server ports are disabled, it is up to the teaming software on each server to take the necessary action to fail over to other working ports (for example, ports connected to the switch in bay 9 if the switch in bay 7 fails). This typically means that the two switches in bays 7 and 9 are configured identically. The BladeOS command script in Example 3-7 can be used to implement this configuration. Example 3-7 BladeOS script for case 4 /c/virt/vnic on /c/virt/vnic/port INT1/vnic 4 ena bw 40 /c/virt/vnic/port INT2/vnic 1 ena bw 33 /c/virt/vnic/vnicgrp 4 ena vnicvlan 727 failover ena addvnic INT1.4 addvnic INT2.1 addport INT3 addport EXT6 apply The iscli command script in Example 3-8 can be used to implement this configuration. Example 3-8 iscli script for case 4 vnic enable vnic port 1 index 4 enable bandwidth 40 vnic port 2 index 1 enable 48 IBM BladeCenter Virtual Fabric Solutions

bandwidth 33 vnic vnicgroup 4 enable vlan 727 failover member 1.3 member 2.1 port INT3 port EXT6 exit The Virtual Fabric Switch BBI can also be used to implement this configuration as follows: 1. Click the Configure tab of the BBI interface. 2. Enable vnic operation by expanding Virtualization VNIC General, and then select On and click Submit Apply Save. Tip: Click Submit Apply Save on every panel to ensure that this setting persists after any switch reboot. Figure 3-20 Case 4 - enable vnic operation (BNT, Reprinted by Permission) Chapter 3. Configuring the Virtual Fabric 49

3. Enable and configure each vnic. Click Virtualization VNICs (Figure 3-21). Figure 3-21 Case 4 - vnics pane (BNT, Reprinted by Permission) 4. For the first vnic, select Enabled, and then type the bandwidth allocation (Figure 3-22). Figure 3-22 Case 4 - vnic INT1.4 Configuration (BNT, Reprinted by Permission) 5. For the second vnic, select Enabled, and then type the bandwidth allocation (Figure 3-23). Figure 3-23 Case 4 - vnic INT2.1 Configuration (BNT, Reprinted by Permission) 50 IBM BladeCenter Virtual Fabric Solutions

6. Enable and configure the vnic group by clicking Virtualization VNIC VNIC Groups (Figure 3-24). Figure 3-24 Case 4 - vnic Groups pane (BNT, Reprinted by Permission) Chapter 3. Configuring the Virtual Fabric 51

7. Click vnic Group 4. Figure 3-25 displays. a. Enable the group. b. Set the VLAN to 727. c. Select the EXT6 uplink port. d. Select the vnics. e. Click Add. f. Select port INT3. g. Click Add. Figure 3-25 Case 4 - vnic Group Configuration (BNT, Reprinted by Permission) 3.7 Case 5: vnics only, one uplink trunk Scenario case 5 is represented as VGRP 5 in Figure 3-1 on page 33. This group consists of vnic ports and one uplink trunk. The configuration is: vnic INT1.1 (blade in slot 1, vnic 1) of the switch in I/O bay 9, set to a maximum bandwidth of 3 Gbps (30% of 10 Gbps) vnic INT2.2 (blade 2, vnic 2) of the switch in I/O bay 9, set to a maximum bandwidth of 3.3 Gbps (33% of 10 Gbps) One vnic group containing these two vnics (vnic Group VLAN 925) One uplink trunk comprising external ports EXT1 and EXT3 52 IBM BladeCenter Virtual Fabric Solutions

The BladeOS command script in Example 3-9 can be used to implement this configuration. Example 3-9 BladeOS script for case 5 /c/l2/trunk 1 ena add EXT1 add EXT3 /c/virt/vnic on /c/virt/vnic/port INT1/vnic 1 ena bw 30 /c/virt/vnic/port INT2/vnic 2 ena bw 33 /c/virt/vnic/vnicgrp 5 ena vnicvlan 925 addvnic INT1.1 addvnic INT2.2 addtrnk 1 apply The iscli command script in Example 3-10 can be used to implement this configuration. Example 3-10 iscli script for case 5 vnic enable portchannel 1 port EXT1 portchannel 1 port EXT3 portchannel 1 enable vnic port INT1 index 1 enable bandwidth 30 vnic port INT2 index 2 enable bandwidth 33 vnic vnicgroup 5 enable vlan 925 member 1.1 member 2.2 trunk 1 exit Chapter 3. Configuring the Virtual Fabric 53

The Virtual Fabric Switch BBI can also be used to implement this configuration as follows: 1. Click the Configure tab of the BBI interface. 2. Enable vnic operation by expanding Virtualization VNIC General, and then select On and click Submit Apply Save. Tip: Click Submit Apply Save on every panel to ensure that this setting persists after any switch reboot. Figure 3-26 Case 5 - enable vnic operation (BNT, Reprinted by Permission) 54 IBM BladeCenter Virtual Fabric Solutions

3. Enable and configure the vnics. Click Virtualization VNICs (Figure 3-27). Figure 3-27 Case 5 - vnic configuration (BNT, Reprinted by Permission) 4. For the first vnic, select Enabled, and then type the bandwidth allocation (Figure 3-28). Figure 3-28 Case 5 - vnic INT1.1 Configuration (BNT, Reprinted by Permission) 5. For the second vnic, select Enabled, and then type the bandwidth allocation (Figure 3-29). Figure 3-29 Case 5 - vnic INT2.2 Configuration (BNT, Reprinted by Permission) Chapter 3. Configuring the Virtual Fabric 55

6. Enable the trunk group by clicking Layer 2 Trunk Groups and Trunk Group 1 (Figure 3-30). Figure 3-30 Case 5 - Trunk Groups pane (BNT, Reprinted by Permission) 7. Enable the trunk group, select EXT1 and EXT3, and click Add (Figure 3-31). Figure 3-31 Case 5 - Trunk Group 1 Configuration (BNT, Reprinted by Permission) 56 IBM BladeCenter Virtual Fabric Solutions

8. Enable and configure the vnic group by clicking Virtualization VNIC VNIC Groups (Figure 3-32). Figure 3-32 Case 5 - vnic Groups pane (BNT, Reprinted by Permission) 9. Click vnic Group 5. Figure 3-33 displays. a. Enable the group. b. Set the VLAN to 925. c. Select the Trunk 1 uplink trunk. d. Select the vnics. e. Click Add. Figure 3-33 Case 5 - vnic Group Configuration (BNT, Reprinted by Permission) Chapter 3. Configuring the Virtual Fabric 57

3.8 Case 6: vnics and pnics, one uplink trunk Scenario case 6 is represented as VGRP 6 in Figure 3-1 on page 33. This group consists of vnic and pnic ports and one uplink trunk. The configuration is: vnic INT1.4 (blade in slot 1, vnic 4) of the switch in I/O bay 7, set to a maximum bandwidth of 2.5 Gbps (25% of 10 Gbps) vnic INT2.4 (blade 2, vnic 4) of the switch in I/O bay 7, set to a maximum bandwidth of 1 Gbps (10% of 10 Gbps) Two 10 Gb physical pnics (port INT4 and INT5) One vnic group containing these two vnics and two pnics (vnic Group VLAN 1010) One uplink trunk comprising external ports EXT3 and EXT4 The BladeOS command script in Example 3-11 can be used to implement this configuration. Example 3-11 iscli script for case 6 /c/l2/trunk 5 ena add EXT3 add EXT4 /c/virt/vnic on /c/virt/vnic/port INT1/vnic 4 ena bw 25 /c/virt/vnic/port INT2/vnic 4 ena bw 10 /c/virt/vnic/vnicgrp 6 ena vnicvlan 1010 addvnic INT1.4 addvnic INT2.4 addport INT4 addport INT5 addtrnk 5 apply The iscli command script in Example 3-12 can be used to implement this configuration. Example 3-12 iscli script for case 6 vnic enable portchannel 5 port EXT3 portchannel 5 port EXT4 portchannel 5 enable vnic port INT1 index 4 enable bandwidth 25 (default - could be omitted) vnic port INT2 index 4 enable bandwidth 10 vnic vnicgroup 6 enable 58 IBM BladeCenter Virtual Fabric Solutions

vlan 1010 member 1.4 member 2.4 port INT4 port INT5 trunk 5 exit The Virtual Fabric Switch BBI can also be used to implement this configuration as follows: 1. Click the Configure tab of the BBI interface. 2. Enable the vnic operation by expanding Virtualization VNIC General, and then select On and click Submit Apply Save. Tip: Click Submit Apply Save on every panel to ensure that this setting persists after any switch reboot. Figure 3-34 Case 6 - enable vnic operation (BNT, Reprinted by Permission) Chapter 3. Configuring the Virtual Fabric 59

3. Enable and configure the vnics. Click Virtualization VNICs (Figure 3-35). Figure 3-35 Case 6 - vnics pane (BNT, Reprinted by Permission) 4. For the first vnic, select Enabled, and then type the bandwidth allocation (Figure 3-36). Figure 3-36 Case 6 - vnic INT1.4 Configuration (BNT, Reprinted by Permission) 5. For the second vnic, select Enabled, and then type the bandwidth allocation (Figure 3-37). Figure 3-37 Case 6 - vnic INT2.4 Configuration (BNT, Reprinted by Permission) 60 IBM BladeCenter Virtual Fabric Solutions

6. Enable the trunk group by clicking Layer 2 Trunk Groups and clicking Trunk Group 5 (Figure 3-38). Figure 3-38 Case 6, Trunk Groups pane (BNT, Reprinted by Permission) 7. Enable the trunk group, select EXT3 and EXT4, and click Add (Figure 3-39). Figure 3-39 Case 6 - Trunk Group 5 Configuration (BNT, Reprinted by Permission) Chapter 3. Configuring the Virtual Fabric 61

8. Enable and configure the vnic group by clicking Virtualization VNIC VNIC Groups (Figure 3-40). Figure 3-40 Case 6 - vnic Groups pane (BNT, Reprinted by Permission) 62 IBM BladeCenter Virtual Fabric Solutions

9. Click vnic Group 6. Figure 3-41 displays. a. Enable the group. b. Set the VLAN to 1010. c. Select the Trunk 6 uplink trunk. d. Select the vnics. e. Click Add. f. Select the ports. g. Click Add. Figure 3-41 Case 6 - vnic Group Configuration (BNT, Reprinted by Permission) Chapter 3. Configuring the Virtual Fabric 63

64 IBM BladeCenter Virtual Fabric Solutions

4 Chapter 4. Example of use In this chapter 1, 2 we show an example of how a virtual fabric can be used. 1 Portions of this chapter are from Blade Network Technologies, Reprinted by Permission. 2 Portions of this chapter are from Emulex, Reprinted by Permission. Copyright IBM Corp. 2010. All rights reserved. 65

4.1 Scenario 1 Figure 4-1 shows a possible configuration using the Virtual Fabric switch and the Virtual Fabric 10Gb NIC. In this example, two vnic groups are in use: vnic group 1 is for iscsi traffic to the storage farm. It is shown as directly connected to the switches in bays 7 and 9, but can also be via one or more switches or routers. vnic VLAN =2001. vnic group 2 is for other data traffic to the blade. The switches in bays 7 and 9 are connected to an upstream router. vnic VLAN = 2002. Bay 7 10-port switch Usage Case 1 Storage Array HS22 Blade #1 w/vnic Blade #2 Blade #3 Router Bay 9 10-port switch Legends: RED vnic Group 1, VLAN 2001 (outer tag) No inner tag default VLAN GREEN vnic Group 2, VLAN 2002 (outer tag) Inner tags for data VLANs 10 and 20 BLUE Non-vNIC ports, tagged 10 and 20 No untagged (native) VLAN Figure 4-1 Usage case 1 (BNT, Reprinted by Permission) The switches are both configured to use one vnic from the blade in each group and one uplink (EXTernal) port in each group. The vnic in group 2 carries data bearing VLANs 10 and 20. This is configured in the blade (such as by the Linux vconfig command) and in the upstream switch, but is not configured on the switches in bay 7 and 9. A second blade is shown that is not using vnic functionality. It also uses data VLANs 10 and 20. In this case the VLANs are configured in the embedded switches as well as on the blade itself and the upstream router. 66 IBM BladeCenter Virtual Fabric Solutions

The connection to the storage farm does not use multiple VLANs. 4.1.1 BNT switch configuration for example 1 The vnic blade is in slot 1. Non-vNIC blades are in slots 2 and 3. The blade in slot 2 is in vnic group 2 even though it is not using vnic functionality. The blade in slot 3 is configured conventionally. Note that if the failover function is to be used, both switches have to be configured identically. Figure 4-1 on page 66 assumes that this is the case. Example 4-1 shows the BladeOS configuration. Example 4-1 BladeOS script /cfg/virt/vnic on /cfg/virt/vnic/port 1/vnic 1/ena bw 50 /cfg/virt/vnic/port 1/vnic 2/ena bw 50 /cfg/virt/vnic/vnicgrp 1/ena failover ena addvnic 1.1 addport EXT8 vnicvlan 2001 /cfg/virt/vnic/vnicgrp 2/ena failover ena addvnic 1.2 addport EXT10 addport INT2 vnicvlan 2002 /cfg/port EXT9/tag ena /cfg/vlan 10/ena def EXT9 INT3 /cfg/vlan 20/ena def EXT9 INT3 Example 4-2 shows the iscli configuration. Example 4-2 iscli script vnic enable interface port EXT9 tagging exit vlan 10! vlan 20 enable name "VLAN 10" member EXT9 member INT3 enable name "VLAN 20" member EXT9 member INT3 Chapter 4. Example of use 67

! vnic port INT1 index 1 bandwidth 50 enable exit! vnic port INT1 index 2 bandwidth 50 enable exit! vnic vnicgroup 1 vlan 2001 enable failover member INT1.1 port EXT8 exit! vnic vnicgroup 2 vlan 2002 enable failover member INT1.2 port INT2 port EXT10 exit 4.1.2 Upstream router configuration (Cisco) The upstream router connects to the embedded Virtual Fabric switch in slot 7 and the one in slot 9 on two ports each. These ports cannot be aggregated together, but they do not need to use Spanning Tree to avoid a loop because one of the two ports is in vnic group 2, and the other is not using vnic functionality. Both ports need to be configured as follows: switchport mode trunk switchport trunk allowed vlan 10,20 Both ports have the same VLAN membership even though they are configured differently on the BNT switch. Note that whereas there is no native VLAN configured, this can be done if desired. Additional configuration of interface VLAN 10 and interface VLAN 20 is required to assign IP addresses for routing to and from these VLANs. 4.1.3 Linux host configuration In this section we discuss a Linux host configuration Blade 1 with vnics In the example, blade 1 has four NICs available, which consist of two sets of vnics corresponding to the two ports on the Virtual Fabric adapter. Each set has a vnic facing the 68 IBM BladeCenter Virtual Fabric Solutions

direct connection to the storage farm, and one facing the router. The one facing the router has an address on VLAN 10 and one on VLAN 20. Assume that eth4 and eth6 are in vnic group 1 and connect to the storage farm. They can be bonded together using mode=1 and given an IP address with an ifconfig bond0 command such as: ifconfig bond0 up <address> <mask> Assume that eth5 and eth7 are in vnic group2, which faces the router, and uses VLANs 10 and 20, both of which are tagged as shown in the switch configuration above. These interfaces can also use bonding mode=1, creating bond1. To create the VLAN interfaces, the following commands can be used: vconfig add bond1.10 10 vconfig add bond1.20 20 < ifconfig bond1.10 up <ip address> <mask> ifconfig bond1.20 up <ip address> <mask> Blades 2 and 3 Blade 2 does not have an address on the subnet (or VLAN), which connects directly to the storage farm. It is possible for blade 2 to reach the iscsi storage farm via the router with additional configuration effort if this is desired. Blade 3 is configured conventionally on the embedded Virtual Fabric switch modules. Port INT3 has the appropriate VLAN membership. Since blades 2 and 3 do not use vnics, they have two conventional 10Gb NIC interfaces, eth2 and eth3. These can be bonded in the same way as the router-facing set of vnics (creating bond1) and are configured the same way: vconfig add bond1.10 10 vconfig add bond1.20 20 ifconfig bond1.10 up <ip address> <mask> ifconfig bond1.20 up <ip address> <mask> 4.2 Troubleshooting commands for vnic There are a variety of condition that can lead to a vnic not passing traffic. For traffic to be successfully passed, all of the following must be true: The Emulex NIC must be installed in all server blades that are going to use vnic functionality. Other 10Gb NIC products are not supported. The appropriate Emulex driver for the operating system on the server blade must be installed. Chapter 4. Example of use 69

The BNT Virtual Fabric switch must be installed in the BladeCenter chassis for use with server blades. Other 10Gb high-speed switches do not support vnic functionality. The BNT switch configuration must include the following: The vnic function must be explicitly enabled. One to four vnics must be defined and enabled for each server blade where vnics are to be used. All of the vnics defined above must be added to vnic groups. The vnic groups must be enabled and must be associated with a vnic VLAN. As described above, the chosen VLAN number cannot be used within the switch for any other purpose. The vnic instances that are visible to the operating system must be configured appropriately for VLAN membership and either given static IP addresses or configured to use DHCP. The commands used to view vnic information about the Virtual Fabric switch are: 1. Display information about vnic definitions and status. If all is specified, all possible vnic instances are dumped whether or not they are completely defined. BladeOS command: /info/virt/vnic/vnic {all} iscli command: show vnic vnic {all} 2. Display information about defined vnic groups. If a group number is specified, only that group is displayed. BladeOS command: /info/virt/vnic/vnicgrp {group number} iscli command: show vnic vnicgroup {group number} 3. Dump the information in the first two steps. If all is specified, information about all possible vnic instances is included. BladeOS command: /info/virt/vnic/dump {all} iscli command: show vnic information-dump Example 4-3 shows sample output from the dump command. Example 4-3 Sample output from the dump command Router#sh vnic information-dump vnic feature: ON vnic vnicgroup Vlan MaxBandwidth Link -------- --------- ------ ------------ --------- INT1.1 1 2001 50 disabled INT1.2 2 2002 50 disabled ------------------------------------------------------------------------ vnic Group 1: enabled ------------------------------------------------------------------------ VLAN : 2001 Failover : enabled vnic Link ---------- --------- INT1.1 disabled Port Link 70 IBM BladeCenter Virtual Fabric Solutions

---------- --------- UplinkPort Link ---------- --------- EXT1 disabled ------------------------------------------------------------------------ vnic Group 2: enabled ------------------------------------------------------------------------ VLAN : 2002 Failover : enabled vnic Link ---------- --------- INT1.2 disabled Port Link ---------- --------- INT2 disabled UplinkPort Link ---------- --------- EXT6 disabled Router# With the all option, the output includes vnic instances that are partially or incorrectly configured (Example 4-4). Example 4-4 Sample output from the all command Router#sh vnic vnic all vnic vnicgroup Vlan MaxBandwidth Link -------- --------- ------ ------------ --------- INT1.1 1 2001 50 disabled INT1.2 2 2002 50 disabled INT1.3 # * 0 disabled INT1.4 # * 0 disabled INT2.1 # * 0 disabled INT2.2 # * 0 disabled INT2.3 # * 0 disabled INT2.4 # * 0 disabled <some ports deleted> INT14.1 # * 0 disabled INT14.2 # * 0 disabled INT14.3 # * 0 disabled INT14.4 # * 0 disabled # = Not added to any vnic group * = Not added to any vnic group or no vlan set for its vnic group Router# The various operating systems show the vnic instances in separate ways. Chapter 4. Example of use 71

Figure 4-2 shows the vnic instances in VMware ESX. Figure 4-2 vnic in VMware Note that the speed of the various vnics is displayed. It can be changed dynamically, and if this is done, a refresh of the display shows the new bandwidth (Figure 4-3). Also, if a vnic is disabled due to failover, it displays as down. Figure 4-3 VMware display after a refresh 72 IBM BladeCenter Virtual Fabric Solutions

Red Hat Enterprise Linux provides a graphical display of network interfaces using the neat utility. Each instance might need to be added to the Devices tab before it can be activated (Figure 4-4). Figure 4-4 vnics as displayed in Red Hat Enterprise Linux Windows Server 2008 displays the various instances (Figure 4-5). It also reflects the bandwidth of each instance. If these bandwidths are changed from the switch (command line or GUI), refreshing the Windows display shows the new bandwidth. vnic instances that are not configured or that are down due to a failover event are shown as network cable unplugged. Figure 4-5 Windows Server 2008 Chapter 4. Example of use 73

74 IBM BladeCenter Virtual Fabric Solutions

Abbreviations and acronyms ACL AFT ALB ASIC BBI BNT BS CEE CIN CLI DAC DCBX DHCP DMA ETS EXT FC FDX FIP FTP GUI HSSM I/O IBM ID IEEE IGMP INT IP IT ITSO KVM LACP LAN LDAP LED Access control list Adapter fault tolerance Adaptive load balancing Application-specific integrated circuit Browser-based interface Blade Network Technologies, Inc. Bachelor of Science Converged Enhanced Ethernet Cisco, Intel, Nuova Command-line interface Dual address cycle Data Center Bridging Exchange Dynamic Host Configuration Protocol Direct memory access Enhanced Transmission Selection External Fibre Channel Full duplex FCoE Initialization Protocol File Transfer Protocol Graphical user interface High speed switch module Input/output International Business Machines Identifier Institute of Electrical and Electronics Engineers Internet Group Management Protocol Internal Internet Protocol Information technology International Technical Support Organization Keyboard video mouse Link Aggregation Control Protocol Local area network Lightweight Directory Access Protocol Light emitting diode LR LSO MAC MSTP NIC NTP OS OSI OSPF PCI PDF PFC POST PVRST PXE RIP RSS RSTP SATA SFP SFT SNMP SOL SQL SR SSH STP TACACS TCP TFTP UDP USB UTP VE VFA VFS VGRP VLAN VM Long range Large send offload Media access control Multiple Spanning Tree Protocol Network interface card Network Time Protocol Operating system Open Systems Interconnect Open Shortest Path First Peripheral Component Interconnect Portable Document Format Priority-based Flow Control Power-on self test Per VLAN Rapid Spanning Tree Preboot execution Environment Routing Information Protocol Receive-side scaling Rapid Spanning Tree Protocol Serial ATA Small form-factor pluggable Switch fault tolerance Simple Network Management Protocol Serial over LAN Structured Query Language Short range Secure Shell Spanning Tree Protocol Terminal Access Controller Access Control System Transmission Control Protocol Trivial File Transfer Protocol User datagram protocol Universal serial bus Unshielded twisted pair Virtualization Engine Virtual Fabric Adapter Virtual Fabric Switch VLAN group Virtual LAN Virtual machine Copyright IBM Corp. 2010. All rights reserved. 75

VNIC VRRP WRR Virtual network interface card Virtual router redundancy protocol Weighted round-robin 76 IBM BladeCenter Virtual Fabric Solutions

Related publications The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this paper. IBM Redbooks For information about ordering these publications, see How to get Redbooks on page 78. Note that certain documents referenced here might be available in softcopy only. IBM BladeCenter Products and Technology, SG24-7523 Emulex Virtual Fabric Adapter (CFFh) for IBM BladeCenter, TIPS0748 BNT Virtual Fabric 10Gb Switch Module for IBM BladeCenter, TIPS0708 Other publications These publications are also relevant as further information sources: Virtual Fabric Adapter Installation and User s Guide http://www.ibm.com/support/docview.wss?uid=psg1migr-5082945 Virtual Fabric Adapter Release Plan http://www.ibm.com/support/docview.wss?uid=psg1migr-5082945 IBM BladeCenter vnic Tutorial http://www.ibm.com/support/docview.wss?uid=psg1migr-5080917 BNT Virtual Fabric 10 Gb Switch Module Application Guide http://www.ibm.com/support/docview.wss?uid=psg1migr-5080917 IBM BladeCenter Interoperability Guide http://www.ibm.com/support/docview.wss?uid=psg1migr-5073016 Online resources These Web sites are also relevant as further information sources: IBM BladeCenter FCoE product page http://www.ibm.com/systems/bladecenter/hardware/openfabric/fcoe.html Emulex Virtual Fabric solution page http://www.emulex.com/solutions/connectivity-solutions/ibm-bladecenter-virtualfabric.html Copyright IBM Corp. 2010. All rights reserved. 77

Emulex Virtual Fabric Adapter product page http://www.emulex.com/solutions/ibm-bladecenter-virtual-fabric/emulex-virtualfabric-adapter.html BNT Virtual Fabric 10Gb Switch Module product page http://www.bladenetwork.net/bnt-virtual-fabric-10g-switch-module.html How to get Redbooks You can search for, view, or download Redbooks, Redpapers, Technotes, draft publications and Additional materials, as well as order hardcopy Redbooks publications, at this Web site: ibm.com/redbooks Help from IBM IBM Support and downloads ibm.com/support IBM Global Services ibm.com/services 78 IBM BladeCenter Virtual Fabric Solutions

Back cover IBM BladeCenter Virtual Fabric Solutions Redpaper Solutions based on the Emulex Virtual Fabric Adapter and the BNT Virtual Fabric 10Gb Switch Concepts and benefits of virtual NICs (vnics) Step-by-step instructions for how to create various vnic solutions The deployment of server virtualization technologies in data centers requires significant efforts in providing sufficient network I/O bandwidth to satisfy the demand of virtualized applications and services. For example, every virtualized system can host several dozens of network applications and services, and each of these services requires certain bandwidth (or speed) to function properly. Furthermore, because of different network traffic patterns relevant to different service types, these traffic flows might interfere with each other, leading to serious network problems, including the inability of the service to perform its functions. The IBM BladeCenter Virtual Fabric virtual network interface controller (vnic) solution addresses these issues. The solution is based on the IBM BladeCenter H chassis with a 10 Gb Converged Enhanced Ethernet infrastructure built on the BNT Virtual Fabric 10Gb Switch Modules in the chassis and Emulex Virtual Fabric Adapters in each blade server. In this configuration, each blade server has up to 20 Gb of available network bandwidth with the ability to split it between up to eight vnics starting from 100 Mb to a maximum of 10 Gb per one vnic with 100 Mb increments. This IBM Redpaper publication is for customers who want to learn how to implement a IBM BladeCenter Virtual Fabric vnic solution. It provides step-by-step instructions for how to configure the switch. This paper assumes that you are already familiar with IBM BladeCenter and with the procedures to configure switches, such as the BNT Virtual Fabric 10Gb Switch Module. INTERNATIONAL TECHNICAL SUPPORT ORGANIZATION BUILDING TECHNICAL INFORMATION BASED ON PRACTICAL EXPERIENCE IBM Redbooks are developed by the IBM International Technical Support Organization. Experts from IBM, Customers and Partners from around the world create timely technical information based on realistic scenarios. Specific recommendations are provided to help you implement IT solutions more effectively in your environment. For more information: ibm.com/redbooks REDP-4673-00