Generic Ethernet Bonding Driver (bonding.o) for Linux

Transcription

1 Generic Ethernet Bonding Driver (bonding.o) for Linux Generic Ethernet Bonding Driver (bonding.o) for Linux 1.0 Overview 1.1 Limitations 2.0 Change History 3.0 Build and Installation Instructions 3.1 Adding the Bonding Module to 2.4.X Kernels 3.2 Adding the Bonding Module to 2.6.X Kernels 4.0 Configuration Information 4.1 Single Bonding Channel Configuration 4.2 Multiple Homogeneous Bonding Channel Configuration 4.3 Multiple Heterogeneous Bonding Channel Configuration 4.4 Configuration for Fail-Over Fail-Over with TX Switches Fail-Over with Fiber Pass-through 4.5 Configuration on Red Hat Enterprise 4.6 Configuration on Suse Enterprise 5.0 WEB Sites and Support Phone Number 6.0 Trademarks and Notices 7.0 License and Disclaimer 1.0 Overview This README is designed to cover installation and configuration of the generic ethernet bonding driver for the IBM Broadcom 5704s Gigabit Ethernet found in the js20 blades. The bonding driver is a kernel module found in the mainline Linux for both the 2.4.x and 2.6.x kernels. The source code used to build the drive can be found in the (kernel base)/drivers/net/bonding directory. The driver provides several advanced features including load-balancing and fault-tolerance for several ethernet adapters. These features are provided by creating virtual bonding interfaces of two or more physical network adapters. The bonding driver supports several different operating modes for these adapter groupings. The bonding driver supports the following modes on the js20 blades: balance-rr: Round-robin load balance active-backup: Active-backup policy balance-tlb: Adaptive transmit load balancing balance-alb: Adaptive load balancing Please refer to the bonding documentation for a more detailed description of each of these modes. The bonding document can be located at (kernel base)/documentation/networking/bonding.txt.

2 1.1 Limitations The current version of bonding has been tested on the latest Red Hat and Suse enterprise distributions for the powerpc 64-bit architecture. This module has been tested up to kernel version Serial-Over-Lan (SoL) issue Because the SoL link is established over the primary ethernet (eth0), any loss of link to eth0 will result in losing your SoL connection. SoL does not fail-over using the bonding driver, this is a hardware limitation. Bladecenter chassis Due to a hardware constraint in the design of the Bladecenter chassis an external link must be made to allow eth0 and eth1 of the same blade to work together on a bonding link. The first solution is to bridge the two external switches found on a bladecenter. This can be accomplished by using an rj45 crossover cable and linking one port of the eth0 switch to another port on the eth1 switch. This has the drawback of consuming one port on each switch. The second solution requires an external switch which both of the eth0 and eth1 switch are connected. There is no way to bridge the bladecenter switches internally. HP2524 switch 802.3ad team member links disconnect and reconnect continuously when connects to the HP2524 switch. This is a 3rd party issue. It is seen only when configuring an 802.3ad team with greater than 2 members on the server and connecting an HP2524 switch, with lacp enabled as passive or active. The HP switch will show an lacp channel being brought up successfully with only 2 members. All other member's links will disconnect and reconnect. This does not occur with a Cisco Catalyst Spanning tree To avoid fail-over issues when using bonding, make sure that spanning tree is disabled on the switch that the network adapter is connected to. Fail-Over mode Arp polling must be used in conjunction with active-backup mode in order for fail-over to occur using the TX switch modules. Arp polling is not necessary if a pass-through fiber module is used. 2.0 Change History Version 1.4 Modified SUSE LINUX configuration in section 4.6. Version 1.3 Added Fail-Over limitation. Updated configuration steps for Fail-Over. Version 1.2 Fix Suse SLES configuration details. Add SoL to limitations. Version 1.1 Use bonding names with the "mode" parameter in the given examples. Minor cleanup and formatting. Version 1.0 Initial document authoring.

3 3.0 Build and Installation Instructions If the command "modprobe bonding" returns no errors and running "lsmod" shows bonding in the table, your kernel already has bonding compiled for it. Please continue to section 4 of this document to configure a bond. If the previous command fails, then bonding is not compiled into your kernel. This can be verified by viewing your kernel build configuration file (kernel base)/.config. The file should contain the line CONFIG_BONDING=m. If it does not, please configure your kernel and enable it as a module. Note The bonding code should not be compiled into the kernel. It is currently not possible in Linux to send loading parameters to modules compiled into the kernel. Please make sure CONFIG_BONDING=m and not CONFIG_BONDING=y. 3.1 Adding the Bonding Module to 2.4.X Kernels Open the kernel configuration (either via make menuconfig or make xconfig) and enable as a module the following path: Network device support ---> < > Bonding driver support Save, exit and build the kernel. 3.2 Adding the Bonding Module to 2.6.X Kernels Open the kernel configuration (either via make menuconfig or make xconfig) and enable as a module the following path: Device Drivers ---> Networking support ---> < > Bonding driver support Save, exit and build the kernel. 4.0 Configuration Information 4.1 Single Bonding Channel Configuration The bonding configuration consists of the following steps (regardless of Linux distribution): 1. Load the bonding.o module with the specific mode. 2. Bring up a bonding device with the proper network configuration. 3. Add the physical adapters to the bond interface. For example, suppose we have two adapters, eth0 and eth1. We wish to have these adapters operate using the active-backup policy using ip address with netmask Here are the commands to run on the root shell prompt: # insmod bonding mode=active-backup On the first line we load the bonding module using active-backup mode. The second line sets up a bonding interface with the name of bond0. The next two lines add adapters to the bonding device. This is the crucial step which actually activates the link. The order the interfaces are added determines fail-over order if the chosen mode does fail-over. In our case above, eth0 is the primary interface and eth1 is our backup interface. Users may also wish to

4 utilize the "primary" option to re-enable a particular interface when it recovers from a failure (this option is described in the standard bonding documentation, as specified in 1.0, above). To remove a bonding interface from active duty, simply use the ifconfig command on the bond interface. In our example above, we would bring down the device with: This will bring the machine back to the initial state prior to any bonding setup. Note that setting the bonding interface's state to down releases all slaves. 4.2 Multiple Homogeneous Bonding Channel Configuration The steps to setup up several bonding channels using the same bonding mode is almost identical to a single channel. The only difference is to use the module parameter max_bonds. This tells the module how many bonding channels of the same type to support. Let's have another example. Suppose we have four adapters, eth0 through eth3. We want eth0 and eth1 on bond0 and eth2 and eth3 on bond1. Both bonding channels are to be run in balance-alb mode. Here are the commands to set this up: # insmod bonding mode=balance-alb max_bonds=2 # ifconfig bond netmask up # ifenslave bond1 eth2 # ifenslave bond1 eth3 Without the max_bonds parameter, the module only supports one interface by default (bond0). Again, bringing down the interfaces is identical to the previous example: # ifconfig bond1 down 4.3 Multiple Heterogeneous Bonding Channel Configuration This is for the user who wishes to run more than one bonding channel in the same box and have at least one of the bonding channels operate in a different manner. For this example, we have four adapters eth0 through eth3. We want bond0 to run eth0 and eth1 in balance-alb and we want bond1 to run eth2 and eth3 in balance-rr. In order to do this, we must load the module twice. When doing so, we have to tell the Linux kernel each copy of the driver must have a different running name. This allows two identical modules run, but using different parameters on each. Here's how we set it up: # insmod -o bonding0 bonding mode=balance-alb # insmod -o bonding1 bonding mode=balance-rr # ifconfig bond netmask up # ifenslave bond1 eth2 # ifenslave bond1 eth3 In this case, doing an lsmod will show both bonding0 and bonding1 as loaded modules. Removing one will not affect the other, so it's safe to bring down bond0 without interrupting traffic on bond1. If we wish to bring everything down from the previous example, we would run the following:

5 # ifconfig bond1 down 1 0 Notice the module names removed are the -o names. There is no longer a module instance named "bonding" in this case. 4.4 Configuration for Fail-Over Fail-Over mode allows multiple ethernet adapters to redundantly support an interface should the currently active one fail. In the case of the js20 there are two modes which can be used depending upon the hardware in the bladecenter Fail-Over with TX Switches If your bladecenter uses the TX switches (the ones with 4 rj45 TX connectors) then this configuration is for you. In this mode, due to a hardware restriction, we must use arp monitoring to determine if our link has gone up or down. You must choose another site on the network to check if the link has been lost. Furthermore, this site must be within the same subnet as the bladecenter; you cannot pass through a gateway or be routed to reach this host. Good examples of sites to monitor in this case would be the local subnet's gateway or the IP of the switch the bladecenter is connected. Let's call this IP address X.X.X.X. Now that we have decided on a host, we can setup the fail-over in the following manner: # insmod bonding mode=active-backup arp_interval=100 arp_ip_target=x.x.x.x This creates a link where eth0 or eth1 will act as with the other interface taking over the job should the first interface lose link or fail. The "arp_interval" is a duration in milliseconds before another arp packet is sent to check if the link is up. The "arp_ip_target" is the address of the machine with which we send our arp packet. As before, we bring down the interface with the same commands: Fail-Over with Fiber Pass-through If your bladecenter uses the TX switches (the ones with 14 SX fiber connectors) then this configuration is for you. In this case, the bladecenter does not have an extra switch to confuse the js20 blades. We will use MII monitoring in this setup. We don't need an IP address to monitor in this case. Here's the setup: # insmod bonding mode=active-backup miimon=100 This creates a link where eth0 or eth1 will act as with the other interface taking over the job should the first interface lose link or fail. The "miimon" is a duration in milliseconds before we check the link status of our adapter. As before, we bring down the interface with the same commands:

6 4.5 Configuration on Red Hat Enterprise Red Hat will not automatically load the network adapter driver unless the ethx device is configured with an IP address. Due to this constraint, users must manually configure a network-script file for all physical adapters that will be members of a bondx link. Network script files are located under /etc/sysconfig/network-scripts. The file name must be prefixed with "ifcfg-eth" and suffixed with the adapter's physical adapter number. For example, to create a network script for eth0, the file name would be /etc/sysconfig/network-scripts/ifcfgeth0. Place the following text in the file: DEVICE=eth0 USERCTL=no ONBOOT=yes MASTER=bond0 SLAVE=yes BOOTPROTO=none Note the DEVICE= line will be different for every ethx device and must correspond with the name of the file. ifcfg-eth8 must have a device line of DEVICE=eth8. This file will ensure the driver is loaded for the bonding layer. The MASTER= line will also depend on the final bonding layer name chosen for your bond. Next, create a bond network script. The file name for this script will be /etc/sysconfig/network-scripts/ifcfg-bondx where X is the number of the bond. For bond0 the file is named "ifcfg-bond0". Within that file, place the following text: DEVICE=bond0 IPADDR= NETMASK= NETWORK= BROADCAST= ONBOOT=yes BOOTPROTO=none USERCTL=no Be sure to change the networking specific lines to match your network. Finally running "/etc/rc.d/init.d/network restart" as root. This will restart the networking subsystem and your bond link should be now up and running. 4.6 Configuration on Suse Enterprise Suse Enterprise's sysconfig configuration system does not support bonding interfaces at this time. There is a way to save a bonding link across reboots, however. As root, you can edit /etc/init.d/boot.local and place the commands you type at a shell in here to create a link. For example, if you wanted to make a simple bonded channel with eth0 and eth1 and have it persist across reboots, do the following: edit /etc/init.d/boot.local and place the following lines in the file: insmod bonding mode=balance-alb # load bonding module insmod bcm5700 # load ethernet driver module ifconfig bond netmask up ifenslave bond0 eth0 ifenslave bond0 eth1 Unfortunately, this will not let you use the familiar "ifup" and "ifdown" commands usually associated with these commands, you can always just re-run the shell script if you want to bring the link back up, as in: # sh /etc/init.d/boot.local

7 And finally, as in the examples above, to bring down the interface at any time, you can use the "ifconfig" and "rmmod" commands. For our example above, you can do the following: # insmod bond0 down # rmmod bcm5700 This will put you in a non-bonding state with all the necessary modules removed. 5.0 WEB Sites and Support Phone Number IBM Support Web Site: IBM Marketing Netfinity Web Site: If you have any questions about this update, or problems applying the update go to the following Help Center World Telephone Numbers URL: Trademarks and Notices The following terms are trademarks of the IBM Corporation in the United States or other countries or both: IBM Netfinity eserver xseries Broadcom and NetXtreme are registered trademarks of Broadcom Corporation. Other company, product, and service names may be trademarks or service marks of others. 7.0 License and Disclaimer Copyright (c) , Broadcom Corporation Copyright (c) , IBM Corporation All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of IBM Corporation nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY

8 OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.