AIX EtherChannel Load Balancing Options Document Author: Cindy K Young Additional Author(s): Jorge R Nogueras Doc. Organization: Document ID: TD101260 Advanced Technical Support Document Revised: 10/28/2003 Product(s) covered: AIX; Ethernet; pseries; RS/6000; TCP/IP Abstract: This article is an overall view of the user-configurable AIX V5 EtherChannel load balancing behavior options, including the new Hash Mode parameter. AIX EtherChannel Load Balancing Options This article is an overall view of the user-configurable AIX EtherChannel load balancing behavior options, including the new Hash Mode parameter. Introduction. A new SMIT menu option called "Hash Mode" appears when AIX users install software APARs available in November, 2003. The menu gives mode EtherChannel users the option of distributing outgoing traffic based upon TCP or UDP port s. Software support is provided in APAR IY48331 for AIX V 5.1 and APAR IY48170 for AIX V 5.2 Load balancing behavior. Two SMIT-selectable load balancing methods for outgoing traffic are offered -- round robin, which spreads the outgoing traffic evenly across all the adapters in the EtherChannel; and, which selects the adapter by algorithm. The Hash Mode parameter, shown in the following SMIT configuration menu, determines the numerical used in the algorithm. Change / Show Characteristics of an EtherChannel / Link Aggregation Type or select s in entry fields. Press Enter AFTER making all desired changes. EtherChannel / Link Aggregation EtherChannel / Link Aggregation Adapters Backup Adapter Delete Backup Adapter Add Main Adapter Delete Main Adapter Add Backup Adapter Enable Alternate Alternate Enable Gigabit Ethernet Jumbo Frames Mode Hash Mode Internet to Ping [] Number of Retries [3] Retry Timeout (sec) [1] Apply change to DATABASE only ent6 ent1,ent2,ent4,ent5 NONE [0x000000000000] The following chart summarizes the valid load balancing option combinations offered in the SMIT screen. 1 of 5 12/10/04 10:27 AM
MODE or 8023ad round robin HASH MODE src_dst_port src_port dst_port OUTGOING TRAFFIC DISTRIBUTION (across adapter ports within the EtherChannel) The traditional AIX behavior. The adapter selection algorithm uses the last byte of the destination IP address (for TCP/IP traffic) or MAC address (for ARP and other n-ip traffic). This mode is typically the best initial choice for a server with a large number of clients. The outgoing adapter path is selected via algorithm using the combined source and destination TCP or UDP port s. Average the TCP/IP address suffix s in the "Local" and "Foreign" columns shown by netstat -an command. Since each connection has a unique TCP or UDP port, the three port-based hash modes provide additional adapter distribution flexibility when there are several, separate TCP or UDP connections between an IP address pair. The adapter selection algorithm uses the source TCP or UDP port. In netstat -an command output, the port is the TCP/IP address suffix in the "Local" column. The outgoing adapter path is selected via algorithm using the destination system port. In netstat -an command output, the TCP/IP address suffix in the "Foreign" column is the TCP or UDP destination port. Outgoing traffic is spread evenly across all the adapter ports in the EtherChannel. This mode is the typical choice for two hosts connected back-to-back (i.e. without an intervening switch). Round Robin. All outgoing traffic is spread evenly across all of the adapters in the EtherChannel. It provides the highest bandwidth optimization for the AIX server system. While round robin distribution is the ideal way to utilize all the links equally, AIX administrators should consider that it also introduces the potential for out-of-order packets at the receiving system. Standard algorithm. The algorithm is used for both and IEEE 802.3ad-style link aggregations. AIX divides the last byte of the "numerical " by the number of adapters in the EtherChannel and uses the to identify the outgoing link. If the is zero, the first adapter in the EtherChannel is selected; one means second adapter, and so on. (The adapters are selected in the order they are listed in the SMIT menu.) The "Hash Mode" selection determines the numerical used in the calculation. By, the last byte of the destination IP or MAC address is used in the calculation, but the source and/or destination TCP or UDP port s may also be utilized. These alternatives allow the AIX administrator to fine-tune the distribution of outgoing traffic across the real adapters in the EtherChannel. In mode, the adapter selection algorithm is applied to the last byte of the destination IP address for IP traffic. For ARP and other n-ip traffic, the same formula is applied on the last byte of the destination MAC address. Unless there is an adapter failure which causes a failover, all traffic between a host pair in mode goes out over the same adapter. For IP traffic, the port selected could be changed by selecting a different hash mode or by implementing a different IP address (perhaps via IP aliases) at the target system. In src_dst_port hash mode, the source and destination port s are added, then divided by two. The resultant whole number ( decimals) is plugged into the algorithm. TCP or UDP traffic is sent on the adapter selected by the algorithm and selected hash mode. Non-TCP or UDP traffic will fall back to the algorithm, meaning the last byte of either the destination IP or MAC address. Incoming traffic distribution. Remember that the AIX adapter selection applies to outgoing packets only. The packets incoming to the AIX system are governed by the algorithm configured at the switch. The traffic distribution schemes used by the two components need t match -- the algorithms can be (and often are) different. For example, the switch might direct packets based on the XOR of source and destination MAC addresses. In this case, the destination MAC address is the same for packets going to the AIX system, but the different source address causes the packets to be sent over different adapters. Understanding packet distribution. Curious about how packets are being distributed over the ports on your system? Use the entstat or netstat command to view aggregate statistics of all the adapters in the EtherChannel. For example, netstat -v ent6 reports the key information about EtherChannel device ent6. Statistics for outgoing packets are a result of AIX's configuration. If round robin is configured, for example, 2 of 5 12/10/04 10:27 AM
statistics will be nearly identical for each underlying adapter. The incoming packet statistics reflect the network switch configuration. Use the netstat -an command to view the TCP or UDP source and destination port address s. Predicting adapter utilization -- Example Environment. The effects of different hash mode s on the adapter selection algorithm are best illustrated in an example. In this fictitious environment, the EtherChannel (ent6) is configured as IP address 10.9.8.1 (the en6 interface) and is comprised of individual adapters ent1, ent2, ent4, ent5. To demonstrate the adapter selection, the example system communicates with six systems with sequential IP addresses ranging from 192.26.91.100 through 192.26.91.105. The outgoing adapter port is selected via the mathematical of the calculation hash mode numerical / the number of real adapters in the EtherChannel. There are four adapters in this EtherChannel, so the numerical is always divided by four. The numerical changes from the last byte of the destination IP address to the source port, then to the destination port, and then to the average of the two port s in the four Hash Modes. It is this change in numerical that causes the variation in the adapter selection. The result of the calculation is the mathematical. If the is zero, the first adapter in the SMIT list is selected. In the example, this is adapter ent1. If the is three, the fourth adapter (ent5 in our sample) is selected, and so forth. The complete matchup of the s, the selection order, and the actual adapters in the example are shown in the following figure. Remainder Adapter in list Actual Adapter 0 1st ent1 1 2nd ent2 2 3rd ent4 3 4th ent5 Figure 1: Adapter selection in the sample environment. The following partial netstat -an command output shows IP addresses and TCP/UDP ports for nine active connections the local system might have with the remote systems 192.26.91.100 through 192.26.91.105. In this case, the "Local " column reflects the EtherChannel's 10.0.8.1 IP address and the "Foreign " column reflects the IP address of the remote system. The numerical s appended to these local and foreign IP addresses are the source and destination ports, respectively, used in three of the four hash modes. # netstat -an Active Internet connections (including servers) Proto Recv-Q Send-Q Local Foreign udp4 0 0 10.9.8.1.36156 192.26.91.105.6000 tcp4 0 0 10.9.8.1.23 192.26.91.102.44070 tcp4 0 0 10.9.8.1.23 192.26.91.103.44075 tcp4 0 0 10.9.8.1.23 192.26.91.105.44078 tcp4 0 0 10.9.8.1.23 192.26.91.101.44078 tcp4 0 0 10.9.8.1.23 192.26.91.104.44089 tcp4 0 0 10.9.8.1.23 192.26.91.100.32776 tcp4 0 3 10.9.8.1.23 192.26.91.100.32777 tcp4 0 0 10.9.8.1.23 192.26.91.104.44411 The following chart shows the mathematical result (the ) of applying the four hash modes to the algorithm. Check the in the Figure 1 chart to identify the real outgoing adapter selected. Line # Default src_port dst_port src_dst_port * 1 105 1 36156 0 6000 0 42156 / 2 2 2 102 2 23 3 44070 2 44093 / 2 2 3 of 5 12/10/04 10:27 AM
3 103 3 23 3 44075 3 44098 / 2 1 4 105 1 23 3 44078 2 44101 / 2 2 5 101 1 23 3 44078 2 44101 / 2 2 6 104 0 23 3 44089 1 44112 / 2 0 7 100 0 23 3 32776 0 32799 / 2 3 8 100 0 23 3 32777 1 32800 / 2 0 9 104 0 23 3 44411 3 44434 / 2 1 * here the address s are first averaged (added, then divided by two). The resultant whole number (excluding decimals) is divided by the four adapters in the example environment. Figure 2. Comparison of results: Standard adapter selection algorithm with various hash mode options. Observations. In the hash mode, the numerical used in the calculation is the last byte in the destination system IP address. Because this number is the same for all connections to the same destination IP address, all traffic between a host pair in mode goes out over the same adapter. Using this hash mode, traffic will be fairly well spread over the adapters if the random nature of the IP addresses result in an even selection and if all of the remote systems do generally the same things (meaning, for instance, one IP address does t have a huge download). In this example, the traffic is unbalanced when using src_port hash mode. Traffic shown on line 1 is sent out adapter ent1 and traffic shown on lines 2-9 will be sent over adapter ent5. Adapters ent2 and ent4 are t used. Why? The source port address is 23 (the well-kwn port for telnet) on all of these connections, hinting at the way the connection was established. This indicates that for the connections on lines 2-9 the remote system telnet'ed to the local system. In dst_port hash mode, the outgoing traffic is fairly well balanced across the adapters in the EtherChannel. If, however, the local system established the connections, you might see the same results skew seen in the src_port hash. In src_dst_port hash mode, the averaging of the fields seems to balance the outgoing traffic fairly well. The skew in s disappears. So, regardless of the direction in which the connection is initiated, the resulting s distribute across the adapter ports. Considerations. The single most important consideration when choosing a load balancing method is nature of the traffic that is being sent across the EtherChannel. Yet, the final choice will probably be a trade-off that provides the best distribution for a subset of this traffic. Depending upon the environment, one administrator may choose to accommodate the majority of the traffic while ather opts to design for the most important form of traffic. There is single choice that will yield a perfect distribution all the time because the type of traffic being handled by the typical EtherChannels varies. It may, for example, handle mixtures of streaming TCP traffic one moment, some ARP traffic the next, then a huge file transfer followed by short-lived UDP exchanges. Administrators can use these key questions begin the process of selecting the mode of operation and hash mode best suited for an environment. Look for patterns and predictability. Network attachment. Is the local system connected to an Ethernet switch? Is it connected directly (back-to-back) with ather host? Direction of connection establishment. Do remote systems typically connect to the local host? Is the local system a server for client systems? Does the local host establish connections to other hosts? IP addresses. Look for IP address patterns. Does the local host have a few, but long-lived TCP connections with the same remote hosts? Conversely, does it have many, short-lived connections to a variety of remote hosts? Traffic type. The main traffic load may be TCP/IP, but ARP and other n-ip traffic is distributed differently in mode. Is there any other significant type of traffic? Is there a key application that generates a predictable load? Conclusions. The results of any mode can be skewed if the numeric s are skewed. In fact, the load balancing options have been added since the initial AIX V4.3.3 EtherChannel implementation for just this reason. There is one-size-fits-all solution to unique environments. The trick is to choose a mode that results in varied numeric s so that the load is distributed across all of the adapters in the EtherChannel. Generally speaking, round robin is ideal for back-to-back connections running jumbo frames. In this environment, there is intervening switch, so there is chance that processing at the switch could alter 4 of 5 12/10/04 10:27 AM
the packet delivery time, order, or adapter path. On this direct cable network path, the receiver sees packets exactly as sent. Jumbo frames (9000 byte MTU) always yield better file transfer performance than traditional 1500 byte MTUs. In this case, however, they add ather benefit. These larger packets take longer to send so it is less likely that the receiving host would be continuously interrupted with out-of-order packets. Round robin mode can be implemented in other environments but at increased risk of out-of-order packets at the receiving system. This risk is particularly high when there are few, long-lived, streaming TCP connections. When there are many such connections between a host pair, packets from different connections could be intermingled, thereby decreasing the chance of packets for the same connection arriving out-of-order. Check for out-of-order packet statistics in the "tcp" section of the netstat -s command output. A steadily-increasing indicates a potential problem in traffic sent from an EtherChannel. If out-of-order packets are a problem on a system that must use traditional Ethernet MTUs and must be switch attached, try the various hash modes offered in mode operation. If particular hash mode is obvious, administrators may try each of them, testing and analyzing the results before moving to the next. Each mode has a particular strength but the "" and "src_dst_port" modes are the logical starting points as they are more widely applicable. The hash mode works well when the local host establishes connections to many different IP addresses. It is a good first choice for most busy systems because the random nature of IP address traffic tends to distribute the traffic in an acceptable manner. If the local host establishes lengthy connections to few IP addresses, look at one of the TCP or UDP hashing options. The "src_dst_port" hash mode option is probably the best initial choice because considers both the source and the destination TCP or UDP port s. In this mode, all of the packets in one TCP or UDP connection are sent over a single adapter so they are guaranteed to arrive in order, but the traffic is still spread because connections (even to the same host) may be sent over different adapters. The results of this hash mode aren't skewed by the connection establishment direction because it uses both the source and destination TCP or UDP port s. Use the "src_port" or "dst_port" hash mode options if port s change from one connection to ather and the "src_dst_port" option is t yielding a desirable distribution. For instance, if the local host is an FTP server, the source TCP or UDP port will be more or less constant across connections. The netstat -s command would reflect 21, which is the FTP well-kwn port. In this case, the "dst_port" hash mode option would be a better choice. If, on the other hand, the local host primarily establishes connections to other machines, the source TCP or UDP port s will be random, making the "src_port" hash mode a better option. Classification: Software Category: Planning and Design Platform(s): RS/6000 or IBM eserver pseries O/S: AIX Keywords: AIX EtherChannel, NIB, network interface backup, load, balancing, traffic, SMIT, configuration, methods, options, hash mode,, algorithm 5 of 5 12/10/04 10:27 AM