CHAPTER 5 This chapter explains the procedures for troubleshooting link bundles and load balancing on the Cisco ASR 9000 Aggregation Services Router. A link bundle is a group of ports that are bundled together and act as a single link. The advantages of link bundles are: Multiple links can span several LCs to form a single interface; thus, the failure of a single link does not cause a loss of connectivity. Bundled interfaces increase bandwidth availability, because traffic is forwarded over all available members of the bundle. Therefore, traffic can move onto another link if one of the links within a bundle fails. This allows you to add or remove bandwidth without interrupting packet flow. This chapter contains the following sections: Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing, page 5-115 Troubleshooting Problems with Link Bundles, page 5-118 Troubleshooting Layer 2 Bundles and Load Balancing, page 5-122 Troubleshooting Layer 3 Bundles and Load Balancing, page 5-124 Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing Cisco Express Forwarding (CEF) uses the path information in the IP routing table to balance traffic over link bundles. For this reason, verifying correct load balancing with CEF begins with confirming the contents of the IP routing table and CEF database. Verifying Routing Table Entries for Parallel Links, page 5-115 Verifying the CEF Database and Measuring Flows, page 5-117 Verifying Routing Table Entries for Parallel Links Perform this procedure to verify the contents of the IP routing table. SUMMARY STEPS 1. show route destination-address 5-115
Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing Chapter 5 2. configure 3. router ospf process 4. maximum paths number 5. end 6. show route destination-address 7. show ospf process interface brief 8. show running-config router ospf process show route destination-address RP/0/RSP0/CPU0:router# show route 10.1.2.1 configure Displays the routes to a destination address. Use a destination address on another host that is reachable through the parallel links. Verify that number of routes in the routing table equals the number of parallel links. If you have fewer routes than expected, continue with this procedure. Enters global configuration mode. RP/0/RSP0/CPU0:router# configure router ospf process Enters configuration mode for the OSPF process. Step 5 Step 6 RP/0/RSP0/CPU0:router# router ospf 200 maximum paths number RP/0/RSP0/CPU0:router# maximum paths 3 end RP/0/RSP0/CPU0:router# end show route destination-address RP/0/RSP0/CPU0:router# show route 10.1.2.1 Configures the maximum number of paths over which to load balance. By default, OSPF balances up to 4 equal-cost paths. Ends the configuration process. Enter yes at the prompt to commit the changes. Displays the routes to a destination address. Verify that number of routes in the routing table equals the number of parallel links. If you have fewer routes than expected, continue with this procedure. 5-116
Chapter 5 Troubleshooting Routing and CEF Issues Related to Bundles and Load Balancing Step 7 Step 8 show ospf process interface brief RP/0/RSP0/CPU0:router# show ospf 200 interface brief show running-config router ospf process RP/0/RSP0/CPU0:router# show running-configuration router ospf process Shows interface information for all routes to the destination address, which displays the cost metric. OSPF balances loads over equal-cost routes only, so verify that the interfaces have equal costs. To load balance over unequal paths, use Enhanced Interior Gateway Routing Protocol or Interior Gateway Routing Protocol (EIGRP/IGRP) as the IGP instead. Displays the running configuration for the OSPF process. This is another way to determine if the interfaces have different costs. Verifying the CEF Database and Measuring Flows SUMMARY STEPS DETAILED STEPS Perform this procedure to verify the contents of the CEF database. 1. show cef ipv4 [prefix [mask]] interface-type interface-path-id] [detail] [location node-id] 2. show cef [ipv4 ipv6] exact-route source-address destination address [protocol type] [source-port source-port] [destination-port destination-port] [ingress-interface type interface-path-id] [policy-class value] [detail location node-id] 3. show interfaces [type interface-path-id all local location node-id] [accounting brief detail summary] show cef ipv4 [prefix [mask]] interface-type interface-path-id] [detail] [location node-id] RP/0/RSP0/CPU0:router# show cef ipv4 10.1.2.1 detail Displays the CEF forwarding table. Verify that it contains the same interfaces that the routing table has for this destination. 5-117
Troubleshooting Problems with Link Bundles Chapter 5 show cef [ipv4 ipv6] exact-route source-address destination address [protocol type] [source-port source-port] [destination-port destination-port] [ingress-interface type interface-path-id] [policy-class value] [detail location node-id] Displays the exact route that a specific flow would take, including the egress interface for a specific source and destination IP. Use this command for several flows to verify that they are distributed equally over the parallel interfaces. RP/0/RSP0/CPU0:router# show cef exact-route 192.168.254.1 10.1.2.1 protocol ospf source-port 5500 destination-port 80 ingress-interface gi0/6/5/4 show interfaces [type interface-path-id all local location node-id] [accounting brief detail summary] Displays the traffic rates by interface. Use this command to verify that the simulated traffic takes the expected egress interface. RP/0/RSP0/CPU0:router# show interfaces accounting rates Troubleshooting Problems with Link Bundles This section explains how to troubleshoot problems with link bundles. It contains the following subsections: Bundle Does Not Come Up, page 5-118 Bundle Member Not Distributing, page 5-119 Bundle Not Using MAC-Address From Backplane, page 5-119 Layer 3 Data Traffic Not Flowing, page 5-120 Ping Failed over Bundle, page 5-120 Layer 3 Packets Not Synching Over Bundle, page 5-121 Layer 2 Traffic Not Flowing, page 5-121 Bundle Statistics, page 5-122 Bundle Does Not Come Up Ensure that the member port is not shutdown. Ensure that the MAC burned-in address (BIA) of the port is valid. RP/0/RSP0/CPU0:router# show interface If running Link Aggregation Control Protocol (LACP), ensure that LACP packets are able to send and receive accordingly. If LACP packets are not able to send and receive accordingly, check interface counters to identify at what stage packets are dropped. RP/0/RSP0/CPU0:router# show lacp counters 5-118
Chapter 5 Troubleshooting Problems with Link Bundles View LACP statistics. RP/0/RSP0/CPU0:router# show lacp Ensure that the other side of the link is up (bundle and members). RP/0/RSP0/CPU0:router# show bundle Bundle Member Not Distributing Ensure that the member is up. RP/0/RSP0/CPU0:router# show interface node-id Ensure that the remote side is up. Ensure that the LACP parameters are the same on both sides. a. If LACP is enabled, check its status. RP/0/RSP0/CPU0:router# show lacp bundle b. Ensure that bundle members have the same characteristics. RP/0/RSP0/CPU0:router# show running interface interface-name c. If the bundle members have different characteristics, make them all the same. d. Ensure that LACP packets are transmitted and received. RP/0/RSP0/CPU0:router# debug bundlemgr local packets port node-id Workaround If the bundle with LACP cannot come up, use one side of the bundle in passive mode and the other in active mode. At least one side must be active. Bundle Not Using MAC-Address From Backplane Ensure that the backplane MAC is programmed. Note that this command has to be run in the admin mode. RP/0/RSP0/CPU0:router(admin)# show diag chassis eeprom-info Display the backplane information. RP/0/RSP0/CPU0:router# show controllers backplane bpe-trace 5-119
Troubleshooting Problems with Link Bundles Chapter 5 Layer 3 Data Traffic Not Flowing Regular Interface (No Subinterfaces) View the Address Resolution Protocol (ARP). RP/0/RSP0/CPU0:router# show arp Verify that the lag table is programmed properly in the hardware. RP/0/RSP0/CPU0:router# show interface bundle-ether bundle-id View the running configuration information. RP/0/RSP0/CPU0:router# show running-config View information about packets forwarded by CEF. Step 5 Step 6 RP/0/RSP0/CPU0:router# show cef RP/0/RSP0/CPU0:router# show cef hardware ingress location node-id RP/0/RSP0/CPU0:router# show cef hardware egress location node-id Subinterface Troubleshoot Layer 3 IPv4 traffic. Ensure that VLAN traffic coming in matches that on the incoming interface. Ping Failed over Bundle View the ARP. RP/0/RSP0/CPU0:router# show arp View the ARP information on the particular LC or RSP. Step 5 Step 6 Step 7 Step 8 Step 9 RP/0/RSP0/CPU0:router# show arp location node-id RP/0/RSP0/CPU0:router# show cef hardware detail location node-id ingress RP/0/RSP0/CPU0:router# show interface Use the hash calculator to determine which bundle member (interface) to test. Remove the interface from the bundle. Assign an IP address to the interface. Ping the interface. Ensure that the ARP is resolved between the router and the node being pinged. 5-120
Chapter 5 Troubleshooting Problems with Link Bundles 0 1 2 3 4 5 Ensure that the MAC address in the ARP table of the other side corresponds to that on the router. Ensure that the MAC address of the bundle is valid. Ensure that the routing and hardware routing table has an entry to the next hop. Check the interface counters to see if ping packets are transmitted and being received on the router member port of the bundle. Check the ucode counters to see where packets are dropped on the incoming or outgoing member of the bundle. Make sure that the table lookup (TLU) entries are allocated and bundle adjacency information is properly programmed. show cef adajacency bundle-type bundle-number hardware egress detail location location-id show cef adajacency bundle-type bundle-number hardware ingress remote detail location location-id Workaround Try a different port. Layer 3 Packets Not Synching Over Bundle View the interface information: RP/0/RSP0/CPU0:router# show interface Turn on the debug of that protocol or look at the protocol counters to see if the protocol packets are being sent and received. If the protocol packets are not being sent or received, check the interface counters to see if interface indicates packets in and out. If the interface level indicates that packets are coming in and out but not reaching protocol, check the ucode counters to see if there are any drops. Layer 2 Traffic Not Flowing VPLS Verify that the AC is up. RP/0/RSP0/CPU0:router# show l2vpn bridge-domain Verify that the bridge domain is up. RP/0/RSP0/CPU0:router# show l2vpn bridge-domain 5-121
Troubleshooting Layer 2 Bundles and Load Balancing Chapter 5 Look for MTU mismatches. RP/0/RSP0/CPU0:router# show l2vpn bridge-domain detail VPWS View brief information on configured cross-connects. RP/0/RSP0/CPU0:router# show l2vpn xconnect summary RP/0/RSP0/CPU0:router# show l2vpn xconnect state RP/0/RSP0/CPU0:router# show controllers bundle bundle-ether bundle-id location node-id Bundle Statistics Layer 2 statistics are not supported in the show interface accounting command for bundle interfaces in the current release. Troubleshooting Layer 2 Bundles and Load Balancing This section describes how to troubleshoot Layer 2 bundles and load balancing. It includes the following topics: Verifying the Bundle Status, IGP Route, and CEF Database, page 5-122 Viewing the Expected Paths and Measuring the Flows, page 5-123 Verifying the Bundle Status, IGP Route, and CEF Database SUMMARY STEPS CEF uses the path information in the IP routing table to balance traffic over multiple links. For this reason, confirming proper CEF load balancing begins with confirming the contents of the IP routing table. When troubleshooting a bundle, verify that the bundle is up and that the IGP route to the desired destination includes the bundle interface. 1. show running-config 2. show bundle {Bundle-Ether Bundle-POS} interface-path-id 3. show interface {Bundle-Ether Bundle-POS} bundle-id 4. show arm router-id 5. show controllers bundle {Bundle-Ether Bundle-POS} bundle-id location node-id 6. show route destination-address 7. show cef ipv4 prefix 5-122
Chapter 5 Troubleshooting Layer 2 Bundles and Load Balancing DETAILED STEPS show running-config RP/0/RSP0/CPU0:router# show running-config show bundle {Bundle-Ether Bundle-POS} interface-path-id RP/0/RSP0/CPU0:router# show bundle bundle-ether 12 show interface {Bundle-Ether Bundle-POS} bundle-id RP/0/RSP0/CPU0:router# show interface bundle-ether 12 show arm router-id Displays the running configuration. Verify that the configuration related to the bundles are correct. Displays the bundle status. Verify that the bundle has the expected number of links. If not, troubleshoot the bundle first. Displays the interface status. Verify that the interface is assigned to the bundle. RP/0/RSP0/CPU0:router# show arm router-id Step 5 show controllers bundle {Bundle-Ether Bundle-POS} bundle-id location node-id Step 6 Step 7 RP/0/RSP0/CPU0:router# show controllers bundle bundle-ether 12 location 0/4/CPU0 show route destination-address RP/0/RSP0/CPU0:router# show route 10.1.2.1 show cef ipv4 prefix RP/0/RSP0/CPU0:router# show cef ipv4 10.1.2.1 Displays the routes to a destination address. Use a destination address on another host that is reachable through the bundle. Verify that the route to the desalination address includes the bundle interface. If not, make sure that the bundle interface is included in the IGP process configuration. Displays the CEF forwarding table. Verify that it contains the same bundle interface that the routing table has for this subnet prefix. Viewing the Expected Paths and Measuring the Flows Cisco IOS XR provides a bundle utility that predicts how Layer 2 loads are balanced across member links. This is an interactive tool prompts for the information that the load balancing algorithm uses to allocate flows to member links. 5-123
Troubleshooting Layer 3 Bundles and Load Balancing Chapter 5 SUMMARY STEPS 1. bundle-hash {Bundle-Ether Bundle-Pos} interface-path-id 2. show interfaces [type interface-path-id all local location node-id] [accounting brief detail summary] DETAILED STEPS bundle-hash {Bundle-Ether Bundle-Pos} interface-path-id Launches the bundle-hash utility. This is an interactive utility that prompts for the necessary information. RP/0/RSP0/CPU0:router# bundle-hash bundle-ether 12 show interfaces [type interface-path-id all local location node-id] [accounting brief detail summary] RP/0/RSP0/CPU0:router# show interfaces tengige 0/5/0/1 Displays interface information, which includes the traffic rates. Use this command for each link in the bundle to verify that the simulated traffic takes the expected link. Use clear counters to make it easier to view the traffic allocation. Troubleshooting Layer 3 Bundles and Load Balancing This section provides commands for troubleshooting Layer 3 bundles and load balancing. Step 5 Step 6 RP/0/RSP0/CPU0:router# show arm router-id RP/0/RSP0/CPU0:router# show bundle RP/0/RSP0/CPU0:router# show interface bundle-ether bundle-id RP/0/RSP0/CPU0:router# show running-config RP/0/RSP0/CPU0:router# show arm router-ids Find out which member is carrying the traffic out. Step 7 RP/0/RSP0/CPU0:router# bundle-hash bundle-ether bundle-id View each member of the bundle to see which member is actually carrying the traffic out. Step 8 RP/0/RSP0/CPU0:router# show interface Display the exact route, including the egress interface for a specific source and destination IP. Use this command for several flows to verify that they are distributed equally over the parallel interfaces. show cef [ipv4 ipv6] exact-route source-address destination address [protocol type] [source-port source-port] [destination-port destination-port] [ingress-interface type interface-path-id] [policy-class value] [detail location node-id] 5-124