Chapter 7 Troubleshooting Network Performance Issues Objectives

Transcription

1 Troubleshooting Network Performance Issues Objectives Explain the process of network application optimization Describe the operation of Cisco base-lining tools. Describe the operation of Cisco network optimization tools. Describe typical switch performance issues. Describe typical router performance issues.

2 Network Application Services 1. Baseline application traffic: establishing a reference from which service quality and application delivery effectiveness can be measured. 2. Optimize the network: Apply optimization or control techniques to enhance application performance. 3. Measure, adjust, and verify: Assess the effectiveness of each successive optimization initiative. 4. Deploy new applications: updates to existing applications to meet changing business needs.

3 Netflow NetFlow efficiently provides a vital set of services for IP applications, including the following: 1.Network traffic accounting 2.Usage-based network billing 3.Network planning 4.Security denial of service monitoring 5.Overall network monitoring Designed by Cisco, NetFlow is now in its ninth version, which is now on the IETF standards track to become an industry-wide standard.

4 Common Netflow Issues The most common issues can be categorized as being related to performance and the exporting process. Limit the number of entries in the cache, or tune the aging timers of NetFlow to prevent performance degradation in the NetFlow-enabled device by due to the consumption of too much memory or CPU cycles. Export problems are typically based on configuration errors or reachability to the NetFlow Collector or server: 1. A destination IP address has not been configured. 2.A source interface has not been configured. 3.A source interface has been configured, but does not have an IPv4 address. 4.A source interface has been configured, but it is not in the up state. 5.The subnet of the destination is unreachable.

5 Common IP SLA Issues Issues related to performance are common because probes will cause a burden if overscheduled, if multiple senders overwhelm one receiver, or if the device is already a bottleneck and its CPU utilization is high. Probe scheduling can be problematic if the clock on the device is out of sync; recommend synchronizing through Network Time Protocol (NTP). Firewalls and access control mechanisms that filter or block traffic.

6 Common NBAR Issues NBAR does not detect traffic that uses nonstandard ports. Check the current NBAR port map using the command show ip nbar port-map. Map multiple ports simultaneously to a single application or protocol, using the command: ip nbar port-map protocol-name [tcp udp] port-number Outdated PDLM

7 Cisco AutoQoS: Automating the Delivery of Network QoS Cisco IOS software offers a portfolio of QoS features that enables customer networks to address voice, video, and data application requirements. Cisco AutoQoS dramatically simplifies QoS deployment by automating Cisco IOS QoS features for voice traffic in a consistent fashion and leveraging the advanced functionality and intelligence of Cisco IOS Software. Cisco AutoQoS provides the user with a simple, intelligent command-line interface (CLI) for enabling campus LAN and WAN QoS for Voice over IP (VoIP) on Cisco switches and routers.

8 Cisco AutoQoS Evolution Cisco AutoQoS VoIP: First phase addressed IP telephony. One command provisions all basic QoS required. Support is provided across broad range of platforms (switches and routers). Cisco AutoQoS for the Enterprise: Second phase extends capabilities (routers only) for data, voice, and video. Two QoS deployment stages: Discovers traffic types, load, and generates policy (NBAR). Implements the generated policy.

9 Configuring Cisco AutoQoS Enterprise: Router Prerequisites It cannot be configured if a QoS policy (service policy) is attached to the interface. CEF must be enabled at the interface or ATM PVC. It classifies an interface as low speed (less than or equal to 768 kbps) or high speed (more than 768 kbps): Correct bandwidth must be configured on all interfaces. If low speed, an IP address must be configured on the interface.

10 Common AutoQoS Issues Mismatched parameters on the two sides of a serial link (e.g. bandwidth). If AutoQoS is disabled on an interface using the no auto qos command, all Cisco AutoQoS generated commands are removed with the exception of those that have been changed/modified.

11 Troubleshooting Performance Problems In general, troubleshooting performance problems is a three-step process: 1. Assessing whether the problem is technical in nature. 2. Isolating the performance problem to a device, link, or component. 3. Diagnosing and resolving the performance degradation at the component level.

12 Troubleshooting Switch Interface Related Performance Problems ALS1#show interfaces FastEthernet 0/1 counters Port InOctets InUcastPkts InMcastPkts InBcastPkts Fa0/ Port OutOctets OutUcastPkts OutMcastPkts OutBcastPkts Fa0/ ALS1#show interfaces FastEthernet 0/1 counters errors Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize OutDiscards Fa0/ Port Single-Col Multi-Col Late-Col Excess-Col Carri-Sen Runts Giants Fa0/ A common cause for interface errors is a mismatched duplex mode between two ends of an Ethernet link.

13 Troubleshooting Switch Port/Interface Issues No cable connected: Connect the cable from the switch to a known good device. Wrong port: Make sure that both ends of the cable are plugged into the correct ports. Device has no power: Ensure that both devices have power. Wrong cable type: Verify that the correct type of cable is being used. Bad cable: Swap the suspect cable with a known good cable. Look for broken or missing pins on connectors. Loose connections: Check for loose connections. Sometimes a cable appears to be seated in the jack, but it is not. Unplug the cable and reinsert it. Patch panels: Eliminate faulty patch panel connections. Bypass the patch panel if possible to rule it out as the problem. Media converters: Eliminate faulty media converters. Bypass the media converter, if possible, to rule it out as the problem. Bad or wrong Gigabit Interface Converter (GBIC): Swap the suspect GBIC with a known good GBIC. Verify hardware and software support for the GBIC.

14 Automatic medium-dependent interface crossover (auto-mdix) Issues This feature automatically detects the required cable connection type (straight-through or crossover) for a connection. This feature is dependent on the speed and duplex auto negotiation - disabling them will also disable auto-mdix for an interface. S1#show interface FastEthernet 0/10 transceiver properties Diagnostic Monitoring is not implemented Name : Fa0/10 Administrative Speed: auto Administrative Duplex: auto Administrative Auto-MDIX: on Administrative Power Inline: N/A Operational Speed: 100 Operational Duplex: full Operational Auto-MDIX: on Media Type: 10/100BaseTX S1(config)# interface FastEthernet 0/1 S1(config-if)# shutdown S1(config-if)# speed auto S1(config-if)# duplex auto S1(config-if)# mdix auto S1(config-if)# no shutdown

15 Troubleshooting Forwarding Hardware Issues: TCAM If frames cannot be forwarded by the TCAM, they will be punted to the CPU for processing. Consequently, if a large amount of traffic is punted to the CPU, the throughput for the traffic concerned will drop and control processes may be effected. SW1#show platform tcam utilization CAM Utilization for ASIC# 0 Max Used Masks/Values Masks/Values Unicast mac addresses: 784/ /99 IPv4 IGMP groups + multicast routes: 144/1152 6/26 IPv4 unicast directly-connected routes: 784/ /99 IPv4 unicast indirectly-connected routes: 272/ /175 IPv4 policy based routing aces: 0/0 30/175 IPv4 qos aces: 768/ /260

16 SW1#show platform ip unicast counts # of HL3U fibs 141 # of HL3U adjs 9 # of HL3U mpaths 2 # of HL3U covering-fibs 0 Troubleshooting Forwarding Hardware Issues: TCAM # of HL3U fibs with adj failures 0 Fibs of Prefix length 0, with TCAM fails: 0 Fibs of Prefix length 1, with TCAM fails: 0 Fibs of Prefix length 2, with TCAM fails: 0 Fibs of Prefix length 3, with TCAM fails: 0 Fibs of Prefix length 4, with TCAM fails: 0 Fibs of Prefix length 5, with TCAM fails: 0 TCAM allocation failures are rare, because switches have more than enough TCAM capacity for the roles that they are designed. MAC attacks which fill up the CAM/TCAM can cause TCAM fails, leading to performance degradation.

17 Troubleshooting Forwarding Hardware Issues: TCAM SW1#sh controllers cpu-interface ASIC Rxbiterr Rxunder Fwdctfix Txbuflos Rxbufloc Rxbufdrain ASIC cpu-queue-frames retrieved dropped invalid hol-block stray rpc stp ipc sw forwarding If the retrieved packet counter in the sw forwarding row is rapidly increasing when you execute this command multiple times in a row, traffic is being switched in software by the CPU instead of in hardware by the TCAM.

18 Troubleshooting Switch CPU Related Performance Problems SW1#show processes cpu sorted CPU utilization for five seconds: 23%/18%; one minute: 24%; five minutes: 17% PID Runtime(ms) Invoked usecs 5Sec 1Min 5Min TTY Process % 0.35% 0.23% 0 IP Input % 0.52% 0.33% 0 HLFM % 0.07% 0.04% 0 HSRP IPv % 0.07% 0.05% 0 Chk heaps Over the past five seconds, the switch consumed 23 percent of the available CPU cycles. 18 percent of CPU cycles were spent on interrupt processing, while only 5 percent was spent on the handling of control plane processes. A switch should not spend a significant amount of CPU time on interrupt processing, because the TCAM should forward the bulk of the traffic. A percentage between 0-5 percent of CPU load spent on interrupts is considered normal, 5-10 percent is deemed acceptable, above 10 percent warrants investigation.

19 An average CPU load of 50 percent is not problematic and temporary bursts to 100 percent are not problematic, as long as there is a reasonable explanation for the observed peaks debugs, SNMP polling, show command, etc. If the high load is mainly caused by processes, identify the responsible processes and see if they can be explained. Processes that can be responsible for high CPU load are the following: 1. IP ARP 2. SNMP Troubleshooting Switch CPU Related Performance Problems 3. IGMPSN 4. IP INPUT

20 PC /24 (VLAN 10) Fa0/11 Fa0/1 S2 PC /24 (VLAN 20) Cisco Storm Control S1#show interfaces accounting vlan10 Protocol Pkts In Chars In Pkts Out Chars Out IP ARP S3 Fa0/3 Fa0/4 Fa0/1 Fa0/2 Fa0/2 Fa0/18 Fa0/1 Fa0/3 Fa0/2 Fa0/3 Fa0/4 Fa0/6 S1 Fa0/4 PC /24 (VLAN 30) Storm control manages how the receiving port handles broadcast traffic. Configures a threshold to drop broadcasts for a certain period of time or until the broadcast flow slows down. In addition, you can shut down the port or send a SNMP trap to an NMS. S1(config)# int range fa0/1 4 S1(config-if-range)# storm-control broadcast level 50 S1(config-if)# storm-control action trap

21 Troubleshooting High CPU Usage Issues on Routers When the router CPU is too busy to forward all packets as they arrive, it may start to buffer packets (increasing latency) or even drop packets. Application traffic passing through the router will be effected, and network performance will suffer. Control plane processes may not be able to get sufficient access to the CPU, which could lead to further disruptions due to failing routing or other control plane protocols. Common router processes that could cause high CPU utilization: 1. ARP Input 2. Net Background 3. TCP Timer

22 Router Switching Operation The newest switching mode is CEF and it is the default, preferred, and recommended switching mode. Note that the switching method used affects the router s performance. The switching method might be altered globally or per interface for several reasons: 1.During troubleshooting, to verify if the observed behaviour is caused by the switching method. 2.During debugging, to direct all packets to CPU for processing. 3.Because some IOS features require a specific switching method.

23 R1(config) #no ip cef R1(config) #int fa0/0 R1(config-if)# no ip route-cache R1#show ip interface fa0/0 fa0/0 is up, line protocol is up < output omitted > Router Switching Operation IP fast switching is disabled IP fast switching on the same interface is disabled IP Flow switching is disabled IP CEF switching is disabled IP Fast switching turbo vector IP multicast fast switching is enabled IP multicast distributed fast switching is disabled IP route-cache flags are Fast Disabling fast switching & CEF increases the load on the system CPU, as every packet is processed by the IP Input process on the router CPU.

24 Router Switching Troubleshooting Tools / /24 Process Switching: R1# sh ip cache CEF: R1 R1# sh ip cef /30 Fa0/0 R1#sh adjacency fa0/0 detail detail R1#sh ip arp /30 Fa0/0 R2

25 Router Packet Forwarding Troubleshooting Tools / /24 R /30 Fa0/ /30 Fa0/0 R2 R1#traceroute R1#show processes cpu exclude 0.00 R1#show ip route R1#show ip cef

26 Troubleshooting Router Memory Issues Memory allocation failures occur when the router has used all available memory, or the memory has been fragmented and the router cannot find a usable available block. Symptoms of memory allocation failures include the following: 1.Messages such as %SYS 2 MALLOCFAIL: Memory allocation of 1028 bytes failed from 0x6015EC84, Pool Processor, alignment 0 being displayed in the router logs. 2.show commands generate no output. 3.Receiving Low on memory messages. 4.Receiving the message Unable to create EXEC no memory or too many processes on the console.

27 Troubleshooting Router Memory - Memory Leak Leak R1#show buffers < output omitted > Middle buffers, 600 bytes (total 17602, permanent 170): 11 in free list (10 min, 400 max allowed) hits, 148 misses, 671 trims, 657 created 0 failures (0 no memory)! Output omitted for brevity There are a total of middle buffers in the router, and only 11 are in the free list. processes are not releasing the buffers. Leak is caused by a software bug and the only solution is to upgrade Cisco IOS Software on the device to a version that fixes the issue.

28 Troubleshooting Router Memory - Buffer Leak R1#show memory allocating-process totals Head Total (b) Used(b) Free(b) Lowest(b) Largest(b) Processor 62A2B2D I/0 ED Allocator PC Summary for: Processor PC Total Count Name 0x6136A5A Init 0x608E TTY data 0x6053ECEC Process Stack 0x Init R1#show interfaces < output omitted > input queue 76/75, 1250 drops Output queue 0/40, 0 drops;! Output omitted for brevity The full input queue (76/75) warns of a buffer leak. Values 76 = number of packets in the input queue, 75 = the maximum size of the input queue; number of packets in the input queue is larger than the queue depth. This is called a wedged interface - the router no longer forwards traffic that enters the affected interface.

29 Troubleshooting Network Performance Issues Objectives Explain the process of network application optimization Describe the operation of Cisco base-lining tools. Describe the operation of Cisco network optimization tools. Describe typical switch performance issues. Describe typical router performance issues.

30 Any Questions?