SolarWinds Technical Reference

Transcription

1 SolarWinds Technical Reference Best Practices for Troubleshooting NetFlow Introduction... 1 NetFlow Overview... 1 Troubleshooting NetFlow Service Status Issues... 3 Troubleshooting NetFlow Source Issues... 4 NetFlow Version 9 Specific Issues... 8 The NTA SQL Server Troubleshooting Vendor Exporters This Technical Reference is intended to help the reader identify and correct issues for an Orion NetFlow Traffic Analyzer deployment. network management simplified - solarwinds.com

2 Copyright SolarWinds. All rights reserved worldwide. No part of this document may be reproduced by any means nor modified, decompiled, disassembled, published or distributed, in whole or in part, or translated to any electronic medium or other means without the written consent of SolarWinds. All right, title and interest in and to the software and documentation are and shall remain the exclusive property of SolarWinds and its licensors. SolarWinds Orion, SolarWinds Cirrus, and SolarWinds Toolset are trademarks of SolarWinds and SolarWinds.net and the SolarWinds logo are registered trademarks of SolarWinds All other trademarks contained in this document and in the Software are the property of their respective owners. SOLARWINDS DISCLAIMS ALL WARRANTIES, CONDITIONS OR OTHER TERMS, EXPRESS OR IMPLIED, STATUTORY OR OTHERWISE, ON SOFTWARE AND DOCUMENTATION FURNISHED HEREUNDER INCLUDING WITHOUT LIMITATION THE WARRANTIES OF DESIGN, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL SOLARWINDS, ITS SUPPLIERS OR ITS LICENSORS BE LIABLE FOR ANY DAMAGES, WHETHER ARISING IN TORT, CONTRACT OR ANY OTHER LEGAL THEORY EVEN IF SOLARWINDS HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Microsoft and Windows 2000 are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. Graph Layout Toolkit and Graph Editor Toolkit Tom Sawyer Software, Oakland, California. All Rights Reserved. Portions Copyright ComponentOne, LLC All Rights Reserved. Document Revised: 10/04/2011

3 Best Practices for Troubleshooting NetFlow 1 Introduction This paper focuses on NetFlow troubleshooting and contains information on troubleshooting other flow analysis technologies, such as sflow, J-Flow and IPFIX. Because of the architectural similarities between these flow technologies the troubleshooting techniques herein apply to all of them. This paper will focus strictly on troubleshooting NetFlow issues with regard to the Orion NetFlow Traffic Analyzer (NTA). For information on the CBQoS feature of NTA see the Orion NetFlow Traffic Analyzer Administrator Guide. For basic information on NetFlow see New to Networks Volume 3 NetFlow Basic and Deployment Strategies. NetFlow Overview The term NetFlow is often used interchangeably between the three major components of NetFlow technology, the NetFlow cache and the NetFlow exporter on the router or switch, and the NetFlow collector used to analyze the flow information. The NetFlow cache is the active monitoring of traffic and only exists on a NetFlow enabled device. The NetFlow exporter involves sending completed flow information from the device to a NetFlow collector. Here s what is happening step-wise: 1. Users, also known as endpoints, are accessing the information via the central router. 2. As the users data flows into a WAN router s interface, the NetFlow cache creates records about the flows and saves the records in the NetFlow cache. 3. As flows expire the WAN router exports the flow information to the NetFlow collector. This architecture is common across all the previously mentioned flow technologies. Before troubleshooting a new installation, be certain that be prerequisites for NetFlow are all in place. These prerequisites include: The NTA module has been installed without any error messages during the installation. The target network devices (exporters) have been configured to for NetFlow export to the Orion NTA server using the proper Orion IP address and port.

4 2 Best Practices for Troubleshooting NetFlow The SQL database has sufficient available space in accordance with the Orion NetFlow Traffic Analyzer Administrator Guide. Problems with NetFlow present themselves at the NetFlow collector, so that is a logical place to begin looking for the source of an issue. The NetFlow sources, NetFlow Collector Status and Last 25 Traffic Analyzer Events resources on the Orion NetFlow Traffic Analyzer Summary screen are typically the best places to see the overall health of NetFlow. By customizing the NTA summary page you can place these three resources together. This provides the ability to assess the NTA status at a glance, as seen below. To quickly review this screen, the Last Received NetFlow (circled) updates each time NetFlow packets are received on a known NetFlow source. The second section shows the status on the NTA receiver. If this is not up NTA cannot received NetFlow at all. The Last NetFlow Collectors resource is a good place to view recent problems with NetFlow. Probably the most common NetFlow troubleshooting method starts with examining the three resources seen above. The Traffic Analyzer events screen can display events regarding the NetFlow service, unknown NetFlow sources and many other events specific to NTA. The Events window is a good place to look for obvious problems. The NetFlow Collector Status indicates obvious problems with the NTA server.

5 Best Practices for Troubleshooting NetFlow 3 Troubleshooting NetFlow Service Status Issues If the NetFlow Service fails, this will be seen as shown below. To troubleshoot the NTA collector service not being in the Started state, as seen below, open Administrative Tools > Services.and check the SolarWinds NetFlow Service is started. If it is not, then highlight the service and click Start on the right side of the services pane. Also check if other SolarWinds services are not in the Started state, particularly the SolarWinds Orion Module and the SolarWinds Information Service services. If the service starts and then stops again, there is an underlying reason causing the service to fail. The most common issues that cause the NetFlow service to shut down are: The NTA module is trial which has expired. Normally this will be seen in a yellow banner on the top of the Orion web console. If you have an expired trial, you can apply a purchased license from the Customer Portal Follow the directions in the NetFlow Administrator Guide, Chapter 2. There is an issue with the connection to the SQL database or with the functioning of SQL. Check that the SQL server is up and that the SQL server has sufficient CPU and memory available. Also check disk queue write length. For more information on where to find these metrics see the Managing Orion Performance technical reference. As a final attempt to reconcile the NetFlow Service being down, run the Configuration Wizard checking all three boxes, Web, Services and Database, as shown below.

6 4 Best Practices for Troubleshooting NetFlow If these steps do not bring the collector back to a stable, up status, and you have an active maintenance contract, you will need to open a ticket with Technical Support. Troubleshooting NetFlow Source Issues NetFlow sources are the devices exporting flows to the Orion NTA server. When sources stop sending data to NTA, or NTA stops processing data this will typically be noticed then one of the NetFlow One common forgotten step with adding NetFlow sources is to manage the device and interface sending flow information into Orion NPM. When this occurs the below will appear in the NetFlow Events screen. Correcting this issue is trivial. Click on the Manage this device text in the event message and NPM will guide you through adding the device and interface. If you add the device but fail to add the interface the following event will be seen. This can be easily corrected by clicking on the Edit this device link and NPM will guide you to add the interface.

7 Best Practices for Troubleshooting NetFlow 5 You may notice that flows are not being received or processes properly when all of the resources on a NetFlow page have messages stating that there is no data for the requested time period, as seen below. There are two typical causes for this. The NetFlow exporter was not configured to export during the time you have requested. There is an issue causing the NTA server to not display the flows. If you believe that the device is correctly exporting flows, but NTA is not displaying data, a packet capture should be taken from the NTA server interface. Wireshark is a commonly used and free tool for capturing packets and analyzing protocols. For information on installing and using packet capture tools see the documentation for that particular package. The packet capture should be run for about 5 minutes to make sure any NetFlow packets are captured. The below capture was run for ten minutes, and then the captured packets were sorted by protocol looking for NetFlow packets. NetFlow is most commonly listed as CFLOW protocol in protocol analyzers. Note that there are no CFLOW packets captured. Typically this is because of one of the following: The exporter is not configured to send NetFlow packets to the NTA server. There is a device between the exporter and the NTA server blocking the flow data. NetFlow device configurations vary by model, NetFlow version, and IOS version. All configurations implement three crucial functions. Turn on the NetFlow cache Apply NetFlow to interfaces Direct flow exports to the collector (NTA server). Examine the device for the required configurations according to your manufacturer, model and software to ensure that each of these functions is in place. For Cisco IOS devices the following should be in place: (config)# ip flow-export version {5/9}

8 6 Best Practices for Troubleshooting NetFlow (config-if)# ip flow ingress (config-if)# ip flow egress Show commands are very helpful in quickly assessing NetFlow configuration and operation on a Cisco device. First we will begin by examining the flow exporter on the device using the abbreviated version of the # show ip flow export command. This screen shows us that two of the three crucial NetFlow configurations are in place. These are: The main cache is enabled for NetFlow v5. The exporter is enabled and exporting to and on port 2055 We can also see that flow records (flows) have been exported in 5101 UDP packets. We can also see that there are no failed exports. While there are no apparent errors one item is missing. The NTA server we are using is at IP address which is not listed as export destination. The address was the old NTA server IP, so the configuration needs to be updated to export to the new NTA server IP. The below commands correct this situation. Notice that we removed the old IP export address prior to adding the new one. NetFlow only allows two export destinations, so adding the new address without deleting the old will result in the below error. The next configuration to check is to see if NetFlow has been applied to the interface we wish to monitor. A simple method is to use the #show ip flow interface command, as seen below.

9 Best Practices for Troubleshooting NetFlow 7 The above screen shows which interfaces have NetFlow applied and the flow direction being monitored. Ingress is defined as into the interface from the network, egress is the opposite. Now that we have corrected the issue with the exporter IP address and have verified that the interfaces are configured, we will look at the packet capture on the NTA server again. We can see that NetFlow packets are being received at the NTA server interface. Looking at the NTA Summary view seen below we can see that NTA is processing these packets by noting the correct time stamp on the Last Received NetFlow column.

10 8 Best Practices for Troubleshooting NetFlow Drilling down to the NTA Interface Details page verifies that NetFlow is being properly received, processed and displayed, as shown below. NetFlow Version 9 Specific Issues Cisco implemented version 9 to create a structure for selective flow field exporting. This feature is known as Flexible NetFlow (FNF). Version 5 exporters send seven key fields and a hand full of non-key fields. Key fields are used to define a unique flow. All fields are part of the IP datagram header. The version 5 key fields are: Source IP Destination IP Protocol Source port Destination port Source interface index Type of Service

11 Best Practices for Troubleshooting NetFlow 9 The below screen shot shows these key fields along with several non-key version 5 fields. The above flow is for an ICMP (protocol 1) packet. ICMP does not use a port number, so the exporter sets the port to 0. Version 9 not only allows you to add many additional non-key fields, but it also allows you to redefine the key fields. This caused issues for the NetFlow collector in attempting to identify unique flows. Using NetFlow version 9 creates no issues unless FNF is configured on the exporter and the seven version 5 key fields are not included in the configuration. Some devices only support version 9 and this is typically not an issue. To ensure that FNF exports are processed by NTA, be certain to include all of the seven version 5 key fields. FNF enabled version 9 exporters send two types of flow information, flow templates and flow records. Both are required for FNF to function. If you have a version 9 exporter and the NetFlow source appears and disappears randomly this may be due to the flow template expiring on the NTA server. The below packet capture shows version 9 FNF data being received but no template is available to define the flow fields. The flow is telling the NTA server to refer to template 265, but the server does not have that template in memory. To fix this issue add the following to the device NetFlow configuration. (config)#flow-export template timeout-rate 1

12 10 Best Practices for Troubleshooting NetFlow This will configure the exporter to send the template every minute. Some device will re-index the interface index numbers frequently. While Orion and NTA do compensate for interface re-indexing through device discovery, some devices will lose their NTA IF index due to rapid re-indexing. This is seen in the NetFlow Events resource as seen below. Creating an automatic rediscovery of these devices to run every 15 minutes will rectify this issue. Only include the devices have the re-indexing issues in this discovery job. The NTA SQL Server NTA is a very data intensive technology. As such, NTA has the possibility of overtaxing some SQL server implementations. The Managing Orion Performance technical reference contains detailed information on Orion and SQL performance. There are also several options you have to improve SQL performance from the NTA server. These include: Plan your NetFlow exporters with the network topology in mind. There is not much use in enabling NetFlow on every NetFlow capable device. Data duplication and unnecessary data will only serve to make flow useful information difficult to find. Use reasonable data retention times. One primary use case for NTA is to quickly identify bottlenecks and the cause behind them. Data overload will eventually slow the NTA server. Make use of the Top Talkers optimization feature in NTA. This feature can greatly enhance NTA performance. Troubleshooting Vendor Exporters Troubleshooting J-Flow and SFlow exporters varies greatly depending on the device model, software level and other factors. Below are useful links for troubleshooting issues on these types of exporters. Juniper J-Flow Setting up J-Flow on a J-Series router Example working J-Flow configuration [Relevant to J-Series and SRX devices] SRX Getting Started - Configure J-Flow HP sflow Implementation HP SFlow