Network Probe Description The MCNC Client Network Engineering group will install several open source network performance management tools on a computer provided by the LEA or charter school to build a Network Probe. Once configured, the Network Probe can be deployed and managed by the LEA or charter school to collect performance data to facilitate problem identification and remediation, and capacity planning. The Network Probe will gather network performance data in four critical areas: 1. Utilization 2. Latency & Packet Loss 3. Throughput 4. Packet Analysis Utilization is a measurement of how much data (bits per second) is being transferred across a network link. Utilization data can be used not only to monitor network links but also to plan for future capacity needs. The Network Pprobe will use SNMP to poll devices on either or both of the endpoints of the link to gather their reported utilization levels. The probe will tabulate the utilization levels and present the data in a graph form. The Network Probe uses a tool called Cacti to measure utilization. Figure 1.1 Cacti Utilization Graph Latency is a measurement of how long packets take to travel across a link. High levels of latency could be indicative of network congestion or a misconfigured device interface. The network probe sends test packets in fixed increments and collects data on their round trip times (RTT). The results are presented in a graph format over a selected time span. The Network Probe uses a tool called SmokePing to collect data on network latency and packet loss. The results are sorted by RTT and the median value is selected as the one to represent with a dark marker. The remaining values are drawn in successively lighter shades of gray stripes based on their increasing delta from the median value. Heavy fluctuation of the RTT sample
shows as smoke and can indicate an overloaded or poorly performing network. If a packet fails to return it is noted as a lost packet. The number of lost packets is indicated by the color of the median marker. Figure 2.1 SmokePing Latency Graph Throughput is a measurement of how much data can be pushed through a network link. Throughput is measured using software running on both a network client and a network server on opposite sides of the link. The network probe acts as the throughput software server. The client software can be run on a mobile device such as a laptop so that throughput can be measured and compared from different points on network. The Network Probe has the server software installed and uses a tool called NDT developed by I2 to measure network throughput performance. NDT measurements are affected by numerous factors including but not limited to network utilization at the time the test was run, TCP network parameters and packet loss.
Testing network path for configuration and performance problems -- Using IPv4 address Checking for Middleboxes.................. Done checking for firewalls................... Done running 10s outbound test (client to server)..... 95.20 Mb/s running 10s inbound test (server to client)...... 91.22 Mb/s The slowest link in the end-to-end path is a 2.4 Gbps OC-48 subnet Information: Other network traffic is congesting the link Server '10.100.1.100' is probably behind a firewall. [Connection to the ephemeral port failed] Client is probably behind a firewall. [Connection to the ephemeral port failed] ------ Web100 Detailed Analysis ------ Web100 reports the Round trip time = 1.45 msec;the Packet size = 1448 Bytes; and There were 6586 packets retransmitted, 14101 duplicate acks received, and 16301 SACK blocks received Packets arrived out-of-order 29.01% of the time. The connection stalled 20 times due to packet loss. The connection was idle 4.02 seconds (40.20%) of the time. This connection is sender limited 26.67% of the time. This connection is network limited 73.04% of the time. Web100 reports TCP negotiated the optional Performance Settings to: RFC 2018 Selective Acknowledgment: ON RFC 896 Nagle Algorithm: ON RFC 3168 Explicit Congestion Notification: OFF RFC 1323 Time Stamping: ON RFC 1323 Window Scaling: ON; Scaling Factors - Server=8, Client=8 The theoretical network limit is 133.83 Mbps The NDT server has a 16384 KByte buffer which limits the throughput to 88581.32 Mbps Your PC/Workstation has a 1898 KByte buffer which limits the throughput to 10260.35 Mbps The network based flow control limits the throughput to 15.29 Mbps Figure 3.1 NDT Throughput Results Packet Analysis involves gathering a sequential collection of networks packets as they pass through a physical connection. A network probe will categorize the captured packets and present various summaries of the traffic in a human readable form. The Network Probe uses a tool called ntop to collect data for packet analysis. Tshark can also be added to the Network Probe which allows for packet captures in a pcap format which are readable by the Wireshark packet analysis tool.
Figure 4.1 ntop Packet Analysis Sample by Host Figure 4.2 ntop Packet Analysis Sample by Traffic Type
Getting Started Using a Network Probe This section covers general suggestions for preparing to add a Network Probe to your environment. Links to documentation for the performance management tools are provided. Specifics questions on setting up a Network Probe are available by emailing cne@mcnc.org. 1. Utilization (Cacti, http:// www.cacti.net) Must enable SNMP on all reporting devices Cacti relies on data reported through the SNMP protocol. Devices (routers, switches, servers, access points, etc.) you wish to poll for data must have SNMP enabled. The community string configured for SNMP must be entered into Cacti s configuration. Do not use default SNMP Community strings Default SNMP strings such as public are easy to guess and users on your network may exploit this vulnerability. Poll statistics on both sides of network link Whenever possible devices on both end of a network link should be polled and graphed. This allows administrators to observe any variance in utilization on either side which would suggest a high level of dropped packets on the link. Poll both MDF and IDF(s) A remote location may have a single MDF hosting the location s edge device or it may also have one or more secondary IDF facilities. Polling MDFs and IDFs allows the administrator to observe problems within the remote location s LAN. 2. Latency (SmokePing, http://oss.oetiker.ch/smokeping/) Must permit ICMP to/from Probe Measuring latency on a network link relies upon the ability to send ICMP ping packets across the network. Access control lists must permit ICMP to be sent both to and from the network probe. Measure both MDF and IDF(s) Measuring latency to and from a remote location s MDFs and IDFs offers the same benefits as measuring utilization to these different facilities. Problems can often be isolated to specific segments of the location s LAN. MCNC monitors LEA NCREN connections MCNC monitors LEA NCREN connections for both latency and utilization. Graphs can be found on an LEA s portal page at https://www.mcnc.org/our-community/k12. 3. Throughput (NDT, http://www.web100.org) Tests are run from remote site back to the core (Probe) The probe should be setup in a centralized location as close to or within the same switch fabric as the LEA s WAN connections.
Non-Intrusive vs. Intrusive testing (Iperf, JPerf) NDT is a non-intrusive test so that it will not interfere with ongoing user traffic. Tests can therefore and run during the workday for realistic results. Iperf (not included with CNE Network Probe) is an intrusive form of testing which can obstruct other network traffic. Java client and command-line web100clt client NDT provides a Java client which can be run from any Java enabled web browser. Command-line clients are also available including a Linux client. Probe uses TCP 3001-3003 for incoming connections Access control lists must allow these TCP ports to be sent to the NDT server while also allowing return traffic from the NDT server which will have a source port in the 3001-3003 range. 4. Packet Analysis (ntop, http://www.ntop.org) Requires mirror port In a switched network a packet analysis tool requires a mirror port to which a switch will send copies of the data being collected. Without a mirror port the ntop server for example will only see traffic sent to or from itself. An alternative to configuring a mirror port is to use a non-switching hub which then serves both the ntop server s interface and the interface the passing the traffic to be analyzed. Uses packet filtering parameters (see tcpdump) ntop will capture a large amount of data in a short time on a busy network. ntop can be configured to filter traffic as it is being collected. Traffic can also be filtered during the analysis stage using ntop s GUI but oftentimes it is desirable to filter out unneeded traffic as it is being captured. ntop uses the same filtering syntax as tcpdump, http://www.rationallyparanoid.com/articles/tcpdump.html Can contain personally identifying information ntop will gather hostnames, traffic types and traffic destinations. The data gathered by ntop could be used to capture personally identifying information. Ideal for identifying immediate issues Since ntop gathers a large amount of data about the network s traffic it often is difficult to discern the cause of an acute problem. Restarting the ntop service or starting the service only when needed allows all of ntop s counters to be cleared so that an immediate problem is often easier to identify. Tshark also available Tshark can also be added to a CNE Network Probe when it is desirable for the network administrator to use the Wireshark packet analysis tool. Packet captures can be saved into a pcap format and downloaded to an administrator s workstation.
Acquiring a Network Probe There are two steps LEA s and charter schools seeking to add a Network Probe: 1. Contact the CNE group at cne@mcnc.org to open a service request ticket. 2. Ship the LEA-owned hardware to MCNC, ATTN: Network Probe The minimum system requirements for the Network Probe are as follows: Intel Core 2 or better 1GB+ RAM SATA/SATA II drive 2 NICs* PXE Bootable *A second network interface card is highly recommended if the probe is to be used for packet analysis Once the Network Probe has been built and the performance management software verified, it will be shipped back to the LEA or charter school. The CNE group is also available to help the LEA or charter school configure the performance management tools for the specific environment to be managed. Prior to contacting the CNE group plug the probe into the network using its primary NIC. Power the probe on and be prepared to provide remote access to the CNE assigned to your probe setup. A screen-sharing service such as join.me or Google Hangouts can be used so that LEA technology staff can see firsthand how to configure the tools listed here.