Milestone MS3.7.5: Report on Passive Monitoring Pilot

Transcription

1 Milestone MS3.7.5: Report on Passive Monitoring Pilot Milestone MS Contractual Date: 31/05/08 Actual Date: 21/08/08 Contract Number: Instrument type: Integrated Infrastructure Initiative (I3) Activity: SA3 Work Item: 7 Nature of Deliverable: R (Report) Dissemination Level PU (Public) Lead Partner CESNET Document Code Authors: Sven Ubik (CESNET), Vladimir Smotlacha (CESNET), Szymon Trocha (PSNC), Simon Leinen (SWITCH), Vedrin Jeliazkov (BREN), Ales Friedl (CESNET), Gina Kramer (DANTE) Abstract This document describes the work carried out during the deployment of the Service Activity (SA) 3 passive monitoring pilot in the GÉANT2 () network. This pilot was conducted to follow up the development of passive network monitoring tools in the Joint Research Activity (JRA) 1 activity, in order to test these tools in a large-scale real network environment, gain experience in using these tools, gather user feedback and draw conclusions about the tool s usability and usefulness.

2 Table of Contents 0 Executive Summary vi 1 Introduction 1 2 Deployed Hardware 3 3 Developed and Deployed Software Architecture 8 4 Passive Monitoring Applications ABW Operation Usage Examples Difference from Related Tools Summary Tbwtools Operation Usage Examples Difference from Related Tools Summary Packetloss Operation Usage Examples Difference from Related Tools Summary Burst Operation Usage Examples Difference from Related Tools Summary 37 Document Code:

3 Contents 5 Service Applications Perfmon Operation Usage Examples Difference from Related Tools Summary Servmon Operation Usage Examples Difference from Related Tools Summary 45 6 The Relation of Applications in the SA3 Passive Monitoring Pilot to perfsonar 46 7 Challenges 48 8 Conclusions and Future Work 50 9 References Acronyms 53 Table of Figures Figure 3.1: DiMAPI architecture 8 Figure 4.1: Integrated access to all monitoring applications. 11 Figure 4.2: Map-based user interface of the ABW application. 13 Figure 4.3: ABW user interface. 14 Figure 4.4: Link load plotted with maximum values (left) and without maximum values (right). 15 Figure 4.5: Examples of link load in L3/L4 protocols (left) and application protocols (right) monitored by ABW. 16 Figure 4.6: SNMP link load monitoring. 17 Figure 4.7: Most detailed link load in perfsonar UI (left) and the ABW application (right). 18 Figure 4.8: Tbwtools architecture. 20 Figure 4.9: Tbwtools user interface. 22 Document Code:

4 Contents Figure 4.10: A test connection summary and performance characteristics. 23 Figure 4.11: A test connection summary and performance characteristics. 24 Figure 4.12: Packetloss application principle. 27 Figure 4.13: The main page of the Packetloss user interface. 29 Figure 4.14: Packetloss status and peer statistics. 30 Figure 4.15: Expired and matched flows from SWITCH to PIONIER. 31 Figure 4.16: Packet loss count and rate or real traffic. 31 Figure 4.17: Hades active loss monitoring. 32 Figure 4.18: User interface of the Burst application. 35 Figure 4.19: Distribution and cumulative distribution functions of frame burst sizes in bytes. 36 Figure 4.20: Time evolution of distribution and cumulative distribution of frame burst sizes in bytes as 3d graphs. 36 Figure 5.1: User interface of Perfmon application. 40 Figure 5.2: History of test events found by Perfmon. 40 Figure 5.3: Table of component states provided by Perfmon application. 41 Figure 5.4: Map-based user interface of the Servmon application. 43 Figure 5.5: Example of CPU usage and load monitoring. 44 Figure 5.6: Example of shared memory usage and DAG card drop count monitoring. 44 Document Code:

5 0 Executive Summary Network monitoring plays an important role in network planning, analysing network usage trends and resolving network usage issues. It also provides an information base for researchers of networking technologies. The JRA1 and SA3 activities have respectively developed and deployed the perfsonar framework in the network. perfsonar primarily uses active monitoring tools to measure network characteristics, such as oneway delay or TCP/UDP throughput. However, many interesting characteristics of real traffic in the network are inherent properties of that traffic and cannot be measured by test packets (for example, link load dynamics, real packet loss, composition of protocol traffic and applications, and trends in their usage). Passive monitoring is in principle much more powerful than active monitoring, because it can calculate any possible characteristics of real traffic. SA3 have developed several passive monitoring applications to address this, and deployed them for evaluation on the -NREN (National Research and Education Networks) links of four partners (ACAD [ACAD], CESNET [CESNET], PSNC [PSNC] and SWITCH [SWITCH]). This is the first phase of passive monitoring deployment in the network 1 : ABW (Available Bandwidth) - Monitors link load in short intervals to detect peak load. It also provides information about protocols in different layers that comprise link load, including multicast, IPv6 (Internet Protocol version 6) and some application protocols that use dynamic ports. Tbwtools - Tests TCP throughput between monitoring stations in the network and the user's PC, monitors this test connection by various techniques and provides feedback about connection health to assist in debugging low throughput. Packetloss - Monitors real packet loss of real network traffic, which is generally unrelated to packet loss that can be measured by test packets. Burst - Quantifies the burstiness of traffic independently from time intervals normally used to calculate average load. Perfmon and Servmon - Service applications that support the reliable operation of monitoring applications. 1 The fifth partner LITNET originally included in the pilot withdrew because their planned upgrade of link from 1 Gb/s to 10 Gb/s was delayed and it was not considered economical to buy a 1 Gb/s monitoring card that would need to be replaced soon. Document Code: v

6 Executive Summary The deployment of the pilot proved to be a valuable experience that made it possible to enhance applications to cope with large volumes of real traffic, to test the behaviour of multiple applications simultaneously sharing monitoring stations and to gather feedback from trial users. Multiple technical challenges were successfully tackled (see Challenges on page 48). The passive monitoring pilot deployment significantly extended the portfolio of useful monitoring applications in the network. It pioneered concepts that were not previously available from perfsonar, such as integrated access to all monitoring applications and map-based user interfaces. However, further steps should be taken in the development and deployment of passive monitoring in the and GN3 networks. Passive monitoring services will become increasingly useful as they are deployed across and beyond most of /GN3-NREN links in participating NRENs. New applications will also provide additional benefits, such as Service Level Agreement (SLA) verification for specified user traffic in projects that use the network for more detailed traffic classification in order to understand network usage trends. Document Code: vi

7 1 Introduction Network monitoring plays an important role in network planning, analysing network usage trends and resolving network usage issues. It also provides an information base for researchers of networking technologies. Most network monitoring methods can be grouped into: Active monitoring. Passive monitoring. Network infrastructure monitoring. In active monitoring, test packets are sent across the network to measure various performance characteristics. Active monitoring is a convenient way to measure, for instance, one-way delay over a network. However, the results of active monitoring are only truly valid for the test packets, not for real user traffic. In passive monitoring, test packets are not sent across the network. Instead the characteristics of real network traffic are observed and analysed. Many important network performance characteristics are inherent properties of real traffic, and can therefore only be obtained from passive monitoring. In principle, passive monitoring is a more powerful approach than using test packets, but it is also more computing-intensive, as large volumes of data need to be analysed in real time. It therefore requires relatively expensive packet capture hardware (monitoring cards). The following are examples of performance characteristics that can only be obtained using passive monitoring or a combination of passive and active monitoring: Link load in a wide range of time scales. Protocols and applications used in the network, and trends in their usage. Real packet loss experienced by users. Detailed feedback about connection health for performance debugging. Various kinds of anomalies. Various kinds of security problems. Passive monitoring also has important properties for the quality of monitoring and network operation: It does not affect user traffic. It can run continuously. It identifies characteristics of real user traffic. Document Code:

8 Introduction Network infrastructure monitoring uses information obtained from network components, such as routers and switches. The two most common protocols of network infrastructure monitoring are SNMP and Netflow. Both protocols are commonly used in reliable monitoring which provides useful statistics about network traffic. However, for both protocols only information provided by specific router and switch makes can be obtained. Common limitations are that SNMP provides only high-level statistics about network traffic in coarse time granularity (such as total volume), and that Netflow records include a fixed set of information fields. Netflow records can also be obtained from passive monitoring, in which case they can be extended with additional information fields. The perfsonar system developed in the JRA1 activity is almost exclusively based on active monitoring. Therefore, it was necessary to extend the range of monitoring methods in the network with passive monitoring to provide key performance characteristics inherent to real user traffic, which could not be obtained from active monitoring. The development of passive monitoring tools and infrastructure in the JRA1 activity was the responsibility of CESNET. As the tools matured enough to be deployed in the network, CESNET passed their experience to the SA3 activity and realised a pilot deployment of passive monitoring in four project partners as described in this deliverable. The pilot aimed to extend the portfolio of monitoring services in the network with additional useful characteristics of real network traffic. Document Code:

9 2 Deployed Hardware To capture and process traffic at multi-gigabit rates, it is necessary to use monitoring cards rather than regular Ethernet adapters. Monitoring cards provide two primary advantages: They transfer packets from the network to the memory of a host PC much more effectively, with almost zero CPU load, leaving CPU for processing of packets. Extensive comparative performance measurements of several monitoring cards and Ethernet adapters [TNC08] confirmed that a modern PC can be used for complex monitoring at multi-gigabit rates, provided that a monitoring card is used to capture packets. They usually provide some hardware acceleration, such as filtering, simple classification and packet statistics. For the pilot deployment, several network monitoring options were explored. The Force10 P-series box, which is primarily designed for 10 Gb/s firewalls, was found not to be suitable for the required monitoring purposes. Napatech s 10 Gb/s monitoring cards were not commercially available. The only commercially available monitoring cards for 10 Gb/s links at the time of procuring hardware for the pilot were Endace DAG cards. Therefore, suitable types of these were used for the -NREN links of pilot partners. The deployed infrastructure consists of 7 monitoring stations and one central station for application support and user interfaces 2. Based on previous experiences and performance tests, Supermicro servers were used in the pilot. The hardware of 5 monitoring stations deployed in the pilot is configured as follows: Supermicro X7DB8+ mainboard 2x Xeon DualCore 3 GHz 2x 73 GB SCSI disk 2 GB FB-DDR2 667 MHz RAM 2 Although two monitoring PCs were installed in CESNET as part of the JRA1 activity before the pilot, the software on these two monitoring stations had to be reinstalled. The other 5 monitoring stations were deployed as part of this pilot. Document Code:

10 Deployed Hardware CESNET monitoring stations: Intel SE7520 mainboard 1x Xeon 3 GHz (single core) 73 GB SCSI disk 1 GB RAM Central station: Supermicro mainboard Xeon DualCore 2 GHz 150 GB SCSI disk 2 GB RAM The monitoring cards are connected to the -NREN links by optical splitters. An exception is the ACAD location, where a mirroring port on a router is used to supply traffic to the monitoring card. This allows the use of twisted-pair transceivers which for logistics reasons are more convenient at this location. The following table summarises all 7 monitoring stations and one central station in CESNET: Partner Locality Monitoring station Monitoring card Monitored link ACAD Sofia, Bulgaria sa3-pm5.acad.bg DAG4.5G4 (4x 1 Gb ethernet) ACAD to, to ACAD CESNET Prague, Czech Republic jra1-1.cesnet.cz DAG6.2S (OC-192/STM-64) CESNET2 to jra1-2.cesnet.cz DAG4.3 (2x 1 Gb ethernet) CESNET2 to to CESNET2 sa3-pm.cesnet.cz, alias perfmon.geant2.net (central station) -- PSNC Poznan, Poland pm1.passive.pionier.net.pl DAG6.2S (OC-192/STM-64) to PIONIER pm2.passive.pionier.net.pl DAG6.2S (OC-192/STM-64) PIONIER to SWITCH Geneve, Switzerland gn2-pm1.switch.ch DAG8.2X (10 Gb ethernet) SWITCH to gn2-pm2.switch.ch DAG8.2X (10 Gb ethernet) to SWITCH Table 2.1: Monitoring station summary. Document Code:

11 3 Developed and Deployed Software Although it was planned to use applications that had previously been developed by the JRA1 activity and the LOBSTER [LOBSTER] project, most applications needed significant improvements to provide more performance stability and better user interfaces. The applications need to run in a large real network environment with much higher traffic volumes than in the local networks where they were developed and initially tested. Two new service applications (Perfmon and Servmon) were developed to support the infrastructure. For details of necessary improvements to individual applications, see their respective sections in Passive Monitoring Applications on page 11 and Challenges on page 48. The following table summarises the passive monitoring applications that were deployed within the pilot: Application Provided network characteristics Added value Users ABW Provides the used capacity in a wide range of time resolutions including short load peaks. It also provides distribution of traffic into protocols and applications. Identifies short-term traffic dynamics that can affect added traffic and protocols and applications that use up capacity or behave differently (e.g., elastic vs. nonelastic traffic), including trends in their usage. These characteristics cannot be obtained from active monitoring or network infrastructure monitoring. PERT NOCs Research projects using the network. Advanced users. Tbwtools Provides TCP throughput test with detailed feedback about the test connection for performance debugging. Helps investigate why TCP connections have low throughput. Uses an integration of active, passive and end-host monitoring. PERT Research projects using the network. Packetloss Provides detailed statistics about packet loss of real user traffic. Provides information about packet loss of real user traffic, which is generally different from packet loss of test packets. Measurements confirmed that real user traffic experiences packet loss which is not detected by active monitoring. PERT NOCs Research projects using the network. Advanced users. Document Code:

12 Developed and Deployed Software Burst Provides the burstiness of real traffic. Helps understand traffic dynamics (useful for researchers) and to evaluate conditions expected by added user traffic. PERT NOCs Research projects using the network. Advanced users. Table 3.1: Passive monitoring applications deployed in the pilot. Notes: The originally planned TCMP (Tracefile Measurement Point) application could not be deployed, because it lacks authorisation necessary for this type of application and the original developer in ARNES left the project. CESNET are currently working on adding authorisation to TCMP and expect to complete this shortly after the pilot. A new Burst application was added to the pilot instead of TCMP. Several support applications and important software components were also developed and deployed during the pilot: Application Provided network characteristics Added value Users Perfmon Monitors the state of hardware and software components of monitoring stations, sends alerts and automatically repairs problems where possible. Checks the status of monitoring stations hardware and software components. Users of measurements results benefit from increased application reliability. Servmon Monitors hardware and software resources on monitoring stations, such as CPU usage, CPU load, allocated shared memory and packet drops reported by monitoring cards. Checks resources on monitoring stations. Users of measurements results benefit from increased application reliability. DiMAPI (middleware) Middleware to support the operation of all passive monitoring applications. Enables the development of portable monitoring applications and their concurrent operation in a heterogeneous environment with transparent hardware acceleration. Necessary infrastructure to support applications. Table 3.2: Support applications and software components deployed in the pilot. Document Code:

13 Developed and Deployed Software Notes: Perfmon and Servmon are service applications developed to support reliable operation of the pilot infrastructure. They can also be used outside of the pilot. DiMAPI (Distributed Monitoring Application Programmable Interface) is middleware, rather than an application, but it forms a part of indivisible application functionality. Document Code:

14 Developed and Deployed Software 3.1 Architecture The architecture of passive monitoring applications based on DiMAPI is shown in Figure 3.1. Packets are processed as follows (from the bottom of the picture upwards): Packets are copied from the network by an optical splitter or a mirroring port on a router or switch. Packets are captured by a network card, which can be (in principle) a regular Ethernet adapter or a monitoring card. Packets are then processed by DiMAPI middleware, which is divided into: The mapid and mapicommd daemons running on monitoring stations with monitoring cards distributed across the network. DiMAPI libraries, which are linked to applications that can run on the same or different PCs than the monitoring stations. Typically on application PCs in order to use their separate processing power. Figure 3.1: DiMAPI architecture Document Code:

15 Developed and Deployed Software The deployed infrastructure has the following particular properties: Monitoring cards (rather than regular Ethernet adapters) are used on all monitoring stations. The ABW and Burst applications run on monitoring stations. Their results are retrieved on demand and sent to the central station that provides the user interface. This ensures that results are not affected by potentially variable communication delay between DiMAPI daemons and libraries in a distributed configuration. The Packetloss application runs on a central station and uses remote communication between DiMAPI daemons and libraries. The Packetloss application requires this configuration to be able to correlate data obtained from multiple monitoring stations and calculate resulting packet loss characteristics. Variable delay is not a problem for this application and it also benefits from running in part on a separate PC from highly loaded monitoring stations. Tbwtools uses an integration of active, passive and end-host monitoring. It has a different architecture (see Document Code:

16 Developed and Deployed Software Tbwtools on page 19). Perfmon and Servmon service applications are also distributed between monitoring stations and the central station, but they do not process packets and do not use DiMAPI mechanism for communication. Document Code:

17 4 Passive Monitoring Applications This chapter describes the passive monitoring applications that were deployed as part of this pilot. An important feature of application deployments in the pilot is that all measurement results are conveniently accessible from a single, central web page [WEBPAGE] (see Figure 4.1). Figure 4.1: Integrated access to all monitoring applications. Document Code:

18 Passive Monitoring Applications The top of the page includes a clearly organised table that lists each application s name, the network characteristics provided for it and a link to its current measurements. Some applications have two user interfaces; a map-based interface for a quick overview of the current state of monitored components or characteristics, and a form to request detailed tables and graphs. The rest of the page includes further details for each application, including information about the type of traffic being monitored, whether monitoring is on demand or continuous, the monitoring time resolution and where documents describing the applications can be found. For convenience, information how to access perfsonar active monitoring applications from the same page is also included. 4.1 ABW Operation ABW is an application to monitor used capacity. It primarily provides the following benefits: It shows used capacity in a wide range of time resolutions (from one second to months). It shows the distribution of used capacity across protocols and applications including the ones using dynamic ports. The ABW application has been developed by the JRA1 activity. It can detect which protocols real traffic on a network link consists of at different OSI (Open Systems Interconnection) hierarchy layers, or monitor which user-specified subsets of traffic based on arbitrary L2 to L4 header fields are present on the link. It uses the trackflib library of DiMAPI to identify many commonly used protocols that use dynamically allocated ports, such as passive FTP (File Transfer Protocol), HTTP (Hypertext Transfer Protocol, which can use other ports than 80 and 443), Skype telephony or file-sharing protocols, such as BitTorrent, Gnutella or DC++. It also monitors the volume of multicast and IPv6 traffic as two interesting subsets of traffic. The volume of traffic (separately the number of packets and the number of bytes) for all configured protocols is measured every second and stored in an RRD (Round Robin Database) where the results are gradually collected. Upon user request, the application generates graphs for specified monitored links, protocols, time intervals and time resolutions (predefined or user-defined). The application is therefore split into two phases: continuous data acquisition and data presentation. The graphs also show the average and maximum link load for the same time period, but with a coarser time resolution. Only a standard web browser is needed, no Java or plugin support is required and, as the user interface is generated on the server, there is no need to download any executable to the client. Generated graphs have one of two possible layouts showing either unidirectional or bidirectional line load. For bidirectional line load, one direction is shown as positive values and the opposite direction as negative values. The selected graph layout depends on the number of ports on the monitoring card. When the card has only one port (such as 10 GE of OC-192 DAG cards), ABW uses unidirectional layout. If the card has two ports, each monitoring one direction, ABW uses bidirectional layout. Document Code:

19 Passive Monitoring Applications Usage Examples ABW provides a map-based and a from-based user interface. The map-based user interface gives a quick overview of the most recent load on all monitored links (see Figure 4.2). Each monitoring station is represented by a coloured circle divided into two halves. The left half shows input load and the right half shows output load, indicated by arrows next to the circle. The colour and the filledin area depend on the average load in the last 15 minutes. If the cursor is placed on a circle s input or output half, a popup box displays more detail about the latest link load. Figure 4.2: Map-based user interface of the ABW application. Document Code:

20 Passive Monitoring Applications If the box is clicked, the form-based user interface is displayed (see Figure 4.3). Users can also invoke this interface directly from the main page of passive monitoring applications. Figure 4.3: ABW user interface. The ABW application s user interface is very similar to that of other passive monitoring applications, making it easy for users to get oriented. Reasonable defaults are preset, so that the user can just click the "Generate graphs" button. Details of requested graphs can be specified using the following steps which are clearly indicated in the user interface. Document Code:

21 Passive Monitoring Applications Step 1: The user can select one or more types of graphs. Currently two types of graphs are available: distribution of L3/L4 protocols including multicast and IPv6 and distribution of application protocols. Step 2: The user can select one or more monitoring stations. Step 3: The user can select the time period and resolution for which link load and protocol distribution should be presented. Several predefined time periods and resolutions are available. Alternatively, the user can specify an arbitrary time period and resolution. Similarly to other applications, one resolution can be specified for the graphs main body and one for envelope lines showing average and maximum values. Step 4: The user can choose if the envelope line for maximum values should be included in the graph. Maximum values are calculated from the original data samples normally received in one second intervals. For graphs with long-term average and maximum aggregation lines, the maximum values can be much higher than values drawn in finer resolution, which then occupy only a small part of the graph and are less clear. The example graphs in Figure 4.4 illustrate this effect. The left graph shows link load with the maximum line, and the right graph shows the same measurement without the maximum line. Figure 4.4: Link load plotted with maximum values (left) and without maximum values (right). Document Code:

22 Passive Monitoring Applications The example graphs generated from the form-based user interface are shown in Figure 4.5. The upper two graphs are for the -CESNET link and the lower two graphs are for the -ACAD link. Figure 4.5: Examples of link load in L3/L4 protocols (left) and application protocols (right) monitored by ABW. In ACAD, traffic in both directions is merged in a mirroring port on a router, monitored together and shown in a unidirectional graph. The left graphs show distribution of L4 protocols and the volume of multicast and IPv6 traffic. The right graphs show distribution of application protocols. Different protocols are shown in different colours. Grey represents traffic whose application protocol could not be determined. All graphs show link load with 1-second time resolution. The light and dark blue lines show 30- second average and maximum values respectively. You can see that short-term load (1-second) is often significantly higher than long-term (30-second) averages. This phenomenon is even more pronounced in longer-term monitoring. Figure 4.4, for instance, shows the link load on the -CESNET link over a one-week period of with 20-minutes resolution and maximum values calculated from 1-second samples. Document Code:

23 Passive Monitoring Applications Difference from Related Tools The ABW application provides several advantages over other approaches to link load monitoring: When compared to SNMP monitoring, passive link load monitoring can provide results at any time scale. This enables the detection of short load peaks that are often much higher than long term averages. These peaks cannot be detected by SNMP monitoring, because routers normally update their MIB (Management Information Base) counters with varying delays of several seconds. For example, Figure 4.6 shows SNMP link load monitoring at 30-second intervals on the left and at 1-second intervals on the right. The 1-second monitoring is unusable because it is distorted by interference between counter update and reading rates. Figure 4.6: SNMP link load monitoring. Another advantage of passive link load monitoring is that it provides detailed information about protocols and applications that use network capacity including usage trends. This information can be used for network planning or traffic engineering. When compared to link load presentation in perfsonar UI (which uses SNMP monitoring), the ABW application can show link load for any time period with any time resolution, provided that requested data is still available and not aggregated. For instance, if a problem is discovered in the network, detailed load dynamics and protocol changes that happened several hours or days ago are available for analysis. ABW can also simultaneously show multiple graphs for different links on one web page for easy comparison, and it allows downloading produced graphs to files, which is not possible with perfsonar UI. Figure 4.7 shows the most detailed link load information available in perfsonar UI on the left (5-minute resolution, most recent), and in ABW on the right (1-second resolution, any selected time period). Document Code:

24 Passive Monitoring Applications Figure 4.7: Most detailed link load in perfsonar UI (left) and the ABW application (right) Summary The ABW application has been deployed on all -NREN links of partners participating in the SA3 pilot. This provides the following benefits: Continuous and 100% non-intrusive link load monitoring. Link load for any time period and in any time resolution, ranging from 1-second peaks to averages over months for evaluation of load dynamics in the latest or previous traffic. Distribution of load into L4 protocols and application protocols, including common protocols that use dynamic ports. Monitoring of multicast and IPv6 traffic. Simultaneous display of multiple links for easy comparison. Fast user interface for a standard web browser. Document Code:

25 Passive Monitoring Applications 4.2 Tbwtools Operation Tbwtools is an application that uses a combination of active monitoring, passive monitoring and end-host monitoring to test and analyse TCP throughput over an end-to-end path. The letters tbw stand for tcpdump, bulk and web100, because these are the three monitoring utilities combined in Tbwtools. The application is similar to iperf [IPERF] or BWCTL (Bandwidth Test Controller [BWCTL]) stress-type throughput tests, but in addition to these tools it also provides connection diagnostics. Tbwtools provide the following primary benefits: The Tbwtools infrastructure deployed in permits the testing of network paths all the way from / to the user's PC, which can run Windows or a Linux operating system. Tbwtools is automatically downloaded from the central web page to the user's PC and started. (A deployed BWCTL infrastructure only permits tests between fixed servers in the network, or from the user's PC with a Linux operating system, where BWCTL must be installed manually.) It provides detailed feedback about connection performance characteristics to help determine the reason why throughput was low. Tbwtools has been initially developed by the LOBSTER [LOBSTER] project. In 2006, it was presented to the JRA1 activity and it was agreed that the further development would be taken over by this activity. Since then the application has been significantly enhanced. The user interface has been improved, support for LS (Lookup Service) has been added, the installation has been simplified and NATs (Network Address Translations), which are often present in local networks, are now supported. Tbwtools consists of a web-based user interface (implemented in Java and started through Java Web Start) and a perfsonar-compatible Monitoring Point (MP). Using the web-based user interface is optional. It is also possible to communicate with the MP by exchanging XML messages using a perfsonar client. However, the user interface allows to make a test connection from a user's PC (if the interface is not used, this is only possible between two MPs), and it presents monitoring results in a graphical form. As Tbwtools also use active and end-host monitoring, it is based on a different architecture to other strictly passive tools (see Figure 4.8). Document Code:

26 Passive Monitoring Applications Figure 4.8: Tbwtools architecture. The Tbwtools operate as follows (see Figure 4.8): 1. Tbw MPs (Measurement Points) are installed on monitoring stations in the network. When each Tbw MP is started during monitoring station boot, it automatically registers its URL with the Lookup Service (LS). Each Tbw MP can belong to a named domain, such as "". 2. A user points their web browser to a web page from which a Java-based user interface is downloaded and started. This user interface sends a query to the LS to obtain URLs of available Tbw MPs. The query to the LS includes the name of the user's domain, such as "", which is obtained from the web page where the user interface was downloaded. This allows Tbw MPs from multiple domains to be registered on the same LS, and the correct set of Tbw MPs of the user's domain to be presented to the user. 3. A user makes a test connection between two MPs or between a selected MP and their own PC. As this test connection is in progress, it is also being monitored by several methods (passive packet capture, web100 and getsockopt() system call). 4. After the test connection finishes, various performance characteristics are calculated and presented to the user in a set of graphs. Document Code:

27 Passive Monitoring Applications Usage Examples The main Tbwtools user interface page is shown in Figure 4.9. The user actions are divided into a few simple steps clearly marked on the user interface. Step 1: Clicking the "Query LS" button queries the LS for a list of available TBw MPs. Step 2: Available Tbw MPs are automatically added to a drop-down list. The user can select one of them or manually enter the URL of another Tbw MP not registered with the LS. Step 3: The user can specify the hostname or IP address of the other end of a test connection. By default this is the user's PC IP address, which means that a test connection is made between a selected Tbw MP and the user's PC. By entering another hostname or IP address, a test connection between two Tbw MPs can be made. A test from / to the user's PC is possible because the Java-based user interface also includes a Tbw MP implementation. Step 4: The user can optionally choose: The direction in which test data is sent during the test connection (to test a network path in a selected direction). The direction in which the test connection is initiated. (If there is a firewall on one side, the test connection is more likely to path through when started from behind the firewall. Note that the TCP port can be specified in step 2.) Step 5: The user can optionally specify: The socket buffer size (when unspecified, operating system defaults are used). The message size sent in one send() call (default 64 kb), web100 sampling rate (default 100 µs). Note that the message size also affects sampling rate for socket options (particularly TCP_INFO) read by getsockopt(), because this call is made after each send() call. The test duration (default 5 s). Finally, the user can click the "Run test" button to make a monitored test connection. Document Code:

28 Passive Monitoring Applications Figure 4.9: Tbwtools user interface. After the test connection finishes, the user is presented with a summary of the connection and a choice of approximately 100 performance characteristics, which were monitored during the connection and can be plotted (see Figure 4.10). A summary includes information about the percentage of time that the connection throughput was limited by a sender, a receiver or a network. This is important initial information about the connection bottleneck s location. The performance characteristics are produced from three monitoring sources: tcpdump - Characteristics calculated from captured packet trace of a test connection, such as instantaneous throughput, flight size, sequence and acknowledgement numbers. bulk - Characteristics calculated from the TCP_INFO socket option, read by the getsockopt() system call after each write of test data by send() call. For example, sender and receiver ssthresh (slow start threshold) variables. Document Code:

29 Passive Monitoring Applications web100 - Characteristics obtained from extended Linux kernel monitoring of TCP connections supported by the web100 kernel patch. Figure 4.10: A test connection summary and performance characteristics. After the user selects characteristics and clicks the "Generate graph" button, the characteristics are calculated and plotted in graphs. A section of a web page with generated graphs is shown in Figure An important benefit provided by Tbwtools is that characteristics obtained from different types of monitoring are timecorrelated and presented together in a common user interface. Document Code:

30 Passive Monitoring Applications Figure 4.11: A test connection summary and performance characteristics. The nature of characteristics requires expert TCP knowledge to analyse results. If a user does now have this knowledge themselves, they can still make a test connection from / to their PC, obtain a data ID from the results and it to expert who can use Tbwtools to retrieve the results for analysis Difference from Related Tools The following tools are most closely related to Tbwtools: iperf - A widely-used stress-type TCP/UDP throughput test tool. A "bulk" component of Tbwtools, it provides a very similar stress-type test (for TCP only). However, Tbwtools also includes test connection monitoring, provides an initial bottleneck location summary and detailed performance characteristics to assist expert users in detailed connection analysis. BWCTL - A wrapper around iperf with added scheduling to enable multiple users to execute tests without interference, deployed in as part of JRA1 and SA3 activities. The same key difference as with iperf applies to BWCTL, which does not provide feedback why measured throughput was potentially low. Tbwtools adds monitoring and detailed feedback about the test connection. Moreover, the installed BWCTL infrastructure measures throughput only between fixed monitoring stations, while Tbwtools measure throughput all the way to / from the user's desktop (measurements between two monitoring stations are also possible). Document Code:

31 Passive Monitoring Applications NDT (Network Diagnostic Tool) - This tool is similar to Tbwtools in that it combines a stress-type TCP throughput test with web100 monitoring. Tbwtools also collects characteristics obtained from passive packet capture and from the TCP_INFO socket option, and presents them correlated in time in a central user interface Summary The Tbwtools application has been deployed on all -NREN links of partners participating in SA3 pilot. This provides the following benefits: On-demand TCP throughput tests between monitoring stations or between monitoring stations and user PCs. Built-in test connection monitoring based on passive packet capture (tcpdump) and end-host monitoring (web100 and getsockopt() ). Connection summary including the achieved throughput and the percentage of time that the connection throughput was limited by a sender, a receiver or a network. Computing and plotting of up to approximately 100 selected performance characteristics obtained from several monitoring sources correlated in time for easy comparison. User-accessible monitoring information enables both self-analysis and forwarding of data IDs for expert monitoring result retrieval and analysis. Able to pass through NATs and some firewalls. 4.3 Packetloss Operation Packetloss is a new application, which is being tested on a large scale in the network for the first time. Packet loss is an important characteristic that affects user experience when communication takes place over a network. As the following example illustrates, it is unfortunately not possible to measure realistic packet loss using active monitoring: If 10 test packets per second were sent between two end points (much more than in the active monitoring tool Hades deployed in ), it would take almost 3 hours to detect packet loss of 10-5 and more than a day to detect However, these are packet loss rates that still significantly affect TCP throughput over fast long-distance networks. Moreover, these calculations are valid for evenly distributed packet loss. When bursts of packet loss occur, which is a common case, it can take an even longer time to detect and realistically measure packet loss rate by test packets. Document Code:

32 Passive Monitoring Applications The problem can also be looked at from another perspective: If a burst of 10 test packets was sent and a loss period was caught, so that 5 out of 10 packets were lost, it would not be possible to determine the time period for which this 50% packet loss rate is valid. Packet loss is a property inherent to traffic in which it is experienced. Due to the volume and dynamics of real traffic, it is not possible to realistically capture its packet loss using test packets. No matter how many and in what patterns they are sent, the outcome is not related to the actual network behaviour experienced by user traffic. This is completely different from one-way delay, which can be realistically measured by test packets in active monitoring. Packetloss is a strictly passive monitoring application for measuring real packet loss experienced by real user traffic. It gathers records about completed flows from monitoring stations on edges of a network (such as NREN - interfaces) and calculates real packet loss of flows passing through the network. It does not require any configuration for the IP prefixes that pass through the edges of a network. The basic idea is to compare the number of packets reported in the same flow from multiple monitoring stations. If the number of packets is different, it is the real packet loss which happened inside the network. This idea is illustrated in Figure However, the implementation is rather complex for achieving required performance properties. The flow record is created on a monitoring station when a flow expires (or completes); that is when no packet of the flow arrives during a specified inactive period (the default is 30 seconds). Flow records are periodically retrieved by the Packetloss application which runs on the central station. A flow reported from multiple monitoring stations is considered the same if it has the same 5-tuple (source IP address, destination IP address, source port, destination port and layer 4 protocol) and if the timestamps of the first packet do not differ more than a specified time (the default is 500 milliseconds). The packets of one connection can go via different routes inside the network and will be counted correctly as long as they arrive at the same edge router. Packets can arrive at different edge routers in case of multicast or where multiple uplinks of an NREN to the network exist and load balancing allows a one single flow to be split into multiple uplinks (which is an unusual configuration). These cases are not supported by the Packetloss application. The Packetloss application was originally developed by the LOBSTER project. However, for the purposes of deployment in the network, its performance and user interface needed to be improved significantly, which resulted in an almost complete redesign of the application including related parts of DiMAPI. Document Code:

33 Passive Monitoring Applications Figure 4.12: Packetloss application principle Usage Examples The main Packetloss user interface page is shown in Figure Its use is similar to other applications. Some reasonable defaults are preset, so that the user can just click the "Generate graphs" button. However, the generated statistics and graphs can be customised using the following steps which are clearly marked in the user interface: Step 1: The user can select NREN pairs between which monitored packet loss is to be shown. On the left, an NREN with a monitoring station is selected. On the right, one or more other NRENs can be selected. Packet loss will be shown between the NREN on the left and all NRENs on the right. Document Code:

34 Passive Monitoring Applications Step 2: The user can select the time period and resolution for which packet loss is to be shown. Several predefined time periods and resolutions are available. Alternatively, a user can specify an arbitrary time period and resolution. Similarly to other applications, one resolution can be specified for the main body of the graphs and one for envelope lines showing average and maximum values. Step 3: The user can select one or more of four graph types: packet loss ratio, packet loss count, number of all expired (completed) flows and number of flows matched between NRENs. Step 4: The user can select one of four available graph sizes. Step 5: The user can generate graphs as draggable to easily access previous or following time periods. Step 6: The user can request a number of characteristics to be listed in tables. Finally, the user can click the "Generate graphs" button to generate tables with statistics in textual form. Although graphs are generated for the requested time period, the tables include statistics for the whole time that the application is running. Statistics can be cleared at any time by clicking the "Reset statistics" button. The user can also enter header filter specification in BPF (Berkeley Packet Filter) format in the field below the "Generate graphs" button. Clicking the "Restart packetloss" button, restarts the application with the specified header filter. Only packets that pass this filter are included in the following monitoring. This feature is useful for users who are interested in monitoring only a subset of traffic, or if the total traffic volume is higher than the amount that the Packetloss application can process. Document Code:

35 Passive Monitoring Applications Figure 4.13: The main page of the Packetloss user interface. Document Code:

36 Passive Monitoring Applications The main page of the Packetloss user interface also includes the status table and the peer statistics table at the bottom (see Figure 4.14). The status table shows valuable information including: The BPF string used for header filtering. By default all packets are monitored. The minimum number of packets in a flow to be processed. By default flows with at least two packets are monitored. A higher value means more restrictive filtering, which reduces the precision of measured packet loss. Ignoring flows of 1-packet size is useful for performance reasons and for better precision. If these flows were not ignored, 1-packet flows reported from one monitoring station would be counted to the total number of flows, but lost 1-packet flows would not be detected and thus not counted to packet loss statistics. The status of all monitoring stations and their monitoring cards, which can be connected, disconnected due to problem on the monitoring station or disabled manually. The peer statistics table shows the total number of matched and lost packets for all selected NRENs pairs. Matched packets exist where flows were detected as passing from one NREN to another NREN. Figure 4.14: Packetloss status and peer statistics. Document Code:

37 Passive Monitoring Applications The left part of Figure 4.15 provides an example of a graph that shows expired (completed) flows reported from two monitoring stations (SWITCH and PIONIER in this case), specifically from the monitoring cards that monitor traffic from one NREN to and from to the other NREN during a period of one week. Matched flows (flows that were flowing from the first NREN to the second NREN) are shown in the right part of Figure Figure 4.15: Expired and matched flows from SWITCH to PIONIER. Figure 4.16 shows the measured packet loss count and packet loss rate. Figure 4.16: Packet loss count and rate or real traffic. Document Code:

38 Passive Monitoring Applications For comparison, Figure 4.17 shows graphs from active packet loss monitoring by the Hades application between the same NRENs during the same time period. There are two Hades monitoring stations in PSNC, shown on the left and right graph respectively. Active monitoring did not detect any packet loss, which could be expected, due to completely different volume and dynamics of a few test packets compared to real traffic. Figure 4.17: Hades active loss monitoring Difference from Related Tools As outlined in Operation on page 25, packet loss is strictly bound to particular traffic and it is not possible to obtain it using other traffic. While test packets in active monitoring can be used to measure delay, which corresponds to delay experienced by real traffic, it is a different case with packet loss. Packet loss provided by active monitoring tools, such as Hades, has no relation to packet loss experienced by user traffic (see Usage Examples on page 27). To identify what packet loss is experienced by user traffic, which may critically affect user satisfaction with network services, it must be monitored directly. It is not possible to extract any meaningful information from packet loss experienced by test packets Summary The Packetloss application has been developed and deployed on all -NREN links of partners participating in the SA3 pilot. This provides the following benefits: Real user traffic packet loss monitoring (rather than unrealistic packet loss that can be measured by test packets). Continuous, 100% non-intrusive monitoring. Plottable number of lost packets and packet loss rate for any user-specified time period with any time resolution. Document Code: