CHAPTER 1 PRINCIPLES OF NETWORK MONITORING

Transcription

1 CHAPTER 1 PRINCIPLES OF NETWORK MONITORING Jawwad Shamsi and Monica Brocmeyer Department of Computer Science, Wayne State University 5143 Cass Avenue, 431 State Hall, Detroit, MI 48202, USA { jshamsi, mbrockmeyer }@wayne.edu Efficient and cost-effective measurement of network characteristics is pivotal for distributed systems deployed on the Internet. The network characteristics are utilized by Internet-based distributed systems to provide better service to the user and enhance performance for the application. This chapter provides a detailed analysis of existing techniques for the measurement of the four important network characteristics which include latency, bandwidth, path detection and loss rate. The chapter describes key concepts related to network measurements, including techniques for clock synchronization, strategies for time stamping of probes, methods for network analysis, difference between active and passive measurements and comparison of round trip vs. one way delay measurements. It elaborates the usefulness of different transport and network layer protocols (i.e. TCP, UDP and ICMP) for obtaining network measurements and continue this discussion to describe some important measurement tools such as Ping, Traceroue, Pathchar, Sting, Scriptroute, Spruce and Paris Traceroute. The chapter explains the effectiveness and limitations of these tools with respect to the measurement of network characteristics. 1. Introduction The effectiveness of a distributed application providing service to its end users depends largely on the characteristics of the underlying network. A web server is of no use if its immediate router fails and all the paths leading to the web server becomes unavailable. Similarly, a VoIP (Voice over IP) service has restricted functionality if the available bandwidth is limited. The two examples mentioned above are not infrequent. Internet based systems observes enough uncertainty and unpredictability that effect the quality and 1

2 2 J. Shamsi and M. Brockmeyer effectiveness of distributed systems. This uncertainty strive the need for monitoring of various network characteristics, both at the level of the service provider and the application. An ISP (Internet Service Provider) may monitor the bandwidth used by its customers, in order to allow a fair share of the bandwidth usage, whereas an application could periodically measure latency to its servers in order to detect congestion and failure. 1.1 Which network characteristics are relevant for measurements? The significance and relevance of network characteristics varies with applications. A content distribution network may be interested in measuring latency from its clients to the application server, whereas a load balancing application may find available bandwidth as a useful criterion to adjust the load on the servers. Other network characteristics such as congestion, queuing delay, network failure, topology discovery, and bottleneck determination also have great significance. However, there are four network characteristics that constitute the basic paradigm of network measurements and could be utilized in the computation of other related network characteristics. They are mentioned below Latency: Latency refers to the time taken by the message to traverse from the sending host to the destination host. It is usually measured in milliseconds (ms). Latency of a path has a great significance for many applications. For many applications latency is used to determine the availability of the servers and select nearest available server in terms of latency, specifically in the content distribution networks. Further, variation in latency (i.e. the deviation between the successive latency measurements) is used to determine the quality of the path and detect network congestion. Packet Loss: If the rate of packets sent from the source host is not equal to the rate of packets received at the destination host then the path experiences packet loss. Loss rate of a path is the rate (in percentage) at which packets are being lost while traversing through the network path. Lost packets affect the performance of the application as they are often required to be retransmitted. Even when the packets cannot be retransmitted (for example, such as the live transmission of audio or video streams) a high packet loss could lead to low application performance. Due to these reasons it is always desirable to select paths with low loss rate.

3 Principles of Network Monitoring 3 Path Detection: When a message is sent across the Internet, it traverses through various intermediate routers to reach the destination. Path detection is the process of determining the actual path taken by the message in reaching from source to destination. Since there are many possibilities for a path from one host to another, path detection enables an application to determine the actual path taken by the message during transmission. Path changes are possible on the Internet, therefore path detection also facilitate in determining a change in the path between two nodes. It can also be used as a sanity check to determine or isolate network failure. Bandwidth: Bandwidth refers to the capacity of the path to transfer data in a unit time. It is measured in bits per second (bps). Bandwidth has great implication for multimedia applications. Such applications generally have high bandwidth requirements. If the available path has limited bandwidth than the multimedia application suffers degraded performance. Bandwidth detection tools can be used to infer the path quality and select appropriate path. The purpose of the chapter is to develop reader skills and proficiency related to network measurements. The chapter elucidates basic concepts about network monitoring and explains several approaches and tools for the measurement of the four basic network characteristics. The chapter also discusses practical issues related to development of network monitoring tools. The chapter assumes that the reader has the understanding of basic networking terms such as the seven layer OSI model and familiarity with the transport layer protocols such as TCP and UDP. 2. Background Information Network monitoring is a very vast and rigorous topic. In order to thoroughly explain the concepts, this section mentions some background information. The information includes description about different types of network monitoring and explanation of underlying protocols used for monitoring. Readers may also refer to Section 9 of this chapter that explains some key terms associated with network monitoring Types of Network Monitoring There are two basic approaches for network monitoring, active monitoring and passive monitoring.

4 4 J. Shamsi and M. Brockmeyer Active Monitoring Active monitoring involves injecting probes in the network, specifically for the purpose of monitoring. Active monitoring bears the cost of sending additional traffic in the network; however, under most scenarios, the packet size is relatively small compared to the actual capacity of the network, therefore the cost of injecting additional traffic is minimal. The cost of injecting traffic can be reduced by decreasing the probing rate; however, this may reduce the quality of the measured characteristics. Active monitoring provides full control with respect to monitoring interval, packet size and the path to be monitored. Further, the data obtained is not specific to any particular application. Passive Monitoring Passive monitoring is the process of observing the existing network traffic and collecting information from it without injecting additional monitoring probes in the network. In a passive measurements based system, network measurement information is retrieved through the packets which are sent as a part of another application. Packets are captured by monitoring application, which is deployed either on the source and the destination hosts or along the path through a tap a. The passive mode of monitoring avoids the overhead of introducing measurement traffic in the network and also evades the use of stale data for measurements. However, in passive mode the monitoring application has little or no control over probing interval, packet size or the path to be monitored. Further, due to increase in the capacity of the core links it is expensive to identify and measure the packet of a particular flow. Passive monitoring could also raise privacy concerns. The preference for the type of monitoring varies with the application. For instance, an application measuring the performance of an existing application such as an intrusion detection system might prefer a passive approach, where as an application employing performance enhancement measures such as load balancing may desire an active scheme. An application may also employ a combination of the passive and active schemes for higher efficiency. The Wren grid monitoring system utilizes such a scheme, in which the passive mode is used a A tap is a hardware device which provides a way to capture data along a path. A tap usually has three connection points. Two connection points are connected to each end of the path, where as the third point is connected to a monitor which listens for the information. Tap is not a sink, i.e., it copies all the information to the monitor and the communication between the two ends of the path remains uninterrupted.

5 Principles of Network Monitoring 5 when an application is sending messages, whereas the active mode is used when the application is silent [18] Network protocol For passive monitoring, the measurement system has no choice for the underlying protocol of network measurement, i.e. the monitoring application monitors packets that are sent as a part of another application. However, in the active mode, the measurement system has to decide about the underlying protocol used for sending probes. Many choices are available and they are discussed below. ICMP ICMP or Internet Control Message Protocol is the underlying protocol for the famous ping utility. It operates at the networking layer. The principle advantage of the ICMP based measurement tools is simplicity and ease of use. ICMP-based schemes do not require connection setup or handshake. Further, they do not require any specific implementation at the recipient end, i.e. only the host sending the active probes implements mechanism for sending the ICMP probes. Since ICMP messages are control messages, an ICMP request is automatically replied by the destination host. However, the automatic response capability of the host could be exploited by attackers and is seen as security vulnerability. ICMP will be discussed further when we analyze the ping utility. TCP TCP (Transmission Control Protocol) is the underlying protocol for many internet applications including WWW and FTP. It operates at the transport layer. Unlike ICMP based tools, TCP-based utilities bear the overhead of the TCP handshake b. TCP-based tools also require that the destination host runs a service on a designated port. The sending host establishes TCP connection to the destination through the TCP handshake and sends probes to the destination host at the server port. The server at the destination replies with a response that is used for network measurement. b TCP handshake is a three-way process. In the first step, the host initiating the connection sends a TCP SYN packet. The recipient host responds with an ACK packet. Finally, the original host sends a SYN ACK packet. All the three packets contain sequence numbers for proper identification.

6 6 J. Shamsi and M. Brockmeyer An important limitation of the TCP based tools is that since TCP is not a packet based protocol, it makes its own decisions about sending packets. By using various optimizations, including Nagle algorithm [34], TCP coalesces packets of small sizes. Since TCP has a large overhead including a 40 byte header (20 byte for IP and 20 for TCP), these optimizations allow TCP to conserve bandwidth. At the user level, it is difficult to force TCP to send packets at a specific interval [44]. In some cases, the requirement of a running service at the destination can be eliminated using TCP ACK and RST. According to RFC 793 [33], if a sender sends a TCP ACK packet to a closed c TCP port, then the TCP at the destination will reply with the TCP RST packet and the same sequence number as of the TCP ACK packet in the request. In such a case, the TCP RST can be used as a response. However, this type of request-response method can be used only if the response from the destination host is not specific to the request, i.e. the response is not generated based on the contents in the request. UDP UDP (User Datagram Protocol) is a datagram-based protocol that operates at the transport layer. Unlike TCP, UDP does not require a handshake. Additionally, UDP has an 8 byte header (plus 20 byte IP header) that results in lower overhead than TCP. Since UDP is a packet protocol it allows sending probe packets at controlled intervals. Moreover, UDP is a connectionless protocol in which a lost request (or a response) can be effectively used to detect packet loss. Like TCP, UDP can also eliminate the need to run a service at the destination port. If a UDP request is received at a port at which the service is not running, then it responds by sending ICMP destination unreachable message to the sender. Many other choices such as DCCP (Datagram Congestion Control Protocol) and SCTP (Stream Control Transmission Protocol) also exist. The selection of underlying protocol for network monitoring is based on the requirements and restrictions from an application. 3. Network Characteristics Having identified the basic principles of monitoring, this section describes various approaches for the measurement of network characteristics. Each of the c A port that is not running any TCP service

7 Principles of Network Monitoring 7 four network characteristics, latency, loss-rate, path detection and bandwidth, are discussed in a subsection which explains the basic concepts about the characteristic and describes various approaches for its measurements Latency Latency has been the most widely measured network characteristic. It refers to the time (in ms) taken by the packet to traverse from the sending host to the destination host. Latency between two hosts could vary due to two reasons: First, variation is possible due to changes in queuing delay or congestion and second due to route changes. When a packet is sent from sender to receiver, it traverses through multiple intermediate routers, each of which has variable queuing delay. Therefore, two successive packets could follow different routes, resulting in differing latency for each.. Route changes are quite common due to congestion or network failure. Therefore, the variation in latency can be utilized to depict congestion between the two end points, where paths with low congestion observes more consistency in latency. The notion of latency mentioned here represents the one-way latency from the sender to the receiver (commonly refers to the One-Way Delay or OWD). It is measured by subtracting the time that packet is sent by the sending host, from the time that packet is received at the recipient host. Besides OWD, the word latency is also sometimes used to refer to the Round Trip Time (RTT), between the two hosts. This involves the time taken by the packet to reach the recipient host from the sending host and the reply from the recipient host to the sending host. Figure 1 depicts the difference between OWD and RTT.

8 8 J. Shamsi and M. Brockmeyer Fig. 1. OWD vs. RTT. Example 1: Assume the round trip time between two Internet hosts is 200 ms. What is the one way latency (OWD) between them? Solution: Unknown. The internet is asymmetric, i.e. the forward and the backward routes between two hosts are not necessarily the same. Therefore OWD is not necessarily equal to one half of RTT. Example 2: What is the RTT between two internet hosts A and B, if the one way latencies of the forward and backward paths between them are 91 ms and 92 ms? The host B takes 200 usec on average to respond to the probes. Solution: RTT = = ms. RTT implies the effective communication delay for applications that are requestresponse applications such as WWW and FTP. In comparison, OWD is useful

9 Principles of Network Monitoring 9 communication tool for applications that have one way communication such as multicast. In case of a large message transfer, the end-to-end latency of a message is higher then the packet latency. Large messages are usually disassembled into small packets and transferred to the destination. At the destination, the message is reassembled into its original form. The processes of disassembling and assembling are called as fragmentation and de-fragmentation, respectively. The capacity of a path to transfer a message without fragmentation depends upon its MTU or Maximum Transmission Unit, where MTU of a path denotes the size of the IP datagram that can be transferred. It usually varies for different networks; however for Ethernet it is equal to 1500 bytes. The end to end path between two internet hosts may compose of links with different MTUs. In such a scenario, intermediate routers may perform the fragmentation according to the capacity of each link. We now describe some important latency measurement techniques. Measurement Strategies The measurement strategy for latency is simple. The sending host sends a packet and waits for the response. When the response is received, it subtracts the sent time stamp from the received time stamp and computes the round trip latency. For OWD, however, the destination host notes the time at which the request is received at the destination and includes it in its response to the original sender which computes OWD by subtracting the sending time from the time at which the request was received at the destination. A major requirement for the measurement of OWD is to synchronize clocks of the two nodes. Note that the underlying protocol could be ICMP, TCP or UDP. The methodology mentioned above is confined to active monitoring. For passive monitoring, the monitoring host can monitor the existing flows form an application and compute RTT. For instance, a web client can compute its RTT with a web server by comparing time stamps of the HTTP request and HTTP response Loss rate The loss rate of a path is the rate at which packets are being dropped while traversing through the path. There could be multiple reasons for packet loss. Most commonly, for a path with limited bandwidth, the intermediate routers buffer the packets in the queue. However, if a queue at the router approaches its

10 10 J. Shamsi and M. Brockmeyer storage capacity then the router drops the packet. Packet loss could also occur due to hardware failure at the router or at the destination host. It can also happen due to the packet being corrupted because of noise in the path. The detection of the packet loss is challenging. In theory, packet loss can be detected if a response to a request is not received by the sender within a certain amount of time. However, the sender (the entity sending the request) faces a challenge in determining the duration of the timeout. A response from the destination host could be delayed due to multiple reasons including high RTT, congested path, heavily loaded server or even low connection speed at the sender. If a sender adopts a safe-passage by implementing a large timeout time then it suffers from poor performance, since a lost packet might be detected earlier; on the other hand if the waiting time is small, then the sender might end up causing Fig. 2. Examples of Packet Loss. congestion in the network by sending multiple requests, or might erroneously conclude that a packet is lost when, in fact, the packet is just delayed. While there is no perfect answer to determine the duration of the waiting time, most of distributed applications adopt a safe approach and keeps a longer waiting time. For instance, the time out for a DNS query is 15 seconds. Measurement Strategies Packet loss is measured by initiating a timer for each probe and noting the probes for which the response is not received. Note that in such a case, the direction of

11 Principles of Network Monitoring 11 packet loss is unknown i.e., if the request from sender is lost or the response from the destination has failed to reach the sender (see Figure 2). This information is important for many applications that use one-way communication such as multicast. In the next section, we will analyze the sting tool [37], which computes loss rate in both directions Path Detection An internet path consists of various hops connected by routers. Path detection is the process of determining information (including name and IP address) about all intermediate routers traversed when a packet is sent across an Internet path. Note that path detection is different from the topology discovery. The former is the process of determining only the intermediate routers that are traversed by a packet when it is sent along the path, whereas the later is the process of determining all the combinations of a path between two Internet hosts. Path detection tools can be utilized to discover topology. Figure 3 illustrates a scenario where several routes exist for a path from client X to the server Y. Fig. 3. Possibilities for a path. The computation of Internet paths is achieved through routing protocols, which are categorized in two broad categories: Internet Gateway Protocols and Exterior Gateway Protocols such as BGP. The former refers to the routing within an autonomous system d, whereas the later is used when the routing is done across autonomous systems. d An autonomous system (AS) is a collection of networks and routers that maintain a single routing policy. An AS could be under control of one or more entities and is uniquely identified by an AS number [35].

12 12 J. Shamsi and M. Brockmeyer Measurement Strategies Path detection strategies are based on using TTL (Time-to-live) values in the IP header. The purpose of the TTL is to avoid infinite looping (IP routing cycle) of IP packets. TTL is an 8 bit field which indicates the number of hops or routers a packet can be traversed, before it must reach its destination or be discarded. When a packet traverses a path, each intermediate router examines the TTL value of the packet. If the TTL value is greater than 1, then the TTL value is decremented by 1 and the packet is sent to the next hop, which could be another router or the final destination host. However, if the TTL valuee is 1, then the router discards the message and sends an ICMP time exceeded message to the original sender. The ICMP time exceeded message indicates that the TTL field has expired, the packet has failed to reach the destination and it has been discarded. The ICMP time exceeded message contains the address of the router that generates the message. Path detection tools work by sending a series of packets with increasing TTL to the final destination. Since the TTL value is gradually increased in the series, each intermediate router once receives a packet having TTL value equal to 1 and sends a reply to the original sender with ICMP time exceeded message. The sender can thus obtain information about all the routers in the path. Example 3: Can path detection tools be used to measure RTT to a specific hop? Solution: Path detection tools can be used to measure RTT. When the original sender receives an ICMP time exceeded message it can compare the send time stamp of the probing packet, with the received time stamp of the ICMP message to compute RTT to a particular hop 3.4. Bandwidth Bandwidth quantifies the data rate at which a network link or a network path can transfer [31]. It is measured in bps or bits per second. Unlike latency, where the end-to-end characteristic of a path is computed by aggregating the latency of the individual links, the bandwidth of a path is the capacity of the link with minimum bandwidth. e When the destination host receives a packet with a TTL value of 1, it delivers the packet to the application.

13 Principles of Network Monitoring 13 The term bandwidth can be used to refer to two different types of speeds. Upload speed denotes the rate at which data is being sent to the destination, while download speed refers to the rate at which data is being received. If a path has similar upload and download speed then it is called as bandwidth-symmetric. Most of the internet backbone is bandwidth symmetric. However, this is not necessarily true for end users, including home users relying on cable or ADSL. The reason for this difference is that most home users are typically interested in downloading, therefore ISPs (Internet Service Providers) allocates more resources for downloading. Note that a bandwidth-symmetric path is not necessarily symmetric in route or vice versa. The two entities are separate and must not be confused with each other. For the measurement of bandwidth two terms are relevant, capacity and available bandwidth. Capacity Capacity refers to the maximum data rate a link can transfer. It is measured as the number of data units (bits) transferred in a unit time (sec). The capacity of a link depends upon the transmission medium and is usually constant at the link-layer (OSI layer2). For instance, for a gigabit Ethernet, the capacity of a link is equal to one gigabit per second. However, this capacity refers to the capability of a link excluding any overhead. Remember, at the IP-layer (layer 3), IP packets are encapsulated in link-layer frames such as Ethernet frames, therefore the actual capacity must also consider the overhead associated in transmitting data in frames. For instance, for Ethernet the overhead consists of 38 bytes in that 14 bytes for Ethernet header, 4 bytes for CRC f (cyclic redundancy check) 8, bytes for the frame preamble g and the 12 bytes for the inter frame gap h [31]. The actual capacity of the path considering the overhead can be expressed as f Cyclic Redundancy Check- Checksum to detect errors in the frame. g A preamble is a 7 byte field, which is a combination of ones and zeros. It is used for synchronization purposes. The preamble field is followed by a 1 byte start of frame delimiter (SFD). The combination of both the preamble and SFD fields takes 8 bytes. h Interframe gap is a delay a network device must wait before it can attempt to transmit a frame. This ensures that a device which has previously transmitted a frame has reverted back to the listening mode. The interframe gap must be 96 bit times. For 10 Mbps, it takes 100 nanoseconds to transmit one bit, therefore 96 bit times is equal to 9.6 micro seconds, which is equal to 12 bytes.

14 14 J. Shamsi and M. Brockmeyer L C = C L (1) O + L where, C is the actual capacity at the IP level. C L is the capacity at the link level, O is the overhead needed to encapsulate IP packet and L is length of the Ethernet frame. Therefore the actual capacity of a link varies with the transmission technology. Example 4 further clarifies the concept. Example 4 Compute the capacity of a path for a 10 base T link with an Ethernet frame size of Solution Given: C L = 10 Mbps, O=38 bytes and L= 1400 bytes Placing values in equation (1), we get C = 9.73 Mbps The capacity of a path that consists of several links is equivalent to the capacity of the link with minimum capacity. Due to this reason minimum capacity link is also called as the narrow link. Available Bandwidth The available bandwidth is the unused capacity of the link. At any moment a link is either transmitting at full speed or is idle [31]. Therefore, the utilization (µ) of a link is expressed as an average which is computed over a series of time intervals. Therefore the available bandwidth (A) is equal to the total capacity minus average usage, where usage is expressed as a fraction of the total capacity. A = C( 1 µ ) (2) The link with the minimum available bandwidth is known as the tight link. The available bandwidth of a path equals to the available bandwidth of the tight link. Measurement Strategies The available bandwidth of a path can be computed by measuring the amount of data transferred over a unit time. Strauss [41] has categorized the existing bandwidth measurement tools into two models.

15 Principles of Network Monitoring 15 Probe Gap Model: The Probe Gap Model (PGM) can be explained as follows. The sender sends two probes that are separated by a time interval of t 1. The two probes arrive at the receiver separated by an interval of t 2. If t 2 > t 1, it implies that the probes have experienced a delay during their traversal. Assuming a single bottleneck that induces delay, t 2 t 1 is the amount of delay experienced at the bottleneck. If the capacity (total bandwidth) of the bottleneck is known to be C, then the utilization at the bottleneck is t2 t1 µ = C( ) (3) t 1 Since the model assumes a single bottleneck, the utilization of the path is also equal to the utilization at the bottleneck. Therefore, the available bandwidth of the path is t2 t1 A = C(1 ) (4) t 1 A major limitation of the PGM is that it assumes that there is a single bottleneck with a known capacity. Probe Rate Model: The Probe Rate Model (PRM) is based on idea of creating congestion in the network. A series of packets are sent along the path. If the sending rate of packets at the sender matches with the arrival rate of packets at the receiver, then it implies no congestion. Therefore, the sending rate is increased to the point where the arrival rate becomes lower than the sending rate. At this point congestion is detected in the network, and the instance is referred as turning point. The sending rate just before the beginning of congestion denotes the available bandwidth. Note that PRM has a larger overhead as compared to the PGM as it requires induction of congestion in the path. 4. Measurements Tools This section describes some important measurement tools. The section is divided in to five subsections. The first four subsections are dedicated to each specific

16 16 J. Shamsi and M. Brockmeyer network characteristic, whereas as the last subsection describes some tools that have multiple usages Latency Measurements Ping Ping is a widely used latency measurement utility. It is built in to most operating systems, including UNIX and Windows. Ping has multiple usages: It can be used to measure RTT between two hosts or to detect packet loss along a path. It works by sending ICMP echo request packets to the destination host and listening to the ICMP echo reply messages. The ICMP echo request is a type of message which must be replied by every host or a router via the ICMP echo reply message and the data of the echo request message must be unaltered in the echo reply message. The sending host sends a series of ping messages that are identified using probe sequence numbers and the corresponding echo reply messages are used to compare the time stamps of the probes and compute RTT. Ping infers a statistical summary of the path latency characteristics including minimum, maximum and average RTT and the packet loss to the destination host. Ping is inexpensive and flexible measurement utility since it does not require the destination host to run any service or listen to any particular port. Fping Fping [9] is an enhancement to the ping utility. Ping is limited as it can only send messages to a single destination host at a time. Fping improves upon this restriction by sending probes to multiple hosts. The hosts can be specified on the command line or through a file. However, the underlying protocol of fping is still ICMP. Both Ping and Fping may demonstrate limited accuracy as many routers or firewalls filter or rate-limit ICMP packets due to security concerns [37]. Recall that we mentioned that every host that receives echo request sends an echo reply. If a firewall filters ping packets then it will drop the packet before it reaches its intended destination. Since the packet will be dropped, the source will not receive echo request message and the echo reply message will not be sent. In such a case, the usability of ping remains limited. Several additional tools which measure latency, as well as other network characteristics are described in Section 4.5.

17 Principles of Network Monitoring Loss rate Latency measurement techniques utilizing active monitoring approach can be utilized to estimate loss rate along the path. Packet loss can be detected by using a timer. Usually, a timer is set whenever a probe request is sent, and is cancelled when the probe response is received. In this manner, a signal is raised whenever a response is not received, which indicates a lost packet. We now describe how the sting tool can be utilized to estimate loss rate in both directions. Sting Sting [37] is a TCP based tool which measures packet loss between two Internet hosts. It can precisely detect the direction of the packet loss i.e. if the loss has occurred in the forward direction (from source to target) or in the reverse direction from (target to source). Sting requires the target host to run a TCPbased service such as a web or telnet server. The Sting protocol is divided into two phases. In the first phase, the source host sends a series of TCP data packets to the service running on the target host. Each packet has a sequence number i one greater than the sequence number of the preceding packet in the series. The target host responds with an acknowledgement for each received packet. In the second phase, the source host initially sends a data packet with a sequence number one greater than the sequence number of the last data packet sent in the first phase. The target host responds by sending a probe with an appropriate acknowledgement number such that if packet loss occurs during the first phase then the acknowledgement number indicates the sequence number of the first lost packet, otherwise the acknowledgement number indicates the next expected packet. In case of packet loss during the first phase, the source responds by retransmitting the lost packet. The process continues until all the lost packets are retransmitted. At the end of the second phase, the loss rate can be computed using the total number of packets sent and the number of retransmitted packets. i Recall that in TCP each packet has a 32 bit sequence number and a 32 bit acknowledgement number. The sequence number uniquely identifies each packet and is used to arrange out of order arrival of packets at the target. The acknowledgment number is used by the target to acknowledge the receipt of the data packet. When a target host receives a packet, it sends and acknowledgement back to the sender with the acknowledgement number equal to the expected sequence number of the next packet. The acknowledgement number also implies that all the packets with the preceding sequence number have been received by the target. The acknowledgement number is not incremented if an out of order message is received. For efficiency, many implementations of TCP delay sending the acknowledgement until they received a next packet or a time out occurs.

18 18 J. Shamsi and M. Brockmeyer The procedure described above is useful for computing packet loss in the forward direction. For the packet loss in the reverse direction, the sender counts the number of acknowledgements received from the target. In order to ensure that target sends acknowledgement for each packet received from the sender and does not delay sending the acknowledgement, the sender sends the packet with some delay. Example 5: Consider sting is used to detect packet loss along a path. If the source sends 100 packets during the first phase and three packets during the second phase then compute the packet loss in the forward direction. Solution: In the first phase, 100 packets are sent. In the second phase, the source sends 3 packets. Therefore, two of the three packets must be the retransmitted packets. Therefore = 98 packets are received at the target. Therefore the forward loss rate is 2/100 =2 % Path Detection Traceroute Traceroute (tracert in Windows) is a path tracing utility that is used for path detection or network failure diagnosis. Traceroute works by sending a series of IP packets with incremental values of the TTL. At each step, three packets with the same TTL value are sent. For instance, the first three packets have a TTL value of one while the next three packets contain a TTL value of two and so on. When the first series of packets (with a TTL of 1) arrives at the first hop, the router discards the packet and sends ICMP time exceeded message with its address as the source address to the original sender. In this manner, a series of traceroute messages are sent in which a TTL value is gradually incremented, the sending host then collects the information about each router along the path and constructs the network path. The traceroute utility prints the information about each hop-router on the screen. During this process, if a message is lost, or if a router is configured not to reply with an ICMP time exceeded message then an asterisk (*) is printed. The choice of the underlying protocol for the traceroute is platform dependent. The original specification proposed by Jacobson [14] uses UDP packets, with

19 Principles of Network Monitoring 19 UNIX utilizing the same choice. The UDP packets are destined for ports from to The ports are used by Traceroute, since they are unlikely to be selected by an application. Thus, when the destination receives a probe at the port at which no service is running it responds with an ICMP destination unreachable message. The Windows implementation of traceroute utilizes ICMP echo request (similar to ping). Note that this does not alter the reply from the router when TTL reaches 1, as in both the cases (UNIX or Windows), the router will send the ICMP time exceeded message to the original sender. Due to security issues, some networks block incoming traceroute packets. In such a scenario, traceroute can only display the router hops up to the firewall router of the network. TCPtraceroute TCPtraceroute is a TCP-based implementation of traceroute. The motivation behind using TCP is to bypass firewalls that block UDP and ICMP, as most firewalls permit TCP-based inbound traffic. Unlike conventional traceroute schemes which send UDP or ICMP echo messages and wait for ICMP time exceeded messages, TCPtraceroute sends TCP SYN message to the destination port 80 (Since port 80 is the known port for the web, most firewalls allow packets intended for port 80). If the target host is running any service on the port 80 then it responds with a SYN ACK, otherwise it will reply with an RST message. If the sender receives a SYN ACK message then it will sends an RST message to reset the connection. Paris Traceroute Paris Traceroute [1] is developed to improve the accuracy of the traceroute utility. Traceroute suffers from two major flaws: First, it can occasionally report false links between routers and second, it can miss existing links between the routers. To understand these two deficiencies, consider the example illustrated in Figure 4.

20 20 J. Shamsi and M. Brockmeyer Figure 1 Traceroute Problems (based on the example mentioned in [1]) The Traceroute client sends traceroute packets with increasing TTL to the server. Router A implements load balancing j and selects either router C or router D based on the load balancing policy. For the first traceroute packet sent by the client with a TTL value of 1, the router A drops the packet and replies with an ICMP time exceeded message. If the second packet with a TTL value of 2 is forwarded to router B (where it is dropped) and the third packet with TTL value of 3 is forwarded to router C (where it is dropped at router E), then it would appear that there is a link between router B and router E. Additionally the link between router C and router E remains unnoticed. The first flaw is the appearance of a false link between C and D, whereas the second deficiency is the failure to observe the link between C and E. Paris traceroute corrects these two problems by maintaining the probe packet header in a manner that permits all probes to follow the same path in case of per flow load balancing [1]. In order to achieve this, Paris traceroute modifies the packet header information such that the load balancing portion of the header remains constant. This implies that the other fields in the header are modified such that the load balancing fields remains constant and the response packets are matched with the probe packets. Paris traceroute achieves this task as follows: In case of ICMP traceroute packets, Paris traceroute modifies the sequence number and the identifier field. For UDP packets, Paris traceroute varies the checksum field. Since checksum is used for data integrity, payload of the packet is also modified. j A Router has multiple interfaces. Load balancing implies that in order to route a message to the destination, a router selects an interface based on the load on the network. Load can be balanced based on per packet, per flow or per destination policy [3] [16]. In per packet balancing, each packet is balanced separately, whereas per flow balancing works by sending all the packets of a particular flow to the same interface. A flow is identified using various fields in the IP header and the transport header. These include the source address, source port, destination address, destination port, the protocol, the type of service (TOS), ICMP code and checksum fields. In per destination policy, load is balanced according to the destination.

21 Principles of Network Monitoring 21 However, since per packet load balancing is random, Paris traceroute is unable to completely eliminate these problems when per packet load balancing is used Bandwidth Pathload Pathload [15] is a bandwidth measurement tool, based on the PRM model. It induces congestion in the network and monitors the OWD between the source and the target. The moment where the OWD increases marks the start of congestion and the bandwidth can be computed. Note in order to compute OWD, Pathload does not require synchronized clocks at both ends as it is only interested in detecting changes in OWD. Pathchar Pathchar [26] is a tool to measure capacity at each hop. Like the traceroute utility, it uses the TTL field in the packet header to get a response from each hop and measure RTT. The path-capacity estimation is based on the principle that the RTT to each hop consist of three components: (i) serialization delay, which is the time utilized to serialize the packet for transmission along a path (ii) propagation delay, the time taken by a packet to traverse along the path, and (iii) queuing delay, the delay encountered by a packet while waiting in a queue at the router before being processed. Of the three delay components, the serialization delay depends upon the size of the packet (L) and the capacity (C) of the path, i.e. L /C, the propagation delay is constant, whereas the queuing delay varies with size of the packet and the available bandwidth of the path. To measure path capacity at hop n, pathchar sends a series of probes with same packet size and with a fixed TTL value of n. The n th hop router responds by sending an ICMP time exceeded message for each received probe. Pathchar obtains the RTT from the ICMP response for each probe. In this manner, a series of RTT values for the n th hop are obtained. Pathchar assumes the minimum RTT observed represents an instance in which the probe and its response did not experience any queuing delay. Therefore, the minimum RTT value consists of only two components: (1) propagation delay and (2) serialization delay. Of the two, the later is a function of path capacity and packet size. Since packet size is

22 22 J. Shamsi and M. Brockmeyer known, the path capacity can be computed via the slope of the minimum RTT up to hop n [31]. One major limitation of pathchar is that it assumes that every router which receives an IP packet with TTL of 1 responds with an ICMP time exceeded message. This assumption works fine for layer-3 routers, however, for layer-2 routers, which operate at the link layer and route messages on MAC addresses, this is not true [32]. Layer-2 devices create serialization delays based on packet size and path capacity, however they do not respond with ICMP error messages. As a result, the estimated capacity is not accurate. Spruce Spruce [41] is a bandwidth measurement tool that is built on the PGM model. It uses a probe pair to estimate the available bandwidth along a path. It utilizes a series of probe pair responses and computes the average. TCP throughput or Bulk Transfer Capacity (BTC) measurement TCP throughput is the capability of the path to transfer TCP traffic in bulk. Note that this capability may differ from the actual capacity of the path due to many reasons. These include TCP flow control, packet loss, TCP congestion avoidance, routing preferences. Since a large portion of Internet activity constitutes TCP traffic, TCP throughput provides an estimate of effective communication capacity in most cases. Iperf [13] is a tool that measures TCP throughput over an Internet path. It works by sending bulk TCP streams over the path and measuring the amount of payload and the delay encountered for transmission. Iperf requires access to two machines that can act as a sender and a receiver. It provides various configuration options to the user. Iperf can also be used for measuring UDP throughput. Alternatively, Iperf can also be used to generate bulk amount of TCP/UDP traffic in the network. It is available for major operating systems including UNIX and windows and also has an available GUI interface Multi-purpose tools In section 4.1, we described the ping tool which can be used to measure RTT and loss rate. We also explained traceroute in section 4.3 which can be used to detect

23 Principles of Network Monitoring 23 route, measure RTT or estimate packet loss. Besides the ping and the traceroute utilities there are several other tools which have multiple usages. Scriptroute Scriptroute [38] [39] is a public network measurement infrastructure. It allows users to run their network measurement scripts through a web interface and measure latency, detect packet loss or estimate a path. The scriptroute facility maintains a list of active servers which runs the scriptroute daemon. Each active server runs a web server through which it accepts the user scripts via the CGI post format. The measurement scripts are written in Ruby a user friendly scripting language. The web server invokes the Ruby interpreter which parses the script and interacts with the scriptroute daemon to facilitate measurement in a controlled environment. The scriptroute daemon sends measurement packets via a raw socket k interface. Scriptroute measurement packets are generated based on the type of request submitted by the user. For instance, if a user desires to execute traceroute to the destination host then TTLlimited UDP messages will be generated. Finally, the response from the destination is matched to the probe request and result is displayed on the web interface. The major advantage of scriptroute is that it aims to provide general availability to the normal users. Another benefit is that the measurements are run in a controlled environment to prevent security threats such as DOS attacks. Pathping Pathping [27] is a multi-purpose network utility. It is used to detect the route to the destination and compute hop-level RTT and loss rate, i.e., latency and loss rate for all the intermediary hops from the source to the destination. It is built-in on Windows operating system. Pathping is a combination of traceroute (tracert) and ping. It first sends the traceroute messages to the destination host and determines all the intermediary routers. It then sends ping messages (through ICMP echo messages) to the intermediary routers and compute latency and loss rate. The advantage of Pathping is that it can detect the packet loss to intermediary routers and thus can k Raw socket is a term used to denote for a socket that is delivered to an application without stripping the headers, i.e., without going through the TCP/IP stack.

24 24 J. Shamsi and M. Brockmeyer be used for network diagnosis. Since the analysis is computed over a longer time period, the results are extensive compared to the normal ping. For this reason, Pathping analysis could also take longer time. 5. Indirect Measurements The measurement aspects explained so far employ direct measurements, i.e., endto-end characteristics of a path are measured directly, either through actively sending probes to the destination and reading the response, or through passively monitoring existing flows sent by some other application. Direct mode provides rapid and accurate mechanism for measurements. However, the schemes based on direct measurements have limited scalability. In a network of n nodes, if each node is to measure its characteristics to every other node then there would be O(n 2 ) end-to-end measurements. Therefore, often indirect measurements are utilized for improved performance. The indirect mode involves use of heuristics or tomography to estimate the network characteristics of a path. The direct measurements from one or more components of a path are utilized to estimate end to end characteristics. As a result, the indirect method is not precise, however it incurs lower overhead. In large networks and in applications where an approximate of measurement is sufficient (such as locating proximity of a node), indirect measurements are utilized. Figure 5 illustrates an example of indirect measurements, where network characteristics of a path between nodes A and C can be estimated if characteristics of paths AB and BC are known. Figure 2 - Example of Indirect Measurement The end-to-end path characteristics are computed through an aggregation function, which varies with the network characteristic. For instance, in case of latency, the overall latency from node A to node C is the sum of the latencies for