Paper C. c Institute of Electrical and Electronics Engineers, Inc Reprinted with permission.

Transcription

1 Paper C Bob Melander, Mats Björkman, and Per Gunningberg. A New End-to-End Probing and Analysis Method for Estimating Bandwidth Bottlenecks. Proceedings of IEEE GLOBECOM 00, San Francisco, CA, USA, November c Institute of Electrical and Electronics Engineers, Inc Reprinted with permission.

2

3 A New End-to-End Probing and Analysis Method for Estimating Bandwidth Bottlenecks Bob Melander Mats Björkman Per Gunningberg Dept. of Computer Systems, Uppsala University Box 325, SE Uppsala, Sweden. Abstract We present a network friendly bandwidth measurement method, TOPP, that is based on active probing and includes analysis by segmented regression. This method can estimate two complementing available bandwidth metrics in addition to the link bandwidth of the congested link. Contrary to traditional packet pair estimates of the bottleneck link bandwidth, our estimate is not limited by the rate at which we can inject probe packets into the network. We also show that our method is able to detect bottlenecks that are invisible to methods such as C-probe. Furthermore, we describe scenarios where our analysis method is able to calculate bandwidth estimates for several congested hops based on a single end-to-end probe session. 1 Introduction In this paper, we present a new end-to-end probing and analysis method, TOPP, that overcomes some of the problems with several existing bandwidth probing methods. These problems include the inability to identify the situation when different types of bandwidth bottlenecks occur on separate links. We call this problem the hidden bottleneck problem, and we present measurements and ns simulations of probing in these situations. In order to clarify the difference between various types of bandwidth bottlenecks, we present definitions of surplus bandwidth, proportional share bandwidth, and protocol dependent available bandwidth. TOPP probing is an extension to the packet pair probing technique for estimating bandwidth bottlenecks. This technique has been studied by a number of researchers [2, 4]. Carter and Crovella [3] propose two packet-pair based techniques, B-probe and C-probe, to measure bottleneck link bandwidth and available bandwidth, respectively. Another variant of the packet-pair technique

4 4 Paper C: A New End-to-End Probing and Analysis... have been suggested by Lai and Baker [6]. They use a receiver-only method based on existing traffic together with a potential bandwidth filtering scheme. In order to quickly adapt to bandwidth changes they introduce a packet window. When estimating the bandwidth, only packets that arrive within the time interval given by the window will be used. Paxson [7] has pointed out a number of weaknesses of the packet pair method, such as the problem with multi-channel links, limitations due to clock resolution and out of order packet delivery. To deal with these shortcomings, he introduces an extension of the packet pair technique called the PBM probing technique. There, estimates for a range of probe bunch sizes (i.e. train lengths) are formed and multiple bottleneck values are allowed. The contributions made in this paper are the following: We present a probing and analysis method, TOPP, that overcomes some of the problems common for many types of existing probing methods. We define three bandwidth metrics that are useful when establishing the somewhat elusive notion of available bandwidth. We show by measurements and simulations how TOPP overcomes the hidden bottleneck problem. 2 Bandwidth Definitions In this section we define and discuss three complementing metrics that are useful when characterizing the bandwidth along a network path. 2.1 Bottleneck Link Bandwidth With the bottleneck link bandwidth, we mean the data transmission rate of the slowest forwarding element of a network path between a sender and a receiver. A sender is seldom alone on a path, but is sharing hops with cross traffic from other sources. Of interest to an application is then the available bandwidth on the path, i.e. the bandwidth the application can achieve when sharing hops with cross traffic. The available bandwidth for an application depends on the behavior of many factors including the application, the protocols, the characteristics of the cross traffic and the routers. The cross traffic will also vary in time, making the available bandwidth an even more elusive goal to measure. We have identified three metrics that can be used to characterize the available bandwidth of a path: the proportional share, the surplus, and the protocol dependent available bandwidth. 2.2 Proportional Share When a link on a path has link bandwidth l and a cross traffic with rate m i and when a sender wants to introduce traffic with rate o such that m < l < m + o, we have an overload situation. What share of the link bandwidth the sender will get depends on the sharing policy of the link multiplexer. The common policy of

5 3 Network Model and Assumptions 5 backbone routers is to use First Come First Serve or some similar policy. Under the assumption of a proportional stateless scheduling policy, such as FCFS, and a random dropping policy of packets at buffer overflow, the connections will get a share of the link bandwidth proportional to their offered rate. The share for new traffic with offered load o is then: f = o m + o l (1) This means that f will increase when o increases. An aggressive sender (e.g. a sender using UDP) could therefore momentarily grab a substantial part of the link bandwidth of a link by increasing the load. 2.3 Surplus Bandwidth With the surplus bottleneck bandwidth of a path we mean the highest possible sending rate of a new sender without affecting cross traffic on the path. With affecting we mean that the cross traffic should not be forced to lower their rate or start to lose packets when the sender starts to send with the surplus rate. Assume that the sum of the cross traffic on link i has a sustained rate of m i. Then the surplus bandwidth on the link is s i = l i m i (l i s i for all i). The surplus bottleneck bandwidth is then: s b = min{s 1, s 2,..., s n } The surplus bandwidth is a lower bound on the available bandwidth. The surplus bandwidth is an important metric, since, with knowledge about the link and surplus bandwidth of a link, a proportional share can be calculated for an arbitrary offered bandwidth. 2.4 Protocol Dependent Available Bandwidth For an application and its protocols, it is interesting to know what sustainable rate the application can achieve and we call this rate the protocol dependent available bandwidth. This rate depends not only on protocols and applications but also on the behavior of the cross traffic. When the sustained rate of the sender reaches the surplus bottleneck rate and beyond, the cross traffic will be affected and may react accordingly to accommodate the new traffic by lowering their rates or not react at all. The protocol dependent bandwidth is hard to predict from probing unless the probe behaves exactly like the application. A probing session may generate a considerable amount of traffic in order to get reliable measurement values. 3 Network Model and Assumptions We assume proportional sharing of links in our paths. Under that assumption, we will get the following proportional share available bandwidth, o n, at the receiver of hop i when the offered load of the sender is o, o 0 = o and the cross traffic at hop i is m i :

6 6 Paper C: A New End-to-End Probing and Analysis... { oj 1 if o j 1 < s j o j = o j 1 m j+o j 1 l j if o j 1 s (2) j The rate after the last hop, i.e. at the receiver will be f = o n. This means that in a scenario where we successively increase the offered rate o, we will get the rate f = o until o reaches the surplus bottleneck bandwidth, s b, at some hop i. Thereafter the rate at the receiver will correspond to the proportional share bandwidth of the link i. While continuing to increase o beyond s b we may eventually reach the second lowest surplus bandwidth on the path. Assume that the surplus links appear at j and at j + p, then we will get: o j = o m j + o l j for o > s j o m j+o l j o j+p = m j+p + o m j +o l j for o j > s j+p (3) The observed proportional share bandwidth at the receiver, f, is thus f = o j+p+1. This model of the observed bandwidth, f as a function of the offered bandwidth, o, will later be used to estimate the link bottleneck bandwidth, surplus bottleneck bandwidth and the proportional share bandwidth of a path. We call a link i congestible for a particular load o if o i > s i. Congestible links are detectable by our probing method. Whether a link is congestible or not depends not only on the surplus bandwidth but also on all links upstream. For a particular load o there is a set of congestible links. Assume the set is ordered in the order they are detected when raising the offered load. We call this order Smallest Surplus First, SSF for short. 4 The Hidden Bottleneck Problem We have discovered a fundamental problem with probing methods based on sequences of back-to-back packets, e.g. C-probe. They can only find the surplus bottleneck if it coincides with the link bottleneck on a path. C-probe sends a train of ICMP echo packets to the destination and measure the inter-arrival times of the reply packets. The probe packets in a train are of equal size and are sent as close as possible. The available bandwidth is calculated as the amount of data in a train divided by the arrival time difference between the last and the first packet in a train. The C-probe packet train only measures the proportional share available bandwidth at the back-to-back transmission rate. For estimating the surplus bandwidth it is necessary to measure the link bottleneck bandwidth with some other probing tool, such as B-probe. Given the link bandwidth l it is possible

7 5 The TOPP Measurement Method 7 to rewrite equation 1 such that: m = o(l f)/f and from the equation derive the surplus bandwidth as s = l m. As an example, assume a two link path, link one is a 155 Mbps link, and the link two is a 10 Mbps link, i.e. l 1 = 155 Mbps and l 2 = 10 Mbps. Cross traffic of 152 Mbps merge with probe traffic into the 155 Mbps link, and the 10 Mbps link has a cross traffic of 3 Mbps, so m 1 = 152 Mbps and m 2 = 3 Mbps. In this configuration the bottleneck surplus bandwidth s b is 3 Mbps (at link 1) and the bottleneck link bandwidth l b is 10 Mbps (at link 2). Assume that we can generate probe traffic at 10 Mbps, i.e. o = 10 Mbps. Sending a probe train with a rate of 10 Mbps will yield o 1 = 10/(152+10) 155 = 9.57 Mbps and then o 2 = 9.57/( ) 10 = 7.61 Mbps. A measurement by e.g. B-probe will detect the link bottleneck bandwidth as 10 Mbps. Using the equation m = o(l f)/f without discrimination then gives: s = l m = 10 (( )/7.61) 10 = 6.86 Mbps. C-probe will hence predict a too high surplus value. We call this phenomenon the hidden bottleneck, since the surplus bottleneck is not visible to a number of probing tools. Our TOPP method is able to accurately find this 3Mbps hidden bottleneck, see later sections. 5 The TOPP Measurement Method To deal with the problems that have been described above we have designed a new method for measuring available bandwidth. One of the ideas in this method is to send trains of packet pairs and for that reason the method goes under the name TOPP (Trains of Packet Pairs). The TOPP measurement method has two separate phases. The first one is the active probing phase where pairs of probe packets are injected into the network. The second phase is the analysis phase, where the bandwidth estimates are calculated based on the reception times of the probe packets. 5.1 Probing Phase One of the aims of the TOPP probing phase is to be network friendly by avoiding to stress the network unnecessarily. This is important for (at least) two reasons; the existing traffic along the probe path will suffer less, and the risk of skewing measurements due to a heavy transient load from the probe traffic is reduced. To achieve these goals the probe traffic is generated in the following way. Starting at some rate o min, n well separated pairs of equally sized probe packets are sent to the destination host. After those n packets have been sent, the offered rate o is increased by o and another set of n probe packets are sent. Then o is increased again (by the same amount o) and another set of n probe packets are sent. This goes on until the offered rate reaches some rate o max which marks the end of the probe phase. Hence, there will be n l = omax o min o offered rate levels. Figures 1 and 2 illustrate the probe phase.

8 8 Paper C: A New End-to-End Probing and Analysis... b t a T p t b T p t c b b b b b Figure 1: Parts of a TOPP packet pair sequence, showing inter- and intra-pair spacing. Probe packets of size b bytes are sent as pairs with decreasing intra-pair spacing, i.e. t a > t b > t c. For each intra-pair spacing, t a, t b, etc, n pairs are sent. The time separation between two probe pairs, T p, is chosen such that the likelihood of more than two probe packets being queued at a node is small. This has the benefit of making the probing phase more network friendly since the nodes along the path will not experience long bursts of probe packets. o o max o min o o min n T p t Figure 2: A TOPP probe sequence, showing the stepwise increase in o. Each black dot corresponds to a pair of packets. On the receiving side, the probe packets are time stamped upon reception. Once all packets have been received (or a timeout is exceeded to deal with lost packets), the time stamps are sent back to the probing host. Hence, all measurements are done one way. The reason for that is that we want to get an estimate for one direction only and thereby avoid the problem of asymmetric paths. 5.2 Analysis Phase The analysis phase relies on the principle of the bottleneck spacing effect. That is, when two packets with time separation S are transmitted over a link with a service time Q b > S, then as the packets leave the link they will separated by R = Q b. Using the size of the packets, b, and the time separation R, the experienced bandwidth across that link can then be estimated as f = b R Should Q b S, then R = S, indicating that the link could service (i.e. transmit) the packets at the rate they arrived. (4)

9 5 The TOPP Measurement Method 9 The outcome of the probing phase is a series of times stamps for all the received probe packets. Since there are n time stamp pairs for each offered rate we calculated the mean of these n values to get one value. As a result, there will be one R i value corresponding to every offered rate, o i. value. Now, using the R i values, the size of the probe packets, b, and equation 4, bandwidth estimates, f i, can be calculated for each of n l offered rate levels. Under the assumptions about the network in section 3, then for each i = 1,..., n l, the relationship between the bandwidth estimate f i and the offered bandwidth o i is given by equations 2. Probe packets sent at an offered rate level o i can only make hops with a surplus bandwidth s o i congested. Since the sequence of offered rates [o 1,..., o n l ] is increasing, several congestible hops may be detected by studying the sequence [ f 1,..., f n l ]. Segmented Regression Given two or more o, f pairs, it is possible to estimate the unknowns l and m in equation 1. A difficulty is that f is not linearly dependent on o so traditional linear regression is not possible. Fortunately, the equation can be converted into a linear form by applying the transformation o/f. The resulting linear equation is then o f = (1 l m ) + 1 l l o = (1 s l ) + 1 o = α + βo (5) l where s is the surplus bandwidth, given by s = l m. Figure 3 shows a plot f 1 τ τ τ o Figure 3: Plot of o/f as a function of o. bandwidth points are indicated. Vertices corresponding to surplus of o/f as a function of o. As can be seen in the figure, the curve is only piecewise linear. This due to the fact that there are more than one congestible hop as explained earlier. Hence, the true model is actually a segmented linear

10 10 Paper C: A New End-to-End Probing and Analysis... model. An overview of this topic can be found in [8]. In the general case, if the breakpoints τ i are be known, then usual linear regression can be applied to each of the segments. Many times these breakpoints can be found by simple inspection but there are more elaborate statistical methods for doing this [5]. Now, assume the τ i s are known (e.g. by studying a plot similar to the one in Figure 3). It is then possible to calculate the link and surplus bandwidths for the congestible hop that affected the probe traffic. However, there is one complication, manifested by equation 2. The rate dictated by one congested link will be the incoming rate to the next congested hop on the path. As a result, an o/f by o plot may be biased since a f i value is plotted against the offered rate o i when in fact it corresponded to a (lower) value o i. If the order of the congestible hops is known, then it is possible to compensate for the dependency of the hops. This is done by calculating what the rate of the incoming probe traffic to a hop j must have been and use that o i value instead of the initially offered rate o i. The calculation of o i is straightforward as it must be the outgoing rate from hop j 1. That rate will, in turn, be determined by the outgoing rate from hop j 2. This recursive unfolding of offered rates can then be traced back to the hop with the lowest surplus bandwidth. But for this hop, the initially offered rates o i can be used to estimate its link bandwidth l and surplus bandwidth s = s b. Once those bandwidths have been estimated, it is possible to calculate the l and s estimates for the remaining congestible links. This is done by traversing them in the order they appear in the network path, i.e. the reverse of the order that they were unfolded and perform the regression for the corresponding segment using the newly obtained o i rates. Normally we will not know the order of the potentially congested hops. Therefore we make the assumption that the congested links are in SSF order, and then proceed with the calculations in same way as described above. 6 Experimental Setup To study the performance of the TOPP method we have made measurements involving three Intel-based hosts running version 2.2 kernels of the Linux operating system. In addition to real world measurements we have performed extensive simulations using the networks simulator ns. In the ns simulations, the topology in Figure 4 has been used. Two 10Mbps Ethernet LAN are connected via a WAN consisting of five routers nodes (the shaded circles). The link bandwidths and delays of the WAN links have been varied depending on simulation. To generate the cross traffic in the WAN, a number of nodes are connected to the router nodes. The ten boxes in the figure illustrate these cross traffic nodes. The nodes that generate and receive probe traffic are drawn as pentagrams in the figure. The cross traffic has been generated by traffic generators sending packets with a uniformly distributed random inter-packet spacing at some given rate. In all measurements, the size of the probe packets have filled an entire Ethernet

11 7 Measurements 11 frame. One reason for making the probe packets as large as possible (without causing fragmentation) is that packet size (together with time resolution) set an upper limit on the rate that can be measured as dictated by equation 4. Another reason is that previous work [1, 3] has indicated that large probe packets lead to more accurate estimates. A B C D E Lan 1 Lan 2 Figure 4: Network topology used in the ns simulations. 7 Measurements In this section, we present results from real measurements and ns simulations, together with theoretical results. 7.1 One Congestible Link Ratio Offered/Measured Offered bandwidth [Mbps] Figure 5: Measurement results from a real network with one congestible link, l i = 10 Mbps and s i = 5 Mbps. If we have a path from sender to receiver with only one congestible link, i.e. only one link i where the surplus bandwidth s i is lower than our probe ceiling

12 12 Paper C: A New End-to-End Probing and Analysis... o max, TOPP is able to detect and estimate the surplus and link bandwidth of this link. Figure 5 shows results from a measurement over a controlled network with one congested link having a link bandwidth l i of 10 Mbps and a surplus bandwidth s i of 5 Mbps, i.e., the cross traffic is 5 Mbps. In the figure, o/f is plotted as a function of the offered bandwidth o. Using our regression analysis method, we can estimate the link bandwidth to 9.19 Mbps and the surplus bandwidth to 4.17 Mbps. Figure 6 shows results from an ns simulation of the same network. For this simulation, regression analysis gives us an estimated link bandwidth of Mbps and a surplus bandwidth estimation of 4.91 Mbps Ratio Offered/Measured Offered bandwidth [Mbps] Figure 6: Measurement results from ns simulations of a network with one congestible link, l i = 10 Mbps and s i = 5 Mbps. Table 1 shows results from ns simulations with varying l i and s i of the congestible link i. Both the correct values l i and s i, and the estimated values l i and s i are shown. As can be seen, the TOPP bandwidth estimations are often quite accurate. 7.2 Two Congestible Links Should two links i and j in the probe path be congestible, both can often be detected and estimated by TOPP. Since TOPP analysis (see the analysis section above) is performed in SSF order, the true order of the links determine the accuracy of TOPP. Should the order be reversed, TOPP analysis in SSF order will introduce a calculation error in the second link. Note however (as already mentioned earlier) that the estimations of the smallest surplus link are always correct, and that the estimations of the other link tend to be conservative, i.e.

13 7 Measurements 13 Table 1: TOPP estimations from ns simulations of one congestible link for varying bandwidths. Correct l i Correct s i Estimated l i Estimated s i we underestimate both l and s for that link. Also, in the case of two congestible links, we can calculate the two alternative β values and then pick the most conservative, i.e. the highest β value (yielding the lowest l and s estimations). We therefore also present the alternative value not presented by TOPP SSF under the heading of ALT (for ALTernative) in the table below. In Table 2, we see theoretical figures for TOPP and train probing estimations for a number of combinations of l and s for two congestible links. The column labeled TOPP SSF contains numbers from SSF analysis as above. The column labeled ALT contains the alternative results as explained above. For train probing, we present an estimation of s calculated from the measured available bandwidth given that the link bandwidth for the link bottleneck link (10 Mbps in the examples) is known. For two of the scenarios presented in the table, there are no values for TOPP estimations of link q. This is because in these scenarios, TOPP can only establish one bottleneck link, p. As can be seen from the figures in Table 2, for the two Table 2: TOPP estimations of l and s for two congestible links, together with train probing estimations. Correct TOPP SSF ALT Train l 1 /s 1 l 2 /s 2 lp / s p lq / s q lq / s q s 155/3 10/7 155/3 10/7 10.3/ /7 155/3 155/3 9.8/6.8 10/ /3 155/7 10/ /7 10/3 10/ /5 155/ /3 10/7 10/ /7 10/3 10/3 7.1/5 10/7 0.92

14 14 Paper C: A New End-to-End Probing and Analysis... first scenarios, TOPP finds bottlenecks that are hidden from train probing. For the next three scenarios, both TOPP and train probing can estimate the lowest surplus bottleneck. For the middle one of these, TOPP is also able to detect the second bottleneck. For the last scenario, train probing results in a notable underestimation. The bottlenecks that TOPP cannot find are obscured because of upstream lower bottlenecks. Note also that when links do not appear in SSF order, TOPP SSF estimates are conservative. Figure 7 shows TOPP measurements from an Ratio Offered/Measured Offered bandwidth [Mbps] Figure 7: Results from a TOPP measurement simulation with two congestible links, see text. ns simulation using l 1 = 155 Mbps, s 1 = 3 Mbps, l 2 = 10 Mbps and s 2 = 7 Mbps, i.e. the same scenario as the top row in the table above. For this data set, TOPP segmented regression yields the estimations l p = Mbps, s p = 2.21 Mbps, l q = 10.7 Mbps and s q = 6.82 Mbps. Figure 8 shows TOPP analysis when the links from the previous example are reversed, i.e. l 1 = 10 Mbps, s 1 = 7 Mbps, l 2 = 155 Mbps, and s 2 = 3 Mbps. For this order of the links, TOPP analysis results in the estimations l p = Mbps, s p = 1.82 Mbps, l q = Mbps and s q = 6.70 Mbps. The conclusion from experiments with two congestible links are that TOPP finds several bottlenecks that are hidden to train probing methods, and that TOPP estimations often can be quite accurate. Also, in the situations where link order assumptions are wrong, TOPP SSF estimates tend to be conservative and thus underestimate s and l bandwidths of the second link.

15 7 Measurements Ratio Offered/Measured Offered bandwidth [Mbps] Figure 8: Results from a TOPP measurement with two congestible links, order reversed compared to previous figure. 7.3 More than Two Congestible Links The ability to detect more than two congestible links depends on the characteristics of the links and the order of them. For links in SSF order, the detection ability depends on our ability to detect vertices on the o/f curve. In theory, any number of congestible links can be detected as long as their surplus bandwidths are all different. Should link surplus bandwidths not be in SSF order, the ability to detect congestible links depends on the order of the links. The same conditions basically hold as in the two-link scenarios above, only with more links involved, i.e. downstream links may be shadowed by limitations in bandwidths of upstream links. As an example of detection of more than two congestible links, Figure 9 shows the analysis of a TOPP probe session over a simulated network having three congestible links with l 1 = 155 Mbps and s 1 = 3 Mbps, l 2 = 34 Mbps and s 2 = 5 Mbps, and l 3 = 10 Mbps and s 3 = 7 Mbps. In the o/f graph, we see the three regression lines corresponding to three detected surplus bandwidths. From the analysis, we obtain the following estimates: l 1 = Mbps, s 1 = 2.03 Mbps, l 2 = 31.9 Mbps, s 2 = 4.02 Mbps, l 3 = 10.1 Mbps and s 3 = 6.37 Mbps. As can be seen, surplus estimates are accurate within 1 Mbps of the correct bandwidths, while link bandwidth estimates are accurate within 8 %.

16 16 Paper C: A New End-to-End Probing and Analysis Ratio Offered/Measured Offered bandwidth [Mbps] Figure 9: Measurement with three congestible links, see text. 8 Conclusions We have demonstrated the feasibility of our TOPP end-to-end probing and analysis method with real measurements and ns simulations. We have defined the terms surplus, proportional share, and protocol dependent available bandwidth in terms of our network model. The TOPP method can estimate what we call hidden bottlenecks (when the surplus bottleneck is not on the link bottleneck). The TOPP regression analysis can not only estimate the surplus bottleneck and link bandwidth but also the surplus and link bandwidths of several hops under some assumptions. TOPP is network friendly. It uses trains of packet pairs and the distance between pairs is variable in order to avoid overloading the network with bursts of packets and to minimize the impact on cross traffic. The set of experiments and simulations show that TOPP can handle single hops and multiple hops with varying bandwidths and cross traffic.

17 References [1] Mats Björkman and Bob Melander. Impact of the Ethernet capture effect on bandwidth measurements. In Networking 2000 Conference Proceedings, pages , Paris, France, May [2] Jean-Chrysostome Bolot. End-to-end packet delay and loss behavior in the Internet. In Proceedings of ACM SIGCOMM, pages , San Francisco, CA, USA, September [3] Robert Carter and Mark Crovella. Measuring bottleneck link speed in packet-switched networks. Technical Report , Boston University Computer Science Department, Boston, MA, USA, March [4] Srinivasan Keshav. A control-theoretic approach to flow control. In Proceedings of ACM SIGCOMM, pages 3 15, Switzerland, September [5] Helmut Küchenhoff. An exact algorithm for estimating breakpoints in segmented generalized linear models. Computational Statistics 12, pages , [6] Kevin Lai and Mary Baker. Measuring bandwidth. In Proceedings of IEEE INFOCOM, pages , New York, NY, USA, March [7] Vern Paxson. End-to-end Internet packet dynamics. In Proceedings of ACM SIGCOMM, pages , Cannes, France, September [8] George A. F. Seber and Christopher J. Wild. Nonlinear regression. John Wiley & Sons, New York,