What do packet dispersion techniques measure?

Transcription

1 Wha do packe dispersion echniques measure? Consaninos Dovrolis Parameswaran Ramanahan David Moore Universiy of Wisconsin Universiy of Wisconsin CAIDA Absrac The packe pair echnique esimaes he capaciy of a pah (boleneck bandwidh) from he dispersion (spacing) experienced by wo back-o-back packes [1][2][3]. We demonsrae ha he dispersion of packe pairs in loaded pahs follows a mulimodal disribuion, and discuss he queueing effecs ha cause he muliple modes. We show ha he pah capaciy is ofen no he global mode, and so i canno be esimaed using sandard saisical procedures. The effec of he size of he probing packes is also invesigaed, showing ha he convenional wisdom of using maximum sized packe pairs is no opimal. We hen sudy he dispersion of long packe rains. Increasing he lengh of he packe rain reduces he measuremen variance, bu he esimaes converge o a value, referred o as Asympoic Dispersion Rae (ADR), ha is lower han he capaciy. We derive he effec of he cross raffic in he dispersion of long packe rains, showing ha he ADR is no he available bandwidh in he pah, as was assumed in previous work. Puing all he pieces ogeher, we presen a capaciy esimaion mehodology ha has been implemened in a ool called pahrae. Keywords Acive nework measuremens, bandwidh monioring, boleneck bandwidh, available bandwidh. I. INTRODUCTION The Inerne is a commercial infrasrucure in which users pay for heir access o an Inerne Service Provider (ISP), and from here o he global Inerne. I is ofen he case ha he performance level (and ariff) of hese nework connecions is based on heir bandwidh, since more bandwidh normally means higher hroughpu and beer qualiy-of-service o an applicaion. In such an environmen, bandwidh monioring becomes a crucial operaion. Users need o check wheher hey ge he access bandwidh ha hey have paid for, and wheher he nework clouds ha hey use are sufficienly provisioned. ISPs also need bandwidh monioring ools in order o plan heir capaciy upgrades, and o deec congesed or underuilized links [4]. Nework operaors are increasingly using ools such as MRTG [5] o monior he uilizaion of heir links wih informaion obained from he rouer managemen sofware. These echniques are based on saisics mainained by he rouers, and hey are normally very accurae. Their drawback, however, is ha hey can be performed only wih access o he rouer, and such an access is usually limied o he nework manager. Insead, in his paper we focus on an end-o-end bandwidh monioring approach ha requires he cooperaion of only he pah end-poins. Even hough end-o-end approaches are usually no as accurae as rouer-based mehodologies, hey are ofen he only feasible approach for monioring a pah ha crosses several neworks. We define a nework pah as he sequence of links ha forward packes from he pah sender (source) o he receiver (sink) 1. Two bandwidh merics ha are commonly associaed wih a pah are he capaciy and he available bandwidh. The This work was suppored in par by he USENIX associaion and by he Naional Science Foundaion under Gran No. NCR We assume ha he pah is fixed and unique, i.e., no rouing changes or mulipah forwarding occur during bandwidh monioring. capaciy is he maximum IP-layer hroughpu ha he pah can provide o a flow, when here is no compeing raffic load (cross raffic). The available bandwidh, on he oher hand, is he maximum IP-layer hroughpu ha he pah can provide o a flow, given he pah s curren cross raffic load. The link wih he minimum ransmission rae deermines he capaciy of he pah, while he link wih he minimum unused capaciy limis he available bandwidh. To avoid he erm boleneck link, ha has been widely used for boh merics, we refer o he capaciy limiing link as he narrow link, and o he available bandwidh limiing link as he igh link. Specifically, if is he number of hops in a pah, is he ransmission rae or capaciy of link, and is he ransmission rae of he source, hen he pah s capaciy is Addiionally, if is he uilizaion of link (wih and =), he unused capaciy in link is, and so he available bandwidh of he pah is Noe ha he available bandwidh definiion requires saionary raffic and sufficienly large imescales so ha he uilizaion erms o be pracically consan. The capaciy and available bandwidh merics are furher discussed in he Appendix. The packe pair echnique is a well-known procedure o measure he capaciy of a pah. When a packe is ransmied in a link, i encouners a ransmission or serializaion delay due o he physical bandwidh limiaions of he link and he hardware consrains of he ransmiing equipmen. In a link of capaciy and for a packe of size, he ransmission delay is. A packe pair experimen consiss of wo packes sen back-o-back, i.e., wih a spacing ha is as shor as possible, from he source o he sink. Wihou any cross raffic in he pah, he packe pair will reach he receiver dispersed (spaced) by he ransmission delay in he narrow link. So, he receiver can compue he capaciy from he measured dispersion, as. Figure 1 illusraes he packe pair echnique in he case of a hree-link pah, using he fluid analogy inroduced in [6]. Even hough simple in principle, his echnique can produce widely varied esimaes and erroneous resuls. The main reason is ha he cross raffic in he pah disors he packe pair dispersion, increasing or decreasing he capaciy esimaes. The main objecive in his paper is o develop a capaciy esimaion mehodology, based on end-o-end measuremens, ha is robus o cross raffic effecs. We show ha a sraighforward applicaion of he packe pair echnique canno, in general, produce accurae resuls when he cross raffic effecs are ignored. (1) (2)

2 Sender S L/3C L/C! =L/C C 1 = 3C C 2 = C C 3 = 3C Receiver Fig. 1. Graphical illusraion of he packe pair echnique. The widh of each link corresponds o is capaciy. The reason is ha he disribuion of bandwidh measuremens is mulimodal, and some local modes, relaed o he cross raffic, are ofen sronger han he capaciy mode. The effec of he probing packe size is also invesigaed, showing ha he convenional wisdom of using maximum sized packe pairs is no opimal in heavily loaded pahs. We hen sudy he dispersion of long packe rains. Increasing he lengh of he packe rain reduces he measuremen variance, bu he esimaes converge o a value, referred o as Asympoic Dispersion Rae (ADR), ha is lower han he capaciy. This sudy shows ha, conrary o previous work [1], he ADR is no he available bandwidh in he pah. For single hop pahs hough, we derive a formula for compuing he available bandwidh from he measured ADR. Finally, we describe a capaciy esimaion mehodology ha has been implemened in a ool called pahrae. This mehodology uses many packe pairs o uncover he mulimodal bandwidh disribuion. The challenge is o idenify he local modes, and o hen selec he mode ha corresponds o he pah capaciy. This laer par is based on he dispersion of gradually longer packe rains. The mehodology is accurae when he capaciy is beween 1-4 Mbps and he specified esimae resoluion is 1 Mbps. For higher capaciy pahs, a larger esimae resoluion is required. The res of he paper is srucured as follows. Secion II summarizes he previous work on bandwidh monioring. Secion III invesigaes he disribuion of bandwidh esimaes using packe pairs, while Secion IV invesigaes he disribuion of bandwidh esimaes using packe rains. An analyical model for he dispersion of long packe rains is given in Secion V. Secion VI focuses on he size of packe pairs. Based on he insigh of he previous secions, Secion VII presens a capaciy esimaion mehodology and he pahrae implemenaion. Some measuremens using pahrae are given in Secion VIII. We conclude and highligh some open problems in Secion IX. II. PREVIOUS WORK R The concep of packe dispersion, as a burs of packes crosses he narrow link of a pah, was originally described in [6]. Jacobson did no consider cross raffic effecs, and so he disincion beween capaciy and available bandwidh was no made. Keshav also sudied he same idea in he conex of congesion conrol [7], bu he recognized ha he dispersion of packe pairs is no relaed o he available bandwidh when he rouer queues are Firs-Come-Firs-Served (FCFS). He showed ha if all rouers use a fair queueing discipline, hen he cross raffic is isolaed and he packe pair echnique can esimae he available bandwidh in he pah. Bolo used packe dispersion measuremens o esimae he capaciy of a ransalanic link and o characerize he inerarrivals of cross raffic [8]. In he pas, he packe pair echnique was simpler o apply. The main reason is ha he possible capaciy values used o be deermined by a few well-known links, such as dial-up modems, ISDN links, T1 s, T3 s, and Ehernes. Today, mainly hrough he use of ATM virual circuis/pahs, he bandwidh given o a pah can be any value up o he physical capaciy of he underlying links. For insance, ISPs ofen pariion an OC-3 link in several fracional virual links, leased in a granulariy of a few Mbps or so [9]. The early works on packe pair dispersion were followed by sophisicaed variaions, focusing on robus saisical filering echniques. Carer and Crovella creaed bprobe, in which several packe pair measuremens, originaing from packes of differen sizes, are processed using union and inersecion filering o produce he final capaciy esimae [1]. Lai and Baker used a kernel densiy esimaor as heir saisical filering ool [3]. In [1] and [3], he underlying assumpion is ha he capaciy of a pah is relaed o he mos common range of bandwidh measuremens, i.e., he mode of he underlying disribuion. Paxson was he firs o observe ha he disribuion of bandwidh measuremens is mulimodal, and he elaboraed on he idenificaion and final selecion of a capaciy esimae from hese modes [1]. He also used packe rains of differen lenghs o deec mulichannel links. The complee mehodology is called Packe Bunch Modes (PBM) [2], bu as Paxson noes in his disseraion [1] (p.267): I is unforunae ha PBM has a large heurisic componen, as i is more difficul o undersand. (..) We hope ha he basic ideas underlying PBM searching for muliple modes and inerpreing he ways hey overlap in erms of boleneck changes and muli-channel pahs migh be revisied in he fuure, in an aemp o pu hem on a more sysemaic basis. The echniques discussed in his paper also rely on some heurisics, bu conrary o Paxson s work, we explain he observed mulimodaliies based on cross raffic effecs. Dispersion echniques using packe rains insead of packe pairs have also been proposed for he esimaion of he available bandwidh in a pah. Carer and Crovella developed a ool called cprobe which esimaes he available bandwidh from he dispersion of rains of eigh packes [1]. Oher researchers have proposed ha he sshresh variable in TCP s slow-sar phase, which should ideally be se o he produc of he connecion s RTT wih he available bandwidh, can be deermined from he dispersion of he firs hree or four ACKs [11], [12]. The underlying assumpion in [1], [11], [12] is ha he dispersion of long packe rains is inversely proporional o he available bandwidh. However, as we show in his paper, his is no rue. Finally, several ools ha measure he capaciy of every link in a pah were recenly developed: Jacobson s pahchar [13], Downey s clink [14], Mah s pchar [15]; for a sudy of hese ools see [16]. The underlying idea here is no based on he dispersion of packe pairs or rains, bu on he variaion of he one-way delay as he packe size increases. Unforunaely, because hese ools require he generaion of IP replies from he rouers, which is a process ha normally follows differen processing pahs in a rouer, he resuling measuremens are ofen quie inaccurae. For example, a 1 Mbps Fas Eherne link

3 in our LAN is always measured in he range of 3-4 Mbps; similar erroneous measuremens are repored in [17]. Recenly, Lai and Baker proposed a echnique called packe ailgaing which avoids he need for IP replies from he pah rouers [17]. However, he repored capaciy measuremens are sill ofen inaccurae. A possible explanaion is ha he errors in he link capaciy esimaes accumulae as he measuremens proceed along he pah. Hop i-1 Hop i " i-1 " i-1 1 2! i-1 1 d i " i 2 di! i " i 1 2 " i-1 1! i-1 1 " i Fig. 2. The wo cases of Equaion 4. " i i d i d " i 1 2! i III. PACKET PAIR DISPERSION Consider an -hop pah defined by he sequence of capaciies. Two packes of size are sen back-o-back from he source o he sink; hese packes are he packe pair or probing packes. The dispersion of he packe pair is he inerval from he insan he las bi of he firs packe is received a a cerain pah poin o he insan he las bi of he second packe is received a ha poin 2. The dispersion is afer he source, and le i be afer link. When he packe pair reaches he sink, he dispersion is and he receiver compues a bandwidh esimae. Since varies in general, if we repea he experimen many imes he values will form a cerain disribuion. Our goal, hen, is o infer a final pah capaciy esimae from he disribuion. Firs, suppose ha here is no cross raffic in he pah. I is easy o see ha he dispersion canno be lower han he dispersion a he previous hop and he ransmission delay a hop, i.e.,. Applying his model recursively from he sink back o he source, we find ha he dispersion a he receiver is Pah Source C Pah Sink H... C1 C2 C3 CH Cross Traffic Sources (a) Pah persisen cross raffic Cross Traffic Sink Cross Traffic Sinks Pah Pah Source Sink H... C C1 C2 C3 CH Cross Traffic Sources (b) One-hop persisen cross raffic Fig. 3. The wo exreme cases of cross raffic rouing. where and are he capaciy and he ransmission delay of he narrow link, respecively. Consequenly, when here is no cross raffic, all he bandwidh esimaes are equal o he capaciy ( ). When here is cross raffic in he pah, he probing packes can experience addiional queueing delays due o cross raffic. Le be he queueing delay of he firs probing packe a hop, and be he queueing delay of he second probing packe a hop afer he firs packe has been ransmied a ha link (see Figure 2). The dispersion afer hop is if oherwise Noe ha when and, he dispersion decreases from hop o hop ( ). This effec can cause a dispersion a he receiver ha is lower han he dispersion a he narrow link, i.e.,, if here are addiional hops afer he narrow link; we refer o such links as pos-narrow links 3. This observaion means ha he capaciy of he pah canno be esimaed simply from he minimum measured dispersion, as ha value could have resuled from a pos-narrow link. We refer o IP packe boundaries. If here are more han one links wih capaciy of hem in he pah. (3) (4), he narrow link is he las In order o examine he properies of he disribuion in a conrollable and repeaable manner, we used he Nework Simulaor [18]. Simulaions allow us o invesigae he cross raffic effecs in packe pair dispersion, avoiding issues such as roue changes, mulichannel links, imesamping accuracy and resoluion, ha can disor he measuremens. We have also verified he repored resuls wih Inerne measuremens 4. The simulaed model follows he descripion given earlier, i.e., he source sends packe pairs and he sink compues bandwidh esimaes from he measured dispersions. The cross raffic (CT) is generaed from sixeen Pareo sources a each hop wih =1.9, i.e., he inerarrivals have infinie variance. The aggregaion of many Pareo sources wih has been shown o produce Long Range Dependen (LRD) raffic [19]. The CT packe size is, which is eiher consan or follows a random disribuion (described laer). The packe scheduling discipline in he simulaion experimens is FCFS. An imporan issue is he rouing of he CT packes relaive o he packe pairs. The wo exreme cases are shown in Figure 3; in Figure 3-a he CT packes follow he same pah as he packe pairs (pah persisen CT), while in Figure 3-b he CT packes always exi one hop afer hey ener he pah (one-hop persisen CT). The effec of CT rouing will be discussed in V; for now, we simulae he one-hop persisen CT case. In he following experimens, he bandwidh disribuion is formed from 1 packe pair experimens. Figure 4 shows he hisogram of, wih a bin widh of 2 The locaions of he measuremen hoss are given in VIII.

4 P={1,75,55,4,6,8}, L=Lc=15B Sub!Capaciy Dispersion Range (SCDR) Capaciy Mode () u=2% Pos!Narrow Capaciy Mode (PN) P={1,75,55,4,6,8}, L=Lc=15B SCDR u=8% PN P={1,75,55,4,6,8}, u=5%, L=15B Fixed CT packe size: Lc=15B SCDR PNs P={1,75,55,4,6,8}, u=5%, L=77B Variable CT packe size: Lc uniform in [4,15]B SCDR PN (a) Ligh load condiions Fig. 4. The (b) Heavy load condiions disribuion in wo differen pah loads. Mbps, for a pah 1,75,55,4,6,8 (all capaciies in Mbps). Noe ha he pah capaciy is =4Mbps, while he pos-narrow links have capaciies of 6 and 8 Mbps, respecively. In Figure 4-a, each link is 2% uilized, whereas in Figure 4- b, all links are 8% uilized. When he pah is lighly loaded ( =2%) he capaciy value of 4 Mbps is prevalen in, forming he Capaciy Mode (), which in his case is he global mode of he disribuion. Bandwidh esimaes ha are lower han he are caused by CT packes ha inerfere wih he packe pair, and hey define he Sub-Capaciy Dispersion Range (SCDR). For insance, he SCDR in Figure 4-a is beween 1 and 4 Mbps; he cause of he local modes in he SCDR is discussed in he nex paragraph. Bandwidh esimaes ha are higher han he are caused in he pos-narrow links when he firs probing packe is delayed more han he second; hese esimaes are referred o as Pos-Narrow Capaciy Modes (PNs). Noe a PN a 6 Mbps, which is he capaciy of he link jus afer he narrow link; his local mode is creaed when he firs probing packe is delayed long enough for he packe pair o be serviced back-o-back in ha link. In heavy load condiions ( =8%), he probabiliy of CT packes inerfering wih he probing packes is large, and he is no he global mode of. Insead, he global mode is in he SCDR, which now dominaes he bandwidh measuremens. A key poin here is ha he pah capaciy canno be always correcly esimaed by saisical echniques ha exrac he mos common bandwidh value or range. Insead, we mus examine he resuling bandwidh disribuion in queueing erms, analyze wha causes each of he local modes, and wha differeniaes he from he res of he local modes. Figure 5 shows for he same pah when he CT packe size is fixed (15 byes) and when i varies uniformly in he range [4, 15] byes ( =5%). In he firs case, he probing packe size is also 15 byes, while in he second case i is 77 byes, i.e., he average of he [4, 15] range 5. When all packes have he same size ( = =15B), i is simpler o explain he local modes in he SCDR. For insance, consider he pah, and assume ha all packes have he same size. A local mode a 3 Mbps can be caused by a packe inerfering wih he packe pair a he 6 Mbps link, since in ha More abou he selecion of in VI. (a) Fixed CT packe size (b) Variable CT packe size Fig. 5. Fixed versus variable CT packe size. C =6Mbps i-1 C =4Mbps i L/6 L/6 L/6 CT 2 1 L/4 L/4 1 2 L/4 + (2 L/6 - L/4) = L/3 Fig. 6. Explanaion of he 3 Mbps local mode in Figure 5-a. case he dispersion afer he narrow link is (see Figure 6). Similarly, a mode a 2 Mbps is caused by a packe inerfering wih he packe pair a he 4 Mbps link or by wo packes inerfering a he 6 Mbps link, and so on. When he CT packe size varies uniformly in he range [4,15]B hough (Figure 5-b), he resuling dispersion is less predicable, since a single packe inerfering wih he packe pair can produce a range of dispersion values, depending on is size. However, he and one or more of he PNs are sill disinc in he disribuion, as hey are caused by he probing packes being serviced back-o-back from he narrow or from posnarrow links, respecively. Several measuremen sudies have shown ha he packe size disribuion in he Inerne is cenered around hree or four values [2], [21]. Specifically, abou 5% of he packes are 4 byes, 2% are 552 or 576 byes, and 15% are 15 byes. These common packe sizes would cause a packe pair bandwidh disribuion ha is more similar o he discree dispersion effecs of Figure 5-a, raher han he coninuous dispersion effecs of Figure 5-b. IV. PACKET TRAIN DISPERSION Exending he packe pair echnique, he source can send back-o-back packes of size o he sink; we refer o hese packes as a packe rain of lengh. The sink measures he oal dispersion of he packe rain, from he firs o he las packe, and compues a bandwidh esimae as. Many such experimens form he bandwidh disribuion. If here is no cross raffic in he pah, he bandwidh esimaes will be equal o he capaciy, as in he packe pair case. Measuring he capaciy of a pah using packe rains is required when he narrow link is mulichanneled [2]. In a -channel link of oal capaciy, he individual channels forward packes in parallel a a rae of and he link capaciy can be measured from

5 P={1,75,55,4,6,8}, u=8%, L=Lc=15B N= P={1,75,55,4,6,8}, u=8%, L=Lc=15B N= jhana (CAIDA) o ren (U!Delaware) L=15B N= jhana (CAIDA) o ren (U!Delaware) L=15B N= (a) Packe pairs ( =2) (b) Packe rains wih =3 (a) Packe pairs ( =2) (b) Packe rains wih = P={1,75,55,4,6,8}, u=8%, L=Lc=15B N= P={1,75,55,4,6,8}, u=8%, L=Lc=15B Asympoic Dispersion Rae R=15Mbps N= jhana (CAIDA) o ren (U!Delaware) L=15B N= jhana (CAIDA) o ren (U!Delaware) L=15B R=29Mbps N= (c) Packe rains wih =5 (d) Packe rains wih =1 Fig. 7. The effec of he packe rain lengh (simulaions). (c) Packe rains wih =12 (d) Packe rains wih =2 Fig. 8. The effec of he packe rain lengh (measuremens). he dispersion of packe rains wih = +1. Packe rains are also required o measure he susainable rae of a raffic shaper 6. I may appear a firs ha using packe rains, insead of packe pairs, makes he capaciy esimaion more robus o random noise caused by cross raffic. One can argue ha his is rue because packe rains lead o larger dispersion values, which are more robus o measuremen noise. However, his is no he case due o he following reason. Alhough he dispersion becomes larger as increases, so does he noise in he measured values of, since i becomes more likely ha CT packes will inerfere in he packe rain. This issue was also briefly menioned in [1] (p.259), noing ha packe rains should be less prone o noise, since individual packe variaions are smoohed over a single large inerval raher han -1 small inervals, bu wih a larger he greaer he likelihood ha a packe rain will be dispersed by cross raffic, leading o bandwidh underesimaion. In his secion, we presen simulaion and experimenal resuls illusraing he effec of in he bandwidh disribuion, and make some general observaions abou his relaion. Figure 7 shows he hisograms of, for four increasing values of, from simulaions of he pah 1,75,55,4,6,8 wih =8% in all links. Figure 8 shows he hisograms of, for four increasing values of, from Inerne measuremens a he pah from jhana (in San Diego CA) o ren (in Newark DE) during June 2. A firs observaion is ha, as increases, he and PC- NMs become weaker, unil hey disappear, and he SCDR pre- Traffic shapers, usually in he form of a leaky bucke, limi he capaciy of a (virual) link from a peak rae o a susainable rae afer a cerain burs size. vails in he bandwidh disribuion. The reason is ha, as increases, almos all packe rains encouner addiional dispersion due o CT packes. This also means ha he bes value of for generaing a srong capaciy mode is =2, i.e., o use packe pairs; anyhing longer han packe pairs is more likely o ge addiional dispersion due o cross raffic. A second observaion is ha, as increases, becomes unimodal. This implies ha, when is large, he dispersion of packe rains by CT packes is no deermined by disinc inerference cases, forming local modes, bu i is deermined by he aggregae amoun of CT inerfering wih he packe rain. A hird observaion is ha he range of he disribuion, which is relaed o he measuremen variance, decreases as increases. This means ha he variance in he amoun of cross raffic inerfering wih he packe rain decreases, as he lengh of he packe rain increases. A fourh observaion is ha, when is sufficienly large and is unimodal, he cener of he (unique) mode is independen on. We refer o he cener of his unique mode as he Asympoic Dispersion Rae (ADR). The fac ha ADR does no depend on he packe rain lengh means ha, for sufficienly large, he dispersion of he packe rain becomes proporional o -1, and hus he packe rain lengh cancels ou from he bandwidh esimae ; his observaion is explained in he nex secion for cerain special cases. V. ASYMPTOTIC DISPERSION RATE In his secion, we presen a model for he dispersion of packe rains, aking ino accoun he cross raffic in he pah. Firs, consider a single hop pah wih, i.e., he

6 link ( link-1 ) is he narrow link. A packe rain of lengh is sen from he source o he sink wih iniial dispersion. Le be he average incoming rae of cross raffic in link-1. The average amoun of cross raffic ha arrives in link-1 during is. Assuming ha he link-1 queue is serviced in a FCFS basis, he cross raffic inerferes wih he packe rain packes, and so he average dispersion a he exi of he narrow link is where is he load (uilizaion) of he narrow link due o cross raffic. Consequenly, he average bandwidh esimae a he receiver, ha we refer o as he Asympoic Dispersion Rae, is which is lower han he pah capaciy. Noe ha he ADR is independen of, as noed in IV, since he amoun of inerfering cross raffic, and hus he overall dispersion, is proporional o -1. As shown in Figures 7-d and 8-d, even wih he bursy Pareo cross raffic or wih he acual Inerne raffic, a value of around 1-2 is normally sufficien o produce a narrow esimae of. Some commens on Equaion 6 follow. Firs, if he capaciies and are known, we can measure from he dispersion of long packe rains, compue he cross raffic uilizaion from Equaion 6, and hen compue he available bandwidh as. So, he available bandwidh of single hop pahs can be esimaed, using he dispersion of packe rains ha are sufficienly long o produce a narrow esimae of. This also implies ha he available bandwidh is no inversely proporional o he dispersion of long packe rains, as was assumed in [1], even for single hop pahs. For example, in he pah of Figure 7-d we have ha =15 Mbps, while =4(1-.8)=8 Mbps. Second, for capaciy esimaion purposes, i helps o injec he probing packes in he pah from a higher bandwidh inerface (higher ), since he cross raffic erm is hen smaller. Third, he erm is equal o, and so, i is equal o he average number of CT byes inerfering wih wo successive probing packes. These resuls can be generalized o an -hop pah wih, for he case of pah persisen cross raffic ( III). Le be he average rae of cross-raffic ha eners he pah in link 7. The average dispersion a he exi of link, hen, is, and he ADR becomes For insance, for he pah : wih Since he cross raffic is pah persisen (see Figure 3-a), he oal cross raffic rae in link is. (5) (6) (7) (8) When he capaciies do no decrease along he pah, he analysis is more complicaed. In he single-hop case wih, here would be an idle spacing of duraion a he exi of link-1 beween any wo probing packes, if here was no cross raffic. The cross raffic can fill in he idle space in he packe rain, or cause addiional dispersion wihou filling in all he idle space. A lower bound on he dispersion can be derived if we assume ha he cross raffic increases he packe rain dispersion beyond only afer i fills in all he idle spacing. When his is he case, he dispersion a he receiver is If he cross raffic load is sufficienly low ( ), he dispersion is no increased a link-1 (i.e., = ), and so. Oherwise, he final dispersion becomes, which gives he same ADR value as Equaion 6. These resuls can be exended for he case of hops, when he cross raffic is pah persisen. Specifically, a lower bound on he dispersion can be derived if we assume ha he cross raffic increases he packe rain dispersion only afer i fills in all he idle spacing beween probing packes. Then, (9) if or (1) Given he capaciies and cross raffic raes in each hop, and since, we can solve recursively for, and hus for. When he cross raffic is no pah persisen, i.e., CT packes exi he pah before he las hop, he dispersion of packe rains is hard o analyze for he same reason: CT packes can inerfere in he packe rain increasing is dispersion, and hen exi he pah leaving idle space, or bubbles, beween probing packes. These bubbles can be filled in by CT packes in subsequen hops, or hey can persis unil he packe rain reaches he sink. For he case of one-hop persisen cross raffic (see Figure 3-b), an upper and a lower bound can be derived for. Noe ha since he cross raffic is assumed o be one-hop persisen in his case, he uilizaion of link is. For an -hop pah in which all links have he same capaciy, i can be shown ha he ADR is (11) The lower bound corresponds o he case ha bubbles are never filled in, while he upper bound corresponds o he case ha he bubbles creaed a he link wih he maximum uilizaion are he only ones ha reach he receiver, and ha he res of he pah links jus fill in (parially) hose bubbles. VI. THE SIZE OF PROBING PACKETS In his secion, we focus on he effec of he packe size in packe pair probing. The convenional wisdom, as refleced

7 P={1,75,55,4,6,8}, u=5% Lc : uniform in [4,15]B L=1B SCDR PN (a) =1 byes P={1,75,55,4,6,8}, u=5% Lc : uniform in [4,15]B SCDR L=15B (b) =15 byes Fig. 9. Small versus large packe size in packe pair probing. for insance in [1] or [1], is ha he opimal is he maximum non-fragmened packe size, i.e., he pah Maximum Transmission Uni (MTU) size. The reason is ha a higher leads o larger dispersion, which in urn is easier o measure, more robus o queueing delay noise, and less sensiive o he imesamping resoluion a he receiver. This previous reasoning, however, does no ake ino accoun he effecs of cross raffic. A larger packe size leads o a wider ime inerval in which a CT packe can inerfere wih he packe pair. Suppose ha a packe pair arrives a a link wih capaciy. If a CT packe arrives a link in he ime inerval beween he arrival of he firs and he second probing packes, which is of lengh, i will inerfere wih he probing packes, increasing he dispersion above. The larger he, he higher he likelihood of an inerfering CT arrival, and hus he SCDR becomes more prevalen in he bandwidh disribuion. This effec is shown in Figure 9, where is shown for a small packe ( =1B), versus a large, Eherne frame sized, packe ( =15B). The narrow link dispersion is 15 imes smaller in he =1B case, causing a much weaker SCDR han he =15B case. A minimum sized packe, however, is no opimal eiher. As decreases, he dispersion decreases proporionally, and hus, i becomes more suscepible o disorion a he pos-narrow links. Suppose ha =1B, = 4,8 and ha a packe pair leaves he narrow link back-o-back, i.e., wih =2 s. I only akes one CT packe, larger han 1 byes, a he 8 Mbps link o delay he firs probing packe so much ha he packe pair dispersion is conrolled by ha link, i.e., =1 s. In oher words, when is small, he formaion of PNs becomes more likely and he becomes weaker. This can be seen in Figure 9-a; noe he srong PN a 6 Mbps, which is acually sronger han he a 4 Mbps. On he oher hand, here are no significan PNs when =15B, as shown in Figure 9-b. Given he previous rade-off in he selecion of he packe size, a value of somewhere in he middle of he range is preferred. For insance, compare Figure 9 wih he bandwidh disribuion in Figure 5-b, where is se o he average of he CT packe size range ( =77B): he is srong in Figure 5-b compared o boh he SCDR and PN pars of he bandwidh disribuion. The empirical conclusion from our Inerne experimens is ha a packe size around 8 byes leads o he sronger in heavily loaded pahs. For lighly loaded pahs, he selecion of he packe size is no so imporan. Finally, we noe a pracical issue ha is relaed o he minimum dispersion ha he receiver can measure. A receiving hos can only measure he dispersion of a packe pair when i is higher han. This lower bound is deermined by he laency o receive a packe in he OS, o move he packe from kernel o user space hrough a recvfrom sysem call, o imesamp he arrival, and o perform any oher operaions of he receiving program before waiing for he second probing packe. For pahrae, we measured in several differen plaforms, including Sun Ulra-1 and Penium-II worksaions running Solaris 2.6 or Free-BSD 3.2, and he minimum dispersion is in he order of 3 o 4 s. Given for a specific receiver, he maximum possible capaciy ha can be measured for a packe size is. For example, wih =4 s and =8B, he maximum capaciy ha can be measured is 16 Mbps. On he oher hand, when a rough esimae of he capaciy is known, he packe size should be a leas. VII. A CAPACITY ESTIMATION METHODOLOGY In his secion, we presen a capaciy esimaion mehodology based on he insigh developed so far in he paper. This mehodology has been implemened in a ool called pahrae. The pahrae mehodology requires he cooperaion of boh he source and he sink, i.e., i is a wo end-poin mehodology. More flexible approaches require access only a he source of he pah, forcing he sink o reply o each received packe using IP, UDP-echo, or TCP-FIN packes. The problem in hose approaches is ha he reverse pah from he sink o he source, hrough which he replies are forwarded, affecs he bandwidh measuremens, making i hard o decouple he characerisics of he wo pahs. We prefer he wo end-poin mehodology, even hough i is less flexible, because i is more accurae. Phase I: Packe pair probing. As shown in IV, one is more likely o observe he capaciy mode using packe pairs han using packe rains. Consequenly, in his phase we use a large number of packe pair experimens o uncover all he local modes of he bandwidh disribuion, expecing ha one of hem is he. Also, as shown in VI, here is a rade-off in he selecion of he probing packe size : smaller packes lead o sronger PNs, while larger packes lead o a more prevalen SCDR. A probing packe size of =8 byes usually leads o he sronges in he resuling bandwidh disribuion. In pahrae, Phase I consiss of =2 packe pair experimens using a packe size of =8 byes. From he resuling disribuion of bandwidh measuremens, we obain all he local modes. The numerical procedure for he idenificaion of he local modes is no described here due o space consrains. I is similar o he algorihm described in [1], bu he user has o specify he hisogram bin widh, which is also he resoluion of he final capaciy esimae. If, for example, he resoluion is =2 Mbps, pahrae will produce a final esimae ha is a 2 Mbps inerval. As will be shown laer, he resoluion is a criical parameer for he accuracy of he final resul. The sequence of local modes, in increasing order, is denoed as. We expec ha one of hese local modes, say, is he (i.e., ), wih he larger modes

8 P={1,7,6,4,55,8,65,9,4,75,9}, u=3% 7 Lc: uniform in [4,15]B 6 L=77B =4Mbps N= P={1,7,6,4,55,8,65,9,4,75,9}, u=3% Lc: uniform in [4,15]B 12 L=77B N= # + =37Mbps jhana (CAIDA) o drwho (Zurich), L=15B N=2 =27Mbps jhana (CAIDA) o drwho (Zurich), L=15B # + =24Mbps N= (a) =2 (b) = =8 Fig. 1. Illusraion of capaciy esimaion (simulaions). (a) =2 (b) = =12 Fig. 11. Illusraion of capaciy esimaion (measuremens). being PCNMs, and he smaller modes being in he SCDR of. If he disribuion is unimodal, which happens in very lighly loaded pahs, he measuremen process erminaes and he capaciy esimae is he unique mode. Oherwise, Phase-II selecs from. Phase II: Packe rain probing. As noed in IV, as increases, he and he PNs are eliminaed from he bandwidh disribuion, and he SCDR accumulaes all measuremens. Gradually, becomes unimodal, cenered a he Asympoic Dispersion Rae, and he widh of his unique mode is reduced as increases. Le be he minimum value of for which is unimodal. Also, le be he range of he unique mode, i.e., he bandwidh inerval ha includes all he significan values in he bell around 8. The heurisic rule wih which he capaciy esimae is seleced is ha he capaciy mode is he minimum mode in which is higher han, i.e., (12) The heurisic is based on he following reasoning. Firs, when is sufficienly large for o be unimodal, almos all packe rains have encounered dispersion due o cross raffic, and so. Second, because is he minimum packe rain lengh ha generaes a unimodal, he range of he unique mode is sill sufficienly wide o cover all he local modes in he SCDR of beween and. This heurisic resuled from long experimenaion, and is evaluaed laer in his secion. In pahrae, Phase II consiss of a =4 packe rain experimens wih =15B for each lengh. If he resuling disribuion is no unimodal, is increased by wo, and he process repeas. Noe ha is significanly lower han, because in Phase II we only check wheher he disribuion is unimodal, insead of esimaing he local modes. When he lengh is reached, he upper hreshold is measured, and he capaciy esimae is deermined from Equaion 12. To illusrae he use of Equaion 12, Figures 1 and 11 show he packe pair disribuion and he unimodal disribuion for a simulaion and a real nework experimen, respecively. The disribuions in Figure 1 resul from simulaing he pah wih onehop persisen cross raffic. The sequence of modes in, wih The exac algorihm for he esimaion of involves heurisics o separae measuremens in he bell from measuremen noise. Capaciy esimae (Mbps) $=1Mbps Acual capaciy (Mbps) (a) One bin widh Capaciy esimae (Mbps) w=1mbps if C<4Mbps, w=2mbps oherwise $ =1Mbps $ =2Mbps Acual capaciy (Mbps) (b) Two bin widhs Fig. 12. Evaluaion of he heurisic of Equaion 12. =1 Mbps, is = 9,14,17,23,26,29,33,4,44,56,75,9. The minimum ha resuls in a unimodal disribuion is =8, and he upper hreshold of he mode is =37 Mbps. Consequenly, from Equaion 12, he esimaed capaciy is 4 Mbps, which is he correc value. The disribuions in Figure 11 resul from experimens in a nework pah from San Diego o Zurich (see VIII). The sequence of modes in is = 9,11,13,15.5, 19.5, 27, 32, 43, =12, and =24 Mbps. So, he esimaed capaciy is 27 Mbps, which is he correc value (see VIII). In order o evaluae he accuracy of he presened heurisic, we simulaed he pahrae mehodology in a number of differen pah and raffic configuraions, comparing he acual capaciy wih he capaciy esimae. The simulaed cases cover a range of values for (3 o 15), (5 o 125 Mbps), (5 o 75 Mbps), (.1 o.9), (consan or uniformly disribued in [4,15]B), and one-hop or pah persisen cross raffic. Figure 12-a shows he resuls when =1 Mbps. The mehodology is quie accurae, leading o, as long as he pah capaciy is lower han abou 4 Mbps. For higher pah capaciies, is lower han, in some cases by almos a facor of wo. I urns ou ha hese erroneously low esimaes are usually he firs local mode in he SCDR of ha is lower han he, and so he assumpion ha he unique mode in includes all he SCDR modes beween and is no always rue. This mainly occurs in pahs wih heavy load (more han 8%) in he narrow or pre-narrow links. Figure 12-b shows he resuls of he same simulaions, bu

9 when he specified bin widh is 1 Mbps for lower capaciy pahs ( 4 Mbps), and 2 Mbps for higher capaciy pahs ( 4 Mbps). Noe ha he esimaes are more accurae for he high capaciy pahs wih he larger bin widh. The few esimaes ha are sill oo low are correced wih an even larger bin widh ( =3 Mbps), a he cos of a wider resoluion. The bin widh has his effec because, as increases, he weak modes in he SCDR of which are close o he, and which cause he underesimaions, end o merge wih he capaciy mode. If is oo large, on he oher hand, he can merge wih he SCDR modes and he final esimae will be one of he PNs (overesimaion). In oher words, he resoluion has o be chosen based on a rough esimae of he pah s capaciy. More work is needed for an adapive selecion of. There are several feaures and issues abou pahrae, ha we only briefly menion here. Before Phase I, pahrae generaes packe rains of gradually increasing lengh o deec mulichannel links; if here is a seep bandwidh decrease when increases from o, we infer ha he narrow link consiss of channels. This iniial se of packe rains is also used o deermine he maximum packe rain lengh ha he pah can ransfer wihou causing buffer overflows a he rouers or he sender/receiver OS. Noe ha we avoid packe rains ha are oo long and cause buffer overflows in order o no affec he cross raffic, which normally responds o losses using he congesion avoidance mechanisms of TCP. pahrae uses UDP for he probing packes. Addiionally, i esablishes a TCP connecion, referred o as he conrol channel, which acknowledges every correcly received packe pair/rain, and is used for exchange of conrol informaion beween he wo end-poins. Any packe pairs or rains ha encounered losses are ignored from he measuremen process. As a simple form of congesion avoidance, pahrae abors he measuremen process when i deecs significan losses in he pah. The ime inerval beween successive packe pairs or rains is se o 5 msec; so, when pahrae sends packe rains wih =15B and =1, he average rae of probing raffic is 24 kbps. Currenly, he receiving par of pahrae uses user-level imesamping. This ofen causes bandwidh esimaes ha are higher han he bandwidh of he nework inerface a he receiving hos, because wo closely received packes can be queued a he kernel and hen delivered o he applicaion wih a small spacing ha is indicaive of he kernel-user bandwidh. These esimaes do no normally cause errors, since hey produce very large modes in which are unlikely o be seleced as. If he receiver s nework inerface bandwidh (ha is ) is known, we know ha he measuremens ha are higher han have been caused a he receiving hos end, and so we can clamp hem o. VIII. CAPACITY MEASUREMENTS In his secion, we presen a few capaciy measuremens using pahrae in a mesh of five hoss in US and Europe. The hos names and heir geographical locaion are shown in Table I. The pahs beween hese hoss cross several academic and commercial neworks, such as he vbns, Abilene, Dane, CalRen2, UUne, Cable & Wireless, Swich, and he local access neworks a each sie. zamboni is conneced o a 1 Mbps Eherne inerface, while he res of he hoss are conneced o Fas Eherne TABLE I MEASUREMENT HOSTS AND THEIR LOCATIONS. Hos sun jhana zamboni ren drwho Locaion Univ.Wisconsin, Madison WI CAIDA, San Diego CA U, Pisburgh PA Univ.Delaware, Newark DE ETH, Zurich-Swizerland TABLE II CAPACITY MEASUREMENTS WITH pahrae sun jhana zamboni ren drwho sun jhana zamboni ren drwho inerfaces (1 Mbps). The pahrae capaciy measuremens are shown in Table II. The measuremens in he row of a hos refer o he capaciies of he pahs ha originae from ha hos. For insance, he capaciy esimae for he pah from sun o drwho is Mbps. The bin widh selecion was an educaed guess, in he sense ha was se o 1 Mbps when he bandwidh measuremens were mosly below 5 Mbps, and o 4 Mbps when he measuremens were higher. Specifically, all pahs ha involve zamboni or drwho were measured wih =1 Mbps, while he res of he pahs were measured wih =4 Mbps. The measuremens were performed during weekdays and dayime a boh ends of he pah. The measuremens ha involve drwho were performed during June 2; a ha ime zamboni was sill conneced o a Fas Eherne. The res of he measuremens were performed during December 2, while preparing he final version of his paper. We verified some of hese measuremens, by conacing he nework managers of he involved sies. Specifically, in June 2 drwho was sill conneced o US hrough a ransalanic 32 Mbps ATM UUne link operaed by Swich 9. Due o he involved AAL5 and ATM header overheads, he IP-layer capaciy of he link is abou 28.3 Mbps for 15B packes, and abou 27.4 Mbps for 5B packes. As shown in Table II, he pahrae measuremens are quie close o his value, in he range Mbps. pahrae accuraely measures he 1 Mbps capaciy of he pahs ha are conneced o zamboni, wih he excepion of he capaciy in he pah from zamboni o ren which is slighly overesimaed ( =13-14 Mbps). Unforunaely, we were unable o verify he res of he capaciy measuremens due o insufficien informaion abou he involved neworks. The pahs beween sun, jhana, and ren hough, lead o resuls in he range 9-11 Mbps, implying ha he corresponding pahs may be limied by he Fas Eherne nework inerfaces (1 Mbps) of he measuremen hoss. This is likely o be he case, since he corresponding In fac, he link consised of wo 32 Mbps ATM virual pahs, bu a cerain microflow could only use one of he wo VPs. Laer in he summer of 2 ha link was upgraded o a POS OC-3.

10 insiuions (Universiy of Wisconsin, CAIDA, and Universiy of Delaware) are conneced o he heir nework providers hrough ATM or POS OC-3 links (abou Mbps). IX. CONCLUSIONS This paper sudied he dispersion of packe pairs and packe rains, focusing on he effecs of he cross raffic. As an applicaion of his sudy we developed a capaciy esimaion mehodology. The insigh gained, hough, can probably be also applied o congesion conrol mechanisms, server selecion algorihms, as well as qualiy of service monioring. A firs ask for fuure work is o improve he capaciy esimaion mehodology, and specifically he heurisic specified by Equaion 12, so ha he underesimaion errors shown in Figure 12 are avoided. This is also relaed o he selecion of he bin widh or resoluion. We also examine he sensiiviy of he resuls o he cross raffic load, running pahrae in differen imes of day. Finally, he ADR meric, which is relaed o he uilizaion of all links in he pah, may be a useful meric for monioring he qualiy of service ha he pah offers, and i would be ineresing o examine is dynamic variaions over boh shor and long imescales. APPENDIX The definiion of he pah capaciy in Equaion 1 is sraighforward. There are wo poins, hough, ha one has o be careful wih. Firs, he use of addiional headers in layer-2 echnologies can resul in an IP-layer capaciy ha is lower han he adverized nominal bandwidh of a link. Second, in cerain mulichannel links he rouer performs hashing based on he desinaion address of he packe, or based on he 5-uple header fields, in order o deermine he specific sub-link ha he packe will be forwarded o. In ha case, all he probing packes will be sen o he same sub-link, and so he measuremen ool will measure he capaciy of only ha sub-link. This is also, however, he maximum hroughpu ha a cerain IP microflow would be able o ge in he pah. Regarding he available bandwidh, defined in Equaion 2, we make he following remarks. is he maximum available hroughpu for a congesion responsive flow, i.e., a flow ha does no aemp o seal bandwidh from he cross raffic. Obviously, a congesion unresponsive flow can ge a higher hroughpu han if i aemps o saurae he pah, causing losses in he TCP par of he cross raffic. Someimes he available bandwidh is defined as he susained hroughpu of a long TCP connecion in he pah [22]. The TCP hroughpu, however, depends on he version and he specific implemenaion of he TCP congesion avoidance mechanisms [23]. Also, he hroughpu of a long TCP connecion ( elephan ) is no he same wih he aggregae hroughpu of a large number of shor TCP connecions ( mice ) in he same pah and load condiions. For hese reasons, we believe ha i is more appropriae o define he available bandwidh in erms of he load (uilizaion) of he pah links, as in Equaion 2. The available bandwidh, hen, has o be inerpreed as he maximum hroughpu ha a congesion responsive flow would ge, if he flow was able o saurae he igh link in he pah, bu wihou causing any reducion in he cross raffic load. ACKNOWLEDGMENTS We are graeful o he following people for providing us wih compuer accouns a heir sies: Tobias Oeiker (ETH), Andy Myers and Hui Zhang (U), Adarsh Sehi and Paul Amer (Univ-Delaware), Hans-Werner Braun (NLANR), David Meyer (Univ-Oregon). We are also graeful o Jambi Ganbar from he vbns nework engineering group for experimening wih pahrae in he vbns, o Simon Leinen from he Swich nework in Swizerland for crucial informaion abou heir ransalanic link, o Daniel Grim for informaion regarding he Universiy of Delaware Inerne access, and o Allen Downey, Kevin Lai, Bob Melander, and he Infocom reviewers for providing useful commens on his paper. Finally, we are graeful o Kimberly Claffy, Tracie Monk, Evi Nemeh, and all he CAIDA elves for heir useful commens and help in compleing his work. REFERENCES [1] R.L. Carer and M.E.Crovella, Measuring Boleneck Link Speed in Packe-Swiched Neworks, Performance Evaluaion, vol. 27,28, pp , [2] V. Paxson, End-o-End Inerne Packe Dynamics, IEEE/A Transacion on Neworking, vol. 7, no. 3, pp , June [3] K. Lai and M.Baker, Measuring Bandwidh, in Proceedings IEEE IN- FOCOM, Apr [4] R. Caceres, N.Duffield, and A.Feldmann, Measuremen and Analysis of IP Nework Usage and Behavior, IEEE Communicaions Magazine, pp , May 2. [5] T. Oeiker, MRTG: Muli Rouer Traffic Grapher, hp://eesaff.ehz.ch/ oeiker/webools/mrg/mrg.hml. [6] V. Jacobson, Congesion Avoidance and Conrol, in Proceedings A SIGCOMM, Sep. 1988, pp [7] S. Keshav, A Conrol-Theoreic Approach o Flow Conrol, in Proceedings A SIGCOMM, Sep [8] J. C. Bolo, Characerizing End-o-End Packe Delay and Loss in he Inerne, in Proceedings A SIGCOMM, 1993, pp [9] UUNET, UUNET Technologies, hp:// Nov. 2. [1] V. Paxson, Measuremens and Analysis of End-o-End Inerne Dynamics, Ph.D. hesis, Universiy of California, Berkeley, Apr [11] J.C. Hoe, Improving he Sar-up Behavior of a Congesion Conrol Scheme for TCP, in Proceedings A SIGCOMM, Sep [12] L. S. Brakmo and L.L.Peerson, TCP Vegas: End o End Congesion Avoidance on a Global Inerne, IEEE Journal on Seleced Areas of Communicaions, vol. 13, no. 8, Oc [13] V. Jacobson, pahchar: A Tool o Infer Characerisics of Inerne Pahs, fp://fp.ee.lbl.gov/pahchar/, Apr [14] A.B. Downey, clink: a Tool for Esimaing Inerne Link Characerisics, hp://rocky.wellesley.edu/downey/clink/, June [15] B. A. Mah, pchar: a Tool for Measuring Inerne Pah Characerisics, hp:// bmah/sofware/pchar/, June 2. [16] A.B. Downey, Using Pahchar o Esimae Inerne Link Characerisics, in A SIGCOMM, Sep [17] K. Lai and M.Baker, Measuring Link Bandwidhs Using a Deerminisic Model of Packe Delay, in Proceedings A SIGCOMM, Sep. 2. [18] Nework Simulaor (ns), version 2, hp://wwwmash.cs.berkeley.edu/ns/. [19] M. S. Taqqu, W.Willinger, and R.Sherman, Proof of a Fundamenal Resul in Self-Similar Traffic Modeling, A Compuer Communicaions Review, pp. 5 23, Apr [2] K. Thompson, G. J. Miller, and R. Wilder, Wide-Area Inerne Traffic Paerns and Characerisics, IEEE Nework, pp. 1 23, Nov [21] S. McCreary and K. C. Claffy, Trends in Wide Area IP Traffic Paerns, Tech. Rep., CAIDA, Feb. 2. [22] M. Mahis, TReno Bulk Transfer Capaciy, Feb. 1999, draf-ief-ippmreno-bc-3.x. [23] V. Paxson, Auomaed Packe Trace Analysis of TCP Implemenaions, in Proceedings SIGCOMM Symposium, Sep