Real-Tme Measurement of End-to-End Avalable Bandwdth usng Kalman Flterng Svante Ekeln, Martn Nlsson, Erk Hartkanen, Andreas Johnsson, Jan-Erk Mångs, Bob Melander, and Mats Börkman Ercsson Research, Stockholm, Sweden Swedsh Insttute of Computer Scence (SICS), Stockholm, Sweden Department of Scence and Technology, Lnköpng Unversty, Norrköpng, Sweden Department of Computer Scence and Electroncs, Mälardalen Unversty, Västerås, Sweden Abstract Ths paper presents a new method, BART (Bandwdth Avalable n Real-Tme), for estmatng the end-toend avalable bandwdth over a network path. It estmates bandwdth quas-contnuously, n real-tme. The method has also been mplemented as a tool. It reles on self-nduced congeston, and repeatedly samples the avalable bandwdth of the network path wth sequences of probe packet pars, sent at randomzed rates. BART requres lttle computaton n each teraton, s lghtweght wth respect to memory requrements, and adds only a small amount of probe traffc. The BART method uses Kalman flterng, whch enables real-tme estmaton (a.k.a. trackng). It mantans a current estmate, whch s ncrementally mproved wth each new measurement of the nter-packet tme separatons n a sequence of probe packet pars. The measurement model has a strong non-lnearty, and would not at frst sght be consdered sutable for Kalman flterng, but we show how ths non-lnearty can be handled. BART may be tuned accordng to the specfc needs of the measurement applcaton, such as aglty vs. stablty of the estmate. We have tested an mplementaton of BART n a physcal test network wth carefully controlled cross traffc, wth good accuracy and agreement. Test measurements have also been performed over the Internet. We compare the performance of BART wth that of pathchrp, a state-of-the-art tool for measurng end-to-end avalable bandwdth n real-tme. Index Terms actve measurement, avalable bandwdth, endto-end, Kalman flter, probng, real-tme IEEE. Personal use of ths materal s permtted. Permsson from IEEE must be obtaned for all other uses, n any current or future meda, ncludng reprntng/republshng ths materal for advertsng or promotonal purposes, creatng new collectve works, for resale or redstrbuton to servers or lsts, or reuse of any copyrghted component of ths work n other works. The orgnal IEEE Xplore lnk s: http://eeexplore.eee.org/xpl/logn.sp?tp=&arnumber=&url=http% A%F%Feeexplore.eee.org%Fxpls%Fabs_all.sp%Farnumber%D Ths work s sponsored by the European Communty s FP Informaton Socety Technologes programme under contract IST-, EVERGROW (www.evergrow.org), by the VINNOVA-funded proect EvaluNet (www.scs.se/cna/evalunet), and by the Swedsh Research Councl. A. Overvew I. INTRODUCTION The capablty of measurng avalable bandwdth s useful n several contexts, ncludng servce level agreement verfcaton, network montorng and server selecton. Measurement of avalable bandwdth n real-tme, wth persample update of the bandwdth estmate (as n BART), opens up for adaptaton based on avalable bandwdth drectly (rather than frst-order measures such as loss or delay) n congeston control and streamng of audo and vdeo. Passve montorng of avalable bandwdth of an end-to-end network path would n prncple be possble, f we could access all the network nodes n the path. However, n practce ths s typcally not possble, and estmaton of avalable endto-end bandwdth s only feasble by actve probng of the network path. By nectng probe traffc nto the network, and then analyzng the observed effects of cross traffc on the probes, we can estmate the avalable bandwdth. Ths knd of actve measurement only requres access to the sender and recever hosts. The man contrbuton of ths work s the technque for applyng a Kalman flter for real-tme estmaton of avalable bandwdth. The measurement model used n the flterng algorthm has a strong non-lnearty, and would not at frst sght be consdered sutable for Kalman flterng, but we show how ths non-lnearty can be handled. We ntroduce the convenent measure nter-packet separaton stran of consecutve probe packets. When there s no congeston, ths stran s zero on average. When the total load starts to become larger than the path s bottleneck capacty, the stran becomes proportonal to the overload. The Kalman flter technque, along wth a sutable probng scheme, comprse a measurement method, BART (Bandwdth Avalable n Real-Tme). We present results obtaned by a BART mplementaton from measurements n a laboratory network as well as over the Internet, demonstratng the vablty of the method.
Some of the features of BART are: t produces an estmate quckly; stablty can be traded for aglty; no communcaton s requred from the recever of the probe packets to the sender; tunng s largely automatc,.e. there are few parameters that need manual adustment. Nevertheless BART may be tuned accordng to the specfc needs of the measurement applcaton, such as aglty vs. stablty of the estmate. In our mplementaton, only about a dozen floatngpont multplcatons and dvsons are needed for the flterng computatons. The memory requrements are mnmal, as only the prevous estmate and the new measurement are needed to calculate the new estmate. B. Related Work A number of papers have appeared n recent years n the feld of avalable bandwdth measurements. Jan and Dovrols have developed Pathload [, ], and Melander et al. TOPP []. Both use probe packet trans sent at varous rates, and attempt to estmate the pont of congeston,.e. the probe rate where the delay starts ncreasng. TOPP fts data to a straght lne n order to arrve at an estmate, whereas Pathload looks for an ncreasng trend n the one-way delay, and performs a bnary search to successvely fnd shorter and shorter ntervals contanng the avalable bandwdth estmate. Both these methods requre a substantal tme for measurement and analyss before producng an estmate, and are not sutable for real-tme trackng of avalable bandwdth. Rbero et al. have developed pathchrp [] for measurng avalable bandwdth. pathchrp uses probe trans wth nternally varyng nter-packet separatons, n order to scan a range of probe rates wth each tran. By analyzng the nternal delay sgnature of each such chrp, an estmate of avalable bandwdth s produced. Estmates are smoothed by averagng over a sldng tme wndow. A Kalman flter has prevously been appled n the context of some other types of traffc measurements, but as far as we know, not to the estmaton of avalable bandwdth. In an early paper [], Keshav dscussed usng a Kalman flter for the estmaton of bottleneck servce rate for endpont flow control purposes, and concluded that ths would not be practcal. However, hs analyss rested on the assumpton that queung servce n network nodes s based on stateful flow-based round-robn nstead of stateless frst-come frst-served (FCFS), whereas typcally the opposte holds n today s Internet. Jacobsson et al. used a Kalman flter for RTT estmaton from the perspectve of TCP []. Km and Noble dd ths for estmatng another type of bandwdth over a path [, ]. However, ther defnton of bandwdth dffers from ours and from that of [,,,, ]. Smlarly to the TOPP method, we use the fact that the stran s approxmately proportonal to the overload. Ths means that there s more nformaton to be ganed from each data pont than merely whether or not the value ndcates overload. In TOPP, a measurement phase where trans of varous rates are sent s followed by an analyss phase, where lnear regresson s performed n order to estmate the parameters of the straght lne. A shortcomng of ths method s that t requres a long tme before arrvng at a measurement value, a value whch then s already old when t s obtaned. Our orgnal contrbuton s a technque for applyng Kalman flterng, mantanng and contnuously updatng the avalable bandwdth estmate for each new samplng. Ths leads to a smple, fast, and easy-to-mplement method. Ths paper reports on development and valdaton of the method frst descrbed n []. C. Paper Organzaton In the next secton of ths paper, we shall descrbe the problem of measurng avalable bandwdth. We ntroduce termnology and defne the basc concepts. In the thrd secton, we frst explan the network model, and then how BART can be appled n order to measure avalable bandwdth n realtme. The fourth secton descrbes valdaton of the method, ncludng measurement results from the Internet as well as a laboratory network, both for synthetc traffc and for a recorded traffc trace. The ffth secton dscusses the results, and concludes the paper. II. MEASURING AVAILABLE BANDWIDTH Whereas several other traffc-dependent network path propertes can be measured drectly, such as packet delay and packet loss probablty, the measurement of the tme-varyng avalable bandwdth over a network path poses more of a challenge. If we are not fortunate enough to have ready access to relevant statstcs from all the network nodes along the path (we almost never are, as the path typcally crosses admnstratve network doman borders, and transt network provders are not compelled to share ths nformaton), there s no other way to measure avalable bandwdth than by actve probng. A. Defnton of Avalable Bandwdth In the lterature, the term avalable bandwdth has been used n dfferent ways. To avod any msnterpretatons, we want to make clear what we denote by avalable bandwdth. For more detals, see the revew artcle []. Each lnk n a network path has a certan capacty, or nomnal bandwdth, C, determned by the network nterfaces n the nodes on each end of the lnk. Ths s smply the hghest possble bt rate over the lnk. The nomnal bandwdth s quasconstant,.e. t typcally does not vary. What s varyng on short tme-scale s the lnk load, or cross traffc, X = X ( t, ). Here, s the tme resoluton at whch we are nterested n descrbng traffc fluctuatons. So, the cross traffc rate s defned by where [ t, t] A X = A [ t, t] () s the number of cross traffc bts
transmtted over lnk durng a tme nterval of length. The tme-varyng avalable bandwdth B = B ( t, ) of lnk s defned as: B = C X () One of the lnks along the path has the smallest value of avalable bandwdth. Ths lnk s called the bottleneck lnk (or tght lnk usng termnology from []), and t determnes the avalable bandwdth of the path. In other words, for a network path from sender to recever, the avalable bandwdth s defned as the mnmum of the avalable lnk bandwdths along the path: B = mn( C X ) () Ths s n lne wth what s denoted by avalable bandwdth n [,,,, ]. An nterpretaton of the avalable bandwdth B s: the smallest ncrease n traffc load from sender to recever at tme t whch causes congeston at some hop on the network path. Ths nterpretaton s closely related to the method of measurng the avalable bandwdth by sendng probe traffc at varous rates, and determnng the threshold rate when the probe traffc n conuncton wth the cross traffc transently experences congeston,.e., measurement by self-nduced congeston. In fact, one mght argue that the rate thus measured can be seen as the defnton of avalable bandwdth. III. APPLYING KALMAN FILTERING TO A PIECEWISE LINEAR NETWORK MODEL A. The Network Model We model a network path from sender to recever as a successon of concatenated hops. A hop conssts of an nput queue and a transmsson lnk. We consder FCFS hops,.e. frst-n frst-out (FIFO) queues. Each hop has a constant capacty C and carres a tme-varyng cross traffc X. We wll be concerned wth the descrpton of nteractons between the regular cross traffc X and the nected probe traffc u, n order to arrve at a formalsm whch allows drawng conclusons about the avalable bandwdth by analyzng the effect that cross traffc has on the probe traffc. For a frst rough dea, let us consder a flud model of traffc flows, and temporarly dsregard the dscrete nature of packet traffc. Frst, consder a sngle hop wth lnk capacty C and cross traffc X. The avalable lnk bandwdth s then C X. Now, assume that the lnk s also subect to probe traffc demand at a rate u. If u C X there s no congeston, and the receved probe traffc rate r extng the lnk s the same as the demanded probe rate u. However, n the case u > C X we have an overload stuaton. In lne wth our model of the hop as havng a FIFO queue, we assume that there s no prortzaton n the router, so that all traffc s served on a FCFS bass. Ths means that the cross traffc and the probe traffc each receve ther proportonate share of the lnk capacty C. So, we have for the receved probe traffc rate: u r = C () u + X Rearrangng, we see that the rato of demanded to receved probe traffc rate s a frst-order polynomal n u: so u X = u + () r C C u ( u C X ) = X () r u + ( u > C X ) C C Thus, by varyng u and tryng to dentfy the pont where u / r starts to devate from unty, we have a way of estmatng the avalable bandwdth of the lnk. Ths s essentally the network model proposed for the TOPP method []. By repeatng ths argument for several concatenated lnks, t was shown n [] that the avalable bandwdth of the entre path from sender to recever can be estmated by studyng the rato of offered probe rate at the sender to receved probe rate at the recever, and determnng the pont where ths rato starts devatng from unty. Ths value of u s the estmate of the avalable bandwdth B of the entre path. Note that when one estmates the parameters of the straght lne, one smultaneously also gets an estmate of the bottleneck lnk capacty C, as ths s the nverse of the slope. Fg.. The asymptotc relaton between avalable bandwdth, probe ntensty and expectaton value of nter-packet stran. Now, consderng the dscrete (packet-orented) nature of network traffc, t s necessary to adapt the above reasonng to take nto account the fact that the traffc s not a contnuous flow, but s composed of dscrete packets. We use the generc multple-hop model presented n [], and the notaton theren.
Consder a sequence of packet pars, where the two packets n par arrve at a hop at tmes τ and τ, and depart (.e. * * arrve at the next hop) at tmes τ and τ. We are not nterested n the absolute arrval tmes, but rather the nterarrval tmes of the packets n par : t * t = τ τ () * * τ = τ () In [], the focus s on the dfference between these * quanttes, the delay varaton δ = t t. Here, we nstead ntroduce the dmensonless quantty nter-packet stran ε, gven by: t * + ε = () t The stran can be expressed n terms of the delay varaton as: δ ε = () t Ths stran provdes a drect analogy n the dscrete descrpton to the flud model rato of demanded to receved traffc rate dscussed above. Wth packet sze s, at the tme resoluton of nter-packet tme separaton level we have for the demanded traffc rate u = s / t and for the receved traffc rate r = s t. We mmedately get: / * u r * * s / t t = = = + ε s / t t () The devaton of u / r from unty s equvalent to the devaton of the stran ε from zero, see Fg.. No stran means no congeston. It should be noted that when gong beyond the flud model and takng packet-level nteractons n router queues nto account, the system measurement model curve s expected to devate somewhat from the form n Fg. []. However, we stll expect the asymptotc behavor to be the slopng straght lne, whch s the man pont of nterest here. The devaton from ths straght lne s taken care of by a nose term n our approach. In our model, observng the nter-packet stran over the path gves a handle on estmatng the avalable bandwdth for the tght lnk, and consequently the avalable bandwdth of the entre path. We assume that for the average nter-packet stran of consecutve probe packet pars n a sequence of a gven rate u, any systematc devaton from zero s domnated by the effect of cross traffc nteracton n the FIFO queue at the bottleneck lnk. Input data for our BART estmaton algorthm s generated by sendng sequences of N probe packet pars wth the same nternal traffc rate u. Ths way, we reduce varance and mprove the statstcal precson n our measurements. For each new samplng,.e. new sequence, the rate u s randomzed. Each probe packet s tme-stamped on sendng and on recepton. The recever calculates the nter-packet stran ε for each probe par =,..., N. When the whole sequence has been receved, the average nter-packet stran ε and ts varance R are computed from ε ( =,..., N). Now, how do we compute an estmate of the avalable bandwdth B usng successve measurements of ε and R? Whle TOPP uses lnear regresson and fts a curve to a large number of measurement ponts, BART keeps a current estmate, and updates t every tme a new probe par sequence s receved, applyng a Kalman flter wth a specal twst. The BART flter s descrbed n the followng subsecton. B. Estmaton by Flterng Suppose that we have a system, a black box, whch s descrbed by a state vector x. Suppose also that we can affect the system wth an nput u. We would lke to know x, but the only way we can observe the system s through measurements z of some aspect of the system. In our case, the black box s the network path, u s the probe traffc rate, and z s the nter-packet stran. The system s also affected by nose w, and the measurements by nose v. A key dea of BART s the choce of the state vector x, enablng the Kalman flter estmaton of x, and consequently of the avalable bandwdth of the network path. All these enttes, ncludng the state vector, evolve over tme. Descrbng the evoluton of the system by and the measurement by x k+ = g( xk, uk, wk ) () z k = f ( xk, uk, vk ) () where f and g are known functons, a flter s a procedure whch takes u k, z k and an estmate xˆ k as nputs, and computes a new estmate x ˆk+. Provded that f ( x, u, v) and g ( x, u, w) are lnear n x, v and w, and that the nose v and w are Gaussan wth zero mean, there s an optmal flter known as a Kalman flter. For a very bref ntroducton, see Appendx. A more thorough ntroducton to ths flter method s gven n [], whch also contans dervatons of the general Kalman flter equatons we use to calculate our estmates. Experence has shown that Kalman flters often work very well, even for non-gaussan nose and slghtly non-lnear models. There are also versons of the Kalman flter that handle non-lnear systems by local lnearzaton, although the flter s not optmal for such stuatons. Unfortunately, they wll not work when the dervatve s dscontnuous, whch s the case we have. We wll return to ths problem shortly. Wth the state vector α x = () β
where α and β are the parameters of the slopng straght lne n the measurement model,.e. ε = α u + β, our system s descrbed by g ( x, u, w) = x + w () and our measurements can be descrbed by (see Fg. ): ε ( u B) = f ( x, u, v) = v + α u + β ( u > B) () In ths way we have provded a relaton between the avalable bandwdth B and the state varables α and β. Estmatng the state wll allow an estmaton of B, whch s our obectve. When we have estmates αˆ and ˆβ, we get the estmate for the avalable bandwdth: ˆ β B ˆ = () ˆ α However, the functon f ( x, u, v) above s only pecewse lnear. The sharp corner at u = B prevents a drect applcaton of a Kalman flter. In order to crcumvent ths, we apply the flter only to measurements for whch u > ˆB, where Bˆ s the current estmate of the avalable bandwdth, whch s known. We argue ths method wll converge f f s a convex functon of u, as s the case n our method. for whch u > Bˆk, and the Kalman flter wll attempt to fnd a new lne L. Due to ε (u) beng convex, ths lne wll ntersect the u-axs at a pont B ˆ k +, where Bˆ ˆ k < Bk+ < B, showng that the new value n the mean wll be an mproved approxmaton. An analogous argument holds for an over-estmate B ˆ k > B. The BART algorthm proceeds as follows:. The recever ntalzes the state vector estmate ˆx, the avalable bandwdth estmate ˆB, and the error covarance matrx P for ˆx.. The sender generates a sequence of probe packet pars wth probe traffc ntensty u, drawn from a probablty dstrbuton. The sender passes the current value of the ntensty n the probe packets.. For each receved probe sequence, the recever recovers u. If u < ˆB, no updatng s performed and the cycle repeats from step. If u > ˆB, the recever computes the average stran ε and ts varance R.. The recever nputs these values to the Kalman flter, and also provdes the flter wth an estmate Q of the process nose covarance matrx (cf. below). The flter then updates the estmates of the state vector xˆ and the matrx P.. The recever uses the updated xˆ to compute a new u-axs crossng, producng a new avalable bandwdth estmate ˆB. The cycle repeats from step. One consderaton when applyng Kalman flters s fndng a sutable Q. Ths matrx descrbes the ntrnsc fluctuatons n the system, and s n our applcaton related to the volatlty of the cross traffc. Ths can not be assumed to be known a pror, but can be treated as an adustable parameter. Ths enables tunng of the BART estmaton characterstcs. Larger values result n low stablty but hgh aglty of the flter, snce the Kalman flter equatons [] gve greater weght to the recent measurement relatve to the prevous estmate. Conversely, smaller values result n hgher stablty n presence of spurous measurement errors, but slower step response []. By defnton, Q s a symmetrc matrx. In our case, t s a -matrx, so t contans ndependent scalar parameters. In ths mplementaton we have taken the smple form: λ Q = λi = () λ Fg.. Convergence of the BART method. The convergence can ntutvely be understood as follows: Gven only ponts for whch u > B, the Kalman flter attempts to fnd a straght-lne approxmaton L to the convex curve ε (u), consder Fg.. The ntersecton wth the u-axs approxmates the avalable bandwdth B. We do not know B, but suppose for now that we have an underestmate B ˆ k < B. The next tme the Kalman flter s appled, we use only ponts As s seen n the next secton, reasonable performance can be produced wthout explotng the full potental of the formalsm. For all traffc cases n the present paper, the BART estmaton results have been produced usng ths smple form for Q, wth λ =. There s plenty of freedom to relax ths constrant, allowng tunng for desred estmaton performance. IV. VALIDATION OF THE METHOD In order to evaluate the method, the performance of a BART mplementaton has been measured n a controlled testbed
envronment as well as over the Internet. Numerous smulatons have also been performed, whch corroborate the evdence from the measurements, but these smulatons are not descrbed n the present paper. We compare the performance of BART wth that of pathchrp, a state-of-the-art tool for measurng end-to-end avalable bandwdth n real-tme []. In ths mplementaton of BART, the sequence of probe packet pars was structured as a tran,.e. the second packet n one par also consttutes the frst packet n the next par. Ths reduces the traffc overhead, at the expense of statstcal accuracy. We consder t concevable that better performance can be obtaned f relaxng ths constrant, although at a hgher overhead. Ths wll be reported on n a forthcomng paper. The dscusson of the measurement results presented n ths secton leads to the concluson that BART yelds estmates wth equal or better accuracy than pathchrp, wth lower overhead of probe traffc nected nto the network. Part A of ths secton descrbes the experment setup, ncludng the testbed and the dfferent traffc cases. Thereafter, we study the mpact that probe packet sze and probe tran length have on BART performance. The remanng parts of the secton present results for BART and pathchrp n several experments, both n the testbed and measurements over the Internet. Observe that throughout ths secton, measured data rates, avalable bandwdth estmates and packet szes refer to the network layer. A. Measurement Testbed Setup A schematc pcture of the testbed s shown n Fg.. It contans two routers, the nterconnecton of whch s the bottleneck lnk n the system. The testbed also ncludes senders and recevers for each probng tool and for the cross traffc. In order to montor the true network traffc, a machne runnng tcpdump (a tool for loggng network traffc) has been connected to hubs. The hosts run Lnux, and are equpped wth processors equal to or better than Pentum III/ MHz. All lnks n the testbed have a capacty of Mbt/s, except the bottleneck lnk between router and router, whch s nomnally confgured to Mbt/s. However, the actually observed capacty was. Mbt/s at the IP layer. Hence, ths value has been used as the true bottleneck capacty of the testbed. In the testbed measurements, both synthetc cross traffc, generated by IPTrafGen (an Ercsson propretary traffc generator), and a traffc trace from MAWI have been used. In the Internet measurements, addtonal cross traffc that orgnates from the same traffc trace has been appled n order to generate extra load on the Internet path. Unless otherwse specfed, BART has used probe packets of bytes and probe trans of probe packets n all the experments. As we see n secton IV B, the accuracy of BART ncreases wth ncreasng probe packet sze and wth ncreasng probe tran length. The probe packet sze s lmted by the Ethernet maxmum transmsson unt. We chose to lmt the tran length to probe packets, as ths gves reasonable accuracy whle not causng excessve probe traffc overhead. The traffc ntensty for each probe tran was randomly chosen from a unform dstrbuton, over the nterval from Mbt/s to Mbt/s. BART was confgured to produce an estmate every second,.e. the nter-departure tme between two consecutve probe trans s one second. Ths mples that BART nects probe traffc wth an average ntensty of. Mbt/s at the IP layer, whch s approxmately.% of the bottleneck lnk capacty n the testbed. The default parameters have been used for pathchrp. Our experments ndcate that pathchrp s producng. estmates per second wth a probe traffc ntensty around. Mbt/s,.e. n the regon of.% of the bottleneck lnk capacty n the testbed. For the testbed measurements usng synthetc cross traffc, the IPTrafGen generator has been confgured for measurements wth hgh and low aggregaton of cross traffc. The two scenaros can approxmately be seen as havng or users smultaneously utlzng a network lnk. New users arrve accordng to a Posson process and reman actve durng a perod obtaned from a Pareto dstrbuton (shape parameter α =., mean =. second). When the users are actve, they transmt packets wth szes accordng to Table I. These packet TABLE I DISTRIBUTION OF SYNTHETIC CROSS TRAFFIC PACKET SIZES Packet Sze: Percentage: B. % B. % B. % B. % B. % szes refer to IP datagrams n bytes, and the dstrbuton roughly corresponds to observatons from Sprnt. Measurements wth synthetc cross traffc have been made usng three dfferent dstrbutons for the nter-arrval tme of the cross traffc packets. These are exponental nter-arrvals and Pareto nter-arrvals wth shape parameter α =. and α =., respectvely. For all nter-arrval scenaros, the dstrbuton parameters have been chosen such that actve users approxmately get the same transmsson rate, and the Fg.. A schematc vew of the measurement testbed. http://tracer.csl.sony.co.p/maw/ (August ) http://pmon.sprntlabs.com/packstat/packetovervew.php (August )
expectaton value of the overall cross traffc ntensty s Mbt/s. To generate cross traffc from a traffc trace, we have used tcpreplay. tcpreplay sends packets from a recorded tcpdump log fle. In our measurements, the cross traffc generated by tcpreplay orgnates from real Internet lnks. The traffc trace from MAWI, whch we use n our experments, was captured durng mnutes startng at. hours on Aprl,. The average bt rate was. Mbt/s. In our testbed, the bottleneck lnk has a capacty of. Mbt/s, as descrbed above. Thus, n order to use the trace obtaned from MAWI, the traffc ntensty has been scaled down to sut the testbed (one of the features of tcpreplay). Most cross traffc packets n the trace have a sze less than bytes whle the larger packets, between and bytes, contan the maorty of the transferred bts. B. Impact of Probe Packet Sze and Probe Tran Length Ths subsecton dscusses the mpact of the probe packet sze and the probe tran length on the avalable bandwdth estmates produced by BART. The measurements have been performed n the testbed, usng tcpreplay to generate cross traffc from a traffc trace. We have used probe packet szes of, and bytes at the IP layer, and probe tran lengths of,, and packets (for each probe packet sze). From these experments, we have observed that the accuracy of BART ncreases both wth the probe packet sze and the probe tran length, see Fg.. The fgures below show the essence of our fndngs by llustratng four dfferent cases. In Fg., the probe packet sze s bytes and the number of probe packets per tran s. Ths s a weak constellaton, and the estmates of BART almost drectly ncrease toward. Mbt/s, correspondng to an almost empty lnk. Ths s due to statstcal effects, whch we wll report on n a forthcomng paper. Fg. shows that the performance mproves f more probe packets are ncluded n each tran, but stll the qualty of the estmates are unacceptable. The same concluson holds for Fg., whch llustrates the result when larger probe packets have been used. Fg. depcts a measurement result when larger probe packets n combnaton wth an ncreased probe tran length have been used. The accuracy mproves, and BART provdes estmates n good agreement wth the true avalable bandwdth. Hence, ths BART mplementaton wll most lkely overestmate the avalable bandwdth when usng small probe packets and short probe packet trans. In these experments, the cross traffc, generated from the traffc trace, s bursty. Consequently, the length of the probe tran and the probe packet sze are crucal for the qualty of the avalable bandwdth estmates produced by BART. An optmzaton of the tool could be to tune these parameters on the bass of observed varatons n the tmestamps of receved probe packets (.e., essentally tune them ndrectly based on the cross traffc characterstcs). [Mbt/s] Fg.. bytes probe packet sze, probe packets per tran. [Mbt/s] Fg.. bytes probe packet sze, probe packets per tran. [Mbt/s] Fg.. bytes probe packet sze, probe packets per tran. http://tcpreplay.sourceforge.net/ (August )
[Mbt/s] [Mbt/s] Fg.. bytes probe packet sze, probe packets per tran. C. Comparson of BART and pathchrp n the Testbed Envronment In ths subsecton, results wll be presented showng the behavor of BART and pathchrp when synthetc cross traffc has been used n the testbed. In total, sx dfferent cross traffc scenaros have been studed n the testbed. For all these experments, the measurements have been runnng for slghtly more than seconds; makng t possble to verfy the behavor n the long term, e.g. consder Fg.. However, n order to provde clear llustratons, the maorty of the followng fgures only show seconds of the measurements, see Fg.. The used nterval ( seconds) was arbtrarly chosen and does not sgnfcantly devate from other second ntervals. Each fgure depcts the estmates of BART and pathchrp. In addton, a thn sold curve ndcates the actual avalable bandwdth. The avalable bandwdth s obtaned by subtractng the average cross traffc from the measured capacty of the bottleneck lnk n the testbed. The average cross traffc s calculated by usng a sldng wndow, whch covers three seconds of cross traffc caught by the tcpdump machne. Fg. shows the long-term behavor of BART and pathchrp when the cross-traffc generator s modelng aggregated traffc of around actve users, where each user s transmttng packets wth exponental nter-arrvals. In Fg., only seconds of Fg. s depcted. It s qute clear that pathchrp over-estmates somewhat, whereas BART estmates the avalable bandwdth more accurately. Fg. llustrates the performance of BART and pathchrp when the cross traffc s less aggregated. actve users make the avalable bandwdth more fluctuatng, and t becomes harder to produce accurate estmates. BART gves decent estmates, although t does not follow the rapd changes n the avalable bandwdth. Ths corresponds to the choce of Q. Cf. the dscusson n secton III B. As prevously, pathchrp s overestmatng. Fg.. Long-term behavor of approxmately actve users wth exponental nter-arrvals between cross traffc packets. [Mbt/s] Fg.. Hgh cross traffc aggregaton ( users) wth exponental nterarrvals between cross traffc packets. [Mbt/s] Fg.. Low cross traffc aggregaton ( users) wth exponental nterarrvals between cross traffc packets. Results from measurements usng Pareto nter-arrvals wth nfnte varance (α =.) can be seen n Fg.. Hgh
aggregaton makes t easy for BART to produce accurate estmates, whle pathchrp shows a fluctuatng behavor wth overestmated values. In Fg., we can agan see that less aggregated cross traffc makes t harder to estmate the avalable bandwdth. However, BART stll appears to provde more stable estmates, whch are on average closer to the actual avalable bandwdth. more bursty the cross traffc s. An nterpretaton s that when a hgh-rate probe tran happens to largely mss the cross traffc, ths results n an underestmaton of the stran ε, whle at the same tme producng a low value for the varance R. The result s a sharp ncrease n the estmate of avalable bandwdth. Ths s why we consder t nterestng to decouple the probe pars from a tran to a sequence of ndependent pars n a future mplementaton. [Mbt/s] [Mbt/s] Fg.. Hgh cross traffc aggregaton ( users) wth Pareto (α =.) nterarrvals between cross traffc packets. Fg.. Hgh cross traffc aggregaton ( users) wth Pareto (α =.) nterarrvals between cross traffc packets. [Mbt/s] [Mbt/s] Fg.. Low cross traffc aggregaton ( users) wth Pareto (α =.) nterarrvals between cross traffc packets. Fg. llustrates the estmates n the case of approxmately actve users and nter-arrvals between cross traffc packets followng a Pareto dstrbuton wth nfnte varance (α =.). The estmates are smlar to the prevous cases wth hgh cross traffc aggregaton,.e. BART s smooth and farly accurate whereas pathchrp s overestmatng and dsplays a hgh varance. In Fg., we can agan make the same observatons as for the other low-aggregaton traffc cases. However, t could be worth notcng the sharp ncreases of the after approxmately seconds. In the current mplementaton of BART, ths s sometmes seen to occur, more markedly the Fg.. Low cross traffc aggregaton ( users) wth Pareto (α =.) nterarrvals between cross traffc packets. The mean squared error (MSE) has been calculated for the sx cases of synthetc cross traffc. See Table II and Table III. The MSE s normalzed wth respect to the bottleneck lnk capacty, and for each MSE computaton, estmates durng seconds have been consdered. Furthermore, the ntal few seconds of the measurements have been excluded, as we wshed to descrbe the steady-state -propertes, and not let varatons caused by arbtrary ntal values nfluence the results. As a benchmark for the estmates, the prevously computed avalable bandwdth was used wth a mnor correcton. In the fgures llustratng the testbed measurements, the avalable bandwdth curve s only based on the generated
cross traffc, although the tools producng estmates also observe a mnor amount of nterferng probe traffc. In order to mprove the qualty of the MSE calculatons, the avalable bandwdth benchmark has been revsed, such that BART MSE computatons take the pathchrp probe traffc nto account, TABLE II NORMALIZED MSE OF BART AND PATHCHIRP ESTIMATES WHEN USING HIGH CROSS TRAFFIC AGGREGATION AND THREE DIFFERENT PACKET INTER-ARRIVAL DISTRIBUTIONS Inter-arrval dstrbuton: BART: pathchrp: Exponental.. Pareto (α =.).. Pareto (α =.).. TABLE III NORMALIZED MSE OF BART AND PATHCHIRP ESTIMATES WHEN USING LOW CROSS TRAFFIC AGGREGATION AND THREE DIFFERENT PACKET INTER-ARRIVAL DISTRIBUTIONS Inter-arrval dstrbuton: BART: pathchrp: Exponental.. Pareto (α =.).. Pareto (α =.).. and vce versa. Based on the values n Table II and III, t can be concluded that BART shows better performance n case of hgh aggregated cross traffc, compared to low aggregaton. The reason for a hgher MSE n case of low aggregaton s that BART does not follow the rapd changes n the avalable bandwdth. However, snce BART s tunable, t s possble to mprove the aglty, smply by adustng the estmate of the process nose covarance matrx Q. The pathchrp MSE s hgher n all traffc cases, manly due to overestmaton. The reason for ths overestmaton s yet unknown, although a smlar observaton has been made n []. D. Comparson of BART and pathchrp usng an Internet Path In ths subsecton, the behavor of BART and pathchrp s llustrated n two measurements performed on the Internet. The experments were performed over an Internet path n Sweden, consstng of layer- hops, accordng to traceroute. Probe traffc was transmtted from a host at Lnköpng Unversty, whch connects to the Swedsh Unversty Network (SUNET), and receved by a host at Ercsson Research n Stockholm, connected to the Internet servce provder Tela. All the hosts n the experments were equpped wth processors equal to or better than Pentum III/ MHz, runnng Lnux (Fedora Core ). They were provded wth Mbt/s Ethernet connectons to the Internet. The probe traffc rate was randomly chosen from a unform dstrbuton, over the nterval from Mbt/s to Mbt/s. As n the prevous experments, the estmated process nose covarance matrx Q was assgned the value of - on the dagonal, whereas the other elements were set to zero. BART transmtted one probe tran every second ( probe packets per tran),.e. the probe traffc ntensty was. Mbt/s. The default parameters were used for pathchrp, except for the J parameter, whch sets the number of packets per Jumbo packet (default = ). In the Internet measurements, pathchrp was usng two packets per Jumbo packet, thus approxmately one estmate per second was produced, whch s n lne wth the behavor of BART. pathchrp loads the network wth a probe traffc ntensty around. Mbt/s. Immedately before the Internet experments were actvated, the throughput between the probe sender at Lnköpng Unversty and the probe recever at Ercsson Research was measured usng Iperf. Several measurements were conducted, and the avalable bandwdth turned out to be farly stable around Mbt/s (at the IP layer). In the evaluaton of the estmaton performance of BART and pathchrp, we made use of ths value as a steady state avalable bandwdth reference. In order to affect the cross traffc on the path between Lnköpng Unversty and Ercsson Research, tcpreplay was used to generate addtonal traffc durng the experments. The extra generated traffc was based on the prevously used traffc trace from MAWI, usng a tme-dependent re-scalng factor to produce the avalable-bandwdth characterstcs of Fg.. By subtractng the known added traffc ntensty from the throughput estmate of Iperf, we constructed a reference avalable bandwdth along the Internet path, to use as a benchmark for BART and pathchrp. Of course, the true avalable bandwdth s perturbed by unknown tme-dependent traffc varatons, and t should be kept n mnd that the reference curve n Fg. cannot reflect ths perturbaton. The results of the frst experment are depcted n Fg. and. The traffc load generated by tcpreplay ncreases from Mbt/s up to approxmately Mbt/s durng a perod of seconds. After that, the added traffc s rather constant for seconds, untl t slowly decreases back to zero. Both BART and pathchrp seem to follow the general trend of the avalable bandwdth durng ths experment, even f t mght be argued that BART has a lower varablty and s somewhat better at followng the mean avalable bandwdth. [Mbt/s] Fg.. ntensty. Internet experments wth a smoothly changng cross traffc http://www.spn.rce.edu/software/pathchrp/ (August ) http://dast.nlanr.net/proects/iperf/ (August )
[Mbt/s] Fg.. Contnuaton of Fg.. [Mbt/s] Fg.. Internet experments wth a rapdly changng cross traffc ntensty. approxmately Mbt/s. Another change occurs after seconds, see Fg.. The cross traffc generator goes off and the avalable bandwdth nstantly ncreases wth roughly Mbt/s. BART s seen to clearly respond to these sudden changes n avalable bandwdth, whle the same s not evdent for pathchrp. V. DISCUSSION AND CONCLUSIONS We have presented BART, a novel method for real-tme estmaton of avalable bandwdth over a network path usng Kalman flterng. We have demonstrated that Kalman flterng s applcable to the problem of measurng avalable bandwdth, despte a strong nonlnearty n the system measurement model. We have also shown, by experments usng an mplementaton of BART both n a laboratory network and over the Internet, that reasonable accuracy can be obtaned wth lttle computatonal efforts and wth low cost n terms of extra traffc load on the network from the probe packets. The method has consderable potental wth regard to tunng for better performance. Notably, the process covarance matrx Q, whch s an nput to the BART estmaton algorthm, contans three degrees of freedom, whereas we have only started explotng one n the measurements reported n ths paper. Also, t s concevable that the estmaton performance could be ncreased by allowng the probe packet pars n a sequence to be transmtted ndependently of each other. Nevertheless, we have already wth ths mplementaton of BART observed reasonable accuracy of the avalable bandwdth estmaton n a wde varety of traffc cases. We conclude that Kalman flterng provdes a promsng engne for fast and effcent estmaton of avalable bandwdth. [Mbt/s] Fg.. Contnuaton of Fg.. The second experment studed rapd changes of the addtonal cross traffc load generated by tcpreplay. As can be seen n Fg., the cross traffc generator was confgured to suddenly ncrease the traffc ntensty after seconds, APPENDIX: AN INTRODUCTION TO THE KALMAN FILTER The Kalman flter s a numercal method for sequentally updatng and mprovng an estmaton of the state of a system, based on a sequence of measurements. For lnear systems where the process nose and measurement nose are Gaussan, the Kalman flter has been shown to be the optmal estmator, n the sense that t mnmzes the expectaton value of the norm of the error. For detals regardng ths, as well as regardng the remander of ths Appendx, see [] and references theren. In our applcaton of estmatng network propertes lke avalable bandwdth, we do not have the detaled knowledge of the system that would allow the method to guarantee optmalty. However, by makng educated guesses or by treatng some quanttes as tunable parameters, the Kalman flter can be appled wth good results even when some of the system propertes are unknown. Our network model system s not lnear. However, t s asymptotcally lnear n the regon of nterest (the regon of congeston), so we solve the problem by applyng a Kalman flter n our estmate of ths regon. In general, a system sutable for Kalman flter estmaton s
descrbed by a state vector x R, the evoluton of whch s governed by a lnear stochastc dfference equaton of the type: k = k + k + k n x Ax Bu w (A-) Here, u s the control nput, and w s the process nose. Ideally, w has a Gaussan probablty dstrbuton N (, Q), where Q s the process nose covarance matrx. Informaton on the state of the system s ganed by m performng a seres of measurements z R, whch depend lnearly on the system state vector: z k = Hxk + vk (A-) Here, the measurement nose v deally has a Gaussan probablty dstrbuton N (, R), where R s the measurement nose covarance matrx. The obectve s to use the measurements to track the state of the system,.e. estmate the state vector x k as t evolves n dscrete tme (k =,,, ). Ths s acheved by dong two operatons n each tme step: frst usng the process equaton to proect the latest estmate ahead n tme ( predcton ), then usng the latest measurement to adust ths predcted state ( correcton ). To do ths we defne the a pror state estmate x ˆ k, as well as the a posteror state estmate x ˆk, the latter of whch also takng the measurement z k at step k nto account. We also need to compute at each step the a pror and a posteror state estmate error covarance matrces where e k s the a pror and estmate error: T P = E( e e ) (A-) k k k T k k P = E( e e ) (A-) k e k s the a posteror state ek = xk xˆ k (A-) ek = xk xˆ k (A-) The crucal pont of the Kalman formalsm s how to make the best of the measurements. Ths s the correcton step mentoned above. The optmal correcton s where the Kalman gan k = k k k k x ˆ xˆ + K ( z Hxˆ ) (A-) Kk s gven by: T Pk = APk A + Q (A-) T T Kk = Pk H ( HPk H + R) (A-) x ˆ ˆ x + K ( z ˆ Hx ) (A-) k = k k k k P k = ( I Kk H ) Pk (A-) REFERENCES [] G. Bshop and G. Welch, An Introducton to the Kalman Flter, n SIGGRAPH, Course,. [] K. Jacobsson, H. Halmarsson, N. Möller, and K H Johansson, Estmaton of RTT and bandwdth for congeston control applcatons n communcaton networks, n Proc. IEEE Conference on Decson and Control (CDC), Paradse Island, Bahamas,. [] M. Jan and C. Dovrols, Pathload: a measurement tool for end-to-end avalable bandwdth, n Proc. Passve and Actve Measurement workshop (PAM),. [] M. Jan and C. Dovrols, End-to-end Avalable Bandwdth: Measurement Methodology, Dynamcs, and Relaton wth TCP Throughput, n Proc. ACM SIGCOMM,. [] S. Keshav, A control-theoretc approach to flow control, n Proc. ACM SIGCOMM', pages --, Zurch, Swtzerland, September. [] M. Km and B. Noble, SANE: stable agle network estmaton, Unversty of Mchgan department of Electrcal Engneerng and Computer Scence, CSE-TR--,. [] M. Km and B. Noble, Moble network estmaton, n Proc. Moble Computng and Networkng (ACM MOBICOM), Rome, Italy,. [] B. Melander, M. Börkman, and P. Gunnngberg, A new end-to-end probng and analyss method for estmatng bandwdth bottlenecks, n Proc. IEEE Globecomm, San Francsco, USA, November. [] A. Pásztor and D. Vetch, The packet sze dependence of packet-par lke methods, n Proc. Tenth Internatonal Workshop on Qualty of Servce (IWQoS ), Mam Beach, USA, May. [] V. Rbero, R. Red, R. Baranuk, J, Navratl, and L. Cottrell, pathchrp: effcent avalable bandwdth estmaton for network paths, n Proc. Passve and Actve Measurement workshop (PAM),. [] A. Shrram, M. Murray, Y. Hyun, N. Brownlee, A. Brodo, M. Fomenkov and K. Claffy, Comparson of publc end-to-end bandwdth estmaton tools on hgh-speed lnks, n Proc. Passve and Actve Measurement workshop (PAM),. [] R. Prasad, M. Murray, C. Dovrols, and K. Claffy, Bandwdth estmaton: metrcs, measurement technques, and tools, n IEEE Network, November/December. [] X. Lu, K. Ravndran, and D. Logunov, Mult-Hop probng asymptotcs n avalable bandwdth estmaton: stochastc analyss, n Proc. Internet Measurement Conference (IMC),. [] S. Ekeln and M. Nlsson, Contnuous montorng of avalable bandwdth over a network path, n Proc. nd Swedsh Natonal Computer Networkng Workshop (SNCNW ), Karlstad, Sweden, November. [] E. Hartkanen, S. Ekeln, and J. M. Karlsson, Adustment of the BART Kalman flter to mprove real-tme estmaton of end-to-end avalable bandwdth, n Proc. rd Swedsh Natonal Computer Networkng Workshop (SNCNW ), Halmstad, Sweden, November. T T Kk = Pk H ( HPk H + R) (A-) To sum up, the Kalman flter equatons are: ˆ x k = Axˆ k + Buk (A-)