Discovering Generl Logicl Network Topologies Mrk otes McGill University, Montrel, Quebec Emil: cotes@ece.mcgill.c Michel Rbbt nd Robert Nowk Rice University, Houston, TX Emil: {rbbt, nowk}@rice.edu Technicl Report TREE-00 Deprtment of EE, Rice University November 5, 00 bstrct We develop method ddressing the tsk of identifying generl logicl network topologies. The method discovers topologies tht include lyer- networking elements nd does not require dministrtive ccess throughout the entire network (it does not use either IMP requests or SNMP informtion). We divide the problem of generl logicl topology discovery into two tsks: () identifiction of the logicl tree topologies from individul sources nd () merging the individul trees. Severl techniques exist tht ddress the first tsk. The problem of uniting the tree topologies into single (non-tree) grph is more chllenging. Our methodology consists of mesurement frmework tht uses novel ctive probing structure nd records only the rrivl order of pckets t the receivers, thus eliminting ny need for precise timing. Successful experiments performed over university LN nd over the Internet verify tht our methodology is verstile nd robust. Introduction The physicl topology of network is the physicl connectivity of the elements tht comprise the network, including switches, routers, hubs nd hosts. Knowledge of the physicl topology of network is extremely importnt for the successful execution of mny network mngement tsks such s fult monitoring nd isoltion, server plcement nd resource shring. The physicl topology cn be depicted s grph, with internl nodes representing switching elements nd edge nodes representing hosts (see the exmple in Figure ()). The logicl topology of network is relted to the physicl topology, nd cn lso be represented s grph. However, over specific period of routing stbility, the logicl topology grph is determined by the pths trversed by pckets sent from the sources to the receivers. Nodes in the physicl topology grph re included only if they represent points in the network where routes diverge or merge, nd single logicl link is used to connect two such nodes if there is (trversed) physicl pth between them. It is cler tht the logicl topology is only stble during period when no routing chnges occur. Figure (b) depicts the logicl topology from the perspective of ech source in Figure (), nd Figure (c) depicts the logicl topology for the multi-source network. In one sense, the logicl topology displys less informtion thn the physicl topology, becuse it does not indicte ll connectivity or switching elements. However, the logicl topology provides some importnt informtion tht cnnot be grnered from the physicl topology: it identifies the pths trversed by pckets sent from ech source to This work ws funded by NSF Grnt no. MIP-97069, RO Grnt no. DD9-99--0349, ONR Grnt no. N004-00--0390, nd Texs Instruments
the receivers nd indictes where these pths diverge nd merge. This ltter informtion is extremely useful for the evlution of the resource shring cpbility of the network under the current configurtion, nd lso cn guide the decisions of source-bsed routing lgorithms. B B b c f d 3 i 4 b c 3 4 B d e f 3 4 g h i 3 4 d = g b c = e i h = f 3 4 () (b) (c) Figure : Physicl nd logicl topologies of n exmple network. () The physicl network showing routing pths. ircles indicte internl network elements (switches nd routers), squres - re sources, nd squre -4 re receivers. Dot-dsh lines re routes from source, dshed lines routes from source B, nd solid lines routes from source. (b) The three logicl tree topologies tht cn be determined from the individul sources (s might be estimted by the lgorithms of [4, 8, 9]). This set of three topologies does not reflect the equivlence, or even reltive position, of the nodes. In this cse, node c is equivlent to node e, node d to node g, nd node f to node h. The unlbeled nodes in the physicl topology do not pper in the logicl topologies. (c) The generlised logicl topology of the multi-source network, showing the correspondence between brnching nodes in the logicl tree topologies. This topology clerly indictes how ech source-destintion pth reltes to ll other pths. In this pper, we describe core component of technique for discovering the (unicst) logicl topology of multiple-sender, multiple receiver network. We refer to such topologies s generl topologies, following [6], thereby distinguishing them from the tree topologies tht hve been the focus of much of the logicl network topology discovery literture [4, 8, 9,, 0,, 7]. Three principles guide our development of the procedure. () The discovered topology should include lyer- elements (switches, bridges); () the procedure should discover topology despite the user not hving dministrtive ccess throughout the entire network; nd (3) the procedure should not query the interior network elements. The first two requirements re importnt if the technique is to be pplicble over network tht includes lyer- elements nd tht is comprised of multiple subnetworks under different ownership nd dministrtion. We include the third requirement becuse we wish to mke the technique resilient to scenrios where dministrtors prevent routers from responding to IMP requests or SNMP queries, nd lso becuse the technique should perform even over legcy networks tht my not be SNMP cpble or complint. The first nd third principles imply tht ny technique is indmissible if it is bsed on trceroute [] or other tools relying on IMP responses (such topology discovery techniques re described in [4,, 3, 7]). These techniques only cpture lyer-3 topology nd rely on router response to IMP requests. The techniques tht re cpble of cpturing lyer- topology [5, 5] invribly rely on SNMP dt from the networking elements, which implies universl dministrtive ccess. We cnnot consider such procedures if we follow the outlined principles. Techniques hve been developed for determining logicl tree topologies (single source networks) both in multicst nd unicst settings [4, 8, 9,, 0,, 7] tht mke use only of mesurements recorded t the senders nd receivers. These techniques do not violte our principles nd re cpble of discovering (logicl) lyer- topology, but re pplicble only to single source networks; there is no mechnism for extending ny of them to generl (multi-source) topologies. lthough the tree topologies of individul sources in generl network cn be discovered, the problem of uniting these tree topologies into single (non-tree) grph is very chllenging. Discovering where the pths from one source to receiver join with the pths from nother source is n open problem (under the principles bove).
We use the tree topology discovery lgorithms s lunching point. We divide the generl logicl topology discovery problem into two tsks: () the discovery of the individul tree topologies, nd () the merging of these trees into single topology. The first tsk is ddressed by the tree topology discovery lgorithms, nd so in this pper, we focus entirely on the second tsk. The nture of these two tsks is indicted in Figure ; solving the first tsk discovers the individul logicl tree topologies of Figure (b), nd solving the second tsk ccomplishes the trnsition from the tree topologies to the generl logicl topology, s depicted in Figure (c). The procedure we present utilises ctive probing, nd records only the ordering of pcket rrivls t receivers. s result, no clock synchroniztion or dely mesurement is necessry, significntly enhncing the pplicbility nd robustness of the scheme. Bsed on the rrivl orderings, the procedure mkes decisions bout the loction where the pths from one source to the receivers join the tree topology of nother source. We hve explored the efficcy of the technique through its ppliction in LN environment nd in network involving sources nd receivers locted t universities in North meric nd Europe.. Relted Work number of uthors hve identified techniques tht rely solely on edge-bsed mesurements to estimte the logicl network topologies tht rise when single source communictes with multiple receivers. The ppers [, 0,, 7] focus on multicst topologies, wheres the ppers [4, 8, 9] investigte unicst topology identifiction. ll of the techniques ssume tht, from the source s point of view, the logicl topology of single source/multiple receiver network is tree nd is stble over the mesurement period (this ssumption cn be violted by lod blncing strtegies nd route chnges). The tree-oriented topology identifiction schemes tht utilize solely end-to-end mesurement involve three min steps. Firstly, end-to-end mesurements re mde (e.g., end-to-end loss, dely, nd dely differences). Secondly, set of end-to-end metrics re estimted bsed on the mesurements. Exmples of previously used metrics include counts of joint zero dely events (the utiliztion metric), counts of joint loss events, dely covrinces, nd shred loss rtes. In the third step of the topology identifiction schemes, inference lgorithms use the estimted metrics to identify the topology. mens of extending these tree identifiction techniques to the multiple-sender cse is not cler. The schemes cn obviously be used to estimte the individul tree topologies observed from ech source in multi-source tree, but the mesurements do not provide enough informtion to enble reconstruction of the correspondence between the trees. In no technique is there logicl extension from the single-source probes to multiple-source probes tht would provide dditionl informtion. In this pper, we develop mesurement frmework nd inference scheme tht permits estimtion of the connections between the single-source trees. There re severl techniques tht re cpble of mpping multiple-source lyer-3 physicl topologies, but they require tht internl routers respond to IMP requests nd identify themselves using their IP ddresses. The Merctor project [4], id s skitter project [], nd the techniques described in [3, 7] ll use trceroute [] in some form to determine the pth from source to receiver. In contrst to the work presented here, these pproches focus on physicl topology identifiction, combining trceroute mesurements collected over very long timefrmes. much more importnt distinction between these techniques nd our proposed procedure is tht the trceroute-bsed methods fil when substntil portion of the topology is comprised of lyer- elements (bridges nd switches) or when routers do not respond to IMP requests. In ddition to the procedures in [4,, 3, 7] tht rely only on IMP responses, there re other pproches tht use SNMP informtion to generte network topology mps. Mny network mngement tools include fetures tht use SNMP informtion to mp lyer-3 physicl topologies, e.g., IBM Tivoli Netview (www.tivoli.com). Other tools such s isco s Discovery Protocol (www.cisco.com) rely on vendor-specific extensions to SNMP MIB (Mngement Informtion Bses) to incorporte lyer- elements; s result they re pplicble only in homogeneous networks (where ll elements re supplied by the sme vendor). Breitbrt et l. [5] nd Lowekmp et l. [5] describe procedures for determining physicl topologies tht include lyer- elements for more heterogeneous networks. These procedures 3
rely only on universlly supported SNMP MIB (Mngement Informtion Bses) informtion. Peregrine Systems Infrtools Network Discovery (www.peregrine.com) is commercil tool tht ddresses the sme tsk. These ltter tools focus primrily on physicl topology, but it is possible to derive logicl topologies using them. However, ll of the SNMP-bsed techniques require dministrtive ccess, which is typiclly only vilble to mchines on the locl network. The techniques cn therefore only generte topology informtion for the component of the network where the user hs dministrtive privileges. Problem Sttement Two key tsks comprise the problem of identifying the unicst logicl topology of network comprised of multiple sources nd multiple receivers. The first tsk is the discovery of the tree topologies perceived by ech source. The second tsk is the merger of the set of trees. It is possible to develop scheme tht jointly ddresses both tsks, but we prefer to focus on sequentil strtegy: the identifiction of individul trees, followed by the merger. Previous work [4, 8, 9] hs ddressed the first tsk. In this pper we focus on the second tsk of merging of trees. For the ske of clrity, we distill the generlized merging tsk into the following simpler problem nd describe n pproch to its solution throughout the reminder of the pper. ssume tht we know (or hve estimted) the logicl tree topology from source to multiple receivers. n we determine (from edge-bsed mesurements) where the pths from nother source to ech receiver enter the source logicl topology? This simple problem lies t the hert of the merging exercise; if we cn ccomplish this, then we cn develop procedure tht merges multiple trees. b c d e f g 3 4 5 6 7 8 9 Figure : Nine receiver exmple network illustrting entry points. The solid lines nd hollow circles depict the tree topology from the perspective of source. The dshed lines nd solid circles indicte where the pths from source to the receivers join the topology (note tht they do not depict the source topology). Figure provides n illustrtion of the problem, depicting nine receiver network. The logicl tree topology from the perspective of source is shown by the solid lines nd hollow, lbelled circles. Our tsk is to identify where the pths from source to ech receiver join this tree, reltive to the hollow nodes. These entry points re shown by the solid circles. s exmples, the pth from to receiver enters t point between nodes d nd e, wheres the pth to receiver 7 enters bove node. Our inference technique ssumes: () Interior switches or routers cnnot be relied upon to respond to queries. If portions of the network (for exmple the IP routers) do respond, then it is strightforwrd to incorporte the informtion in the discovery procedure. () The topology perceived from ech source is tree. This requires tht ny lod blncing or routing chnges over the mesurement period do not ffect the logicl multiple-source topology. In order to mke this ssumption more resonble, we seek to limit probing nd keep the mesurement period s short s possible. (3) 4
The routers nd switches in the topology obey first-in first-out policy for pckets of the sme clss. This is necessry to ensure tht probe pckets do not frequently experience reorderings when trversing the sme route.. Orgniztion In Section 3 of the pper, we describe the methodology, commencing with description nd idelised nlysis of simplified two-receiver scenrio. The section proceeds to conduct more detiled nlysis with more relistic ssumptions, nd extends the frmework to multiple receiver networks. Potentil extensions to the methodology re lso described. Section 4 presents results from n experiment conducted on LN nd n experiment over the Internet, two scenrios tht present very different types of chllenges. Section 5 discusses some limittions of the procedure nd includes concluding remrks. 3 Methodology nd Mesurement Frmework 3. Simplified Description nd nlysis In this first description of the frmework, we will perform nlysis ssuming no cross-trffic nd clock synchroniztion between the sources, in order to motivte the technique nd highlight the intuition behind it. In Section 3.3, we will relx these ssumptions nd conduct more creful nlysis with cross-trffic effects included. () (b) (c) (d) Figure 3: The four possible entry cses for two-sender, two-receiver network. The blck circles indicte entry points. lthough depicted s lying in the middle of links in the -topology, these entry points cn coincide with the children nodes. For exmple, in (), the entry point cn be the node, but it must lie below the node. The dshed lines re used to indicte entry pths only, so the topology of the source tree is not depicted except in (). se () hs common brnching point for the two sources; in cses (b), (c) nd (d), the brnching points differ. We begin by exploring the simple cse of two-sender, two-receiver network. In such network, under the ssumptions outlined bove, there re four possible entry scenrios, s depicted in Figure 3. Our mesurement frmework in this simple cse proceeds s follows (in trees with more receivers, the frmework is strightforwrd extension). To mke the n-th mesurement, we send two pckets from source, spced some smll time difference t prt, with the first pcket being sent t time t n. The first pcket, which we lbel p,, is destined for receiver ; the second, p,, for receiver. We lso send two pckets from source, gin spced by t. The first pcket of this pir is sent t time t n + v n, where v n is n offset time. The first pcket, p,, is sent to receiver nd the second, p,, to receiver. Figure 4() depicts this setup for the scenrio in which the brnching point is common to both sources (we will cll this the shred scenrio). Denote the fixed portion of the dely (trnsmission nd propgtion) of pcket p, from source to the joining point d,, nd tht of pcket p, from source by d,. Denote the corresponding quntities for the second pckets sent by ech source by d, nd d,, respectively. Since the joining point is the sme in the 5
t n + t t n + v n + t t n t n + v n d, d, d = d = d,, d, = d, = d d, d, () (b) Figure 4: The mesurement process. () Mesurement for topology in which the brnching point is common. The pckets next to ech source re lbelled with send times. The d nd d lbels correspond to the fixed dely component (trnsmission nd propgtion) of the indicted pths. (b) In this cse the brnching points re not common, the joining points differ, so the fixed dely components d, nd d, re unlikely to be equl. shred scenrio, d, = d, = d nd d, = d, = d. The rrivl time of pcket p, t the joining point is t n + d,, wheres tht of pcket p, is t n + v n + d,. The rrivl times of pckets p, nd p, re t n + t + d, nd t n + v n + t + d,. If we now exmine the rrivl order of pckets t the two receivers, we see tht p, rrives before p, if v n > (d, d, ). Similrly pcket p, rrives before p, if v n > (d, d, ). We sy tht mesurement records reverse-ordering event if the order of pcket rrivls (compring the pcket from to the pcket from ) is not the sme t the two receivers. In the shred brnching point scenrio, since d, = d, nd d, = d,, the order of rrivls t the two receivers will be exctly the sme, irrespective of the offset v n. There will be no occurrences of reverse-ordering events. Now consider one of the unshred scenrios in which the brnching is not common (cse (b) in Figure 3). In this cse, the joining points differ, so the fixed delys re (lmost lwys) not equl, i.e., d, d, nd d, d, (see Figure 4(b)). If the probes re sent t the sme times s bove, then pcket p, rrives t its joining point t t n + d,, nd pcket p, rrives t time t n + v n + d,. Pcket p, rrives t its joining point t time t n + t + d, nd pcket p, t time t n + v n + t + d,. Let d = (d, d, ) nd d = (d, d, ). If we compre the rrivl orderings t the two receivers, we see tht the orderings differ when d < v n < d if d < d, or when d < v n < d if d < d. In either cse, there is n offset region of mgnitude d d where different orderings rise t the two receivers. The result is the sme for the entry scenrios depicted in Figure 3(c) nd 3(d). The mesurement process consists of repeting the mesurement described bove mny times for n =,..., N, with v n drwn from uniform distribution over the rnge [ D, D] (with D chosen to be much lrger thn ny mesured round-trip-time). In the idel world nlysed thus fr, we observe no reverse-ordering events in the shred entry scenrio depicted in Figure 3()). In the unshred scenrios of Figures 3(b)-(d), the frction of reverse-ordering events pproches d d /D for lrge N. To be more precise, the number of such events obeys binomil distribution Bi(N, d d /D). In prctice, we implement this mesurement procedure by hving one source send its ( t-seprted) pcket-pirs t stedy rte. The rte must be sufficiently slow to void network flooding nd probe interference. The second source sends it pirs t the sme rte, but dds rndom offset time (drwn from uniform distribution over D,..., D). The two receivers record the orderings of pckets, nd send the results bck to the sources. The motivtion for the spcing t between the two pckets from ech source is to ensure tht they do not bunch up becuse of trnsmission dely. If this bunching occurs, pcket p, experiences dditionl dely reltive to p, in its trversl to the joining point, so tht even in the shred cse d, d,. Similrly, d, d,. The discrepncies here re determined by the bottleneck bndwidths from the sources to the joining point; if these re not equl, then 6
d d even in the shred cse, nd reverse-ordering events will occur. The vlue t should thus be sufficiently lrge to ensure tht bunching does not occur in the bsence of cross trffic. We need t > p/(min(b, B )), where p is the probe size, nd B nd B re the bottleneck bndwidths of the pths from the respective sources to the joining point. s n exmple, for p = 40 bytes nd B = Mbps, we hve t > 30 microseconds. In prctice, we set t substntilly lrger thn this to void s much s possible the bunching effects of cross-trffic. The procedure just described enbles us to distinguish between entry scenrio () nd entry scenrios (b)-(d) (referring to Figure 3). However, we cnnot determine from these mesurements exctly which of (b)-(d) is in effect. In Section 3.5, we will see tht when there re more receivers in the network, it is often possible to combine the results of pirwise tests to resolve the uncertinty. We estblish conditions for identifibility (locliztion to single link) of the entry points. 3. Timing issues The two min timing tsks involve performing n pproximte synchroniztion of the sources t the beginning of the experiment nd in keeping them on trck during the experiment. Timing is not n issue t the receivers, becuse they simply record pcket orderings. Unless some form of synchroniztion is performed, the sending times of the sources will be offset from one nother s result of clock differences [6]. There will be constnt offset c (in ddition to the rndom offset v ) between the sending times of the very first probes due to the offsets between the clocks of the two sources. In turn, the effective rnge of the totl rndom offset distribution becomes D + c,..., D + c rther thn D,..., D. If we choose D such tht this rnge still encompsses the much smller offset region where reverse-ordering events potentilly occur then the results of the experiment re unffected by the constnt offset. If the send times re clculted nively from system clocks, then network timing protocols cn induce lrge, unexpected shifts in reltive offsets when reclibrtions occur. lock skew lso rises from the physicl mchines hving different internl system clock rtes. The technique described in [6] cn eliminte these problems, but s yet our procedure does not incorporte it. Over n experiment lsting few minutes, clock drift cn men tht the n-th probes re (pproximtely) seprted by v n + c + c n, nd for the finl (N-th) probe, c N is of the order of severl hundred microseconds. The drift mens tht the true offset distribution is not completely uniform, but for sizeble D, it is sufficient pproximtion. In fct, the use of uniform distribution is not criticl to the nlysis; distribution suffices if it stisfies the property tht the rtio of the density t ny two points in the rnge is sufficiently close to one. side from the initil constnt offset c, nd the drift offset c dditionl (nd quite substntil) offsets cn be incurred if the operting system swps out the source process during n experiment. We overcome this by ssigning ech probe sequence number bsed on the difference between the time when the experiment begn nd when the probe is being sent. We find tht the mount of time necessry to perform some system tsks is not necessrily deterministic, but lwys within smll rnge (on the order of microseconds). While discrepncies between send-times of the first pckets in corresponding pirs re evded by choosing prmeter D to be sufficiently lrge, it is importnt tht the t vlues t the two sources re pproximtely the sme. However, since t is only of the order of few milliseconds, clock skew induces mximum discrepncy of few microseconds. In the nlysis tht follows, we bsorb ll errors incurred by ll timing discrepncies in noise terms tht lso include cross-trffic delys. n dditionl fctor to consider in more thorough nlysis is the potentil for reorderings of successive probes trversing the sme pth cn, rising, for exmple, s result of multiple prllel physicl connections between routers. We ssume tht these events re rre, becuse they cn only occur when the pckets re very closely spced, sitution tht is common in our mesurement frmework for only very smll rnge of offsets. Such reorderings hve the effect of very slightly incresing the probbility of reverse-ordering event. 7
3.3 More Detiled nlysis We now revisit the nlysis of the rrivl times for the shred scenrio of the two-receiver network, incorporting cross-trffic effects. The rrivl times t the joining point(s) re: p, (n) : t, (n) = t n + d, + g, (n) p, (n) : t, (n) = t n + v n + d, + g, (n) p, (n) : t, (n) = t n + t + d, + g, (n) p, (n) : t, (n) = t n + t + v n + d, + g, (n) Here g, (n) nd g, (n) represent the combintion of timing errors nd cross-trffic delys experienced by the first pckets sent by the source, nd g, (n) nd g, (n) re the corresponding quntities for the second pckets. These terms include only the delys incurred on the pth(s) to the joining point(s). Let us first consider the shred scenrio. If pcket p, (n) rrives before p, (n) then d, + g, (n) < v n + d, + g, (n). Setting d = d, d, s before, nd g (n) = g, (n) g, (n), we hve v n > d + g (n). In order for reverse-ordering event to occur, pcket p, (n) must rrive fter p, (n). With d = d, d, s before, nd g (n) = g, (n) g, (n), reverse-ordering event occurs only when v n < d + g (n), since d = d in the shred scenrio. By reversing the inequlities, we obtin the expressions for the requirements for reverse-ordering event when pcket p, rrives first. If we consider fixed offset v, the probbility tht reverse-ordering event occurs is: r(v) = v d v d v d v d p(g, g ) dg dg + p(g, g ) dg dg. () The nture of this integrtion is depicted in Figure 5(). t ech offset point v, there is region where n (g, g) combintion cuses n reverse-ordering event. The totl probbility of reverse-ordering event is then: f = D r(v)dv. () D D Figures 5(c) nd (d) disply n estimtion of the integrl for common brnching point in the LN experiment described in 4.. The figure indictes the very smll offset region (reltive to D = ms) where reverse-ordering cn occur. The probbility of reverse-ordering event cn be estimted by numericlly pproximting (). For the depicted scenrio, the estimted probbility is 0.007. Similr vlues were observed for ll common brnching points encountered during the LN experiment described below. In the unshred scenrios, the rrivl times t the joining points remin the sme s bove, but we must tke into ccount the fct tht d, d, nd d, d,. Proceeding s before, if pcket p, (n) rrives before p, (n) then v n > d + g (n). In order for reverse-ordering event to occur, pcket p, (n) must rrive fter p, (n). This requires tht v n < d +g (n). By reversing the inequlities, we obtin the expressions for the requirements for reverseordering event when pcket p, rrives first. If we consider fixed offset v, the probbility tht reverse-ordering event occurs is: r(v) = v d v d v d v d p(g, g ) dg dg + p(g, g ) dg dg (3) The nture of this integrtion is depicted in Figure 5(b). 8
Define P (t) t p g (x) dx, where p g is the probbility distribution of g, nd equivlently, P (t) t p g (x) dx. If t is sufficiently lrge, then g (n) nd g (n) re pproximtely independent. In the shred cse, the probbility distributions re the sme (ssuming semi-sttionrity), so p g (x) = p g (x). Under the ssumptions bove we cn write the following expression for f in the shred brnching point scenrio: f(0) = P (t)[ P (t)] dt. (4) D In the unshred scenrios, defining d = d d : f(d) = D t [ P (t)]p (t d) + t P (t)[ P (t d)] dt. (5) These expressions demonstrte tht the probbility of different ordering event is usully much lrger in the unshred cse compred to the shred cse. Suppose tht g (n) nd g (n) re zero-men noises nd re well concentrted (in the noise-free cse they re point-mss (Dirc delt) functions locted t the origin). Then P (t) nd P (t) re pproximtely step functions, being ner zero for t < 0 nd close to for t 0. If this is the cse nd the brnching proint is shred, then f(0) 0, since the integrnd of (4) is zero except for very smll intervl bout the point t = 0. In the unshred cse, d 0 nd f(d) 0. To see this, note tht if 0 < t < d (or d < t < 0), then the integrnd of (5) is equl to on quite lrge intervl (the size of the intervl depends on the difference d). onsequently the totl integrl f(d) is strictly greter thn zero, nd moreover f(d) is monotoniclly incresing function of d the lrger the difference d, the more distinguishble re the shred nd unshred cses. 9
g v - d p(g, g ) v - d v g () v - d g p(g, g ) v - d g v (b) Estimted g + d ( µ s) 00 50 f(v) 0.3 0. 0. 0 40 60 80 00 0 Estimted g + d (µ s) (c) 000 500 000 500 0 500 000 500 000 Offset v (µ s) (d) Figure 5: ross-trffic nd timing effects on ordering observtions. () n exmple of how the likelihood of n ordering offset is determined for the shred scenrio ccording to (). The contours depict the joint probbility distribution p(g, g ), which re the dely differences due to cross-trffic nd timing error. For n offset v, the probbility of reverse-ordering event r(v) is determined by the frction of the distribution lying in the hshed regions. s v vries, the meeting point of the two subregions of integrtion trverses the dshed line, which psses through the origin nd hs slope. (b) The determintion of the probbility of reverse-ordering event in the unshred cse ccording to (3). In this cse, s v vries, the meeting point of the subregions of integrtion trverses line of slope offset from the origin by d d. (c) In the LN experiment described in Section 4., dely differences were mesured t common joining point. Bsed on these dely differences, we estimte g (n) + d nd g (n) + d, nd disply them using sctter plot. Here the hshed regions re the res where n ordering difference would occur when the offset v = 80 + d microseconds. (d) We estimte f(v) s the frction of points lying within the equivlent regions for ech v. The estimted function f(v) is displyed for D = milliseconds. In this experiment, the estimted probbility of reverse-ordering event is 0.007. 0
3.4 Mking the Decision fter N mesurements hve been performed, the number of reverse-ordering events in the two receiver network test is recorded s x,. Bsed on this vlue, decision must be mde s to whether the brnching point is shred or not. This decision would be simpler to mke if we knew how mny reverse-ordering events we could resonbly expect if the brnching point were shred. We cn obtin n indiction of this number using the following procedure. We collect mesurements in exctly the sme mnner s the two-receiver mesurement described bove, except tht ll four pckets re sent to the sme receiver. We re thus mking mesurements cross Y -shped topology. We perform N mesurements of this form to both receiver nd receiver nd record the number of reverse-ordering events s x nd x, respectively. If the brnching point to the receivers is shred, then the upper brnches of the Y-topologies tested in these experiments coincide with the pths to the common merging point. In this cse, the probbility of reverse-ordering events should be the sme in ll three experiments, i.e., x, x nd x, re ll drwn from the sme binomil distribution. If the brnching point is not shred, then we expect x, to be drwn from different binomil distribution thn either x or x, nd moreover, the proportion prmeter of the former distribution should be significntly lrger thn for either of the ltter distributions. The decision s to whether brnching point is shred or unshred cn now be formulted s hypothesis test. Let x = mx(x, x ). Denote the proportion prmeter of the binomil from which this mesurement ws drwn p, nd the proportion prmeter of the binomil from which x, ws drwn p,. We wnt to test whether these prmeters re equivlent (the distributions re the sme), so the hypothesis test becomes: H 0 : p, = p For resonbly lrge N, we cn perform this test s Z-test, with: H : p, > p (6) Z = p, p p( p)/n. (8) (7) where p, = x, /N, p = x /N nd p = (x, +x )/N. For resonbly lrge N, distributions cn be pproximted s norml, nd we cn set threshold for Z such tht the probbility of declring brnching point unshred when it is in fct shred is equl to specified level α. In our experiments, we set α = 0.05. 3.5 Multiple Receiver Networks Thus fr, we hve concentrted on describing the mesurement frmework for two-receiver network. In the two receiver network, ech mesurement consists of pir of pckets sent from ech source. The first pcket from ech source is destined for receiver, nd the second for receiver, nd there is spcing between them of t. The frmework for n r-receiver network is the nturl extension of this. For ech mesurement, the two sources send stripe of r pckets, using the terminology of [3], with spcing of t between successive pckets. The i-th pcket in this stripe is destined for the i-th receiver. Ech such mesurement provides ( r ) pirwise mesurements of the form described bove, nd counts of reverse-ordering events re collected for ech pir of receivers. We perform the test described bove for ech pir of receivers to determine if there is only one brnching point for both sources. Let s(r, r ) be binry vlue, indicting whether receivers i nd j shre common brnching point from the two sources (0 indicting no, indicting yes). In the simple two-receiver network, if we determined tht the brnching point ws not shred, then it ws impossible to distinguish between the three unshred entry scenrios of Figure 3(b)-(d). However, when we hve multiple pirwise test results, n unshred test result cn be useful informtion when used in conjunction with nother shred test result.
Now, we consider the scenrio where the tree topology is known for single source. For instnce, suppose we hve identified source s tree using the techniques described in [8] nd [9]. We sy tht the two-source network is resolved with respect to source s tree from the mesurements if ech point t which pth from source joins pth from source cn be loclized to certin logicl link in the source tree topology. We will now describe n lgorithm for determining whether the network hs been resolved with respect to source s tree, nd then we verify tht the stipultions of the lgorithm re indeed necessry nd sufficient. Before stting the lgorithm, we introduce some nottion. Let R be the set of receivers, nd let P(i, r) be the pth from n internl node i to one of its descendnt receivers r in source s tree. Let b(r, r ) denote the brnching point of the pths from to receivers r nd r. Finlly, ssocite mrker bit, m k, with ech internl node (brnching point) in the tree. Step : Set m k = 0 for ll k. Merging lgorithm Step : For ech pir of receivers r, r R, if s(r, r ) =, set m k = for k corresponding to b(r, r ) nd lso set the mrker bit to for ech internl node encountered long the pths P(b(r, r ), r ) nd P(b(r, r ), r ). The results of this lgorithm cn then be used to determine whether the network hs been resolved with respect to source s tree s stted in the following Theorem. Two key ssumptions re mde. We ssume tht once the pths from source to two different receivers brnch prt, the remining sub-pths from the brnching point to the receivers do not shre ny nodes or links in common. Similrly, we ssume tht once the pths from two different sources to unique receiver hve joined, they trverse the sme set of nodes nd links from the joining point to the receiver. Theorem Let source s tree topology to the set of receiver R, nd the set of mesurements, {s(r, r )} for ll pirs r, r R be given. The two-source network is resolved with respect to source s tree from the mesurements if the mrker bit, m k, is set to for every internl node of source s tree. Proof: Necessity of the condition stted bove follows from the fct tht our test only distinguishes between shred nd unshred brnching points nd does not distinguish between the different unshred cses. Suppose tht there is n internl node whose mrker bit is still set to 0. This internl node is brnching point by definition nd so there re t lest two links, l nd l, connecting to sets of descendnts D nd D respectively, nd one link l 3 entering the node from its prent. Becuse the node is unmrked, there is uncertinty s to whether or not the pths from to receivers in D join s tree t link l 3 or l, so the network is not resolved. The condition is sufficient bsed on the following logic. If ll internl nodes re mrked, then ech link the tree is isolted nd so the pths from source to ny of the receivers re resolved to single link. This link either lies bove the highest shred brnching point in subtree or is the link directly entering the receiver node. For networks consisting of lrge number of receivers, it is not prcticl to send stripes of mny closely spced pckets into the network. This my result in flooding the network. In prctice, we hve found tht breking the set of receivers into groups of 5-7 nd then using stripes of this length gives ccurte results without overwhelming other trffic on the network. While the most efficient probing scheme would be to use stripes of length r, we find tht using smller stripes nd repeting the experiment over different subsets of receivers llows us to inject miniml mount of probe trffic nd still get ccurte results. 3.6 Extensions The methodology nd nlysis presented in this pper focused on the two-source topology identifiction problem. Extensions to multiple source scenrios re strightforwrd. Beginning with single-source tree, second source s
topologicl reltionships re incorported s described bove. The topologies of subsequent sources cn be joined to this topology, one source t time. For ech new source, the probing nd merging lgorithms operte in similr mnner s before, but in this cse probing cn be performed from the new source nd ny one (or ll) of the other sources in the current topology. The shredness indictors s(i, j) tke non-zero vlue if the new source shres the i, j brnching point with ny one of the other sources, in which cse vlue indicting which source shres the brnch cn be ssigned. The merging lgorithm uses the shredness indictors s well s their non-zero vlues nd employs similr cycling procedure to loclize (s much s possible) the joining points for the new source. Theorem gives conditions under which the cquired mesurements provide full identifibility. If these conditions re not met, then certin joining points will only be loclized to within sequence of two or more consecutive links. It my be possible to employ more informtive probing of the portion of the network in question tht cn help to further resolve such cses. dditionl informtion, reflective of link bndwidths, cn be glened by performing the procedure used to set the thresholds (Y -topology probing) but mking the second pcket from source consistently much lrger. When ll the pckets re the sme size, the number of reverse-ordering events cn be used to estimte f(0). When one pcket is much lrger, however, the number of reverse-ordering events cn be used to form n estimte of metric of the pth from to the joining point. This pth metric is the sme s the pth metric generted by the mesurement procedure used in the identifiction of single source topologies in [9] (it is reflective of the bndwidths of the links on the pth). The mesurement frmework in [9] cn be used to determine the pth metric from the source to ny brnching point in the source topology. By simply compring the metrics of pths to brnching nd joining points, the reltive position of ll entry points cn be determined. However, forming ccurte estimtes of the metrics cn require intensive probing. For this reson, we envision tht these extended mesurements could form potentil secondry step, utilized only fter the ppliction of the simple nd undemnding probing mechnism we hve presented. 4 Experimentl Results 3 4 5 6 7 8 9 0 3 4 5 6 Figure 6: The true (nd lso discovered) logicl topology of the LN network. The hollow interior circles represent switches or routers where the pths from source to different receivers brnch prt. The filled circles indicte the nodes (the joining points) where the pths to given receiver from sources nd merge. In this figure, they re depicted s seprte nodes, but our lgorithm only resolves the loction of these nodes to single logicl link of the source- topology. If filled node is positioned on link in the source- topology, then the node must lie below the prent node of tht link but cn either coincide with or lie bove the child node. 3
Our msprobe multiple-sender probing progrm implements the techniques discussed bove. There re two source components nd receiver component. Source sends UDP pcket probes to the receivers t regulr period. Source controls the experiment nd sends t the sme period but dds uniform rndom offset to ech sending time. The receiver component simply trcks the order in which probes rrive, nd then sends the results bck to source when the experiment hs reched completion. Becuse the only importnt metric is pcket rrivl order, no specil timing infrstructure is required. fter the probes hve been sent the results re collected nd processed t source. This source lso keeps trck of the offset used for ech tril. This informtion cn lter be used, long with the outcome for ech tril, to djust the bounds of the distribution from which the offsets re chosen. Receiver Index 4 6 8 0 4 s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s s 4 6 8 0 4 6 Receiver Index 0.00.00.00 3.00 4.00 5.00 6.00 Figure 7: Results of the LN experiment. The x- nd y-xes correspond to receivers s lbeled in Figure 6. The shde of gry of the squre t position (i, j) indictes the observed rtio of different ordering events to totl mesurements for the receiver pir (i, j). If the squre t (i, j) is lbelled with n s, then the pths from the two sources to receivers i nd j shre common brnching point in the true topology. When the detection threshold is set to.00, the vlue determined by the procedure outlined in Section 3.4, then ll test decisions re correct. To explore the efficcy of our technique we hve run experiments in two very different networking environments. The first is deprtmentl LN. The second consists of hosts locted t cdemic nd reserch institutions throughout the United Sttes nd Europe. Ech scenrio presents its own set of difficulties. In the LN, the fixed dely differences cn be very smll nd RTTs re of the order of hundreds of microseconds, so timing issues re importnt nd the decision-mking component of the lgorithm must perform well. ross-trffic in the LN does not produce such extreme dely vritions s we observe in the Internet-wide experiment. In the Internet experiment, fixed dely differences re much lrger, nd RTTs of the order of tens or hundreds of milliseconds, so timing nd thresholds re not so importnt. However, the dely vritions re much lrger, inducing lrger noise effect due to cross-trffic. 4. LN Experiment The first set of experiments were run over US University deprtmentl LN. For this experiment there were 6 receivers with IP ddresses from two different subnets. Both subnets reside over the sme physicl network, which consists of single lyer-3 router nd multiple lyer- ethernet switches. Figure 6 depicts the logicl network connectivity of the LN. The router is isco model 6509MSF nd switches re 3om SuperStck models 3300 nd 000. Note tht some of the switches tht interconnect hosts re store-nd-forwrd switches nd others re cut-through. Our technique resolves shred pths regrdless of the switching technology implemented t joining or brnching points. Ech probe is 68 bytes, including pylod, UDP, nd IP heders. We conservtively set spcing prmeter t to 4
3 4 5 6 7 8 9 Figure 8: True logicl topology of the Internet experiment testbed. Shred brnching points only occurred when both receivers were physiclly locted on the sme cmpus, i.e. receiver pirs (,), (3,4), (5,6) nd (8,9). In this cse the network topology is not identifible in the sense we defined bove. In this experiment, we cnnot completely resolve the entry points of the pths from, but we do correctly identify shred brnching points. be 600 microseconds bsed on the ssumption tht the minimum link bndwidth is Mbps. Using 600 microseconds for the rndom offset bound D is sufficient to encompss the rnge of possible delys for the short pths of the LN. In our experiments on this topology, ll of the decisions (shred or unshred brnching points) were correct in the sense tht they greed with the known logicl connectivity. The decisions were mde using the methodology for setting thresholds described in Section 3.4. Figure 7 grphiclly depicts the results of one experiment. We correctly identify the set of shred pths. In this cse, the results re sufficient to completely resolve (to the logicl link level) where the pths from source to the receivers join those from source. 4. Internet Experiment In order to explore lgorithm performnce in n environment very different from the LN, we performed nother set of experiments using Internet hosts locted in North meric nd Europe. For these experiments there were 9 receiving hosts locted t 5 different cdemic estblishments. The two sources were both situted in North meric. Figure 8 shows the logicl connectivity between sources nd receivers. The mjor network properties tht ffect prmeter selection for our technique re minimum link bndwith nd mximum end-to-end dely. Becuse these properties differ gretly between the LN nd Internet scenrios, softwre prmeters need to be djusted ccordingly. The sme 68 byte UDP probes re used in either cse. To ccount for potentilly lower minimum link bndwidth we increse the pcket spcing prmeter t from 600 microseconds to millisecond. Likewise, to djust for the much lrger rnge of possible end-to-end delys the rndom offset is drwn from uniform distribution spnning 90 milliseconds. In this experiment we re ble to correctly identify pirs of receivers with shred pths from the two sources, but not completely resolve entry points. Figure 9 shows the results. In the Internet experiments, the set of results is insufficient to resolve the entry points of the pths from source to single link. More receivers re required to produce more complete picture. 5 Discussion nd onclusions We hve presented technique for identifying shred pths from multiple senders to receiver. ombined with knowledge of one of the sender s connectivity to the receivers, this informtion cn be used to determine portions of, 5
s 0.0 0.0 3 s 0.0 Receiver Index 4 5 s 30.0 40.0 6 50.0 7 60.0 8 s 70.0 3 4 5 6 7 8 9 Receiver Index Figure 9: Results of n Internet experiment. Note tht in comprison to the LN experiment, the rtio reverseorderings spns much greter rnge. This cn be ttributed to two fctors: () the fixed dely differences d nd d re much lrger in the Internet nd () the rnge of end-to-end delys experienced by pckets on the Internet is much lrger thn in LN. nd in some cses completely resolve, the generl topology. The frmework we propose only records pcket rrivl orderings. Without the need for precise timing mesurements, our scheme is very prcticl to implement. Through Internet nd LN experiments we demonstrte the verstility nd robustness of the technique. The experiments we report involve reltively smll number of receiver hosts. Our technique scles resonbly well, in tht the number of pckets sent per tril increses linerly with the number of receiver hosts. For lrge number of hosts, it becomes imprcticl to inject such lrge mounts of trffic onto the network. The durtion of the procedure should be limited to ensure tht the routing topology is stble for the length of the experiment. For run time of pproximtely 5 minutes nd without overwhelming the network with trffic, our procedure scles to pproximtely 50 receivers. In future work, we will explore the development of less intensive multiple sender probing methods tht monitor for chnges to n initilly estblished bseline topology nd cn ddress lrger topologies. We will lso investigte the integrtion of the multiple sender techniques presented in this pper with the techniques of [4, 8, 9] so tht we cn infer generl topologies without ny prior knowledge. References [] Skitter. http://www.cid.org/tools/mesurement/skitter. [] trceroute. http://www.trceroute.org. [3] P. Brford,. Bestvros, J. Byers, nd M. rovell. On the mrginl utility of network topology mesurements. In Proc. IEEE/M SIGOMM Internet Mesurement Workshop, Sn Frncisco,, Nov. 00. [4]. Bestvros, J. Byers, nd K. Hrfoush. Inference nd lbeling of metric-induced network topologies. Technicl Report BUS-00-00, omputer Science Deprtment, Boston University, June 00. [5] Y. Breitbrt, M. Groflkis,. Mrtin, R. Rstogi, S. Seshdri, nd. Silberschtz. Topology discovery in heterogeneous ip networks. In Proc. IEEE INFOOM 000, Tel viv, Isrel, Mr. 000. 6
[6] T. Bu, N. Duffield, F. L. Presti, nd D. Towsley. Network tomogrphy on generl topologies. In Proc. M Sigmetrics, Mrin Del Rey,, Jun. 00. [7] H. Burch nd B. heswick. Mpping the Internet. IEEE omputer, 3(4):97 98, 999. [8] R. stro, M. otes, nd R. Nowk. Mximum likelihood identifiction of network topology from end-to-end mesurements. In DIMS Workshop on Internet nd WWW Mesurement, Mpping nd Modeling, Pisctwy, NJ, Feb. 00. [9] M. otes, R. stro, M. Gdhiok, R. King, Y. Tsng, nd R. Nowk. Mximum likelihood network topology identifiction from edge-bsed unicst mesurements. In Proc. M Sigmetrics, Mrin Del Rey,, Jun. 00. [0] N. Duffield, J. Horowitz, nd F. L. Presti. dptive multicst topology inference. In Proceedings of IEEE INFOOM 00, nchorge, lsk, pril 00. [] N. Duffield, J. Horowitz, F. L. Presti, nd D. Towsley. Multicst topology inference from end-to-end mesurements. In IT Seminr on IP Trffic, Mesurement nd Modelling, Monterey,, Sep. 000. [] N. Duffield, J. Horowitz, F. L. Presti, nd D. Towsley. Multicst topology inference from mesured end-to-end loss. IEEE Trns. Info. Theory, 48():6 45, Jnury 00. [3] N. G. Duffield, F. L. Presti, V. Pxson, nd D. F. Towsley. Inferring link loss using striped unicst probes. In INFOOM, 00. [4] R. Govindn nd H. Tngmunrunkit. Heuristics for Internet mp discovery. In Proc. IEEE INFOOM 000, Tel viv, Isrel, Mr. 000. [5] B. Lowekmp, D. O Hllron, nd T. Gross. Topology discovery for lrge ethernet networks. In Proc. M SIGOMM 00, Sn Diego,, ug. 00. [6]. Pásztor nd D. Veitch. precision infrstructure for ctive probing. In Proc. Workshop on Pssive nd ctive Newtorking, msterdm, pr. 00. [7] S. Rtnsmy nd S. Mcnne. Inference of multicst routing trees nd bottleneck bndwidths using end-to-end mesurements. In Proceedings of IEEE INFOOM 999, New York, NY, Mrch 999. 7