MATE: MPLS Adaptive Traffic Engineering

Transcription

1 MATE: MPLS Adaptive Traffi Engineering Anwar Elwalid Cheng Jin Steven Low Indra Widjaja Bell Labs EECS Dept EE Dept Fujitsu Network Communiations Luent Tehnologies Univ. of Mihigan Calteh Pearl River, NY Murray Hill, NJ Ann Arbor, MI Pasadena, CA Abstrat Destination-based forwarding in traditional IP routers has not been able to take full advantage of multiple paths that frequently exist in Internet Servie Provider Networks. As a result, the networks may not operate effiiently, espeially when the traffi patterns are dynami. This paper desribes a multipath adaptive traffi engineering mehanism, alled MATE, whih is targeted for swithed networks suh as MultiProtool Label Swithing (MPLS) networks. The main goal of MATE is to avoid network ongestion by adaptively balaning the load among multiple paths based on measurement and analysis of path ongestion. MATE adopts a minimalist approah in that intermediate nodes are not required to perform traffi engineering or measurements besides normal paket forwarding. Moreover, MATE does not impose any partiular sheduling, buffer management, or a priori traffi haraterization on the nodes. This paper presents an analytial model, derives a lass of MATE algorithms, and proves their onvergene. Several pratial design tehniques to implement MATE are desribed. Simulation results are provided to illustrate the effiay of MATE under various network senarios. A. Motivation I. INTRODUCTION Internet Servie Providers (ISPs) are faing the hallenge of designing their networks to satisfy ustomers demands for fast, reliable, and differentiated servies. Internet traffi engineering is emerging as a key tool for ahieving these goals in a ost-effetive manner. Aording to the IETF, Internet traffi engineering is broadly defined as that aspet of network engineering dealing with the issue of performane evaluation and performane optimization of operational IP networks [2]. More speifially, traffi engineering often deals with effetive mapping of traffi demands onto the network topology, and adaptively reonfiguring the mapping to hanging network onditions. It is worth noting that traffi engineering is more general than os routing in the sense that traffi engineering typially aims at maximizing operational network effiieny while meeting ertain onstraints, whereas the main objetive in os routing is to meet ertain os onstraints for a given souredestination traffi flow. The emergene of MultiProtool Label Swithing (MPLS) with its effiient support of expliit routing provides basi mehanisms for failitating traffi engineering [7], [4]. Expliit routing allows a partiular paket stream to follow a pre-determined path rather than a path omputed by hop-by-hop destinationbased routing suh as OSPF or IS-IS. With destination-based routing as in traditional IP network, expliit routing an only be provided by attahing to eah paket the network-layer address of eah node along the expliit path. However, this approah generally makes the overhead in the paket prohibitively expensive. In MPLS, a path (known as a label swithed path or LSP) is identified by a onatenation of labels whih are stored in the nodes. As in traditional virtual-iruit paket swithing, a paket is forwarded along the LSP by swapping labels. Thus, support of expliit routing in MPLS does not entail additional paket header overhead. Several researhers have proposed to add traffi engineering apabilities in traditional datagram networks using shortest path algorithms (e.g., see [5], [9]). Although suh shemes have been shown to improve the effiieny of the network, they suffer from several limitations inluding: load sharing annot be aomplished among paths of different osts, traffi/poliy onstraints (for example, avoiding ertain links for partiular soure-destination traffi) are not taken into aount, modifiations of link metris to re-adjust traffi mapping tend to have network-wide effets ausing undesirable and unantiipated traffi shifts, and traffi demands must be preditable and known a priori. The ombination of MPLS tehnology and its traffi engineering apabilities are expeted to overome these limitations. Expliit LSPs and flexible traffi assignment address the first limitation. Constraint-based routing has been proposed to address the seond limitation. Furthermore, network-wide effets an be prevented when LSPs are pinned down. A hange in LSP route will only ause the disturbane of the traffi for the orresponding soure-destination pair. The objetive of this paper is to address the last limitation. In MPLS, traffi engineering mehanisms may be timedependent or state-dependent. In a time-dependent mehanism, historial information based on seasonal variations in traffi is used to pre-program LSP layout and traffi assignment. Additionally, ustomer subsription or traffi projetion may be used. Pre-programmed LSP layout typially hanges on a relatively long time sale (e.g., diurnal). Time-dependent mehanisms do not attempt to adapt to unpreditable traffi variations or hanging network onditions. An example of a time-dependent mehanism is a global entralized optimizer where the input to the system is a traffi matrix and multilass os requirements as desribed in []. When there are appreiable variations in atual traffi that ould not predited using historial information, a timedependent mehanism may not be able to prevent signifiant imbalane in loading and ongestion. In suh a situation, a statedependent mehanism an be used to deal with adaptive traffi assignment to the established LSPs aording to the urrent state of the network whih may be based on utilization, paket delay, paket loss, et. In this paper, we assume that LSP layout has been determined using a long-term traffi matrix. The fous is on load balaning short-term traffi flutuations among multiple

2 2 BI AI Swithing Cloud AE BE Inoming Data Pakets Filtering and Distribution Measurement and Analysis Traffi Engineering Probe Pakets Data Pakets LSPs to one egress LSP LSP2 LSP Fig.. A Transit Network Running MATE LSPs between an ingress node and an egress node. B. Overview We propose a state-dependent traffi engineering mehanism alled Multipath Adaptive Traffi Engineering (MATE). Some of the features of MATE inlude: distributed adaptive load-balaning algorithm, end-to-end ontrol between ingress and egress nodes, no new hardware or protool requirement in intermediate nodes, no knowledge of traffi demand is required, no assumption on the sheduling or buffer management shemes at a node, optimization deision based on path ongestion measure, minimal paket reordering, and no lok synhronization between two nodes. MATE operational setting assumes that that several expliit LSPs (typially range from two to five) between an ingress node and an egress node in an MPLS domain have been established using a standard protool suh as CR-LDP [] or RSVP-TE [], or onfigured manually. This is a typial setting whih exists in an operational ISP network that implements MPLS. The goal of the ingress node is to distribute the traffi aross the LSPs so that the loads are balaned and ongestion is thus minimized. The traffi to be balaned by the ingress node is the aggregated flow (alled traffi trunk in [2]) that shares the same destination. MATE is intended for traffi that does not require bandwidth reservation with best-effort traffi being the most dominant type. Figure shows an example of a network environment where there are two ingress nodes, AI and BI, and two egress nodes, AE and BE, in an MPLS domain. MATE would be run on AI and BI to balane traffi destined to AE and BE, respetively, aross the LSPs onneting from AI to AE and from BI to BE. Note that the LSPs onneting the two pairs may be overlapping and sharing the same resoures. However, this paper shows that stability an be guaranteed even though the pairs operate asynhronously. Figure 2 shows a funtional blok diagram of MATE loated at an ingress node. Inoming traffi enters into a filtering and distribution funtion whose objetive is to failitate traffi shifting among the LSPs in a way that redues the possibilities of having pakets arrive at the destination out of order. The mehanism does not need to know the statistis of the traffi demands or flow state information. The traffi engineering funtion deides on when and how to shift traffi among the LSPs. This is done based on LSP statistis whih are obtained from measurement using probe pakets. The traffi engineering funtion onsists of two phases: a monitoring phase and a load balaning phase. In the monitoring Fig. 2. MATE Funtions in an Ingress Node phase, if an appreiable and persistent hange in the network state is deteted, transition is made to the load balaning phase. In the load balaning phase, the algorithm tries to equalize the ongestion measures among the LSPs. One the measures are equalized, the algorithm moves to the monitoring phase and the whole proess repeats. The role of the measurement and analysis funtion is to obtain one-way LSP statistis suh as paket delay and paket loss. This is done by having the ingress node transmit probe pakets periodially to the egress node whih returns them bak to the ingress node. Probing may be done per lass, i.e., probe pakets have the same type of servie header information as the traffi lass being engineered. Based on the information in the returning probe pakets, the ingress node is able to ompute the oneway LSP statistis. Estimators of LSP statistis from the probes may be obtained reliably and effiiently using bootstrap resampling tehniques. These tehniques provide a dynami mehanism for sending the probe pakets so that the smallest number is automatially seleted as the traffi onditions hange to provide a given desirable degree of auray. Reent measurements in the Internet indiate little variations of aggregate traffi on links in 5-min intervals [7]. This quasi-stationarity ondition where traffi statistis hange relatively slowly (muh longer than the round-trip delay between the ingress and egress nodes) failitates traffi engineering and load balaning based on measurement of LSP statistis. C. Paper organization The rest of the paper is organized as follows. Setion II presents an analytial model for multipath load balaning, derives a lass of MATE algorithms, and proves their stability and optimality. Setion III details the overall MATE sheme and disusses several implementation tehniques, suh as traffi filtering and distribution, traffi measurement, bootstrapping, et. Setion IV desribes an experimental setup to verify the effetiveness of the proposed sheme. Setion V presents the performane results and illuminates the behaviors of the algorithm in different networking environments and with traffi models ranging from Poisson to traffi models with longer dependene. The Appendix provides analytial support and proofs for the results presented. II. MATE ALGORITHMS In this setion we present an analytial model of multipath load balaning, derive a lass of asynhronous MATE algorithms, and prove their stability and optimality.

3 : : \ A. Model We model a MATE network by a set of unidiretional links. It is shared by a set of ingress egress (IE) node pairs, indexed of LSPs. Eah of these IE pairs has a set available to it. Note that, by definition, no two (distint) IE pair uses the same LSP, even though some of their LSPs may share links. Hene are disjoint sets. An IE pair has a total input traffi of rate and routes amount of it on LSP suh that Let "%#'(". /! " # $ for all )+, be the rate vetor of, and let -#, the vetor of all rates. The flow on a link 2 has a rate that is the sum of soure rates on all LSPs that traverse link : 45# 687 $98 Assoiated with eah link is a ost :., as a funtion of the link flow. We assume that, for all, : <;=, is onvex (and hene ontinuous). Our objetive is to minimize the total ost :>(",?#A@ by optimally routing the traffis on LSPs in B : CEDGF H :>.",I# subjet to K #L $ NPO for all / : (, :.JK, () for all M (2) A vetor is alled a feasible rate if it satisfies (2 ). A feasible rate is alled optimal if it is a minimizer to the problem ( ). As observed in [5, Chapter 5], the derivative of the objetive funtion with respet to is.",r# :S $.4K, We will interpret : S (, as the first derivative length of link, and :UT.V, as the (first derivative) length of LSP. The following haraterization of optimal rate is a diret onsequene of the Kuhn Tuker theorem (see also [5, Chapter 5]). It says that at optimality a pair splits its traffi only among LSPs that have the minimum (and hene equal) first derivative lengths. Theorem : The rate vetor XW is optimal if and only if, for eah pair, all LSPs YZ with positive flows have minimum (and equal) first derivative lengths. B. Asynhronous algorithm A standard tehnique to solve the onstrained optimization problem ( ) is the gradient projetion algorithm. In suh an algorithm routing is iteratively adjusted in opposite diretion of the gradient and projeted onto the feasible spae defined by (2 ). Eah iteration of the algorithm takes the form: [.\^],_# ` +.\a,zb%xde:>(\a,fehg () where ji O is a stepsize and should be hosen suffiiently ", th element is the first small, d>:>.\a, is a vetor whose derivative length ` d>:>(\a,fe(k# :T "K of LSP at time \, and ` le g is the projetion of a vetor l onto the feasible spae. The algorithm terminates when there is no appreiable hange, i.e., mgm +.\^],Zbn[(\a,mGm oqp for some predefined p. Note that the above iteration an be distributively arried out by eah pair without the need to oordinate with other pairs: (\],_# `.\a,byxd>: (\a,fe g (4) where ".\a,y#5.vj.\a, rszt, is s rate vetor at time \, and d>: (\a,u#v :UT.+.\a,a, wx, is the vetor of first derivative lengths of LSPs in. However (4) is not realisti, for two reasons. First (4) assumes all updates are synhronized. Seond it assumes zero feedbak delay. Speifially (4) assumes that as soon as the IE pairs have alulated a new rate vetor +.\a,, it is refleted immediately in all the link flows:.\a,y# $9$! and in all the first derivative lengths: "Kz(\a, (5) V.+.\a,a,R# : S $..\a,a, (6) Moreover all pairs have available these new values in d>:{.\a, for omputation of the rate vetor in the next period. In pratie the IE pairs update their rates asynhronously and in an unoordinated manner. Moreover the first derivative length of a LSP an only be estimated empirially by averaging several measurements over a period of time. We now extend the model to take these fators into aount. Let -} 86 ~t be a set of times at whih IE pair adjusts its rate based on its urrent knowledge of the (first derivative) lengths of LSPs. At a time \ ƒ, alulates a new rate vetor (\] and, starting from time \[],_# `.\a,byx (\a,fe g (7), splits its traffi along its LSPs in aording to.\+], until after the next update time in X. Here ".\a, is an estimate of the first derivative length vetor at time \, and is alulated as follows. The new rates alulated by the IE pairs may be refleted in the link flows after ertain delays. We model this by (f. (5)) (\a,?# $$9.\ S \a,< (\ S, (8). Gˆf J.Š K (\<S \a,# 8 Œ ŽŒ (9) ˆXf J.Š.\a, represents the flow rate available at link at In the above time \ and is an weighted average (onvex sum) of past soure \a, and rates (\ S,. The weights.\ S \a, depend on ( an be different between eah soure and link, on different

4 ] \ \ \ o LSPs, and at different times \. This model is very general and inludes in partiular the following two popular types: Latest data only: only the latest rate 4,, for some (typially unknown) /Y} ẗ \[bn\, is used in the measurement of.\a,, i.e., Kz(\ S \a,# if \ S # and otherwise. Latest average: only the average over the latest rates is used in the measurement of (\a,, i.e., (\ S \a, i O for ẗ \ S # /b and otherwise, for some (typially unknown). } \+bn\ An IE pair estimates the first derivative length of an LSP by asynhronously olleting a ertain number of measurements (using probe pakets, see below), and forming their mean. Hene (f. (6)).\a,# $.\<S \a,a:s Ja.\<S,a, (). ˆf ".Š K (\<S \a,?# 8 Œ <Œ / () ˆf ".Š Again the estimate is obtained by averaging over the past values of LSP lengths, and an depend on \a,. The model is very general and inlude the speial ases of using only the last reeived measurement or the average over the last values, as disussed above. The interpretation in both ases is that the S,a, for \ S i have not been re- $ : S eived by by time \, and the measurements for \ S o (latest data only) or for \ S b (latest average) have been disarded. This onludes the desription of our algorithm model (equations 7 ). The model is similar to that in [8], with two differenes. First their model distinguishes between the desired rate +.\a, as alulated by the projetion algorithm and the atual realized soure rate +.\a, is a onvex ombination of the urrent desired rate [(\a, and the previous atual rate [(\b,. This models the fat that a desired rate +.\a, may not be realized immediately, as in a virtual iruit network where virtual iruits may persist over several update yles. We are however only dealing with IP datagrams and hene it is reasonable to assume that eah ingress node an shift its traffi among the LSPs available to it immediately after eah update. +.\a,. The atual rate Seond their model assumes that, at time \, eah has available the urrent first derivative $ : S (\a,a, and uses it in plae of the gradient in the update algorithm. We however assume that, at time \, may only have outdated first derivative lengths (see ( )); moreover uses a weighted average over several past lengths in the update algorithm. This is beause, in our ase, an only estimate the first derivative lengths through noisy measurement. The next result states that the algorithm onverges to an optimal routing, provided the following onditions are satisfied: C: The ost funtions : (l, are twie ontinuously differentiable and onvex. C2: Their derivatives : S (l, are Lipshitz over any bounded sets, i.e., for any bounded set suh that for all l there exists a onstant l S, we have m : S l!,tb : S (l S,m m ljb{l S m. C: For any onstant the sets } lvm : (l, are bounded. C4: The time interval between updates is bounded. Theorem 2: Under onditions C C4, starting from any initial vetor [ O,, there exists a suffiiently small stepsize suh that any aumulation point of the sequene } +.\a, 4 generated by the asynhronous algorithm is optimal. A more areful aounting shows that the stepsize, and hene the speed of onvergene, depends on the degree of asynhronism as expressed by the parameter \ defined in (8), the steepness of the ost funtion as expressed by the Lipshitz onstant in ondition C2, and the size of the network. For ease of exposition, suppose the ost funtions are uniformly globally Lipshitz, i.e., for all links and all l, l S, we have m :S l!,+b :S (l!sg, m m lb l!skm Theorem : An upper bound in Theorem 2 is: ]J [ where is the total number of LSPs in the network, is the number of hops in the longest (maximum hop) LSP, is the maximum number of LSPs going through a link, and \, defined in (8), measures the degree of asynhronism. The theorem suggests that the larger the degree of asynhronism measured by \, the smaller the stepsize and hene slower the onvergene. C. Example ost funtion The hoie of ost funtions determines the parameters to be measured and equalized in arrying out MATE. A natural hoie for the link ost is delay. Then Theorem implies that the derivatives of the LSP delay are the ongestion measures to be equalized. If we take the delay to be the average delay of an T T queue :.,U# T bƒ,, representing the link apaity then : satisfies the onditions of Theorem 2 and hene the algorithm will onverge to an optimal traffi split. Note that the link apaity typially flutuates randomly. Hene the delay derivatives annot be omputed and must be measured. Loss may be inorporated into the ost by treating eah paket loss as a fixed (large) delay. Another alternative is to use the produt of loss and delay as the ost funtion. In summary, the basi goal is to steer the network towards a desired operating performane based on a hosen ost funtion. \ ] III. MATE IMPLEMENTATION TECHNIUES In this setion, we provide further elaboration on the tehniques employed in our implementation of the MATE funtions. A. Traffi filtering and distribution The traffi filtering and distribution funtion first distributes the traffi to be engineered for a given ingress-egress pair equally among bins, where the number of bins determines the minimum amount of the traffi that an be shifted. If the total inoming traffi to be engineered is of rate bps, eah bin would reeive an amount of u# T bps. The traffi from the bins is then mapped into the LSPs aording to the MATE algorithm desribed in the last setion. The engineered traffi an be filtered and distributed into the bins in a number of ways. A simple method is to distribute the traffi on a per-paket basis without filtering. For example, one,a,

5 , 5 may distribute inoming pakets at the ingress node to the bins in a round-robin fashion. Although it does not have to maintain any per-flow state, the method suffers from potentially having to reorder an exessive amount of pakets for a given flow whih is undesirable for TCP appliations. On the other extreme, one may filter the traffi on a per-flow basis (e.g., based on o soure IP address, soure port, destination IP address, destination port, IP protooli tuple), and distribute the flows to the bins suh that the loads are similar. Although per-flow traffi filtering and distribution preserves paket sequening, this approah has to maintain a large number of states to keep trak of eah ative flow. Another method is to filter the inoming pakets by using a hash funtion on the IP field(s). The fields an be based on the soure and destination address pair, or other ombinations. A typial hash funtion is based on a Cyli Redundany Chek (CRC). The purpose of the hash funtion is to randomize the address spae to prevent hot spots. Traffi an be distributed into the bins by applying a modulo-n operation on the hash spae. Note that paket sequene for eah flow is maintained with this method. After the engineered traffi is distributed into the bins, a seond funtion maps eah bin to the orresponding LSP aording to the MATE algorithm. The rule for the seond funtion is very simple. If LSP is to reeive twie as muh traffi as LSP, then LSP should reeive traffi from twie as many bins as LSP. The value should be hosen so that the smallest amount of traffi that an be shifted, whih is equal to T of the total inoming traffi, has a reasonable granularity. B. Traffi measurement and analysis The effiay of any state-dependent traffi engineering sheme depends ruially on the traffi measurement proess. MATE does not require eah node to perform traffi measurement. Only the ingress and egress nodes are required to partiipate in the measurement proess. For the purpose of balaning the loads among LSPs, the available bandwidth appears to be a desirable metri to measure. The methods for measuring the available bandwidth of a given path have been desribed in the past (e.g., see [], [8]). Based on our experiene, this metri turns out to be diffiult to measure aurately using the minimal requirements assumed in MATE. To this end, we found that paket delay is a metri that an be reliably measured. The delay of a paket on an LSP an be obtained by transmitting a probe paket from the ingress node to the egress node. The probe paket is time-stamped at the ingress node at time and reorded at the egress node at time. If the ingress lok is faster than the egress lok by, then the total paket delay (i.e, queueing time, propagation time, and proessing time) is b {]w. A group of probe pakets sent one at a time on an LSP an easily yield an estimate of the mean paket delay ` bƒ e"]q. The reliability of the estimator an be evaluated by bootstrapping (see details below) to give the onfidene interval for the mean delay. One important point to note is that the value of is not required when only the marginal delay is needed. MATE exploits this property by relying only on marginal delays rather than absolute delays. Therefore, lok synhronization is not neessary. Paket loss probability is another metri that an be estimated by a group of probe pakets. In general, only reasonably high paket loss rates an be reliably observed. Paket loss probability an be estimated by enoding a sequene number in the probe paket to notify the egress node how many probe pakets have been transmitted by the ingress node, and another field in the probe paket to indiate how many probe pakets have been reeived by the egress node. When a probe paket returns, the ingress node is able to estimate the one-way paket loss probability based on the number of probe pakets that has been transmitted and the number that has been reeived. The advantage of this approah is that it is resilient to losses in the reverse diretion. The bootstrap is a powerful tehnique for assessing the auray of a parameter estimator in situations where onventional tehniques are not valid [9]. Most other tehniques for omputing the variane of parameter estimators or for setting onfidene intervals for the true parameter assume that the size of the available set of sample values is suffiiently large, so that asymptoti results (entral limit theorem) an be applied. However, in many situations the sample size is neessarily limited, suh is the ase in traffi engineering mehanisms like MATE, where the probe paket load should not onsume signifiant network resoures. In MATE, we an use the bootstrap to obtain reliable estimates of the ongestion measures of the mean delay and ell loss rate from a given set of measurements obtained via the probe pakets. By seleting a desirable onfidene interval, we get a dynami way of speifying the number of observations needed. This provides a built-in reliability estimator whih automatially selets the required number of probe pakets to send. We have found this quite useful in our implementations, in omparison with shemes where the number of probe pakets is set in an ad-ho manner, and the number of probes may be too small or too large. The following is a basi proedure for omputing a onfidene interval: Step : Suppose the original sample is #k}. Step : Draw a random sample of m values, with replaement, from. This produes the bootstrap resample. Step 2: Calulate the mean for (say, ). Step : Repeat steps and 2 a large number of times to obtain n bootstrap estimates. Step 4: Sort the bootstrap estimates into inreasing order ^,. Step 5: The desired for the mean is, b]. bz, O8O bootstrap onfidene interval,a,, where #P T IV. EXPERIMENTAL METHODOLOGY, and # In this setion, we use simulations to evaluate the effetiveness of MATE. The objetive of our simulation study is to show that within a network that has multiple LSPs between some ingress and egress nodes, the traffi distribution under the MATE algorithm is stable, and load balaning is ahieved. We onentrate on two network topologies: one with a single ingress-egress pair onneted by multiple LSPs, and the other with multiple ingress-egress pairs where some links are shared among the LSPs from different pairs. Note that in the latter ase, there is a onsiderable interation between the pairs. In the fol-

6 b Aggregate Traffi on Link Aggregate Traffi on Link 2 I E Aggregate Traffi on Link Aggregate Traffi on Link Aggregate Traffi on Link 5 Aggregate Traffi on Link 6 Fig.. Experiment Network Topology I L E I2 L2 E2 Fig. 5. Offered load under Poisson traffi for network topology I L Fig. 4. Experiment Network Topology 2 lowing, we will present a desription of the simulator, the test networks we use, and the data olletion. We wrote a paket level disrete-event simulator, whih supports entities suh as paket queues, swithed LSPs, network onnetions. We onsider networking environments where the traffi onditions vary due to hanges in network load (link utilization), for example, due to rush hour onditions, or some LSP failures, and traffi variations due to orrelations and dependenies. We realize that we an not distribute pakets belonging to short lived network onnetions. E As a result, we speify two types of traffi in our simulator: engineered traffi and ross traffi. The engineered traffi is the traffi that needs to be balaned, and the ross traffi is the bakground traffi that we have no ontrol over. We assign a lifetime to eah traffi soure so we are able to simulate the dynami behaviors of a network by swithing on and off ross traffi soures. We onsider a traffi model whih exhibits short-range dependenies, suh as Poisson, and another model whih an be tuned to model a large degree of dependenies. For the latter we use the DAR( ) proess (disrete autoregressive proess of order ) [6]. The parameter determines the time-sale over whih traffi dependeny and orrelation are exhibited. If is, the proess is a standard Markov proess. In our experiments we set to a value of ; this leads to a substantial degree of orrelation in the generated traes. Figure and Figure 4 are the two network topologies used in our experiment. The first topology onsists of a single pair of ingress-egress nodes. There are 6 LSPs onneting the ingress node to the egress node, and all links are idential so that the LSPs have the same bottlenek link bandwidth. In the seond network topology, we have ingress nodes, I, I2, and I, and three egress nodes, E, E2, and E. Altogether, they form three pairs. The links in this network are again all idential. Eah ingress-egress pair has two LSPs for traffi balaning. We set up this senario so there is a ommon link for every two pairs. In eah of our simulations, the engineered traffi for eah pair flows from the ingress node to the egress node. The ross traffi enters at the intermediate node and exits at egress node(s). We onsider two implementations of the Loss Rate Loss Rate on Path Loss Rate on Path 2 Loss Rate on Path Loss Rate on Path 4 Loss Rate on Path 5 Loss Rate on Path Fig. 6. Loss under Poisson traffi for network topology basi algorithm. In the first one, a small random delay is introdued before the algorithm moves from the monitoring phase to the traffi engineering phase upon detetion of hange in traffi onditions. This damping mehanism redues synhronization among multiple ingress nodes. In the seond implementation, there is a oordination among the ingress nodes so that only one ingress node at a time enters the traffi engineering phase. This obviously requires a speial oordination protool. We omit the details in this paper. In order to do data olletion, we reord the total offered load and the loss rate on eah link. We ompute the loss rate on eah LSP from the link loss rates. The loss rate on an LSP an be omputed by assuming that the link loss rates are independent as follows: \ $\ $ # b $,, where the produt is taken over all links in the LSP. V. SIMULATION RESULTS In this setion, we show the results from the simulation of the two networks in the previous setion. These results are enouraging in that they show our algorithms have good stability and onvergene properties. First we present the results from a single ingress-egress pair. We show two sets of data for this senario. Figure 5 and Figure 6 show the results of an experiment with Poisson traffi on the network in Figure. Initially, all of the engineered traffi streams are routed on one of the LSPs, and ross traffi enter the

7 Aggregate Traffi on Link Aggregate Traffi on Link 2 Aggregate Traffi on Link Aggregate Traffi on Link 4 Aggregate Traffi on Link 5 Aggregate Traffi on Link 6 Loss Rate Loss Rate on Link Loss Rate on Link 2 Loss Rate on Link Fig. 7. Offered load under DAR traffi for network topology Fig.. Loss under Poisson traffi for network topology 2.5 Loss Rate on Path Loss Rate on Path Aggregate Traffi on Link Aggregate Traffi on Link 2 Aggregate Traffi on Link. Loss Rate on Path Loss Rate on Path 4 Loss Rate 5 Loss Rate on Path 5 Loss Rate on Path Fig. 8. Loss under DAR traffi for network topology Fig. 2. Offered load under Poisson traffi with oordination for network topology Cross Traffi on Link Cross Traffi on Link 2 Cross Traffi on Link Fig. 9. Cross Traffi for network topology 2 network at the intermediate nodes onneting the ingress and egress nodes. We have an unbalaned situation with one heavily ongested LSP and five lightly loaded LSPs. As shown in the plot, the algorithm is able to suessfully redue the engineered traffi from the overloaded link and distribute them to the underutilized links. The loss urve shows learly that the loss rate on the first LSP dropped from 4% to a value that is too small to observe. The loss rates on the other LSPs are maintained at negligible levels throughout the simulation. The final traffi distribution onverges to a steady state, where utilizations are very Note that loss rates on the order of % to 2% are not atypial in the Internet..4.2 Aggregate Traffi on Link Aggregate Traffi on Link 2 Aggregate Traffi on Link.5 Loss Rate on Link Loss Rate on Link 2 Loss Rate on Link..8.6 Loss Rate Fig.. Offered load under Poisson traffi for network topology Fig.. Loss under Poisson traffi with oordination for network topology 2

8 : Aggregate Traffi on Link Aggregate Traffi on Link 2 Aggregate Traffi on Link and marginal delays that are easily measurable and do not require lok synhronization. Our analytial models prove the stability and optimality of MATE. Our simulation results show that MATE an effetively remove traffi imbalanes among that may our among multiple LSPs. We observe that, in many ases, high paket loss rates an be signifiantly redued by properly shifting some traffi to less loaded LSPs. This should benefit many appliations suh as TCP. For future work we will onsider more realisti networking environments and examine the impat of MATE on the appliation level Fig. 4. Offered load under DAR traffi with oordination for network topology 2 Loss Rate Loss Rate on Link Loss Rate on Link 2 Loss Rate on Link Fig. 5. Loss under DAR traffi with oordination for network topology 2 lose on all LSPs. We observe similar behavior in Figures 7 and 8 where the Poisson streams are here replaed with DAR traffi streams that possess orrelation and dependene. We point out that the probe traffi required in the eah phase of the algorithm is around O of the engineered traffi, thereby ensuring the salability of the overall approah. The Figures - show the simulation senario for Figure 4 under the two implementations mentioned earlier. Again the engineered traffi streams travel from the ingress node to the egress node, and the ross traffi enters through the intermediate nodes and exit at the egress nodes. The ross traffi dynamis are shown in Figure 9. There is a derease in ross traffi on link right before 2 seonds and a inrease in ross traffi on link 2 around 6 seonds. In order to balane traffi, the algorithms must shift traffi into link and possibly out of link 2. Both implementations essentially ahieve the same performane, where utilizations and loss rates on three LSPs are omparable. Figure 4 and Figure 5 show the same simulation with DAR traffi instead of Poisson traffi where oordination among ingress node is onsidered. VI. CONCLUSIONS Our fous on this paper was to apply adaptive traffi engineering to utilize network resoure more effiiently and minimize ongestion. We have proposed a lass of algorithms alled MATE, whih tries to ahieve these objetives using minimal assumptions through a ombination of tehniques suh as bootstrap probe pakets, whih ontrol the amount of extra traffi, REFERENCES [] D. O. Awduhe, J. Malolm, J. Agogbua, M. O Dell, and J. MManus, Requirements for traffi engineering over mpls, RFC 272, Sep [2] D. Awduhe, A. Chui, A. Elwalidn, I. Widjaja, and X. Xiao, A framework for Internet traffi engineering, Internet draft draft-ietf-tewgframework-.txt, Mar. 2. [] D. O. Awduhe et. al., RSVP-TE: extensions to RSVP for LSP tunnels, Internet draft draft-ietf-mpls-rsvp-lsp-tunnel-5.txt, Feb. 2. [4] D. Bertsekas, Nonlinear programming, Athena Sientifi, 995. [5] D. Bertsekas and R. Gallager, Data networks, Prentie-Hall In., 2nd ed. edition, 992. [6] Dimitri P. Bertsekas and John N. Tsitsiklis, Parallel and distributed omputation, Prentie-Hall, 989. [7] R. Callon, P. Doolan, N. Feldman, A. Fredette, G. Swallow, and A. Viswanathan, A framework for multiprotool label swithing, Internet draft draft-ietf-mpls-framework-5.txt, Sep [8] R.L. Carter and M.E. Crovella, Measuring bottlenek link speed in paket-swithed networks, Tehnial Report BU-CS-96-6, Boston University, Mar [9] B. Fortz and M. Thorup, Internet traffi engineering by optimizing OSPF weights, Proeedings of INFOCOM 2, Tel-Aviv, Israel, Mar. 2. [] B. Jamousi et. al., Constraint-based LSP setup using LDP, Internet draft draft-ietf-mpls-r-ldp-.txt, Sep [] S. Keshav, A ontrol theoreti approah to flow ontrol, Proeedings of SIGCOMM 9, ACM, August 99. [2] T. Li and Y. Rekhter, Provider arhiteture for differentiated servies and traffi engineering (PASTE), RFC 24, Ot [] D. Mitra and K.G. Ramakrishnan, A Case Study of Multiservie, Multipriority Traffi Engineering Design for Data Networks, Pro. Globeom 99, De 999. [4] E. C. Rosen, A. Viswanathan, and R. Callon, Multiprotool label swithing arhiteture, Internet draft draft-ietf-mpls-arh-5.txt, Sep [5] M. A Rodrigues and K. G. Ramakrishnan, Optimal routing in shortestpath networks, ITS 94, Rio de Genero, Brazil. [6] B. K. Ryu and A. Elwalid, The importane of long-range dependene of VBR video traffi in ATM traffi engineering, In Proeedings of SIG- COMM 96, pages 4, Aug [7] K. Thompson, G.J. Miller, and R. Wilder, Wide-area internet traffi patterns and harateristis, IEEE Networks, 6(6), De [8] John N. Tsitsiklis and Dimitri P. Bertsekas, Distributed asynhronous optimal routing in data networks, IEEE Transations on Automati Control, (4):25 2, April 986. [9] A. M. Zoubir and B. Boashash, The bootstrap and its appliations in signal proessing, IEEE Signal Proessing Magazine, Jan Proof of Theorem APPENDIX Sine the ost funtion is onvex the first order optimality ondition is both neessary and suffiient: ^W is optimal if and only if W is feasible and there exist onstants suh that for all V, V. W,Y# :S $. W, N (2) with equality if W i O. Hene all LSPs nn with W i O have their first derivative lengths equal to.

9 : b ˆ ˆ ˆ 9 Proof of Theorem 2 Its proof is adapted from that in [8]. Let lj(\a,i#k+.\[],b we have for [(\a,. Using a first order Taylor expansion for : 2 some rate vetorv(\a, :>.[(\[],a,_# :>.+.\a,a,+] d>:>([.\a,,<l".\a,+] l".\a,ad :>.\a,,<l".\a, :>.+.\a,a,+] [.\a,alj(\a, ]>m m d>:>.+.\a,a,b [.\a, m m;!mgm l".\a, m m ] m m lj(\a,mgm () where ^.\a,e# K (\a, Y, and the onstant depends on the initial vetor [ O,. We next show that [.\a,<l".\a, m m d>:>.[(\a,a,?b [.\a, m m;8mgm l".\a, m m ˆXf Š for some onstant that depends on [ O,. mgm lj(\a,mgm m m l".\ S,mGm First, note that (4) holds if the following holds for all : (\a,<l.\a, b mgm l.\a,mgm (4) (5) (6) For \ n X (6) trivially holds. For \ % [ apply the projetion theorem [4] to (7) to obtain ([(\a,zbyx [.\a,zbn[(\[],a,t([(\a,zb%+.\[] Rearranging terms yields (6). To show (5) note that sine all norms in there exist onstants and suh that m m d>:>.\a,b [.\a,mgm,, O are equivalent (7) C C "K ([(\a,a,b.\a, C C : S..\a,a, $ b 4.\<S \a, :S 4.\<Sh,a, C ˆf J C 8 Š C C $ :S.J.\a,,Zb :S 4 f " (\<SG, Š, (8) # C C :S (4a(\a,a,Zb :S 4.\<Sh,a, f J.Š. Let =# } )m:>.", :>.+ O,, and =# } $98 V, for some -. In words, is the set of rate vetors at whih the total ost :>.V, is no greater than the initial ost. As will be seen, provided the stepsize is suffiiently small, :>.+.\a,, :>([ O,a, for all \ (see (2)). That is, is the set of all possible rate vetors given the initial [ O,. (This For simpliity we write instead of the more orret notation for the inner produt of two vetors and. We usually use to denote the Eulidean norm, but sometimes for emphasis. an be made more rigorous by indution.) Then is the set of all possible link flows on link. By ondition C2, we have for some onstants! Hene mgm d>:>(\a,b ^(\a,mgm C 4.\a,Zb 4a(\<Sh,! C f J.Š C. f J Š $98 (\a,zb.\<s S \<S,Ž.\<S S, C. Gˆ. h ".Š C C $! m.\a,b%.\<s S,m! C C $! mf " "4(\a,Zb%VJ.\+b Š J Š,m ] ; ;; ] m "4.\+b \ ],+bn"4(\+b \,m C C f ˆXf Š (9) m m l".\ S,mGm (2) m m d>:>.[(\a,a,?b [.\a, m m;8mgm l".\a, m m f. ˆf.Š mgm lj(\<sg, m m$; m m l".\a,mgm mgm lj(\<sg, m m where the last inequality ˆf follows Š from the fat that the onvex "]# $ attains its minimum of zero over }% Xm N O N O at the origin. This ompletes the proof of (5). Substituting (4 5) into () we have :>.+.\^] :>.+.\a,,b ] ˆf Summing over all \ we have :>.[(\[] :>.[ O,,b,a, ]. ˆ :>.[ O,,b m m l"4,mgm,a, Š b' m m lj(\ S, m m b' )(.Š m m l".\ S,mGm b' b' )( m m lj(\a,mgm m m lj 4,mGm \ I] Choose small enough suh that + b, {b,!, ( (2) \ ],Mi O. Sine [.\a, is in a ompat set and : is ontinuous, :>([(\a,a, is

10 o ˆ, # lower bounded. Then sine :>([(\a,a, is bounded for all \ we must mgm lj 4,mGm o, whih implies m m lj(\a,mgm O as \ (22) Substituting this into (2) we onlude that [.\a, d>:>.\a, as \ (2). One exists sine is in a ompat set. By (2) and the fat that : is ontinuously differentiable we have Let W be an aumulation point of } +.\a, } +.\a, ^(\a, DGC de:>(\a,%# d>:>. W, (24) Sine the time interval between updates is bounded, for any, we an find a subsequene } [.\, \n that onverges to VW, i.e., DGC ++.\8,# VW. Applying again the projetion theorem [4] to (7) we have for any feasible X..\,+b%X (\,Zbn (\ ],a,t( bn (\{] (l$(\,x] ".\,a,t(" bn"(\ ],a, O,a, N O Taking we have by (22) and (24) that for any feasible, d>: ( W,t( b D C.\ ],a, N O Sine lj(\a,#x+.\ ], b +.\a,,# D C +.\,# VW, and hene d>:2. W,t.V b% W, N O O by (22), we have DGC [.\ ] for any feasible. Summing over all, we have for any feasible whih, sine : be optimal. Proof of Theorem d>:>. W,. bn W, N O is onvex, is neessary and suffiient for [W to Sine the ost funtions : are globally Lipshitz uniformly in, the onstant in () equals the Lipshitz onstant. For any tuple l, m m lvm m?mgm lvm m, and hene the onstant in (7) is. Similarly, sine m m l"mgm?mgm lvm m, the onstant in (8) is. By Lipshitz ontinuity, the onstant in (9) is # #, the onstant following is # #. Finally, sine mgm lvm m mgm lvm m, we have ƒ#! # in (2). Hene from (2) an upper bound for the stepsize is ] \ I] ]J [ \ ],a,