The Synchrnizatin f Peridic Ruting Messages Sally Flyd and Van Jacbsn, Lawrence Berkeley Labratry, One Cycltrn Rad, Berkeley CA 9470, flyd@eelblgv, van@eelblgv T appear in the April 994 IEEE/ACM Transactins n Netwrking Abstract The paper cnsiders a netwrk with many apparently-independent peridic prcesses and discusses ne methd by which these prcesses can inadvertently becme synchrnized In particular, we study the synchrnizatin f peridic ruting messages, and ffer guidelines n hw t avid inadvertent synchrnizatin Using simulatins and analysis, we study the prcess f synchrnizatin and shw that the transitin frm unsynchrnized t synchrnized traffic is nt ne f gradual degradatin but is instead a very abrupt phase transitin : in general, the additin f a single ruter will cnvert a cmpletely unsynchrnized traffic stream int a cmpletely synchrnized ne We shw that synchrnizatin can be avided by the additin f randmizatin t the traffic surces and quantify hw much randmizatin is necessary In additin, we argue that the inadvertent synchrnizatin f peridic prcesses is likely t becme an increasing prblem in cmputer netwrks Intrductin A substantial, and increasing, fractin f the traffic in tday s cmputer netwrks cmes frm peridic traffic surces; eamples include the peridic echange f ruting messages between gateways r the distributin f real-time audi r vide Netwrk architects usually assume that since the surces f this peridic traffic are independent, the resulting traffic will be independent and uncrrelated Eg, even thugh each ruting prcess might generate a packet at fied, 30 secnd intervals, the ttal ruting traffic bserved at any pint in the netwrk shuld be smth and unifrm since the prcesses are n separate ndes and started with a randm relative phase Hwever, many netwrk traffic studies [Pa93a, SaAgGuJa9, Ja9, BrChClP93] shw that the ttal traffic is nt unifrm but instead is highly synchrnized This paper argues that the architect s intuitin that independent surces give rise t uncrrelated aggregate traffic is simply wrng and shuld be replaced by epectatins mre in line with bserved This wrk was supprted by the Directr, Office f Energy Research, Scientific Cmputing Staff, f the US Department f Energy under Cntract N DE-AC03-76SF00098 This paper is an epanded versin f [FJ93]
reality There is a huge bdy f research n the tendency f dynamic systems t synchrnize in the presence f weak cupling [Bl88] As far back as the mid-seventeenth century, Huygens nticed that tw unsynchrnized pendulum clcks wuld keep in time if hung n the same wall, synchrnized by the barely-perceptible vibratins each induced in the wall As reprted in [Bl88], synchrnizatin has been studied in electrnic circuits, a wide range f mechanical bjects, and bilgical systems such as cell ppulatins and cmmunities f fireflies Mst f these systems ehibit a tendency twards synchrnizatin that is independent f the physical cnstants and initial cnditins f the system [En9] This research suggests that a cmple cupled system like a mdern cmputer netwrk evlves t a state f rder and synchrnizatin if left t itself Where synchrnizatin des harm, as in the case f highly crrelated, bursty ruting traffic, it is up t netwrk and prtcl designers t engineer ut the rder that nature tries t put in This paper investigates ne means by which independent surces f peridic traffic can becme synchrnized An analytic mdel is develped that shares many f the features bserved in simulatins and in real traffic measurements There are tw main results frm this mdel: The transitin frm unsynchrnized t synchrnized behavir is very abrupt The traffic des nt gradually clump up and becme mre synchrnized as netwrk parameters change Instead, fr each set f prtcl parameters and implementatin interactin strengths there eists a clearly defined transitin threshld If the number f surces is belw the transitin threshld, the traffic will almst certainly be unsynchrnized and, even if synchrnized by sme eternal frce it will unsynchrnize ver time Cnversely, if the number f surces is abve the threshld, the traffic will almst certainly be synchrnized and, even if placed in an unsynchrnized state by sme eternal frce, will evlve t synchrnizatin ver time The amunt f randmness that must be injected t prevent synchrnizatin is surprisingly large Fr eample, in the Xer PARC internal netwrk, measurements [De93] shw their cisc ruters require rughly 300 ms t prcess a ruting message ( ms per rute times 300 rutes per update) Frm the results in Sectin 5, the ruters wuld have t add at least a secnd f randmness t their update intervals t prevent synchrnizatin There are many eamples f unanticipated synchrnized behavir in netwrks: TCP windw increase/decrease cycles A well-knwn eample f unintended synchrnizatin is the synchrnizatin f the windw increase/decrease cycles f separate TCP cnnectins sharing a cmmn bttleneck gateway [ZhCl90] This eample illustrates that unless we actively engineer t avid synchrnizatin, such as by injecting randmness int the netwrk, synchrnizatin is likely t be the equilibrium state As an eample f injecting randmness, the synchrnizatin f windw increase/decrease cycles can be avided by adding randmizatin t the gateway s algrithm fr chsing packets t drp during perids f cngestin[fj9] (This randmizatin has the advantage f aviding ther unintended phase effects as well) Synchrnizatin t an eternal clck Tw prcesses can becme synchrnized with each ther simply by bth being synchrnized t an eternal clck Fr eample, [Pa93a] shws DECnet traffic peaks n the hur and half-hur intervals; [Pa93b] shws peaks in ftp traffic as several users fetch the mst recent weather map frm Clrad every hur n the hur Client-server mdels Multiple clients can becme synchrnized as they wait fr service frm a busy r recvering server Fr eample, in the Sprite perating system clients check Eg, by restarting all the ruters at the same time because f a pwer failure
with the file server every 30 secnds; in an early versin f the system, when the file server recvered after a failure, r after a busy perid, a number f clients wuld becme synchrnized in their recvery prcedures Because the recvery prcedures invlved synchrnized timeuts, this synchrnizatin resulted in a substantial delay in the recvery prcedure [Ba9] Peridic ruting messages Unlike the client/server mdel r the eternal clck mdel, the synchrnizatin f peridic ruting messages invlves seemingly-independent peridic prcesses There are many ruting prtcls where each ruter transmits a ruting message at peridic intervals Assuming that the ruters n a netwrk are initially unsynchrnized, at first glance it might seem that the peridic messages frm the different ruters wuld remain unsynchrnized This paper eplres hw initially-unsynchrnized ruting messages can becme synchrnized We eamine the details f ruter synchrnizatin t give a cncrete eample f inadvertent synchrnizatin, t underline the necessity f actively designing t avid synchrnizatin, and t emphasize the utility f injecting randmizatin as a methd f breaking up synchrnizatin When a particular instance f synchrnizatin is bserved, it is usually easy t suggest prtcl changes that culd prevent it This misses the pint Synchrnizatin is nt a small prblem caused by minr versights in prtcl design The tendency f weakly-cupled systems t synchrnize is quite strng and changing a deterministic prtcl t crrect ne instance f synchrnizatin is likely t make anther appear Varius frms f peridic traffic are becming an increasingly-large cmpnent f Internet traffic This peridic traffic includes nt nly ruting updates and traffic resulting frm the increasing use f peridic backgrund scripts by individual users [Pa93a], but realtime traffic (such as vide traffic) that has a peridic structure Althugh the peridic structure f vide traffic is generally nt affected by feedback frm the netwrk, there are still pssibilities fr synchrnizatin Fr eample, individual variable-bit-rate vide cnnectins sharing a bttleneck gateway and transmitting the same number f frames per secnd culd cntribute t a larger peridic traffic pattern in the netwrk As peridic traffic increases in the Internet, it becmes increasingly imprtant fr netwrk researchers t cnsider questins f netwrk synchrnizatin We use bth simulatin and analysis t eplre the synchrnizatin f peridic ruting messages The first gal f the analysis is t eamine the rle that randm fluctuatins in timing play in the synchrnizatin f ruting messages These randm fluctuatins cntribute t bth the frmatin f synchrnizatin and t the breaking up f synchrnizatin after it ccurs One way t break up synchrnizatin is fr each ruter t add a (sufficiently large) randm cmpnent t the perid between ruting messages A secnd gal f ur analysis is t investigate this eplicit additin f a randm cmpnent t the ruting timer, and t specify the magnitude f the randm cmpnent necessary t prevent synchrnizatin Sectin gives eamples f peridic traffic patterns in the Internet; Sectin 3 describes ur mdel f peridic ruting messages n a netwrk Sectin 4 eplains the results f ur simulatins Sectin 5 describes a Markv chain used t analyze sme aspects f the Peridic Messages mdel Sectin 7 presents cnclusins and discusses alternatives fr preventing ruting message synchrnizatin 3
Peridic traffic patterns in the Internet This sectin gives an eample f synchrnized ruting messages, and several eamples f peridic traffic patterns in the Internet (sme f which are caused by peridic ruting messages) While we d nt have direct evidence f peratinal prblems in the Internet related t synchrnized ruting messages, we shw indirect evidence that such prblems culd eist In general, there are significant patterns f peridic packet drps and delays in the Internet We began this investigatin in 988 after bserving synchrnized ruting messages frm DECnet s DNA Phase IV (the DIGITAL Netwrk Architecture) [VMS88] n a lcal Ethernet at LBL (Lawrence Berkeley Labratry) Each DECnet ruter transmitted a ruting message at 0- secnd intervals Within hurs after bringing up the ruters n the netwrk after a failure, the ruting messages frm the varius ruters were cmpletely synchrnized In May 99, in the curse f investigating packet lss rates in the Internet, we cnducted eperiments sending runs f a thusand pings each, at rughly ne-secnd intervals, frm Berkeley and ther sites t destinatins acrss the Internet Fr all f the runs t destinatins at Harvard r MIT, at least three percent f the ping packets were drpped, regardless f the time f day Figure shws a particular run f a thusand pings frm Berkeley t MIT; the -ais shws the ping number and the y-ais shws the rundtrip time Drpped packets are represented by a negative rundtrip time Figure shws the autcrrelatin functin fr the rundtrip times in Figure, where the drpped packets are assigned a rundtrip time f tw secnds (higher than the largest rundtrip time in the eperiment) The pattern f peridic packet drps at 90-secnd intervals is illustrated in bth figures Further eperiments determined that these packet drps were ccurring at the NEARnet (New England Academic and Research Netwrk) cre ruters Earlier investigatin f Internet behavir had als reprted a degradatin in service with a 90-secnd peridicity n paths t MIT [SaAgGuJa9] These packet drps were determined t be caused by IGRP (the Inter-Gateway Ruting Prtcl [He9]) ruting updates at the NEARnet ruters [Sc9] The ruters were unable t frward ther packets while large ruting updates were being prcessed The particular prblem f peridic packet lsses n NEARnet has since been reslved; the ruter sftware has been changed s that nrmal packet ruting can be carried ut while the ruters are dealing with ruting update messages Althugh it has been speculated that these packet drps were als cnnected with synchrnizatin, it is unclear, and there is n direct evidence [Sc9, L93] Peridic packet drps have been demnstrated assciated with RIP (the Ruting Infrmatin Prtcl [He88]) as well as with IGRP Figure 3 shws audi packet lsses during an audicast 3 f the December 99 Packet Vide wrkshp [Ja9] The -ais shws the time in secnds; the y-ais shws the duratin f each audi utage in secnds The little blips mre-r-less randmly spread alng the time ais represent single packet lsses The larger lss spikes are strngly peridic; they ccur every 30 secnds and last fr several secnds at a time During these events the packet lss rate ranges frm 50 t 85% and there are frequent single utages f 00-500 ms These peridic lsses are almst certainly due t the surce-ruted (tunneled) multicast packets cmpeting with ruting updates and lsing Because 30 secnds is the default update time fr RIP, these lng intervals f packet lsses are cnjectured t result frm RIP ruting updates; it is nt knwn if this prblem invlves synchrnizatin In ther instances peridic 30-secnd audi 3 Fr a reprt n the first such audicast, see [Ca9] 4
Rundtrip time (secnds) -05 00 05 0 5 0 00 400 600 800 000 Ping number Figure : Peridic packet lsses frm IGRP ruting messages Autcrrelatin 00 0 04 06 0 50 00 50 00 Pings Figure : The autcrrelatin f rundtrip times Audi lst (secnds) 00 05 0 5 0 00 00 300 400 500 600 Time (secnds) Figure 3: Peridic packet lsses at 30-secnd intervals packet lsses have been cnclusively traced t RIP ruting updates [De93], and there is sme indirect evidence f synchrnizatin In ur ping eperiments f the Internet in May 99 we fund many eamples f peridic packet drps fr which we have n eplanatin Fr eample, we fund paths with packet drps 5
every 38 secnds, paths with packet drps every 5 secnds, and paths with large delays every 45 secnds We fund different peridic patterns n the lcal path frm LBL t the UC Berkeley campus at different times f the day Frm ur ping eperiments, we cnjecture that a significant number f packet drps in the Internet are assciated with peridic prcesses f ne type r anther 3 The Peridic Messages mdel This sectin describes a general mdel f peridic ruting messages n a netwrk; we call this the Peridic Messages mdel This mdel was initially patterned after DECnet s DNA Phase IV, but ther ruting prtcls that can cnfrm t this mdel include EGP (Eterir Gateway Prtcl) [M84], Hell [Mi83], IGRP, and RIP In these ruting prtcls, each ruter n a netwrk transmits a ruting message at peridic intervals This ensures that ruting tables are kept up-t-date even if ruting update messages are ccasinally lst The Peridic Messages mdel behaves as fllws: The ruter prepares and sends a ruting message In the absence f incming ruting messages, the ruter resets its timer secnds after step begins Other ruters receive the first packet f this ruter s ruting message secnds after step begins If the ruter receives an incming ruting message (r the first packet f an incming ruting message) while preparing its wn utging ruting message, the ruter als prcesses the incming ruting message The ruter takes secnds t prcess an incming ruting message 3 After cmpleting steps and, the ruter sets its timer The time until the timer net epires is unifrmly drawn frm the interval [, ] secnds, where is the average perid and represents a randm cmpnent; this culd be a (small) randm fluctuatin due t unavidable variatins in perating system verhead r a (larger) fluctuatin due t a randm cmpnent intentinally added t the system When the timer epires, the ruter ges t step 4 If the ruter receives an incming ruting message after the timer has been set, the incming ruting message is prcessed immediately If the incming ruting message is a triggered update caused by a majr change in the netwrk such as the failure f a link, then the ruter ges t step, withut waiting fr the timer t epire Because the ruter resets its timer nly after prcessing its wn utging ruting message and any incming ruting messages, the timing f ne ruter s ruting messages can be affected by the ruting messages frm ther ndes This gives the weak cupling between ruters, allwing the synchrnizatin f ruting messages frm several ruters The Peridic Messages mdel ignres prperties f physical netwrks such as the pssibility f cllisins and retransmissins n an Ethernet The Peridic Messages mdel is nt intended t replicate the eact behavir f peridic ruting messages, but t capture sme significant characteristics f that behavir RIP and IGRP are intradmain ruting prtcls that use peridic ruting messages In RIP each ruter transmits peridic ruting messages every 30 secnds In IGRP, ruters send ruting messages at 90-secnd intervals EGP (Eterir Gateway Prtcl) is used in sme places between the NSFNET backbne and its attached reginal netwrks; EGP ruters send update messages every three minutes 4 In the 4 With BGP (Brder Gateway Prtcl), which runs n tp f TCP, incremental update messages are sent as the 6
988 LBL netwrk, DECnet ruters implementing DNA Phase IV sent ruting messages every tw minutes IGRP, RIP, and DECnet s DNA Phase IV all incrprate triggered updates, where ruting messages are sent immediately in respnse t a netwrk change such as the remval f a rute The first triggered update results in a wave f triggered updates frm neighbring ruters Nt all implementatins f these ruting prtcls crrespnd t the Peridic Messages mdel in this paper The RFC fr RIP [He88] mentins that when there are many gateways n a single netwrk, there is a tendency fr the peridic ruting messages t synchrnize The RFC specifies that in rder t avid this synchrnizatin, either the ruting messages must be triggered by a clck that is nt affected by the time required t service the previus message, r a small randm time must be added t the 30-secnd ruting timer each time, thugh the magnitude f the randm time is nt specified As an eample f implementatins that dn t cnfrm t the Peridic Messages mdel, in sme implementatins f IGRP and RIP ruters reset their ruting timers befre the utging ruting message is prepared, and ruters dn t reset their ruting timers after triggered updates [Li93] Thus the Peridic Messages mdel illustrates nly ne pssible mechanism by which ruting messages can becme synchrnized Wherever there are interactins between ruters, r between a ruter and the netwrk, there culd eist mechanisms that lead t synchrnizatin 4 Simulatins This sectin describes simulatins f the Peridic Messages mdel These simulatins shw the behavir f a netwrk with N ruting ndes n a single bradcast netwrk, fr 0 In the first set f simulatins the peridic ruting messages fr the N ndes are initially unsynchrnized; in the secnd set the peridic messages are initially clustered The simulatins shw that the behavir f the Peridic Messages system is determined by the randm verhead added t each nde s peridic timer As the level f randmizatin increases, the system s ability t break up clusters f synchrnized ruting messages als increases Definitins: T p, T r, T c, T c, and T d The time is the cnstant cmpnent f the peridic timer and is the magnitude f the randm cmpnent Each ruter s ruting timer is drawn at each rund frm the unifrm distributin n [ -, ] secnds Each ruter requires secnds f cmputatin time t prcess an utging ruting message, and secnds f cmputatin time t prcess an incming ruting message; each ruting message culd cnsist f multiple packets In this paper we assume that and are the same secnds after a ruter s ruting timer epires, ther ruters receive the first packet f the ruting message Fr the simulatins in this sectin, is secnds, is 0 secnds, and is set t zer; fr the initial simulatins in this sectin is set t 0 secnds The average timer-value f secnds was chsen t give a minimum timer-value cmparable t the 0-secnd timer used by the DECnet ruters n ur lcal netwrk The value f 0 secnds fr was chsen smewhat arbitrarily t mdel an estimated cmputatin time f 0 secnds and transmissin time f 00 secnds fr a ruter t cmpute and transmit packets fr an utging ruting message after a timer epiratin; these values are nt based n any measurements f actual netwrks Sectin 53 discusses hw the results scale with different values fr the varius parameters ruting table changes 7
Time (secnds) Time ffset (secnds) 0 0000 40000 60000 80000 00000 0 0 40 60 80 00 0 (86 runds in all) Figure 4: A simulatin shwing synchrnized ruting messages Time (secnds) Time ffset (secnds) 35500 36000 36500 37000 37500 38000 38500 455 460 465 470 : timer is set : timer epires A: B: Figure 5: An enlargement f the simulatin abve When a nde s ruting timer epires, the nde takes secnds t prepare and transmit its ruting message We call this time the busy perid Fr each ruting message received while a nde is in its busy perid, that nde s busy perid is etended by the secnds required t prcess an incming ruting message Fr simplicity, in the simulatins in this sectin is set t zer; that is, when nde A s timer epires the ther ndes immediately receive the first packet f nde A s ruting message Thus in the simulatins, when nde A s timer epires nde A immediately spends secnds preparing and transmitting its ruting message, and at the same time the ther ruting ndes each spend secnds receiving and prcessing the ruting message frm nde A This assumptin mst plausibly reflects a netwrk with lw prpagatin delay, where a ruter s ruting message cnsists f several packets transmitted ver a -secnd perid Sectin 54 shws the results f simulatins with 0 The first set f simulatins investigates the prcess by which initially-unsynchrnized ruting 8
Largest cluster 5 0 5 0 0 0000 40000 60000 80000 00000 Time (secnds) Figure 6: The cluster graph, shwing the largest cluster fr each rund messages becme synchrnized The ruting messages fr the ndes are initially unsynchrnized; fr each nde the time at which the first ruting message is sent is chsen frm the unifrm distributin n [0, ] secnds Fr the simulatin in Figure 4, is set t 0 secnds Each jittery line in Figure 4 is cmpsed f hundreds f pints, and each pint represents ne ruting message sent by a ruting nde The -ais shws the time in secnds at which the ruting message was sent, and the y-ais shws the time-ffset, ie, the time mdul, fr secnds This time-ffset gives the time that each ruting message was sent relative t the start f each rund The simulatin in Figure 4 begins with unsynchrnized ruting messages and ends with the N=0 ruters transmitting their ruting messages at essentially the same time each rund At the left-hand side f the figure the twenty jittery lines represent the time-ffsets f the transmit times fr the twenty ndes In the absence f synchrnizatin each ruter s timer epires, n the average, secnds after that ruter s previus timer epiratin These successive timer epiratins give a jittery but generally hrizntal line fr the timer epiratins fr a single ruter Hwever, as we eplain belw, when ruters becme synchrnized this increases the time interval between successive ruting messages frm a single ruter At the end f the simulatin the ruting messages are fully synchrnized, and all f the ndes set their timers at the same time each rund In this case each ruter has a busy perid f 0 secnds rather than f secnds, increasing the time interval between successive ruting messages Figure 5 is an enlargement f a small sectin f Figure 4 This figure illustrates the synchrnizatin f ruting messages frm tw ruters; each marks a timer epiratin, and each marks the timer being reset In the first five runds f Figure 5 the tw ndes are independent, and each nde sets its timer eactly secnds after its previus timer epires Hwever, in the sith rund, nde A s timer epires, say, at time, and nde A begins preparing its ruting message Befre nde A finishes preparing and sending its ruting message, nde B s timer epires; nde A has t finish sending its wn ruting message and t prcess nde B s ruting message befre it can reset its wn timer These tw tasks take secnds, s nde A resets its timer at time In ur mdel nde B begins prcessing nde A s ruting message at time, and in the simulatin in Figure 5 is set t zer While nde B is receiving and prcessing nde A s ruting message, nde B s wn timer epires; nde B has t prepare and send its wn utging ruting message and finish prcessing nde A s ruting message befre resetting its timer These tasks take secnds, s fr 0 nde B als resets its timer at time At this pint nde A and nde B are synchrnized and we say that they frm a cluster; nde A and 9
nde B set their timers at the same time The tw ndes remain synchrnized, setting their timers at rughly the same time, as lng as the timers epire within secnds f each ther each rund The cluster breaks up again when, because f the randm cmpnent, nde A and nde B s timers epire mre than secnds apart Mre generally, a cluster f size refers t a set f ruting messages that have becme synchrnized Each f the ndes in a cluster is busy prcessing incming ruting messages and preparing its wn utging ruting message fr secnds after the first timer in the cluster epires Fr 0, the ndes in a cluster reset their timers at eactly the same time One way t think f the simulatin in Figure 4 is as a system f N particles, each with sme randm mvement in a ne-dimensinal space Fr a particle in a lne cluster (a cluster f size ne), each timer-ffset differs frm the previus rund s timer-ffset by an amunt drawn frm the unifrm distributin n [-, ] secnds In Figure 4 the successive timer-ffsets fr an unsynchrnized ruting nde (the mvement f a single particle) are represented by a jittery but generally hrizntal line Fr particles (r ruting ndes) in a cluster f size, secnds are spent prcessing ruting messages after the first timer f the cluster epires; then the ndes in the cluster all reset their timers A cluster f particles mves ahead a distance f rughly secnds in each rund In Figure 4 the mvement f a cluster is represented by an irregular line with psitive slpe; the larger the cluster, the steeper the slpe When tw clusters meet, the ndes in the tw clusters all reset their timers at the same time; the tw clusters merge, fr the mment, int a larger cluster As Figure 4 shws, a cluster f particles can smetimes break up int tw smaller clusters Even thugh the ndes set their ruting timers at the same time, it is pssible fr ne nde s ruting timer t epire mre than secnds befre any f the ther ndes in the cluster, because f the randm cmpnent in the timer interval fr each nde When this happens, the first nde breaks ut f the cluster, as discussed further in Sectin 5 The break-up f a cluster can be seen in Figure 5 where a cluster f size tw frms and then breaks up again The first part f the simulatin in Figure 4 shws small clusters ccasinally frming and breaking up Twards the end f the simulatin a sufficiently-large cluster is frmed, mving rapidly acrss the space and incrprating all f the unclustered ndes that it encunters alng its path As the cluster size increases, the average perid f the cluster als increases; the larger the cluster, the mre quickly it bumps int and incrprates the smaller clusters A simulatin at any pint in time can be partially characterized by the size f the largest cluster f ruting messages Figure 6 shws a cluster-graph f the simulatin in Figure 4 The -ais shws time and the y-ais shws the size f the largest cluster in the current rund f N ruting messages Figure 7 shws a simulatin identical t that in Figure 4, ecept that the simulatin was started with a different randm seed Unlike the simulatin in Figure 4, the simulatin in Figure 7 ends with unsynchrnized ruting messages Fr the simulatin in Figure 7, a cluster as large as five ccasinally frms but each time the cluster breaks up again Figure 9 shws the cluster graphs frm several simulatins that start with unsynchrnized ruting messages The parameters are the same as the previus simulatins, ecept that the randm cmpnent ranges frm 0 6 t 4 Nte that the time scale is different frm the cluster graphs n previus pages; in Figure 9 the simulatins run fr 0 7 secnds (5 days) instead f 0 5 0
Time (secnds) Time ffset (secnds) 0 0000 40000 60000 80000 00000 0 0 40 60 80 00 0 (86 runds in all) Figure 7: A simulatin shwing unsynchrnized ruting messages Time (secnds) Largest cluster 0 0000 40000 60000 80000 00000 5 0 5 0 Figure 8: The cluster graph, shwing the largest cluster fr each rund secnds (just ver day) As the randm cmpnent increases, the simulatins take lnger and lnger t synchrnize These simulatins d nt specifically include triggered updates, triggered by a change in the netwrk We can instead begin ur simulatins with synchrnized ruting messages, which can result frm triggered updates These simulatins are shwn in Figure 0; the randm cmpnent ranges frm 3 t 8 As the randm cmpnent increases, the simulatins return mre quickly t the unsynchrnized state Our simulatin results are cnsistent with simulatins f the same mdel in [Tr9] In additin t simulatins, preliminary results frm eperiments by Treese have shwn synchrnizatin f systems n an Ethernet [Tr94] The eperiments use an algrithm similar t the Peridic Messages mdel The results suggest that the Peridic Messages mdel captures a realistic pssible behavir f real cmputer systems
Tr = 06 Tc Time Largest cluster 0 e0 4 6 8 e7 5 0 5 0 synchrnizatin after 498 runds (7 hurs) Tr = Tc Time Largest cluster 0 e0 4 6 8 e7 5 0 5 0 synchrnizatin after 7,798 runds ( days) Tr = 4 Tc Time Largest cluster 0 e0 4 6 8 e7 5 0 5 0 n synchrnizatin Figure 9: Simulatins starting with unsynchrnized updates, fr different values fr Tr = 3 Tc Time Largest cluster 0 e0 4 6 8 e7 5 0 5 0 synchrnizatin nt brken Tr = 5 Tc Time Largest cluster 0 e0 4 6 8 e7 5 0 5 0 synchrnizatin brken after 4,79 runds (7 days) Tr = 8 Tc Time Largest cluster 0 e0 4 6 8 e7 5 0 5 0 synchrnizatin brken after 300 runds (0 hurs) Figure 0: Simulatins starting with synchrnized updates, fr different values fr 5 The Markv chain mdel This sectin uses a Markv chain mdel t further eplre the behavir f the Peridic Messages system The Markv chain eplres the behavir f a system f N ruters that each implement the Peridic Messages mdel described in the previus sectin The Markv chain mdel assumes that each ruter receives a peridic ruting message frm every ther ruter; this wuld be the case, fr eample, fr N ruters n a bradcast netwrk Sectin 56 discusses the issues in etending these results t N ruters cnnected in an arbitrary tplgy, Sectin 54 discusses the effects f a nnzer transmissin and prpagatin delay between ruters, and Sectin 6 discusses ther analytical appraches t synchrnizatin The Markv chain mdel is used t cmpute the epected time fr the system t mve frm an unsynchrnized state t a synchrnized state, and vice versa This Markv chain mdel uses several simplifying assumptins, and therefre nly apprimates the behavir f the Peridic Messages mdel Nevertheless, the Markv chain mdel illustrates sme significant prperties f the simulatins f the Peridic Messages mdel The Markv chain has N states; when the largest cluster frm a rund f N ruting messages is f size, the Markv chain is defined t be in state Figure shws the Markv chain, alng with the transitin prbabilities The transitin prbability is the prbability that a Markv chain in state mves t state in the net rund The Markv chain mdel is based n several simplifying assumptins: The first simplifying assumptin f the Markv chain mdel is that the future behavir f
p, p, pi,i p N,N p, p,3 p pi,i p size size size size i N p, p3, p pi,i p Figure : The Markv chain the system depends nly n the current state and is independent f past states This assumptin is clearly nt true fr the Peridic Messages mdel, where the future behavir f the system depends nt nly n the size f the largest cluster but n the transmit times f the ther ruting messages The secnd simplifying assumptin is that the size f the largest cluster changes by at mst ne frm ne rund t the net Again, this assumptin is nt strictly accurate, particularly fr large values f r Fr eample, in the Peridic Messages mdel it is pssible fr tw clusters f sizes and respectively t merge and frm a cluster f size in the net rund The analysis f the Markv chain mdel assumes that ecept fr the largest cluster f size, all ther clusters are lne clusters f size ne; again, this cnservative assumptin is nt strictly accurate Given a cluster f size, the fllwing cluster is defined as the cluster that fllws the cluster f size in time At each rund, we assume that the distance between the largest cluster f size and the fllwing lne cluster is given by an epnential randm variable with epectatin This distance is defined as the wait between the time when the ndes in the cluster f size set their timer and the time when the timer epires fr the nde in the fllwing lne cluster This epected value is based n the average distance between clusters As in the Peridic Messages mdel, we assume that each nde s timer epires after a time drawn frm the unifrm distributin n [ -, ] secnds Fr a nde in a cluster f size, the nde takes secnds t prcess the incming and utging ruting messages in the cluster, and ther ndes receive the first packet f the ruting message secnds after the timer epires In this sectin we assume that ; if nt, then a cluster never breaks up int smaller clusters The net tw sectins define the transitin prbabilities fr the Markv chain Given these transitin prbabilities, we cmpute the average time fr the Markv chain t mve frm state t state N, and the average time fr the Markv chain t mve frm state N back dwn t state This analysis shws that when is sufficiently large, the Markv chain mves quickly frm a synchrnized state t an unsynchrnized state 5 Cluster breakup and grwth This sectin estimates, the prbability that the Markv chain mves frm state t state in ne rund The secnd half f this sectin estimates In the Markv chain, a cluster f size can break up t frm a cluster f size either by breaking up int a cluster f size ne fllwed by a cluster f size, r vice versa Because the first f the tw cases is mre likely, 5 fr simplicity we nly cnsider this case We say that the first nde breaks away frm the head f the cluster 5 The secnd f the tw cases ccurs nly if the last nde transmits its ruting message after it has had time t prcess ruting messages frm all previus ndes in the cluster 3
Thus is the prbability that the nde whse timer epires first, nde A, resets its timer befre it receives any ruting messages frm any f the ther ndes in the cluster Fr ndes in a cluster, the timers are all set within secnds f each ther; in this analysis we estimate that the timers are all set at the same time and the timers epire at times unifrmly distributed in a time interval f length Let L be the time frm the epiratin f the first timer until the epiratin f the secnd f the timers In the absence f incming messages, nde A resets its timer secnds after its timer epires, and receives ntificatin f a ruting message frm anther nde in the cluster secnds after its timer epires Because we assume that, there is always a nnzer prbability that a cluster f size breaks up int smaller clusters Frm [F6, p], () fr Nw we estimate, the prbability that the system mves frm state t state in ne rund We leave as a variable; depends largely n, the randm change in the timer-ffsets frm ne rund t the net Fr simplicity, this analysis assumes that The prbability that a cluster f size tw r mre incrprates additinal ruting ndes, frming a larger cluster, depends largely n the fact that larger clusters have larger average perids than smaller clusters After sme time the larger cluster cllides with a smaller cluster, and the tw clusters merge Fr a cluster f size, each nde in the cluster sets its timer secnds after the first timer in the cluster epires (r after it receives the first packet frm that nde s ruting message) Fr 0, each f the timer epiratins is unifrmly distributed in the interval [, ] Given events unifrmly distributed n the interval [0, ], the epected value f the smallest event is [F6, p4] Thus the first f the timers epires, n average, secnds after the timers are set The average ttal perid fr a nde in a cluster f size is therefre secnds In ne rund the timer-ffset fr a cluster f size mves an average distance f secnds relative t the timer-ffset fr a cluster f size ne Fr simplicity, in estimating we assume that the timer-ffset fr a cluster f size mves in each rund eactly secnds relative t the timer-ffset fr a cluster f size ne (This assumptin ignres the smewhat remte pssibility that a cluster f size culd jump ver a smaller cluster) What is the prbability that, after ne rund, the timer-ffset fr a cluster f size mves t within secnds f the timer-ffset fr a cluster f size ne? The Markv chain mdel assumes that the distance between a cluster f size and the fllwing small cluster is an epnential randm variable with epectatin Thus fr a Markv chain in state, is the prbability that an epnential randm variable with epectatin is less than Fr, this gives Fr all, ( )( ) () 4
5 Average time t cluster, and t break up a cluster This sectin investigates the average time fr the Markv chain t mve frm state t state N, and vice versa Definitins: t i j and f i Let be the epected number f runds until the Markv chain mves frm state t state, given that it mves frm state directly t state Let be the epected number f runds until the Markv chain first enters state, given that the Markv chain starts in state We leave as a variable We give a recursive definitin fr fr The epected number f runds t first reach state equals the epected number f runds t first reach state, plus the additinal epected number f runds, after first entering state, t enter state After state is first reached, the net state change is either t state, with prbability, r t state, with prbability The epected number f runds t reach state, after first entering state, is This leads t the fllwing recursive equatin fr : Thus fr, (3) Frm Appendi A, equatin 3 has the slutin 6 : 3 3 3 Cnsider, the epected number f runds t mve frm state t state, given that the Markv chain in fact mves frm state t state Let be the prbability that the Markv chain in state first mves t state n rund, given that the Markv chain mves frm state t state The equatin fr is as fllws [R85, p37]: (4) Similarly, the equatin fr is as fllws: (3) 6 This slutin culd als be verified by the reader by substituting the right-hand side f Equatin (4) int Equatin 5
Net we investigate the average time fr the Markv chain t mve frm state N t state Definitins: g i Let be the epected number f runds fr the Markv chain t first enter state, given that the Markv chain starts in state N Thus 0 and Fr, this gives the recursive equatin (5) Equatin 5 has the slutin belw: (6) The derivatin f this equatin is similar t that f n the values f r f Nte that this equatin des nt depend Cluster size 0 5 0 5 0 0 00000 400000 600000 Time (in secnds) t reach given cluster size, frm size (Slid line frm analysis, dtted lines frm simulatins) Figure : The epected time t reach cluster size, starting frm cluster size, fr 0 secnds The slid line in Figure shws, cmputed frm Equatin 4, fr 0, secnds, 0 secnds, 0 secnds, 0 secnds, and 9 runds (This value fr is based bth n simulatins and n an apprimate analysis that is nt given here 7 ) The -ais shws the time in secnds, cmputed as The y-ais shws the cluster size ; a 7 The dynamics fr mving frm a cluster f size tw r mre t a larger cluster is based largely n the fact that larger clusters have larger average perids In cntrast, the dynamics fr mving frm a cluster f size ne t a cluster f size tw depends n hw frequently tw clusters f size ne cllide, where all clusters f size ne have the same average perid; this requires a different analysis 6
Cluster size 0 5 0 5 0 0 50000 00000 50000 00000 50000 300000 Time (in secnds) t reach given cluster size, frm size 0 (Slid line frm analysis, dtted line frm simulatins) Figure 3: The epected time t reach cluster size, starting frm cluster size N, fr 0 3 secnds mark is placed at cluster size when the system first reaches that cluster size The results f twenty simulatins are shwn by light dashed lines Each simulatin was started with unsynchrnized ruting messages, with the values fr,,,, and described abve; these simulatins differ nly in the randm seed The heavy dashed line shws the results averaged frm twenty simulatins The slid line in Figure 3 shws, cmputed frm Equatin 6, fr the same parameters fr,,, and as in Figure, and fr 0 30 secnds; fr the value f in Figure, the system takes a lng time t unsynchrnize, making simulatins unrealistic The heavy dtted line averages the results frm twenty simulatins Figures and 3 shw that the average times predicted by the Markv chain are tw r three times the average times frm the simulatins This discrepancy is nt surprising, because the Markv chain is nly a rugh apprimatin f the behavir f the Peridic Messages system Aside frm the difference in magnitude, hwever, the functins predicted frm the Markv chain and cmputed frm the simulatins are reasnably similar Thus the Markv chain mdel des in fact capture sme essential prperties f the Peridic Messages system 53 Results frm the Markv chain mdel This sectin eplres the general behavir f the Markv chain mdel We cmpute the epected time fr the Markv chain t synchrnize and t unsynchrnize, fr a range f values fr,, and, and cmpare these analytical results t the results f simulatins This cmparisn shws that the Markv chain mdel is eplanatry rather than predictive; the Markv chain mdel and the simulatins ehibit the same qualitative behavir, and the Markv chain mdel can be used t eplain the behavir f the simulatins, but the Markv mdel is nt sufficiently accurate t predict the eact results f the simulatins The analysis in this sectin, alng with the simulatins, shws that fr a wide range f parameters, chsing as a small multiple f ensures that the system is almst always unsynchrnized The analysis further shws that fr fied values fr and, the transitin t synchrnizatin is an abrupt functin f the number f ndes Finally, in this sectin we cnsider a system f ruters 7
in an arbitrary tplgy, where each ruter nly receives peridic messages frm its immediate neighbrs We suggest that the mdel f synchrnized ruting messages in this paper is likely t hld in arbitrary tplgies nly fr cnnected subsets f ndes with similar degree Figure 4 cnsiders the epected time fr the Markv chain t synchrnize r t unsynchrnize, as a functin f the parameter Figure 4 gives, frm Equatin 4, and, frm Equatin 6, fr ranging frm zer t 4 5, given 0, = secnds, = 0 secnds, and 0 secnds The slid line n the right shws the epected time fr the Markv chain t mve frm state N t state ; the slid line n the left shws the epected time fr the Markv chain t mve frm state t state N The dtted line n the left was cmputed using values fr based n an apprimate analysis that is nt given here; the slid line n the left uses set t zer Fr 0 5, clusters never break up nce they have frmed, and the time t synchrnize depends largely n the time t first frm a cluster f size tw; this time increases as appraches 0 8 Nte that the y-ais is n a lg scale, and ranges frm less than 0 3 secnds (rughly 6 minutes) up t 0 secnds (ver 3 thusand years) e0 lw mderate high Average time e9 e8 e3 e7 e5 e4 0 3 4 Randm Nise Tr (as a multiple f Tc) Tc,Tc=00 secnds, Td=0 secnds, N=0 Figure 4: Epected time t g frm cluster size t cluster size N, and vice versa, as a functin f Figure 4 can be used as a general guide in chsing a sufficiently large value f, given the values fr the ther parameters in a system, s that the system mves easily frm state N t state and rarely mves frm state back t state N The figure shws the regins f lw, mderate, and high randmizatin In the regin f lw randmizatin the system mves easily frm state t state N; in the regin f high randmizatin the system mves easily frm state N t state In the regin f mderate randmizatin the system takes a significant perid f time t mve either frm state t state N, r frm state N back t state In the lw and mderate regins, the epected time fr the Markv chain t mve frm state t state N, grws epnentially with The X marks n Figure 4 shw simulatins that start with unsynchrnized ruting messages and the marks shw simulatins that start with synchrnized ruting messages Figure 5 shws the same analytical results as in Figure 4 fr the number f ndes N ranging frm 0 t 30, and fr a range f values fr These simulatins were perfrmed t check hw 8 As Figure 4 shws, fr etremely small values f there is little randmness in the system, and it can take sme time fr tw ndes t first frm a cluster 8
Average time e e9 e8 e7 e5 e4 e3 N=0 N=30 N=0 # # # N=0 N=30 # N=0 * * * * * * * * * * * 0 4 6 8 Randm Nise Tr (as a multiple f Tc) Tc=000 secnds (N=0: *, # N=0:, N=30: ) Average time e e9 e8 e7 e5 e4 e3 N=0 N=0 N=30 N=30 - - - - - - - - - - N=0 * * * N=0 * * * * * * * * * * * 0 4 6 8 Randm Nise Tr (as a multiple f Tc) Tc=00 secnds (N=0: *, - N=0:, N=30: ) Average time e e9 e8 e7 e5 e4 e3 N=0 N=30 # # # # N=0 N=0 # # N=30 # # # # # # # # * * * N=0 * * * * * * * * 0 4 6 8 Randm Nise Tr (as a multiple f Tc) Tc=0500 secnds (N=0: *, # N=0:, N=30: ) Figure 5: Epected time t g frm cluster size t cluster size N, and vice versa, as a functin f N and f accurately the analytical results predict the simulatin results fr a range f parameters Nte that fr larger values f and f, the analytical results significantly verestimate the time required by the simulatins t g frm state N t state The analytical results use the simplifying assumptin that the size f the largest cluster changes by at mst ne frm ne rund t the net 9
As the parameters and increase, this assumptin becmes less applicable The figures shw that fr a wide range f parameters, chsing at least ten times greater than ensures that clusters f ruting messages will be quickly brken up Fr any range f parameters, chsing as shuld eliminate any synchrnizatin f ruting messages This wuld be equivalent t setting the ruting timer each time t an amunt frm the unifrm distributin n the interval [0 5 5 ] secnds This intrduces a high degree f randmizatin int the system, yet ensures that the interval between ruting messages is never t small r t large 54 Incrprating delays between ruters The analysis and simulatins in the paper s far have assumed that 0; that is, that when a nde s timer epires, ther ruters are immediately ntified f the timer epiratin While small values fr accurately reflect a mdel f ruting ndes where prpagatin delay is lw and each ruting message cnsists f a number f packets, it is physically impssible fr t be zer In this sectin we eplre simulatins with small nnzer values fr Recall that, in the absence f incming ruting messages, ruter A resets its timer secnds after its timer epires, and ruter B is ntified f ruter A s incming ruting message secnds after ruter A s timer epires If (fr eample, because ruter A resets its timer befre it transmits the first packet f the ruting timer), then there is little cupling between adjacent ruters In this case, if tw ruters timers epire at the rughly same time, then each ruter resets its timer befre receiving a ruting message frm the ther ruter, and clusters break up quickly In this sectin we eplre simulatins with 0 This reflects a mdel where each ruting message cnsists f multiple packets, and neighbring ruters receive the first packet f a ruting message befre the surce ruter resets its timer e0 e9 Average time e8 e3 e7 e5 e4 0 3 4 Randm Nise Tr (as a multiple f Tc) Tc,Tc=00 secnds, Td=00 secnds, N=0 Figure 6: Time t g frm a cluster f size t a cluster f size N, and vice versa, fr 0 0 secnds Figure 6 shws the results f simulatins with 0 0 secnds The lines shw the same analytical results given in Figure 4, but cmputed fr 0 0 secnds Frm the analysis in Sectin 5,, the prbability that a cluster f size breaks int a cluster f size in ne rund, can be estimated by (7) 0
e9 Average time e8 e3 e7 e0 e5 e4 0 3 4 Randm Nise Tr (as a multiple f Tc) Tc,Tc=00 secnds, Td=004 secnds, N=0 Figure 7: Time t g frm a cluster f size t a cluster f size N, and vice versa, fr 0 04 secnds fr and As this equatin describes, the main effect f increasing is t increase In general, five simulatins were run fr each value f, and each simulatin was terminated after 0 8 secnds The X marks shw simulatins that start with unsynchrnized ruting messages and the marks shw simulatins that start with synchrnized ruting messages Nte that with set t 00 secnds rather than t zer, the simulatins take lnger t synchrnize and less lng t unsynchrnize Nevertheless, the basic behavir f synchrnizatin is preserved As Figure 7 shws, increasing frm 00 secnds t 004 secnds further increases the time required fr synchrnizatin The simulatins and analysis shw that the time t synchrnize increases as increases After a nde s timer epires, this is the time between when ther ndes are ntified f the timer epiratin, and when the nde resets its wn timer (in the absence f incming ruting messages) This interval can be affected by a number f factrs, such as the prpagatin delay, the number f packets in the ruting message and the timing between the transmissin f these packets, and the prmptness with which ndes reset their ruting timers 55 Steady state behavir One quantity f interest is the fractin f time that the Markv chain spends with lw cluster sizes We were nly able t estimate the equilibrium distributin fr the Markv chain by further apprimating the transitin prbabilities Hwever, ne simple way t estimate the fractin f time that the Markv chain spends in synchrnized states is t cmpute Recall that is the epected number f runds fr the system t mve frm state t state N; fr mst f this time the system is largely unsynchrnized Similarly, is the epected number f runds fr the system t mve frm state N t state ; fr mst f this time the system is largely synchrnized In Figure 8 the -ais shws ; the ther parameters are 0, = secnds, = 0, and = 0 secnds The y-ais is, the estimated fractin f time fr which the system is synchrnized As Figure 8 shws, as is increased, the system makes a sharp transitin frm predminately-synchrnized t predminately-unsynchrnized The simulatins and analysis in Figure 5 shw that fr a wide range f values fr and, the transitin frm
fractin f time synchrnized 00 0 04 06 08 0 almst always synchrnized almst always unsynchrnized 0 5 0 5 Randm cmpnent Tr (as a multiple f Tc) Figure 8: The fractin f time synchrnized vs the randm cmpnent fractin f time synchrnized 00 0 04 06 08 0 almst always synchrnized 5 0 5 0 5 Number f ndes Figure 9: The fractin f time synchrnized vs the number f ndes, fr = 0 secnds fractin f time synchrnized 00 0 04 06 08 0 almst always unsynchrnized 5 0 5 0 5 Number f ndes Figure 0: The fractin f time synchrnized vs the number f ndes, fr = 030 secnds predminately-synchrnized t predminately-unsynchrnized ccurs fr a small multiple f Figures 9 and 0 shw the estimated fractin f time that the Markv chain spends synchrnized as a functin f the number f ndes in the netwrk The parameters fr these figures are = secnds, = 0, and = 0 secnds; is 0 secnds in Figure 9 and 03 secnds in Figure 0 Fr each figure, as the number f ndes is increased the system makes a sharp transitin frm predminately-unsynchrnized t predminately-synchrnized This crrespnds in practice t a netwrk that mves frm an unsynchrnized t a fully synchrnized state when ne additinal
ruter is added t the system Figures 9 and 0 shw that the number f ndes where the transitin t synchrnizatin takes place is a functin f the ther parameters f the system; in Figure 9 the transitin ccurs fr 9, and fr Figure 0 the transitin ccurs fr Recall that because f the simplifying assumptins, the analysis is eplanatry rather than predictive; the analysis eplains why the transitin t synchrnizatin is an abrupt functin f the number f ndes, but the analysis des nt necessarily accurately predict the eact number f ndes at which this transitin takes place 56 Tplgies with pint-t-pint links The analysis in this paper applies t a netwrk f N ruters where each ruter receives a peridic ruting message frm each ther ruter Cnsider instead N ruters in an arbitrary tplgy, where the ruters are cnnected by pint-t-pint links and each ruter nly receives peridic messages frm its immediate neighbrs It is still pssible fr all N ruters in an arbitrary tplgy t becme synchrnized, using the mechanisms described in this paper, but the synchrnizatin dynamics depend strngly n the particular netwrk tplgy A B D C Figure : A ring tplgy f ruters E B A D C Figure : A star tplgy f ruters Fr eample, fr the tplgy f ruters in Figure each ruter receives a ruting message frm tw neighbring ruters Thus each ruter has t prepare its wn utging ruting message and t prcess tw incming ruting messages If all fur ruters becme synchrnized, then each ruter s timer epires n the average 3 secnds after its busy perid begins, and each ruter s busy perid begins when either its timer r ne f its neighbrs timers epires The prcess f synchrnizatin in a netwrk where each ruting nde has cnstant degree is similar t the prcess described in Sectin 5, and fr small values f such a netwrk is likely t remain synchrnized nce synchrnizatin ccurs Fr the star tplgy f ruters in Figure, hwever, given the Peridic Messages mdel discussed in this paper, the synchrnizatin f ruting messages is less likely t ccur In Figure 3
Ruter A has t prcess fur incming ruting messages, while each f the ther ruters prcesses nly ne incming ruting message Assume that all five ruters transmit a ruting message at the same time Then ruter A s timer epires n the average 5 secnds later, while each f the ther ruters timers epires n the average secnds later This difference in perid shuld break up synchrnizatin fairly rapidly Thus fr ruters in an arbitrary tplgy the tendency twards synchrnizatin can depend heavily n the details f the tplgy Current internet tplgies where mst ruters have a similar and lw degree shuld lend themselves t synchrnizatin 6 Other appraches t synchrnizatin The Markv chain mdel in Sectin 5 was designed t capture sme key prperties f the ruting messages mdel such as the rle f randmizatin in prducing an initial cluster f size tw, the rle f changes in perid in enabling the frmatin f larger clusters, and the rle f randmizatin in breaking up clusters nce they have frmed In additin, this Markv mdel seems (t us) fairly simple and straightfrward Hwever, there are many pssible appraches t the analysis f synchrnizatin; in this sectin we discuss briefly cupled nnlinear scillatrs and cupling methds fr Markv prcesses The apprach t synchrnizatin in [Bl88] cncerns cupled nnlinear scillatrs and uses the tls frm nnlinear dynamic systems The bk shws that the tendency f synchrnizatin is characteristic f a brad class f dynamic systems Because [Bl88] des nt treat the questin f synchrnizatin in the presence f randmizatin, the results are nt immediately applicable t the synchrnizatin f ruting messages Mathematical mdels have recently been develped t analyze pulse-cupled scillatr systems [SS93] such as cmmunities f Thai fireflies, where the peridic pattern f ne firefly s flashing is affected by the thers nearby At dusk male fireflies gather in trees by the edge f the river and flash n and ff in an unsynchrnized fashin but, as the night prgresses, whle trees f fireflies will flash in synchrnizatin fr hurs These pulse-cupled scillatr systems are similar t ur mdel f ruting messages where the timing f ne ruter s ruting messages can be affected by the arrival f ruting messages frm a neighbring ruter A different apprach t synchrnizatin cmes frm the literature n cupling methds fr Markv prcesses [G78] In the classical cupling [G78, p3], tw independent cpies f a Markv prcess evlve until they reach a cmmn state then, frm that pint, the tw Markv prcesses use the same transitin mechanism and fllw the same path It might be pssible t epress ur mdel fr the synchrnizatin f ruting messages as an eample f cupled stchastic prcesses, with each ruting nde a separate Markv prcess, but the analysis that we use des nt cme frm the literature f cupled Markv prcesses 7 Cnclusins and pen questins In this sectin we give sme specific cnclusins abut the synchrnizatin f ruting messages, then discuss cnclusins and pen questins abut the mre general prblem f unanticipated structure in the Internet 4
As the simulatins and analysis in this paper demnstrate, peridic ruting messages frm a system f ruters in a netwrk can easily becme synchrnized The simulatins and analysis bth shw that this synchrnizatin is an emergent prperty at a particular scale with an abrupt transitin frm unsynchrnized t synchrnized behavir Thus the behavir f the system with a small number f ruters can nt necessarily be etraplated t eplain the behavir f the system with a large number Synchrnizatin can be avided with apprpriate implementatin f the ruting prtcls One pssible methd is the additin f a randm cmpnent t the ruting timer intervals Our analysis prvides general guidelines n determining the magnitude f the randm cmpnent necessary t avid synchrnizatin In particular, setting the timer each rund t a time frm the unifrm distributin n the interval [0 5 5 ] secnds is a simple way t avid synchrnized ruting messages 9 Adding a randm cmpnent t the ruting timers changes the strict peridicity f the ruting messages but des nt affect the cnvergence prperties f the underlying ruting prtcls A secnd methd fr aviding synchrnizatin is t implement a ruting timer that is independent f eternal events (as mentined in the specificatins fr RIP) [He88, p3] If each ruter resets its timer immediately after the timer epires (regardless f its activities when the timer epires) and if ruters dn t reset their timers after triggered updates, then the prcess f timer synchrnizatin described in this paper might be avided There are, hwever, drawbacks t this apprach: if ruters are initially synchrnized (either by chance, r because they were restarted at the same time) then they will remain synchrnized since there is n mechanism t break up synchrnizatin if it des ccur It is als pssible t reduce the negative impact f synchrnized ruting updates by mdifying ruters t give acceptable perfrmance in the presence f large r synchrnized ruting updates While it is ften mre efficient t eplit traffic structure rather than t engineer it ut, this des nt seem t be the case fr synchrnized ruting messages Even with tuned ruter implementatins, synchrnized ruting messages place an unnecessary burden n the netwrk, and a preferable slutin is t avid synchrnizatin in the first place Peridic ruting messages are nt the nly eample f unintended structure in the Internet Our ping eperiments suggest that many peridic prcesses are at wrk Frm these eperiments we cnjecture that a significant number f the packet drps in the Internet are assciated with sme frm f peridic prcess, but the causes f structure in current Internet traffic are nly beginning t be eplred One mdel f large-scale structure cmes frm the kinetic thery f autmbile traffic [PH7] Individual drivers, each seeking t ptimize their wn gals, can prduce cllective traffic patterns akin t the crdinated mtin f flcks f birds r schls f fish Individual strategies f reducing speed during cngestin result in a cllective decrease in traffic flw as the vehicle cncentratin increases past a certain density [KK93] shws that given an initially hmgeneus traffic flw, regins f high density and lw average velcity (clusters f cars) can spntaneusly appear These high-density regins can mve either with r against the flw f traffic, and tw clusters with different velcities, widths, and amplitudes merge when they meet, resulting in a single cluster We believe there are analgus interactins between packet streams and we intend t eplre these mechanisms fr the emergence f large-scale structure in packet-switched netwrks 9 Pseud-randm numbers can be efficiently generated by a randm number generatr [Ca90] 5
As the Internet epands t new types f traffic (eg, vice and vide), new ruting patterns (multicast distributin), and new gateway scheduling disciplines (Quality-f-Service fr realtime traffic), it is imprtant t anticipate the large-scale structure that might be intrduced by these changes But, as ur analysis f ruting shws, large scale structure is ften an emergent prperty that cannt be bserved r inferred frm small scale simulatins r measurements Althugh they d nt currently eist, we feel that large scale simulatrs are a necessary tl fr eplring questins f large scale structure Large scale simulatins can help us build intuitin abut the behavir f large-scale netwrks, better understand behavir f current Internet traffic, and predict hw this behavir might change as Internet traffic types, ruting patterns, and gateway scheduling disciplines evlve 8 Acknwledgements We thank Tny Li, Steven McCanne, Mark Mran, Vern Pasn and Liia Zhang fr their cmments n earlier drafts f this paper Sme f the simulatin and analysis in this paper was dne several years ag, after bserving synchrnized DECnet ruting messages in 988 After a reprt in 99 by Prfessr Agrawala n packet drp rates in the Internet referred t lng delays at peridic 90- secnd intervals n paths t MIT, we reprted t the End-t-End Research Grup that we fund a similar prblem f packet drps at 90-secnd intervals n paths t MIT We thank Dave Clark fr pursuing these prblems further; initial cnjectures that the prblems f peridic packet drps at MIT were related t synchrnized ruting messages pushed us t finish ur draft f this paper We have enjyed discussins with Win Treese, wh has als investigated the synchrnizatin f ruting messages References [Ba9] Baker, Mary, private cmmunicatin, 99 [Bl88] Blekhman, II, Synchrnizatin in Science and Technlgy, ASME Press Translatins, 988 [BrChClP93] Braun, H, Chiny, B, Claffy, K, and Plyzs, G, Analysis and Mdeling f Wide-Area Netwrks: Annual Status Reprt, CSL, University f Califrnia at San Dieg, February 993 [Ca90] Carta, D, Tw Fast Implementatins f the Minimal Standard Randm Number Generatr, Cmmun ACM, V33 N, January 990, p87-88 [Ca9] Casner, S, and Deering, S, First IETF Internet Audicast, Cmputer Cmmunicatin Review, V N3, July 99, p9-97 [De93] Deering, S, private cmmunicatin, 993 [En9] Engel, EMRA, A Rad t Randmness in Physical Systems, Springer-Verlag, 99 [F6] Feller, W, An Intrductin t Prbability Thery and Its Applicatins, V II, Jhn Wiley & Sns, 966 [FJ9] Flyd, S, and Jacbsn, V, On Traffic Phase Effects in Packet-Switched Gateways, Internetwrking: Research and Eperience, V3 N3, September 99, p5-56 [FJ93] Flyd, S, and Jacbsn, V, The Synchrnizatin f Peridic Ruting Messages, SIG- COMM 993, September 993, pp 33-44 6
[Gr8] Greene, DH, and Knuth, DE, Mathematics fr the Analysis f Algrithms, Birkhauser, 98 [G78] Griffeath, D, Cupling Methds fr Markv Prcesses, Advances in Mathematics Supplementary Studies, V, Academic Press, 978, pp -43 [He88] Hedrick, C, Ruting Infrmatin Prtcl, Request Fr Cmments (RFC) 058, June 988 [He9] Hedrick, C L, An Intrductin t IGRP, August 99, available by annymus ftp frm shivacm in /dc/igrppsz [Ja9] Jacbsn, V, Audi lsses during yesterday s packet vide wrkshp audicast, Message- ID 9039AA4349@r7eelblgv archived in nicesnet:ietfrem-cnf, Nv, 99 [KK93] Kerner, BS, and Knhauser, P, Cluster effect in initially hmgeneus traffic flw, Physical Review E, V48 N4, pp 335-338 [Li93] Li, Tny, private cmmunicatin, 993 [L93] Lng, Dan, private cmmunicatin, 993 [Mi87] Mickens, R E, Difference Equatins, Van Nstrand Reinhld, 987 [Mi83] Mills, DL, DCN Lcal-Netwrk Prtcls, Request Fr Cmments (RFC) 89, Dec 983 [M84] Mills, DL, Eterir Gateway Prtcl Frmal Specificatin, RFC 904, April 984 [Pa93a] Pasn, V, Empirically-Derived Analytic Mdels f Wide Area TCP Cnnectins, LBL Tech Reprt LBL-34086, May 993 [Pa93b] Pasn, V, Grwth Trends in Wide-Area TCP Cnnectins, submitted t IEEE Netwrk, 993 [PH7] Priggine, I and Herman, R, Kinetic Thery f Vehicular Traffic, American Elsevier, 97 [R85] Rss, S M, Intrductin t Prbability Mdels, Academic Press, 985 [SaAgGuJa9] Sanghi, D, Agrawala, A, Gudmundssn, O, and Jain, B, Eperimental Assessment f End-t-End Behavir n Internet, University f Maryland Reprt UMIACS- TR-9-6, June 99 [Sc9] Schiller, Jeffrey, private cmmunicatin, Sept 99 [SS93] [Tr9] Strgatz, S, and Stewart, I, Cupled Oscillatrs and Bilgical Synchrnizatin, Scientific American, December 993, pp 0-09 Treese, W, Self-Synchrnizatin Phenmena in Cmputer Netwrks, MIT class prject, Dec 99 [Tr94] Treese, W, private cmmunicatin, 994 [VMS88] VMS Netwrking Manual, Versin 50, AA-LA48A-TE, Digital Equipment Crpratin, Maynard Massachusetts, April 988 [ZhCl90] Zhang, L, and Clark, D, Oscillating Behavir f Netwrk Traffic: A Case Study Simulatin, Internetwrking: Research and Eperience, Vl N, 990, pp 0-7
A Slutin t Recursive Frmula This sectin gives the slutin fr the fllwing recursive equatin fr : where We epress this, using peratr ntatin, as Fr an eplanatin f peratr ntatin in slving recursive equatins, see [Gr8, p9] r [Mi87, p5] Let This gives the first rder difference equatin Therefre fr with the initial cnditin 3, and fr 0 This has the slutin [Gr8, p8] 3 3 4 4 Nw slving fr we have that 4 4 4 4 with the initial cnditin This has the slutin 3 4 3 4 Substituting fr, we get 3 3 3 8