Synchronizing Time Servers. Leslie Lamport June 1, 1987

Transcription

1 Synchronizing Time Servers Leslie Lamort June 1, 1987

2 c Digital Equiment Cororation 1987 This work may not be coied or reroduced in whole or in art for any commercial urose. Permission to coy in whole or in art without ayment of fee is granted for nonrofit educational and research uroses rovided that all such whole or artial coies include the following: a notice that such coying is by ermission of the Systems Research Center of Digital Equiment Cororation in Palo Alto, California; an acknowledgement of the authors and individual contributors to the work; and all alicable ortions of the coyright notice. Coying, reroducing or reublishing for any other urose shall require a license with ayment of fee to the Systems Research Center. All rights reserved.

3 Author s Abstract A time service in a distributed system may be used both for multirocess synchronization and for simly finding out what time it is. For synchronization, the time rovided by different servers should be closely synchronized. For telling time, the time rovided by each server should be a close aroximation to Universal Time (the international time standard). Algorithms are resented for imlementing a fault-tolerant time service that meets both requirements. Casule Review Users of electronic mail are not surrised to see messages that are timestamed after they were received. The naive blame the robably nonexistent oerator, who didn t get the time right. The wary know that many comuter systems go for long eriods without stoing; during those eriods adjustments are difficult or imossible because almost all timedeendent rocesses (for examle, file systems, mailers, audit trails, back-u mechanisms, etc.) assume that the time rovided by the local oerating system is increasing smoothly. Hence a good time service not only must be close to the real time, but also must increase and maintain only a bounded rate of change. For some alications (for examle, distributed data bases), there is a third requirement: the times maintained by various rocessors within a network must be very close to each other. This requirement is so stringent that it is not enough simly to ensure that the time rovided by each rocessor is within a certain limit of the real time. Resynchronizations with the real time must therefore be coordinated. The final requirement of a good time service is that the resynchronization rotocol must allow a certain number of faulty links or faulty rocessors since network rotocols ought to work in the resence of artial failures. Previous authors have resented algorithms that satisfy some of these requirements, but the algorithm described here is the first that satisfies all four simultaneously. Another attribute of the aer is that both the roblem and the solution are recisely formulated.

4 iv

5 Contents 1 Introduction 1 2 Notation and Assumtions Intervals Clocks and Clock Ranges The Network Obtaining Universal Time 9 4 Providing The Service Time Ideal Time Synchronization and Time-Correctness of Ideal Clocks Broadcasting Universal Time The Comlete Algorithm Glossary 27 Acknowledgments 29 References 31 Index 33

6 vi

7 1 Introduction A time service rovides the current time to its users. functions: Telling a user the current time and date. It erforms two Allowing different users to synchronize their activities. Though related, these two functions are distinct. The first requires that the time rovided be aroximately equal to Universal Time the ideal standard that is aroximated by the National Bureau of Standards broadcasts on station WWV. The second requires that the time rovided to different users be aroximately the same. Marzullo [6] devised algorithms for roviding an accurate time and date, and a number of fault-tolerant synchronization algorithms have been roosed [2, 4, 5], but there has aarently been no revious work that considered both functions at the same time. In this aer, I consider the roblem of imlementing a fault-tolerant time service that rovides a single time value to erform both functions more recisely, a time value that ermits close to otimum synchronization and is a reasonable aroximation to the correct time and date. The roblem to be solved is stated somewhat informally in this introduction. Section 2 states the assumtions and conditions more recisely and defines some helful notation. (A glossary is rovided at the end of the aer to hel the reader follow the notation.) Section 3 describes Marzullo s algorithm for comuting the best ossible aroximation to the correct time and date, and Section 4 develos an algorithm for a time service that rovides the two functions. The algorithms assume a network of rocesses, in which each node has a local clock that runs at aroximately the correct rate, and some nodes also have direct access to Universal Time, erhas obtained by listening to WWV. A time server is imlemented by synchronizing all the nodes clocks, using the available information about Universal Time. The nodes roviding the time service may be a subset of the nodes in the comlete system the other nodes interrogating the time servers to obtain time information. However, nodes that do not act as time servers or roviders of Universal Time are ignored. The time service must function roerly desite the failure of some network comonents. A time service could rovide two distinct values to satisfy its two different functions. For the function of roviding the current time and date, a rocess 1

8 would rovide a time interval I that is its best aroximation to Universal Time. More recisely, I would be the smallest interval that knows to contain UT, the current Universal Time. (It is most common to write a value with error bound in the form t ± ɛ. However, it is more convenient to work with the interval [t ɛ, t + ɛ] within which the correct value is known to lie.) For the second time-service function, must rovide a time T, called service time, that is close to the service time T q rovided by any other node q. More recisely, it should rovide T and, for each other node q, anumber δ q such that T T q <δ q. However, this condition is not sufficient, since it is trivially met by letting each T always be zero. To reresent time, T should change at aroximately the same rate as UT, sothevalueoft increases by about one second with the assage of one second of Universal Time. While not strictly necessary, it is convenient to have T not only change at aroximately the same rate as UT, but be aroximately equal to it. Consider, for examle, the roblem of generating creation times for files. One might want to use the creation time to decide which version of a file is the current version. Since versions may be created at different nodes, a file generated at node should use T as its creation time to minimize the likelihood that a version created at one node receives an earlier creation time than a version created before it at a different node. However, one might also want a creation time to tell the Universal Time at which the file was created, so the user can determine the actual date and time of creation. This could be accomlished by recording a searate universal creation time derived from I. However, this additional value is not needed if T rovides an accetable aroximation to UT. We therefore state the following three requirements for the time T rovided by node, whereκ, ɛ,andtheδ q are values rovided by. (They could be constants that are announced when the system is turned on, or they could be rovided in resonse to user requests.) correct rate The rate of change of T with resect to UT lies between 1 κ and 1 + κ. synchronization For every other node q: T q T <δ q. correct time UT T <ɛ. The synchronization requirement follows from the correct-time requirement by letting δ q = ɛ + ɛ q. However, this may not rovide close enough 2

9 synchronization. Universal Time is of interest mainly to humans; synchronization algorithms deend only uon the differences between the values of T at the different nodes. Humans seldom need to know the value of UT to better than a few seconds, so an ɛ of several seconds is accetable. On the other hand, synchronization algorithms may have more stringent requirements for δ q. When nodes and q are using the time service for synchronization, generally incurs a delay of O(δ q ) seconds because of the lack of synchrony of T and T q. For examle, if q announces that it will release a resource at time t (that is, when T q equals t), then must wait until T reaches t + δ q before acquiring the resource. Some alications might require that this delay be ket to within a few microseconds. While it is not necessary to kee the ɛ as small as the δ q,therequired synchronization condition can be achieved if ɛ could be ket small enough. However, this is not always ossible. The closeness with which clocks at different nodes can be directly synchronized deends uon the uncertainty in message transmission time between those nodes. Modern large networks are heterogeneous, and the uncertainty in transmission times may be very different for different airs of nodes. Tyically, a large network consists of a collection of local area networks (LANs) that are interconnected by oint-to-oint links. Nodes on a single LAN may be directly connected by a fiber-otic link, in which the uncertainty in transmission time can be as small as a few microseconds if the timing functions are erformed at a low enough system level. The nodes in different LANs may communicate with one another by a store-and-forward rotocol that could have an uncertainty of a second or more in transmission time. 1 If a articular LAN does not include a direct source of Universal Time (such as a WWV receiver), so nodes in the LAN must base their knowledge of UT on messages received from outside the LAN, then the values of ɛ and ɛ q for and q in the LAN could be several orders of magnitude greater than the best achievable value of δ q. Marzullo [6] resented algorithms for obtaining a clock value from a set of clocks, some of which may be faulty. These algorithms can be used to rovide the intervals I that best aroximate Universal Time, but they do not satisfy the synchronization condition if δ q <ɛ + ɛ q. Several Byzantine 1 Such a large value results not from uncertainty in the hysical transmission times, but because the communication involves higher-level rotocols, searated from the hysical messages by many layers of software. It is quite likely that the timing of transmission delays can be done at a lower system level for intra-lan messages than for inter-lan messages. 3

10 clock synchronization algorithms have been resented that can be used to satisfy the correct-rate and synchronization conditions in the resence of failures [2, 4, 5]. However, to my knowledge, there have been no ublished algorithms to achieve all three of the conditions above. The major art of this reort concerns algorithms for achieving the synchronization condition when δ q may be much smaller than ɛ + ɛ q.thisis a nontrivial roblem in the resence of failures, because even very simle kinds of failure act like malicious, Byzantine failures. For examle, suose a node is sending its clock value to all other nodes in the network. It does this by sending a message saying something like my clock now reads 11:47. Suose, through some hardware or software error, it auses for five minutes in the middle of this broadcast. While it sends the same message to all nodes, it has essentially sent descritions of two clocks that differ by five minutes. Thus, this simle error results in a two-faced clock that rovides different clock values to different nodes. The correct-time requirement does not mention the interval I,which reresents s knowledge of UT. One might be temted to relace this requirement by the condition that T be in I. However, such a condition would be inconsistent with the other two requirements for T. It is inconsistent with the correct-rate requirement because new knowledge of the correct value of UT, such as the receit of a message from a node with a WWV receiver, could suddenly reduce the width of the interval I. Keeing the value of T within the interval I could require a sudden change to T,whichis rohibited by the correct-rate requirement. It can also be shown that, with malicious failures, requiring T to be within I could require violating the synchronization requirement if δ q <ɛ + ɛ q. 2 Notation and Assumtions In the introduction, the term node was used to emhasize that each node in the network rovides a time service for user rocesses running at that node. To be consistent with the terminology commonly used in discussing clock synchronization, the term rocess will be used instead of node. 2.1 Intervals The term interval is used to denote a closed interval on the real line that is, an interval of the form [x, y] forx y. The width of the interval R is denoted by R, so [x, y] = y x. The sum of two intervals is defined by 4

11 [x, y]+[z,w] =[x + z,y + w]. Arealnumberz is considered to be the same as the interval [z,z], so [x, y]+z is defined to be the interval [x + z,y + z] obtained by translating the interval [x, y] to the right a distance of z. For any interval U and number δ>0, U ± δ is defined to equal U +[ δ, δ], so [x, y] ± δ =[x δ, y + δ]. A seudo-metric d on intervals is a nonnegative, real-valued function on airs of intervals satisfying the following roerties for all intervals U, V,andW : (i) d(u, V )=d(v,u), (ii) d(u, V )+d(v,w) d(u, W ), and (iii) d(u, U + ɛ) =ɛ for any ɛ 0. A seudo-metric satisfying the additional roerty that d(u, V ) = 0 imlies U = V is called a metric. (A metric is sometimes called a distance function.) Two imortant seudo-metrics are The midoint seudo-metric d m,where d m (U, V ) equals the distance between the midoints of U and V. The uniform metric d u,whered u ([x, y], [v, w]) equals the maximum of v x and w y. As its name imlies, the uniform metric is a metric. Note that for any intervals U and V, d m (U, V ) d u (U, V ). A real-valued function F on m-tules of intervals is said to satisfy the Lischitz condition for a seudo-metric d if, for any intervals U i and V i and any number δ > 0: d(u i,v i ) <δfor all i imlies F (U 1,...,U m ) F (V 1,...,V m ) < δ. The function F is said to be translation invariant if F (U 1 + x,...,u m + x) = F (U 1,...,U m )+x for any intervals U i and real number x. Satisfying the Lischitz condition is a stronger requirement than continuity and a weaker requirement than having a bounded derivative. Translation invariance asserts that translating all arguments by a fixed amount causes the value to be translated by the same amount. We exect any functions aearing in a clock synchronization algorithm to be translation invariant, since increasing all inut clock values by a fixed amount should roduce a corresonding increase in the clock values comuted by the algorithm. Observe that if F satisfies the Lischitz condition for the seudo-metric d m, then it also satisfies the condition for the metric d u. 2.2 Clocks and Clock Ranges A time-deendent value is any real-valued function of a real variable. If v is a time-deendent value, we interret v(t) to be the value of v at Universal 5

12 Time t. Aclock is a nondecreasing time-deendent value. If V is a clock, the value V (t) reresents the value read by clock V at Universal Time t. The identity function, denoted by UT (so UT(t) = t), is a clock. The service time T rovided by rocess is a clock, where T (t) reresentsthetime value rovided by to a request for the current service time received at Universal Time t. (Ofcourse, does not need to know the current value of Universal Time to comute the value of T at that time.) Let V 1,...,V n be clocks. (Think of V as a clock maintained by rocess.) The following definitions exress the correctness conditions introduced informally in the introduction. They describe conditions on these clocks for an interval of time [u, v], where u and v reresent clock values that is, times indicated by the clocks themselves. Thus, the conditions exress roerties of the clocks over intervals [s, t] of Universal Time such that V (s) =u and V (t) =v for a rocess. The conditions are exressed in this form because clock times are directly observable, Universal Times are not. The bounds κ, δ q,andɛ are time-deendent values. (In many cases, they will be constants.) correct rate with bounds κ : For each and any x and y, x y such that V (x) andv (y) arein[u, v]: y y (1 κ (t)) dt < V (y) V (x) < (1 + κ (t)) dt x synchronization with bounds δ q : For each and q, q, andeacht such that V (t) isin[u, v]: V q (t) V (t) <δ q (t). correct time with bounds ɛ : For each and each t such that V (t) isin [u, v]: UT (t) V (t) <ɛ (t). The correct-rate condition is defined in terms of integrals to avoid requiring that the V be differentiable functions of time. When no interval is secified, these conditions are assumed to hold for all intervals. Each nonfaulty rocess is assumed to have a clock C, called its local clock. 2 It is assumed that the local clocks of all nonfaulty rocesses satisfy the correct rate condition with bounds ρ 1, where the ρ are constants. (AnerrorinthelocalclockC is considered to be a failure of rocess.) A -clock is a clock of the form C + v for some constant v that is, a clock whose value at time t is v+c (t). A -clock is one that runs at the same 2 It is sufficient for to have a cyclic timer, since one can construct a monotonic clock from such a timer. In fact, the algorithms are easily modified to work with only a timer. x 6

13 rate as s local clock. Of course, C is a -clock. A -clock V is determined by its value at any single time t 0,sinceV (t) =V (t 0 ) C (t 0 )+C (t). Note that T, the time-service clock rovided by, will not in general be a -clock. The following result is an easy consequence of the assumtion that C satisfies the correct-rate condition. It asserts that the uncertainty ρ in the running rate of s local clock causes its knowledge of Universal Time to degrade at a rate of ρ seconds er second of elased time on its local clock. Proosition 1 If rocess is nonfaulty and R is an interval such that UT (t 0 ) R, then for all t t 0 : UT(t) R +(1± ρ )(C (t) C (t 0 )) where terms of order ρ 2 (t t 0 ) are neglected. A clock range is an interval-valued function on the reals of the form [x, y] where x and y are clocks. In other words, R is a clock range if there exist clocks x and y such that R(t) =[x(t),y(t)] for all times t. A -clock range is a clock range of the form [x, y] such that x and y are -clocks. A -clock range can be written as U + C for some interval U. Since a real number x is identified with the interval [x, x], a clock is a secial case of a clock range, and a -clock is a secial case of a -clock range. A -clock range is determined by its value at any single time. If F is a real-valued function on m-tules of intervals, then alying F to an m-tule of clock ranges roduces a time-deendent value. Let R 1,..., R m be -clock ranges with R i = U i + C for intervals U i.iff is translation invariant, then F (R 1,...,R m )=F (U 1,...,U m )+C Thus, if the R i are -clock ranges, then F (R 1,...,R m )isa-clock. 2.3 The Network I assume a network of rocesses connected by channels, where a channel may connect more than two rocesses. The two kinds of channels that are of interest are a oint-to-oint channel that has a single sender and a single receiver, and a broadcast channel that connects a set of rocesses so that any one of them can broadcast a message over it to all other rocesses on 7

14 the channel. A two-way communication line is a broadcast channel that connects just two rocesses. Certain of the rocesses, called Universal Time roviders, are assumed to have a direct source of Universal Time. Let UT (j) denote the value of UT obtained by rocess j. This value is a clock range that is known to contain the correct value of Universal Time, so UT (t) UT (j) (t) for any time t. Process j will eriodically broadcast UT (j) to other rocesses. For each channel c, I assume values τmin c and τ max c such that a message sent over c at time t is received between times t + τmin c and t + τ max c if the sender, the receiver, and c are all nonfaulty. More recisely, if an event, such as the receit of another message, that occurs at the sender at time t causes the sending of a message M over channel c, thenm will be received between times t + τmin c and t + τ max c. Thus, the minimum and maximum delays τ min c and τmax c include the time needed to generate and send the message as well as the time the message was actually in transit along channel c. Thevalues of τmin c and τ max c may vary with time, but they are assumed to be known to the receiver. Let γ c denote τmax c τmin c. Delivering a message with a delay less than τmin c or greater than τ max c constitutes a channel failure. If there is unredictable variance in transmission delay, due, for examle, to variation in the channel loading, then τmin c and τmax c should be chosen conservatively to reduce the robability of such a failure. (Note that τmin c can always be taken to be zero.) However, the time required for fault-tolerant synchronization algorithms deends uon the values of τmax c, not on the actual delays, and the bounds δ q on clock synchronization deend uon γ c, so tradeoffs between reliability and efficiency must be made when choosing the values of τmin c and τ max c. A ath is a sequence of rocesses and channels, each channel connecting successive rocesses. The null ath connects rocess to itself. If π is a ath from to q and ψ is a ath from q to r, thenπψ denotes the obvious ath from to r via q. For a ath π from to q, letτmin π and τ max π denote the sum of all τmin c and τmax, c resectively, for all channels c in π. Thus,τmin π and τ max π reresent the minimum and maximum transmission delays for a message relayed from to q along π. Define γ π to be τmax π τ min π, the uncertainty in transmission delay along π. Fault-tolerant synchronization algorithms require that a rocess know the values τmin π, τ max π,andγπ for messages it receives over the ath π, which usually requires knowledge of the values of τmin c and τ max c for each channel c in the ath. If these values can change, then new values can be broadcast 8

15 using the method of [3], which assures that the same values are used by all rocesses. For simlicity, I assume that the τ π min, τ π max,andγπ are constants for each fixed ath π. If R is an interval and π aath,thenr π is defined to be the interval R+[τ π min,τπ max]. Suose that π is a nonfaulty ath from rocess to rocess q, andr reresents s knowledge of Universal Time at a certain time t that is, knows that UT (t) R. If sends a message with the value R along π to q and that message arrives at time t, then since the transmission time of the message is in the interval [τ π min,τπ max ], q knows that UT (t ) R π. Observe that if πφ is a ath, then R πφ =(R π ) φ. Synchronization algorithms require rocesses to send clock ranges to one another. (Remember that a clock is a secial case of a clock range.) A rocess sends a -clock range by sending a message with the clock range s current value R along a ath π. The receiving rocess q interrets this message as the receit of a q-clock range whose value, at the time the message is received, is R π. The following result asserts that transmitting a clock range in this way causes an initial erturbation by a distance of u to γ π, after which the two clock ranges drift aart at a rate of at most ρ + ρ q, where distance is measured by the uniform metric d u on intervals. This result is a simle consequence of the correct-rate assumtion for the local clocks and the assumed bounds on message-transmission times. Proosition 2 Let R be a -clock range, and suose that rocess sends a message at time t that is received at time t over a nonfaulty ath q, and let R q be the q-clock range such that R q (t )=R (t) π. Then for any t 0: d u (R (t + t),r q (t + t)) γ π +(ρ + ρ q ) t where terms of order ρ q τ π max are neglected. 3 Obtaining Universal Time Let us now consider how a time server could rovide the best ossible value of I, an interval known to contain UT. Assume that each Universal Time rovider j maintains a clock range UT (j) that reresents its current knowledge of Universal Time. If j is nonfaulty, then UT (t) UT (j) (t) for all times t. At various times, rovider j broadcasts the current value of UT (j). Let UT (j) denote a clock range that reresents s knowledge of the current value of UT (j). More recisely, assume that, if j and are nonfaulty, then 9

16 UT (t) lies within the interval UT (j) (t) for all times t. If receives a message at time t 0 informing it that UT (t 0 ) lies in the interval R, thenproosition1 imlies that we can define UT (j) by UT (j) (t) =R +(1± ρ )(C (t) C (t 0 )) for t t 0. In fact, this is how UT (j) (t) should be defined if did not receive any information about UT (j) during the time interval [t 0,t]. The roblem of how j broadcasts the value UT (j) is considered later. The algorithm for comuting the best aroximation to UT from a set of m intervals, each asserted to contain UT, was first obtained by Marzullo. It aears as Algorithm 4-2 in his thesis [6]. Define M f m (U 1,...,U m )tobe the largest interval whose endoints belong to at least m f of the intervals U i. It is not hard to show that, if one knows only that UT lies in all but at most f of the intervals U j,thenm f m(u 1,...,U m ) is the smallest interval known to contain UT.ThevalueofM f m(u 1,...,U m ) can be comuted from the set of m intervals U i in O(m log m) time by sorting their endoints. It can be recomuted in O(m) timeifjustoneoftheu i changes. If the intersection of U 1 with M f m (U 1,...,U m ) is emty, then M f m(u 1,...,U m )=M f 1 m 1 (U 2,...,U m ). In this case, U 1 is known to be one of the intervals that does not contain UT (there are at most f of them), so it may be thrown away when comuting the best aroximation to UT. After throwing it away, we are left with m 1 intervals, all but at most f 1 of them containing UT. More generally, if k of the U i have an emty intersection with M f m (U 1,...,U m ), then M f m (U 1,...,U m )equalsthevalue obtained by throwing away those k intervals and alying M f k m k to the remaining intervals. Assume that u to f of the m values UT (j) may be incorrect, where an interval UT (j) is correct if UT always lies within it. The obvious way to choose I, an interval that rocess knows to contain UT, is to let it equal (1) (UT,...,UT (m) ). However, suose that at some time when C has M f m the value C, M f m(ut (1),...,UT (m) ) equals the interval U. When C has the value C + C, UT must lie in the interval U +(1±ρ ) C. However, the intervals UT (j) are sreading out at a rate ρ, which could cause the value of M to sread out at a faster rate in fact, to make large, discontinuous jums. Thus, when C has the value C + C, M(UT (1),...,UT (m) )could be a larger interval than U +(1± ρ ) C. In Marzullo s algorithm, comutes an initial value I (t 0 )frominitial 10

17 values UT (j) (t 0 )oftheut (j) as follows. It throws away any of the UT (j) (t 0 ) that it decides are incorrect. If m k intervals are currently believed to be correct, then the interval I (t 0 )equalsm f k m k alied to the m k correct intervals UT (j) (t 0 ). If no new values are received from the Universal Time roviders between times t 0 and t, then, ignoring terms of order ρ 2 (t t 0 ), I (t) is defined by I (t) =I (t 0 )+(1± ρ )(C (t) C (t 0 )) In other words, when receives no new information, the interval I advances (moves right along the number line) at s clock rate and widens at the rate of 2ρ seconds er second of clock time. When a new value for UT (j) arrives, rocess adds the value to the set of intervals that it resumes to be correct, throwing away the revious value of UT (j). Process next comutes I by alying M f k m k to the m k intervals currently resumed correct. It then declares to be incorrect any of these intervals (the ones it had resumed to be correct) that have emty intersections with I. When no new information is received from the Universal Time roviders, the value of I deteriorates at the rate of ρ seconds er second. To maintain the accuracy of I, it is necessary for the Universal Time roviders to broadcast their values of UT (j) sufficiently often. Suose that a rovider j sends its clock range UT (j) to at time t by simly sending a message with the interval UT (j) (t) along some ath π. If receives this message at time t,itsetsut (j) (t )tout (j) (t) π. Suose that each rovider j sends UT (j) in this way at least once every J seconds over a ath π with γ π γ. It is then easy to show that if UT (j) <ɛfor every nonfaulty rovider j and at least m f of the time roviders and their aths π are nonfaulty, then Marzullo s algorithm guarantees that for all times t, I (t) iscontained within the interval UT (t) ± (ɛ + γ + ρ J). 4 Providing The Service Time 4.1 Ideal Time A time server eriodically receives information allowing it to refine the clock T that reresents the service time it rovides to its clients. Information comes in discrete lums usually through the receit of a message. To maintain the continuity of T more recisely, to maintain the boundedness 11

18 of its rate of change the value of T cannot change instantly in resonse to new information. Instead, the rate of change of T is modified discontinuously. This section resents a method for comuting T s rate of change. Process comutes T from its clock C by the formula: T = a C + b where a and b are constants that are changed discontinuously that is, they are iecewise constant functions of time. The value b reresents a zero-oint correction; 3 the value a reresents a correction to the running rate of C. I will discuss the way a changes; the change to b is determined by the requirement that T be continuous and will be ignored. There are two corrections embodied in the value of a : a correction to comensate for the measured inaccuracy of the local clock C,andacorrection to bring T into synchrony with UT and with the times T q rovided by other servers q. The comonent of a that comensates for the inaccuracy of C effectively reduces the error ρ in its rate. It can be obtained by comaring C with I for a long enough eriod of time. I will ignore this comonent and assume that any difference between a and 1 is meant as a correction to achieve synchronization. Such a correction actually increases the error κ in the rate of change of T. This increase is unavoidable. If clock synchronization is to be maintained, κ must be allowed to become larger than ρ, the inherent error in the rate of s local clock. It is easiest to describe synchronization algorithms in terms of discontinuously resetting the time. There are a sequence of resynchronization times T (0), T (1), T (2),... at which rocesses resynchronize. Every rocess changes a when T = T (i),sothet (i) reresent service times. For convenience, I assume that all rocesses resynchronize at every time T (i) a rocess that does nothing at that time can be thought of as erforming a resynchronization in which the new value of a equals its old value. Time T (0) reresents the service time at which the system is started. Resynchronization may actually be erformed by having every rocess agree when their time T should read T (i). However, for this discussion, it is more convenient to have a rocess convert this into the inverse information: the time that T should read when it actually reads T (i). Assuming ρ 1, if discovers that T should read T (i) when it actually reads T (i) + T, then it knows that T should read aroximately T (i) T when it actually reads T (i). 3 If C is actually a cyclic timer instead of a monotonic clock, then b is incremented every time C is reset to zero. 12

19 At the i th resynchronization, rocess thus learns what the correct service time should be when T reads T (i), and uses this information to reset a. (The resynchronization must be carried out in such a way that receives this information before its clock reaches T (i).) Here, the correct time is the one that is agreed uon by all rocesses as the one to which they want to resynchronize. The most convenient way to describe resynchronization is to, that kees the correct time learned during the i th resynchronization in other words, C (i) is the -clock that has the correct time when T equals T (i). Hence, C (i) reresents s knowledge, as of time T (i), of what the current service time should be. This knowledge becomes obsolete at service time T (i+1), when the resynchronization rovides with more recent information about the correct time. Initially a =1 have maintain an ideal -clock C (i) from the time T (0) when the system is started until T (1). To resynchronize T at service time T (i), resets the rate of change of T and T = C (0),soT equals C (0) so that, in the absence of any further resynchronization, T would equal C (i) after exactly J seconds, where J is some fixed constant. In other words, if learned that, when T reads T (i), the service time should be T + T,then sets a equal to 1 + ( T/J). Thus, T is always chasing the current ideal clock C (i). It is convenient to assume that there is at least one resynchronization every J seconds. (We can always add a null resynchronization in isthesameastheoldonec (i).) Thus, which the new ideal clock C (i+1) if there is a nonzero correction during each resynchronization, then T is always chasing the current ideal clock but never catches it. Finally, let us make one minor change to this algorithm. Instead of erforming the i th resynchronization when T equals T (i), we erform it when C (i 1) equals T (i). This is a minor difference, since we exect T and C (i 1) to be close together when either of them reads T (i). However, as will be a little more closely synchronized to one another than the service times T,soitis slightly better to use them to control the resynchronization. We can now restate our algorithm formally as follows, where J is a fixed arameter. Proosition 5 below indicates, the different ideal clocks C (i) Resynchronization Algorithm: Let T (0), T (1),... be an unbounded increasing sequence of times with T (j+1) T (j) J for all j, letc (0), C (1),... be a sequence of -clocks; for i > 0, let t (i) be the Universal Time such that C (i 1) (t (i) )=T (i) ;andlett (0) be the Universal Time such that 13

20 C (0) (t (0) )=T (0). Then the service clock T is defined for t t (0) by T (t) =a (t)c (t)+b (t) where a and b are defined as follows: For t (0) t t (1) : a (t) =1andb (t) =T (0) C (t (0) ). For t (i) <t t (i+1) b (t) =b (t (i 1), i>0: a (t) =1+(C (i) (t (i) ) T (t (i) )+(a (t (i 1) ) a (t (i) ))C (t (i) ). ))/J and What conditions are required of the ideal clocks C (i) to guarantee that the T defined by the Resynchronization Algorithm satisfy the correct-rate, synchronization, and correct-time conditions? Since each C (i) is a -clock, running at the same rate as C, we know that the ideal clocks C (i) 1,..., C n (i) satisfy the correct-rate condition with bounds ρ. We exect that they must also satisfy the synchronization and correct-time conditions. If the ideal clocks C (i) satisfy these conditions, then the service clocks T will too, rovided that each T remains close to the current ideal clock C (i). (Of course, the actual bounds in these conditions will not be the same for the T as for the C (i).) Since T is always chasing the current C (i), we need a bound on how fast the ideal clocks can change as a result of resynchronization. The required condition is that there exist a constant σ such that, during any time interval of length J, the total amount by which s ideal clocks are changed is less than σ. (The constant J is the arameter of the Resynchronization Algorithm.) bounded correction with constants σ : For all and all j, k: ifj<k and T (k) T (j) <J,then k 1 C (i+1) i=j C (i) <σ The following result shows that the bounded-correction condition ensures that T stays close to C (i). The aearance of e (which equals ) is somewhat surrising. 14

21 Proosition 3 If the C (i) satisfy the bounded-correction condition with constants σ, then the Resynchronization Algorithm ensures that for every Universal Time t such that T (i) C (i) (t) T (i+1) : C (i) (t) T (t) <eσ /(e 1) where terms of order σ 2 /J are neglected. Proof : Define the time-deendent value C by C(t) =C (i) (t), for i determined by the condition T (i) C (i) t <T (i+1). In other words, C(t) isthe time read at Universal Time t by the ideal clock C (i) being used at time t. Observe that C(t) advances at the same rate as C (t) excetthatitis incremented by C (i) C (i 1) at the resynchronization (Universal) times t (i). The bounded-correction condition means that the sum of the absolute value of all such corrections erformed during an interval of length J as measured by C is less than σ. For convenience, I assume that this condition holds when the length of the interval is measured by s local clock C rather than by UT. This introduces an error in the length of the interval of at most σ, which will introduce an error of order at most σ/j 2 in our bounds. Let α(t) equal C(t) T (t). We must show that α(t) <eσ /(e 1). A rigorous, straightforward roof is obtained by comuting α(t + t) asa function of α(t) and the resynchronizations erformed during the interval (t, t + t). Such a roof is tedious and unenlightening. Instead, a less rigorous but more intuitive roof is given. Since C satisfies the correct-rate condition with bound less than one, we can aroximate it arbitrarily closely on the interval [t, t + t] bya differentiable function C with a strictly ositive derivative. We can then.this substitution leads to the same formulas we would have if C were a erfect clock, with C (t) =t for all t. (In other words, the substitution effects a change of coordinates from Universal Time to local-clock time.) Therefore, we may assume without loss of generality that C is the identity function, so C is the Universal Time clock UT. Let t = t 0 <t 1 < <t n = t + J, and assume that the only resynchronizations with nonzero corrections to C in the interval [t, t + J] occur at the times t i.letc i be the change to C at time t i,andlet t i = t i t i 1.Then ni=1 t i = J and, by the bounded-correction hyothesis, n i=1 c i σ. In addition to the resynchronizations at time t i, we can have an arbitrary number of resynchronizations with zero correction. Such a resynchroniza- relace all functions of Universal Time by their comosition with C 1 15

22 tion has the effect of slowing the rate at which T converges towards C. The maximum value for α is achieved by doing as many such resynchronizations as ossible. The value of α obtained by any finite number of resynchronizations is less than the value obtained in the limiting case of continual resynchronization, in which dα/dt = α/j on each interval (t i 1,t i ). A bit of calculus then shows that from which we deduce α(t i ) < (α(t i 1 )+c i 1 )e t i/j α(t + J) <σ +(α(t)/e) It is easy to show from this that if α(t) <eσ /(e 1) then α(t + J) < eσ /(e 1). To comlete the roof of the roosition, we need only show that α(t) <eσ /(e 1) holds for all t in the initial interval [T (0),T (0) + J]. However, this follows from the bounded-correction hyothesis and the fact that T initially equals C (0),soα(T (0) )=0. I leave it as an exercise for the reader to show that the bound of Proosition 3 is the best ossible one. (Consider a scenario in which there is a resynchronization every J seconds that advances the ideal clock by almost σ and a large number of zero resynchronizations.) Proosition 3 immediately imlies the following two results. Proosition 4 If the C (i) satisfy the bounded-correction condition with constants σ, then the T chosen by the algorithm above satisfy the correct-rate condition with bounds ρ +eσ /(e 1)J (neglecting terms of order σ 2/J 2 ). If, for each fixed i, thec (i) also satisfy the correct-time condition with bounds ɛ on the interval [T (i),t (i+1) ], then the T satisfy the correct-time condition with bounds ɛ + eσ /(e 1) (neglecting terms of order σ 2/J). Proosition 5 If, for each fixed i, thec (i) satisfy the synchronization condition with bounds δ q on the interval [T (i),t (i+1) ], then the T chosen by the algorithm above satisfy the synchronization condition with bounds δ q +(2eσ /(e 1)) (neglecting terms of order σ/j). 2 These results are based uon the assumtion that T initially equals C (0). Suose this is not the case, so T initially differs from the ideal clock C (0) by some quantity T 0. The argument used in the roof of Proosition 3 shows that at time T (0) + J, T will differ from its ideal clock by a quantity 16

23 T 1 that is less than σ + T 0 /e, attimet (0) +2J, T will differ from its ideal clock by T 2 <σ + T 1 /e, and so on. Thus, T will kee getting closer to the ideal clock until it is within eσ /(e 1). Similarly, the synchronization condition assumes that initially T q T < δ q. If this condition is not met, then the bound on T q T will kee getting smaller until it eventually reaches a value less than δ q +(2eσ /(e 1)). 4.2 Synchronization and Time-Correctness of Ideal Clocks Proositions 4 and 5 show that the T satisfy the synchronization and correct-time conditions if the ideal clocks C (i) satisfy these conditions during the interval [T (i),t (i+1) ] and the sequence of ideal clocks C (0), C (1),... satisfies the bounded-correction condition. Moreover, suose that the C (i) satisfy the synchronization and correct-time conditions with bounds δ q and ɛ just at service time T (i) that is, on the interval [T (i),t (i) ]. In other words, suose only that the ideal clocks are synchronized to within δ q and lie within ɛ of Universal Time when they read T (i). It is easy to see that must then satisfy the synchronization and correct-time conditions ontheentireinterval[t (i),t (i+1) ] with bounds δ q +(ρ + ρ q )(T (i+1) T (i) ) and ɛ + ρ (T (i+1) T (i) ), resectively. the C (i) The requirement that the C (i) satisfy the correct-time condition laces and C (i+1) may differ. In articular, if C (i) a bound on how much C (i) satisfies the correct-time condition on the entire interval [T (i),t (i+1) ]with bound ɛ + ρ (T (i+1) T (i) ), and C (i+1) at time T (i+1) with bound ɛ,then C (i+1) satisfies the correct-time condition C (i) < 2ɛ + ρ (T (i+1) T (i) ). These inequalities together with the roositions above easily imly the following result, where the hyothesis asserts that there is at least one and at most r resynchronizations erformed every J seconds. Proosition 6 If, for each i, the clocks C (i) satisfy the synchronization condition with bounds δ q and the correct-time condition with bounds ɛ at time T (i), and there is at least one and at most r of the T (i) in any interval of the form [t, t + J), then the Resynchronization Algorithm guarantees that the clocks T satisfy: the correct-rate condition with bounds ( 1+ e ) ρ + 2re e 1 (e 1)J ɛ (neglecting terms of order (rɛ /J) 2 and ρ 2 ) 17

24 the synchronization condition with bounds (( δ q e ) ) ρ + ρ q J + 4re e 1 e 1 ɛ (neglecting terms of order (rɛ ) 2 /J and ρ 2 J). the correct-time condition with bounds ( 1+ 2re ) ɛ + e 1 ( 1+ e e 1 (neglecting terms of order (rɛ ) 2 /J and ρ 2 J). ) ρ J In order to comute I and I q, rocesses and q can use the values for Universal Time rovided by different subsets of the Universal Time roviders. Process does not care which values are used by rocess q. However, the following result indicates that to achieve the synchronization condition with δ q <ɛ + ɛ q, it is necessary for and q to agree to comute the values C (i) and C q (i) using values of UT obtained from the same set of roviders j. To aly this roosition in our case, let F and G be the functions used to comute T and T q from the values UT (j) (t) broadcast by the Universal Time roviders. As I observed earlier, we exect these functions to be translation invariant. (Recall that U is the width of the interval U.) Proosition 7 Let F and G be translation-invariant functions on n-tules of intervals such that for any intervals U 1,..., U n : if, for each j, U j < ɛ and the intersection of all the U j is nonemty, then G(U 1,...,U n ) F (U 1,...,U n ) <δ. If δ<ɛ, then there is some j such that the values of both F (U 1,...,U n ) and G(U 1,...,U n ) deend uon the value of U j. Proof : We assume that there is no such j and show that ɛ δ. This assumtion imlies that we can renumber the arguments so that, for some k, thevalueoff deends only uon its first k arguments and the value of G deends only uon its last n k arguments. Let F (U, V )= F (U,...,U,V,...,V)andG (U, V )=G(U,...,U,V,...,V), with k coies of U and n k coies of V. The value of F deends only on its first argument and the value of G deends only on its second argument. Let U be an interval of width ɛ, where ɛ < ɛ. Without loss of generality, we can assume that F (U, U) G (U, U). The hyothesis imlies that G (U + ɛ,u) F (U + ɛ,u) <δ. Since the value of G does not deend uon its first argument, G (U + ɛ,u)=g (U, U); similarly, F (U + ɛ,u)= 18

25 F (U + ɛ,u + ɛ ). Hence, G (U, U) F (U + ɛ,u + ɛ ) <δ. However, the translation invariance of F imlies that F (U + ɛ,u + ɛ )=F (U, U)+ɛ. Since F (U, U) G (U, U), this allows us to conclude that ɛ δ. This is true for any ɛ <ɛ, which imlies the desired result ɛ δ. Thus, and q must agree uon a set of Universal Time roviders whose values they will use in comuting C (i) and C q (i). This set may change for different values of i (different resynchronizations). The method described in [3] can be emloyed to obtain agreement on the current set of Universal Time roviders that are to be used. Here, let us assume that the values UT (1),..., UT (m) from m roviders are used. To erform the i th resynchronization, each rocess obtains a set of intervals U (1),...,U (m),whereu (j) is the value obtained from Universal Time rovider j. (It is the current value of UT (j) smeared out by uncertainties in message-transmission time.) Process sets its -clock C (i) to equal F (U (1),...,U (m) )(whenc (i 1) = T (i) ), where F is some real-valued function of m intervals. What roerties must F have? As we indicated above, we exect F to be translation invariant. We also require that F satisfy the Lischitz condition for some seudo-metric. Recall that the Lischitz condition means that changing each argument by less than δ changes the value of F by less than δ. Translation invariance imlies that moving each interval a distance of δ in the same direction changes the value of F by δ, so the Lischitz condition is the strongest continuity bound that can be achieved. The Lischitz condition imlies that we can satisfy the synchronization condition for the C (i) by ensuring that d(u (j) q, U (j) ) <δ q for all j. Intuitively, the Lischitz condition ensures that and q will be closely synchronized if, for each Universal Time rovider j, the values they obtain from j are almost the same. Marzullo s function M f m was defined so that Mf m (U 1,...,U m )isthe largest interval whose endoints lie within at least m f of the intervals U 1,..., U m. One might be temted to define the function F by letting F (U 1,...,U m ) be the midoint of M f m(u 1,...,U m ). However, this function does not satisfy a Lischitz condition; in fact, it is not even continuous. Its discontinuity is illustrated in Figure 1, where m = 4, f = 1, I = M 1 (1) 4 (U,...,U (4) ), and I q = M 1 (1) 4 (U q,...,u (4) q ). In this examle, the values of U (j) and U (j) q are ones that could be obtained if Universal Time rovider 1 is faulty. Processes and q see the same values of U (2), U (3),and 19

26 U (1) U (1) q U (2) U (3) U (2) q U (3) q U (4) U (4) q I I q (a) Process s comutation. (b) Process q s comutation. Figure 1: Examle of discontinuity of Marzullo s function. U (4), and they see values of U (1) that are almost the same. However, when they aly Marzullo s function to these sets of intervals, they comute very different intervals I and I q. Recall that Marzullo s function comutes the otimal value of I that is, the smallest interval that knows to contain UT. It is this discontinuity in the otimal I that makes it imossible, in the resence of malicious faults, to satisfy the synchronization condition with the extra requirement that T lies within I. More recisely, it can be shown that if a nonfaulty rocess may assume that it is nonfaulty, then the synchronization condition is incomatible with the requirement that each T lies within I when the δ q are smaller than half the widths of the intervals UT (j). There are a number of functions F that are translation invariant and satisfy the Lischitz condition for a suitable choice of seudo-metric. Two such functions are obtained by letting F (U 1,...,U m ) equal the average or the median of the midoints of the U i. (These functions satisfy the Lischitz condition for the midoint seudo-metric and therefore for the uniform metric.) A class of functions that includes both of these is defined as follows. Let A f (U 1,...,U m ) be the average of the multiset of m 2f numbers obtained by taking the midoints of all the U i and omitting the f lowest and f highest of them. Each A f (with m>2l) is translation invariant and satisfies the Lischitz condition for the midoint seudo-metric. (This follows from the result that, if the numbers x i and y i,with1 i m, satisfy 20

27 x i y i <δ, then for any s, thes th largest of the x i and the s th largest of the y i differ by at most δ.) The functions A f actually give us a somewhat stronger bound on δ q than that obtained simly from the Lischitz condition. If d m (U (j), U (j) q ) <δ(j), then using A f to comute C (i) and C q (i) gives an algorithm that satisfies the synchronization condition with bounds δ q equal to the average of the m 2f largest of the δ(j). If the worst-case difference δ(j) between the values that and q obtain from Universal Time rovider j deends uon j, then the Lischitz condition guarantees only that δ q is no larger than the maximum of the δ(j), while the averaging function A f can do better. However, the different values of δ(j) will robably be almost the same in a ractical alication, so this is not significant. <ɛ. Suose that at most f of the Universal Time roviders may be faulty. If m 2f, so at least half the Universal Time roviders are faulty, there is not much hoe of finding any algorithm that satisfies the correct-time condition, since all the faulty roviders could give the same incorrect value. 4 Therefore, we can assume m>2f. It is then easy to show that there exist nonfaulty roviders Next, we consider the correct-time condition: UT C (i) j and j such that A f (U (1),...,U (m) ) lies between the midoints of U (j) and U (j ). Combining this with the result above for the synchronization condition, it is easy to rove the following result. Proosition 8 With the notation of the Resynchronization Algorithm, let U (j) be intervals such that, for all j: 1. For all and q: d m (U (j), U (j) q ) <δ q. 2. For all : if Universal Time rovider j is nonfaulty, then t (i) U (j). where each C (i) is chosen so that C (i) (t (i) ) = A f (U (1),...,U (m) ). If there are at most f faulty Universal Time Providers, then the C (i) satisfy the synchronization condition with bounds δ q (neglecting terms of order (ρ + ρ q )δ q ) and the correct-time condition with bounds max{ U (j) /2 : rovider j nonfaulty} at time T (i). Observe that A f (U 1,...,U m ) deends only uon the midoints of the U i,soa f (U 1,...,U m )=A f (U 1 ±ξ 1,...,U m ±ξ m ) for any numbers ξ j. While 4 However, even with more than half the roviders faulty, it is still ossible to satisfy the synchronization condition. 21