LiMoSense: live monitoring in dynamic sensor networks

Transcription

1 Dstrb. Comput. DOI /s LMoSense: lve montorng n dynamc sensor networks Ittay Eyal Idt Kedar Raphael Rom Receved: 7 Aprl 2013 / Accepted: 19 March 2014 Sprnger-Verlag Berln Hedelberg 2014 Abstract We present LMoSense, a fault-tolerant lve montorng algorthm for dynamc sensor networks. Ths s the frst asynchronous robust average aggregaton algorthm that performs lve montorng,.e., t constantly obtans a tmely and accurate pcture of dynamcally changng data. LMoSense uses gossp to dynamcally track and aggregate a large collecton of ever-changng sensor reads. It overcomes message loss, node falures and recoveres, and dynamc network topology changes. The algorthm uses a novel technue to bound varable sze. We present the algorthm and formally prove ts correctness. We use smulatons to llustrate ts ablty to uckly react to changes of both the network topology and the sensor reads, and to provde accurate nformaton. A prelmnary verson of ths paper appears n the proceedngs of the 7th Internatonal Symposum on Algorthms for Sensor Systems, Wreless Ad Hoc Networks and Autonomous Moble Enttes ALGOSENSOR) [7]. Ths work was partally supported by the Israel Scence Foundaton ISF), Technon Funds for Securty Research, the Technon Autonomous Systems Program TASP), the Intel Collaboratve Research Insttute for Computatonal Intellgence ICRI-CI), and the Hasso-Plattner Insttute for Software Systems Engneerng HPI). I. Eyal B) Department of Computer Scence, Cornell Unversty, Ithaca, NY, USA e-mal: [email protected] I. Kedar R. Rom Department of Electrcal Engneerng, Technon, Hafa, Israel e-mal: [email protected] R. Rom e-mal: [email protected] 1 Introducton To perform montorng of large envronments, we can expect to see n years to come sensor networks wth thousands of lght-weght nodes montorng condtons lke sesmc actvty, humdty or temperature [2,19]. Each of these nodes s comprsed of a sensor, a wreless communcaton module to connect wth close-by nodes, a processng unt and some storage. The nature of these wdely spread networks prohbts a centralzed soluton n whch the raw montored data s accumulated at a sngle locaton. Specfcally, all sensors cannot drectly communcate wth a central unt. Fortunately, often the raw data s not necessary. Rather, an aggregate that can be computed nsde the network, such as the sum or average of sensor reads, s of nterest. For example, when measurng ranfall, one s nterested only n the total amount of ran, and not n the ndvdual reads at each of the sensors. Smlarly, one may be nterested n the average humdty or temperature rather than mnor local rregulartes. In dynamc settngs, t s partcularly mportant to perform lve montorng,.e., to constantly obtan a tmely and accurate pcture of the ever-changng data. However, most prevous solutons have focused on a statc sngle-shot) verson of the problem, where the average of a sngle nput-set s calculated [4,14,16,15]. Though t s n prncple possble to perform lve montorng usng multple teratons of such algorthms, ths approach s not adeuate, due to the nherent tradeoff t nduces between accuracy and speed of detecton. For further detals on prevous work, see Sect. 2. In ths paper we tackle the problem of lve montorng n a dynamc sensor network. Ths problem s partcularly challengng due to the dynamc nature of sensor networks, where nodes may fal and may be added on the fly churn), and the network topology may change due to battery decay or weather change. The formal model and problem defnton appear n Sect. 3.

2 I. Eyal et al. In Sect. 4 we present our new Lve Montorng for Sensor networks algorthm, LMoSense. Our algorthm computes the average over a dynamcally changng collecton of sensor reads. The algorthm has each node calculate an estmate of the average, whch contnuously converges to the current average. The space complexty at each node s lnear n the number of ts neghbors, and message complexty s that of the sensed values plus a constant. At ts core, LMoSense employs gossp-based aggregaton [14,16], wth a new approach to accommodate data changes whle the aggregaton s on-gong. Ths s trcky, because when a sensor read changes, ts old value should be removed from the system after t has propagated to other nodes. LMoSense further employs a new technue to accommodate message loss, falures, and dynamc network behavor n asynchronous settngs. Ths s agan dffcult, snce a node cannot know whether a prevous message t had sent over a faulty lnk has arrved or not. In Sect. 5, we prove the correctness of the algorthm, showng that once the network stablzes, n the sense that no more value or topology changes occur, LMoSense eventually converges to the correct average, despte message loss. To demonstrate the dynamc behavor of LMoSense, we present n Sect. 6 results of smulatons of representatve scenaros that demonstrate the dynamc reactons. We observe the algorthm s uck reacton to dynamc data read changes and fault tolerance. 2 Related work To gather nformaton n a sensor network, one typcally reles on n-network aggregaton of sensor reads. The vast majorty of the lterature on aggregaton has focused on obtanng a sngle summary of sensed data, assumng these reads do not change whle the aggregaton protocol s runnng [4,15,14,16]. For obtanng a sngle aggregate, two man approaches were employed. The frst s herarchcal gatherng to a sngle base staton [15]. The herarchcal method ncurs consderable resource waste for tree mantenance, and results n aggregaton errors n dynamc envronments, as shown n [10]. The second approach s gossp-based aggregaton at all nodes. To avod countng the same data multple tmes, Nath et al. [17] employ order and duplcate nsenstve ODI) functons to aggregate nputs n the face of message loss and a dynamc topology. However, these functons do not support dynamc nputs or node falures. Moreover, due to the nature of the ODI functons used, the algorthms accuracy s nherently lmted they do not converge to an accurate value [9]. An alternatve approach to gossp-based aggregaton s presented by Kempe et al. [14]. They ntroduce Push-Sum, an average aggregaton algorthm, and bound ts convergence rate, showng that t converges exponentally fast n fully connected networks where nodes operate n lock-step. Fangan and Zamper [8] analyze the exact convergence rate for a fully connected network, and Boyd et al. [5] analyze ths algorthm n an arbtrary topology. Jelasty et al. [11,12] perodcally restart a symmetrc verson of the push-sum algorthm to handle dynamc settngs, tradng off accuracy and bandwdth. Although these algorthms do not deal wth dynamc nputs and topology as we do, we borrow some technues from them. In partcular, our algorthm s nspred by the Push-Sum construct, and operates n a smlar manner n statc settngs. The aforementoned analyses therefore apply to our algorthm f and when the system stablzes. Another approach [6] utlzes broadcast to expedte convergence, however unlke our soluton t does not allow for message loss, and wth a dynamc topology nodes cannot montor who got ther messages. Wuhb et al. ntroduce G-GAP [20], a sngle-shot algorthm for robust aggregaton of averages. Ther message acknowledge mechansm has smlartes to ours, however t addresses a system wth much stronger falure detecton assumptons: They assume no message loss that s, a node knows whether ts message are receved), and no smultaneous crashes. We are aware of two approaches to aggregate dynamc nputs. The frst, by Brk et al. [3], s lmted to unrealstc settngs, namely a statc topology wth relable communcaton lnks, falure freedom, and synchronous operaton. The second approach, called flow updates FU) solves aggregaton n dynamc settngs, overcomng message loss, dynamc topology and churn, albet n synchronous settngs only, runnng n rounds. In [1,13], the authors also solve aggregaton n dynamc settngs, overcomng message loss, dynamc topology and churn. In [13] they provde an emprcal evaluaton and n [1] they prove correctness and convergence rate for statc-nput scenaros. However, they consder only synchronous settngs, and they do not prove correctness nor analyze the behavour of ther algorthm wth dynamc nputs. Note that aggregaton n sensor networks s dstnct from other aggregaton problems, such as stream aggregaton, where the data n a sldng wndow s summarzed. In the latter, a sngle system component has the entre data, and the dstrbuted aspects do not exst. 3 Model and problem defnton 3.1 Model The system s comprsed of a dynamc set of nodes sensors), partally connected by dynamc undrected communcaton lnks. Two nodes connected by a lnk are called neghbors,

3 LMoSense: lve montorng and they can send messages to each other. These messages ether arrve at some later tme, or are lost. Messages that are not lost on each lnk arrve n FIFO order. Lnks do not generate or duplcate messages. The system s asynchronous and progresses n steps, where n each step an event happens and the approprate node s notfed, or a node acts spontaneously. Spontaneous steps occur nfntely often. In a step, a node may change ts nternal state and send messages to ts neghbors. Nodes can be dynamcally added to the system, and may fal or be removed from the system churn). The set of nodes at tme t s denoted N t and ther number n t.thesystem state at tme t conssts of the nternal states of all nodes n N t, and the lnks among them. When a node s added nt event), t s notfed, and ts nternal state becomes a part of the system state. When t s removed remove event), t s not allowed to perform any acton, and ts nternal state s removed from the system state. Each sensor has a tme varyng data read n R. A node s ntal data read s provded as a parameter when t s notfed of ts nt event. Ths value may later change change event) and the node s notfed wth the newly read value. For a node n N t, we denote 1 by r t, the latest data read provded by an nt or change event at that node before tme t. Communcaton lnks may be added or removed from the system. A node s notfed of lnk addton addneghbor event) and removal removeneghbor event), gven the dentty of the lnk that was added or removed. We call these topology events. 2 For convenence of presentaton, we assume that ntally, nodes have no lnks, and they are notfed of ther neghbors by a seres of addneghbor events. We say that a lnk, j) s up at step t f by step t, both nodes and j had receved an approprate addneghbor notfcaton and no later removeneghbor notfcaton. Note that a lnk, j) may be half-up n the sense that the node was notfed of ts addton but node j was not, or f node j had faled. A node may send messages on a lnk only f the last message t had receved regardng the state of the lnk s addneghbor. If ths s the case, the node may also receve a message on the lnk receve event). Global stablzaton tme We defne global stablzaton tme, GST, to be the frst tme from whch onward the followng propertes hold: 1) The system s statc,.e., there are no change, nt, remove, addneghbor or removeneghbor events. 2) If the latest topology event a node N GST has receved for another node j 1 For any varable, the node t belongs to s wrtten n subscrpt and, when relevant, the tme s wrtten n superscrpt. 2 There s a rch lterature dealng wth the means of detectng falures, usually wth tmeouts. Ths subject s outsde the scope of ths work. s addneghbor, then node j s alve, and the latest topology event j has receved for s also addneghbor.e., there are no half-up lnks). 3) The network s connected. 4) If a lnk s up after GST, and nfntely many messages are sent on t, then nfntely many of them arrve. 3.2 The lve average montorng problem We defne the read average of the system at tme t as R t = 1 N t N t r t. Note that the read average does not change after GST. Our goal s to have all nodes estmate the read average after GST. More formally, an algorthm solves the Lve Average Montorng Problem f t gets tme-varyng data reads as ts nputs, and has nodes contnuously output ther estmates of the average, such that at every node n N GST, the output estmate converges to the read average after GST. 4 The LMoSense algorthm In Sect. 4.1 we descrbe a smplfed verson of the algorthm for dynamc nputs but statc topology and no falures. Ths smplfed verson demonstrates our novel approach for handlng dynamc nputs. However, ths smplfed verson s unable to accommodate topology changes, churn, and message loss. To overcome these, we present n Sect. 4.2 a robust algorthm, n whch each node mantans for each of ts lnks a summary of the data communcated over that lnk thereby enablng t to recover after these changes. These summares, however, are aggregates of all exchanges on the lnks, and ther sze grows, unboundedly. In Sect. 4.3, we descrbe the complete LMoSense algorthm, whch also mplements a clearng mechansm that results n bounded szes of all ts varables and messages, wthout resortng to atomcty or synchrony assumptons. 4.1 Falure-free dynamc algorthm We begn by descrbng a verson of the algorthm that handles dynamcally changng nputs, but assumes no message loss or lnk or node falures. The pseudocode s shown n Algorthm 1. The base of the algorthm operates lke Push-Sum [4,14]: Each node mantans a weghted estmate of the read average a par contanng the estmate and a weght), whch s updated as a result of the node s communcaton wth ts neghbors. As the algorthm progresses, the estmate converges to the read average. In order to accommodate dynamc reads, a node whose read value changes must notfy the other nodes. It not only needs to ntroduce the new value, but also needs to undo the

4 I. Eyal et al. Algorthm 1: Falure-Free Dynamc Algorthm 1 state 2 est,w ) R 2 3 prevread R 4 on nt ntval) 5 est,w ) ntval, 1) 6 prevread ntval 7 on receve v n,w n )) from j 8 est,w ) est,w ) v n,w n ) 9 perodcally send ) 10 f w 2 then 11 Choose a neghbor j farly 12 w w /2 13 send est,w )) to j 14 on change newread) 15 est est + 1 w newread prevread ) 16 prevread newread effect of ts prevous read value, whch by now has partally propagated through the network. The algorthm often reures nodes to merge two weghted values nto one. They do so usng the weghted value sum operaton, whch we defne below and concsely denote by. Subtracton operatons wll be used later, they are denoted by and are also defned below. The and operatons are undefned when the sum resp. dfference) between the weghts of the operands s zero. We note that the operaton s commutatve and both operatons are assocatve. ) v a,w a ) v b,w b ) = va w a + v b w b,w a + w b. 1) w a + w b v a,w a ) v b,w b ) = v a,w a ) v b, w b ). 2) The state of a node lnes 2 3) conssts of a weghted value, est,w ), where est s an output varable holdng the node s estmate of the read average, and the value prevread of the latest data read. We assume at ths stage that each node knows ts set of neghbors. We shall remove ths assumpton later, n the robust LMoSense algorthm. Node ntalzes ts state on ts nt event. The data read s ntalzed to the gven value ntval, and the estmate s ntval, 1) lnes 5 6). The algorthm s mplemented wth the functons receve and change, whch are called n response to events, and the functon send, whch s called perodcally. Perodcally, a node shares ts estmate wth a neghbor j chosen farly lne 11). Farly means that each neghbor s chosen nfntely often. Node transfers half of ts estmate to node j by halvng the weght w of ts locally stored estmate and sendng the same weghted value to that neghbor lnes 12 13). When the neghbor receves the message, t merges the accepted weghted value wth ts own lne 8). Nodes keep ther weghts larger than some small arbtrary sze, by performng a send only f the node s weght s larger than 2. A small value of therefore ncreases the weght sendng freuency among nodes, but t does not affect the accuracy of estmaton. Correctness of the algorthm n statc settngs follows from two key observatons. Frst, safety of the algorthm s preserved, because the system-wde weghted average over all weghted-value estmate pars at all nodes and all communcaton lnks s always the correct read average; ths nvarant s preserved by send and receve operatons. Thus, no nformaton s ever lost. Second, the algorthm s convergence follows from the fact that when a node merges ts estmate wth that receved from a neghbor, the result s closer to the read average than the furthest of the two. We proceed to dscuss the dynamc operaton of the algorthm. When a node s data read changes, the read average changes, and so the estmate should change as well. Let us denote the prevous read of node by r t 1 and the new read at step t by r t. In essence, the new read, r t, should be added to the system-wde estmate wth weght 1, whle the old read,, ought to be deducted from t, also wth weght 1. But snce the old value has been dstrbuted to an unknown set of nodes, we cannot smply recall t. Instead, we make the approprate adjustment locally, allowng the natural flow of the algorthm to propagate t. We now explan how we compute the local adjustment. The system-wde estmate should shft by the dfference between the read values, factored by the relatve nfluence of a sngle sensor,.e., 1/n. So an ncrease of x ncreases the system-wde estmate by x/n. However, when a node s read value changes, ts estmate has an arbtrary weght of w, so we need to factor the change of ts value by 1/w to obtan the reured system-wde shft. Therefore, n response to a change event at tme t, f the node s estmate before the r t 1 change was est t 1 and ts weght was w t 1, then the estmate s updated to lnes 15 16) est t = estt 1 + r t r t 1 )/w t 1. Note that the value of n t does not appear n the euaton, as t s unknown to any of the nodes. 4.2 Addng robustness Overcomng falures s challengng n an asynchronous system, where a node cannot determne whether a message t had sent was successfully receved. In order to overcome message loss and lnk and node falure, each node mantans a summary of ts conversatons wth each of ts neghbors. Each node mantans the aggregates as weghted sums) of the messages receved from and sent to node j n the varables recevedtotal j) and senttotal j), respectvely. Nodes nteract by sendng and recevng these summares, rather than weghted values as n the falure-free algorthm. The data n each message subsumes all prevous value

5 LMoSense: lve montorng exchanges on the same lnk. Thus, f a message s lost, the lost data s recovered once an ensung message arrves. When a lnk fals, the nodes at both of ts ends use the summares to retroactvely cancel the effect of all the messages ever transferred over t. A node falure s treated as the falure of all ts lnks. The resultng algorthm, whose pseudocode s gven n Algorthm 2, s robust to message loss, lnk falure, and churn. To send, node adds to senttotal j) the weghted value t wants to send, and sends senttotal j) to j lnes 18 20). When recevng ths message, node j calculates the newly receved weghted value by subtractng ts recevedtotal j) varable from the newly receved aggregate lne 22). After actng on the receved message lne 23), node j replaces ts recevedtotal varable wth the new weghted value lne 34). Thus, f a message s lost, the next receved message compensates for the loss and brngs the recevng neghbor to the same state t would have reached had t receved the lost messages as well. Whenever the most recent message on a lnk, j) s correctly receved and there are no messages n transt, the value of senttotal j s dentcal to the value of recevedtotal j. In order to overcome message loss, a node sends ts summary to ts neghbor j even f ts current weght s smaller than 2, and the message carres no new nformaton. Upon notfcaton of topology events, nodes act as follows. When notfed of an addneghbor event, a node smply adds the new neghbor to ts neghbors lst lne 29). When notfed of a removeneghbor event, a node reacts by nullfyng the effect of ths lnk, clearng the state varables, removng the neghbor from ts neghbors lst, and dscardng ts lnk records lnes 31 35). Here, the neghbor set of node may be dfferent on each call to send. Far n ths case means that f n nfntely many calls of send a node j s a neghbor j neghbors ) nfntely often, then j s chosen nfntely often. The effects of sent and receved messages are summarzed n the respectve senttotal and recevedtotal varables. When a node dscovers that lnk, j) faled, t adds the outgong lnk summary senttotal j to ts estmate, thus cancellng the effect of ever havng sent anythng on the lnk. The ncomng lnk summary, however, s not drectly subtracted from the estmate, n order to prevent ts weght from becomng negatve. Instead, the node adds the ncomng lnk summary recevedtotal j to a buffer the weghted value unrecvval, unrecvweght). It then lazly subtracts t from ts estmate, preservng the estmate weght postve lnes 13 16). The node thus cancels the effect of everythng t has receved from that neghbor. After a node jons the system or leaves t, ts neghbors are notfed of the approprate topology events, addng lnks to the new node, or removng lnks to the faled one.thus, when a node fals, any part of ts read value that had propagated through the system s annulled, and t no longer contrbutes to the system-wde estmate. 4.3 LMoSense The summary approach of Algorthm 2 causes summary szes, namely the weghts of recevedtotal j) and senttotal j), to ncrease unboundedly as the algorthm progresses. To avod that, we devse a channel reset mechansm that prevents ths wthout resortng to synchronzaton assumptons. Instead of storng the aggregates of all receved and sent weghts, we store only ther dfference,whch can be bounded, and we store the receved and sent aggregates only for lmted epochs, thereby boundng them as well. Algorthm 2: Robust Dynamc Algorthm wth Unbounded State 1 state 2 est,w ) R 2 3 prevread R 4 neghbors N, ntally 5 senttotal : N R 2, ntally j : senttotal j) = 0, 0) 6 recevedtotal : N R 2, ntally j : recevedtotal j) = 0, 0) 7 unrecvval, unrecvweght ) R 2, ntally 0, 0) 8 on nt ntval) 9 est,w ) ntval, 1) 10 prevread ntval 11 perodcally send ) 12 Choose a neghbor j farly 13 f w 2 then 14 w tounrecv mnunrecvweght,w ) 15 est,w ) est,w ) unrecvval,w tounrecv ) 16 unrecvweght unrecvweght w tounrecv 17 f w >= 2 then 18 senttotal j) senttotal j) est,w /2) 19 est,w ) est,w /2) 20 send senttotal j) to j 21 on receve v n,w n ) from j 22 dff v n,w n ) recevedtotal j) 23 est,w ) est,w ) dff 24 recevedtotal j) v n,w n ) 25 on change r new ) 26 est est + 1 w r new prevread ) 27 prevread r new 28 on addneghbor j) 29 neghbors neghbors {j} 30 on removeneghbor j) 31 est,w ) est,w ) senttotal j) 32 unrecvval, unrecvweght) unrecvval, unrecvweght) recevedtotal j) 33 neghbors neghbors \{j} 34 senttotal j) 0, 0) 35 recevedtotal j) 0, 0)

6 I. Eyal et al. Algorthm 3a: LMoSense part 1 1 state 2 est,w ) R 2 3 prevread R 4 neghbors N, ntally 5 totaldff : N R 2, ntally j : totaldff j) = 0, 0) 6 unrecvval, unrecvweght ) R 2, ntally 0, 0) 7 sent : N R 2, ntally j : sent j) = 0, 0) 8 outsn : N {0, 1}, ntally j : outsn j) = 0, 0) Seral number of outgong messages) 9 receved : N R 2, ntally j : receved j) = 0, 0) 10 nsn : N {0, 1}, ntally j : nsn j) = 0 Expected seral number of ncomng messages) 11 cleared : N R 2, ntally j : cleared j) = 0, 0) Weght receved wth prevous seral number) 12 on nt ntval) 13 est,w ) ntval, 1) 14 prevread ntval 15 perodcally send ) 16 Choose a neghbor j farly 17 f w >= 2 then 18 w tounrecv mnunrecvweght,w ) 19 est,w ) est,w ) unrecvval,w tounrecv ) 20 unrecvweght unrecvweght w tounrecv 21 f w >= 2 and weght of totaldff j)> 2 bound and weght of sent j)<2 bound then 22 sent j) sent j) est,w /2) senttotal j) senttotal j) + est,w /2)) 23 totaldff j) totaldff j) est,w /2) 24 est,w ) est,w /2) 25 send sent j), outsn j), nsn j) 1 mod 2, cleared j)) to j Ack cleared vals and seral of prevous epoch) 26 on receve v n,w n ), msgsn, clearsn, clearval) from j 27 f clearsn = outsn j) then Relevant clear) 28 outsn j) outsn j) + 1 mod 2 29 sent j) sent j) clearval sentcleared j) sentcleared j) + clearval) 30 f msgsn = nsn j) then Relevant message) 31 dff v n,w n ) receved j) 32 est,w ) est,w ) dff 33 totaldff j) totaldff j) dff 34 receved j) v n,w n ) recevedtotal j) recevedtotal j) + dff) 35 f weght of receved j)) > bound then Reset the channel) 36 nsn j) nsn j) + 1 mod 2 37 cleared j) receved j) recevedcleared j) recevedcleared j) + receved j)) 38 receved j) 0, 0) The result s the full LMoSense algorthm, shown as Algorthms 3a, 3b, where the state nformaton of Algorthm 2 s replaced wth a more elaborate scheme. Messages are aggregated n epochs, and the aggregate s reset on epoch change. Epochs are defned per node, per lnk, and per drecton, and are dentfed by bnary seral numbers, so each node mantans an ncomng and an outgong seral Algorthm 3b: LMoSense part 2 39 on change r new ) 40 est est + 1 w r new prevread ) 41 prevread r new 42 on addneghbor j) 43 neghbors neghbors {j} 44 nsn j) 0 45 outsn j) 0 46 on removeneghbor j) 47 f Weght of totaldff j)<0 then 48 est,w ) est,w ) totaldff j) 49 else 50 unrecvval, unrecvweght ) unrecvval, unrecvweght ) totaldff j) 51 neghbors neghbors \{j} 52 totaldff j) 0, 0) recevedcleared j) 0, 0), sentcleared j) 0, 0)) 53 sent j) 0, 0) 54 receved j) 0, 0) 55 cleared j) 0, 0) number per lnk. Node mantans for ts lnk wth node j the seral numbers nsn j) and outsn j) for the ncomng and outgong weghts, respectvely. Epochs on dfferent drected lnks are ndependent of each other. Neghborng nodes reset ther aggregates for ther connectng drected lnk and proceed to the next epoch after reachng consensus on the aggregate values sent n the current epoch. Ths approach s smlar to the classcal stop-and-wat message exchange protocol [18]. However, here the recevng end of the lnk ntates the transton to the next epoch, after recevng multple messages. Intutvely, the stop-and-wat s performed for the ACKs, each of whch acknowledges a set of weght transfers. For a lnk, j), node mantans n sent j) and receved j) the aggregate sent and receved values n the current epoch rather than the entre hstory as n the falurefree algorthm). In addton, t mantans n totaldff j) the dfference between the sent and receved aggregates over the entre hstory. A channel reset for a lnk, j) s ntated by the recever j when t notces that the weght n receved j) reaches a bound bound lne 35). Ths s an arbtrary large bound, for example n, of the weght stored. Node j then 1) ncrements modulo 2 the seral number on that lnk, 2) adds the aggregate receved values n the completed epoch to ts totaldff j) summary of the lnk, and 3) clears the aggregate by storng receved j) n cleared j) and settng receved j) to zero lnes 36 38). Node j wll not accept future messages for the prevous seral number t wll smply gnore them. On ts next send to n the nverse drecton), node j s message wll update of the epoch reset by sendng the ndex and fnal aggregate of the completed epoch lne 25).

7 LMoSense: lve montorng When notfed of the channel reset, the sender resets the aggregate for that channel, and ncreases modulo 2 the seral number as well. Note that may have sent messages wth the old seral number after the recever reset the lnk, but these messages are gnored by j. To prevent ths weght from beng lost, node does not reset ts aggregate to zero, but rather to the aggregate of messages sent wth the old seral number but not cleared lne 29). Upon notfcaton of topology events, nodes act as follows. When notfed of an addneghbor event, a node adds the new neghbor to ts neghbors lst and resets the epoch seral numbers for the lnk lnes 43 45). When notfed of a removeneghbor event, a node removes the neghbor from ts neghbors lst and dscards ts lnk records. Addtonally, t subtracts totaldff from ts estmate, thus cancellng the effect of ever havng communcated over the lnk. Unlke Algorthm 2, we cannot separate here the sent from the receved weghts. To prevent the estmate weght from beng negatve, we check f the weght n totaldff s postve. If t s, we add t to an aggregate buffer unrecvval, unrecvweght), whch s later subtracted n stages from totaldff on send events), as before. We follow n comments the behavor of four vrtual varables, the total sent and receved aggregates n senttotal j) and recevedtotal j), respectvely, and the aggregates of everythng that was ever cleared from sent j)and receved j) n sentcleared j) and recevedcleared j), respectvely. These vrtual varables all grow unboundedly as the algorthm progresses and we wll use them for provng correctness n Sect Correctness In ths secton, we show that the LMoSense algorthm Algorthms 3a, 3b) adapts to network topology as well as value changes and converges to the correct average. We start n Sect. 5.1 by provng that when there are no half-up lnks, a combnaton of the system s varables euals the read sum. Then, n Sect. 5.2, we prove that after GST the estmates at all nodes eventually converge to the average of the latest read values. 5.1 Invarant We denote by R t, n t ) the read sum at tme t, asshownn E. 3. R t, n t ) = n =1 r t, 1) 3) We denote by E t, n) the weghted sum over all nodes at tme t of ther 1) weghted values, 2) outgong lnk summares n ther sent varables, 3) the nverse of ther ncomng summares n ther receved varables, and 4) the latest cleared receved aggregate, f ther neghbor has not yet receved the reset message. The sum s shown n E. 4 E t, n t ) est t,wt ) unrecvvalt, unrecvweghtt ) n = sent t j) receved t =1 j)) cleared t j) j neghbors t j neghbors t s.t. nsn t j) = outsnt j ) We show that f there are no half-up lnks n the system each lnk s known to be up or down by both ts nodes), then R t = E t. Lemma 1 For any tme t, f for any nodes and j, ether j neghbors t neghborst j or j neghbors t neghborst j, then R t = E t. We begn by analyzng the effect of communcaton steps, then of dynamc steps, and then conclude by provng the statement. Statc behavor Frst, we consder send and receve events. Note that message loss s not an event and does not affect the state of the system. In partcular, t does not affect the correctness of ths lemma or the followng ones. Lemma 2 Statc operatons) If step t s ether send or receve at some node, then R t E t = R t 1 E t 1. Proof Frst, consder a send step. If the weght n s below the threshold of 2, no varables are changed lnes 17 and 21), so the lemma trvally holds. If the weght s above the threshold, then a certan weght s subtracted from the pars unrecvval t 1, unrecvweght t 1 ) and est t 1,w t 1 ) lnes 18 20). Snce the two pars appear wth opposte sgns n E. 4, thevalueofe s unchanged. If the weght n s stll above the threshold of 2, then the weghted value est t 1, 2 1 wt 1 ) s subtracted from the weghted value of node, and added to sent j g Et, n) accordng to E. 4 unchanged. Next consder a receve step. Lnes handle the outgong lnk to j. If the message s clearsn s the same as the current outsn j) lne 27), t causes a reset. Node reacts by resettng sent j) and ncrementng outsn j). Incrementng 4)

8 I. Eyal et al. outsn j) makes t eual to ts counterpart nsn j ),removng the negatve cleared j ) element from the sum n E. 4. Decreasng sent j) by the same value, leaves E t n E. 4 unchanged. Next, f the ncomng message carres the approprate seral number, the ncomng value deductng the prevously receved value from the ncomng aggregate) s added to the weghted value of node j, and the same weghted value s added to receved j. Snce the latter s subtracted n E. 4, ths leaves E t, n) unchanged. Fnally, f the weght n the ncomng message s too hgh, the recever ntates a channel reset. Note that the ncomng message seral number euals outsn j ). The node ncrements nsn j), causng cleared j) to be counted n the sum, snce after the change t becomes dfferent than outsn j ). Then t stores the value of receved j) n cleared j), and nullfes receved j), both wth negatve sgn n E. 4, leavng E t, n) unchanged. None of these events changes the read sum, therefore, snce nether the read sum nor E t, n) change, R t E t = R t 1 E t 1. Dynamc values When the value read by node changes from r t 1 to r t,the node updates ts estmate n a manner that changes E, n) correctly, as shown n the followng lemma. Lemma 3 Read value change) If step t s change at node, then R t E t = R t 1 E t 1 Proof After the change of the read value, the new read average s R t = R t 1 + r t t 1 r, and the weghted value ) 3 n t 1 est t 1 + r t t 1 r w,w replaces the weghted value of node. We show that the new E, n) changes just lke the read sum: E t, n t ) = E t 1, n t 1 ) est t 1 est t 1,w t 1 + r t r t 1 w t 1,w t 1 = E t 1 + r t r t 1 n t 1, n t 1 ), leavng the dfference between R and E unchanged. Dynamc topology When a lnk s added, the node addng t starts to keep track of the messages passed on the lnk. When a lnk s removed, the 3 Note that the weght at a node never drops below, so the expresson s vald. ) ) node retroactvely cancels the messages that passed through ths lnk, as f t never exsted. In both cases, both E t and R t are unchanged, as we now show. Lemma 4 Dynamc topology) If step t s addneghbor at node, then R t E t = R t 1 E t 1, and f the lnk between nodes and j fals and ts nodes receve removeneghbor at tmes t and t j respectvely), wth t < t j, then E t, n t ) E t 1, n t 1 ) = E t j 1, n t j 1 ) E t j, n t j ). Proof The addneghbor functon does not affect the read sum or E t, so the clam holds. We proceed to handle lnk falure. When the falure s dscovered at t by, the weghted value totaldff t 1 j) s subtracted from est or added to unrecvval, unrecvweght ) at node, and the varables sent j), receved j) and cleared j) are nullfed. The same happens n j at t j. We note that totaldff j) does not drectly appear n E. 4. We decompose totaldff j) to the dfference between the vrtual varables recevedtotal j) and senttotal j), defned above. totaldff j) = recevedtotal j) senttotal j). We also note that summng vrtual varables sentcleared j) and recevedcleared j), together wth the real varables sent j) and receved j) respectvely) results n senttotal j) and recevedtotal j), respectvely. Therefore, when subtractng totaldff j) n s sde, we subtract totaldff j) = recevedtotal j) senttotal j) = recevedcleared j) receved j) sentcleared j) sent j). Now, the receved j) and sent j) cancel each other on s sde, as they are subtracted and added respectvely) drectly lnes 53 54) when clearng totaldff j) lne 52). On the other hand, recevedcleared j) and sentcleared j) must be canceled by an nverse change on j s sde. Note that f nsn j) = outsn j ), wehave recevedcleared j) = sentcleared j ), and cleared j) s not counted n E. 4, whereas f nsn j) = outsn j ) then cleared j) s counted, and we have recevedcleared j) + cleared j) = sentcleared j ). In both cases, the change s canceled,.e., nverse weghted values are subtracted from/added to est and unrecvval, unrecvweght) respectvely) at t and t j, and the euaton n the lemma holds. Dynamc node set When a node s added, ts state s added to the system. When t s removed, ts state s removed.

9 LMoSense: lve montorng Lemma 5 Dynamc node set) Ifstep t s nt or remove, then R t E t = R t 1 E t 1 Proof An addton of a node wth ntal estmate r t results n R t, n t ) = R t 1, n t 1 ) r t, 1) and Et, n t ) = E t 1, n t 1 ) r t, 1), so ther dfference s unchanged at step t. We model the falure of a node as the falure of all ts lnks, followed by ts removal from the system. The falure of the lnks leaves wth ts most recent read value and a weght of one, r t 1, 1), and all other state varables empty totaldff j), sent j), etc.), wth E t, n) unchanged. Removng the node thus results n R t, n t )=R t 1, n t 1 ) r t 1, 1) and E t, n t ) = E t 1, n t 1 ) r t 1, 1), so ther dfference s unchanged at step t. We are now ready to prove Lemma 1. Proof Intally, at t = 0, the clam holds, snce for any node, the component of the read sum s dentcal to that of E t : r t, 1) = estt, 1). Accordng to Lemmas 2 5, the dfference between R t and E t changes only due to lnk falure events. Snce there are no half-up lnks, then f a node detected the falure of ts lnk wth j before t, then j has also detected the falure of the lnk before t. Lemma 4 shows that the resultng operatons by and j compensate each other, resultng n the reured eualty at t. 5.2 Convergence We show that after GST the estmate at all nodes converges to the read average. Snce after GST messages are not lost, we can smplfy our proof by abstractng away the fact that messages contan aggregated values; nstead, we consder each message to delver only the delta from the prevous one, as translated n the code to the dff varable upon recept lne 31). Frst, we prove n Sect that connected nodes send each other values nfntely often. Then, n Sect , we defne the trackng of the propagaton of the weghted value from a node at tme t at any tme later tme. We proceed to show n Sect that there exsts a tme t after t such that the rato of the weght propagated to any node j from any node, relatve to the total weght at j, s bounded from below. In Sect we construct a seres of such tmes, where n each tme t x the values from t x 1 have propagated and match ths bound. Ths allows us to prove convergence, as reured Far schedulng We begn by provng the followng lemma. Lemma 6 Every node sends weght to each of ts neghbors nfntely often. Proof We prove by contradcton. Assume that a node never sends a message to ts neghbor j. Snce neghbors are chosen farly,.e., each neghbor s chosen nfntely often, ths means the condton of lne 21 evaluates to false. The last part of the condton may evaluate to false only f weght was sent to j but not receved. Once ths weght s receved, node j changes epochs lne 36 38), whch wll reset sent j) once the next message arrves from j to, and the condton wll evaluate to true. Therefore, ether the frst or the second part of the condton do not hold. Assume frst that the frst part does not hold,.e., the weght n s always smaller than 2. Ths means that none of s neghbors ever sends t weght from some tme). Otherwse, eventually unrecvweght would drop to zero, and subseuently est would rse above zero. Assume that none of s neghbors ever sends t weght also due to ther weghts beng smaller than 2, and contnue smlarly,.e., all nodes n the system hold a weght smaller than 2. Snce the entre weght n the est varables s at least n possbly more, f nodes have non-zero unrecvweght varables), at least one node must hold a weght larger than one,.e., larger than 2, and we reach a contradcton. Mantanng our ntal assumpton, we conclude that there exsts some node that never sends weght to a neghbor j snce the second part of the condton holds,.e., t already sent to j much more than t got back. Once a message from successfully reaches j,thevalueof j s totaldff j ) s correctly updated to totaldff j), so t s postve. Therefore the second part of the condton s true n j. However, f j sends weght to, thevalueoftotaldff j) eventually becomes postve, contradctng our assumpton, and we conclude that j stops sendng weght at some pont, snce ts weght never rses above 2. For that to happen, j must not receve weght from any of ts neghbors. So each of j s neghbors ether has a weght less than 2, or has already sent to j more than t got back. If all of j s neghbors ncludng ) have sent t more than they got back, than j s weght would be more than 2, whch we already ruled out. Therefore at least one neghbor k receved from j more than t sent, but t does not send weght to j because ts own k s) weght s too small. Now, the same logc that held for j holds for k, and we contnue ths, formng a chan of nodes. Each node n the chan holds a weght smaller than 2. At the end of such a chan and there s an end, snce the number of nodes s fnte) there s a node z that does not send weght to any of ts neghbors, but has receved from each of them more weght than t has sent. The weght at node z s therefore larger than one, and ts totaldff for all ts neghbors s postve, So the condton n lne 21 holds, and t should have sent weghts to ts neghbors, leadng to a contradcton.

10 I. Eyal et al. After GST, no lnks falures are detected. Snce weghts are sent nfntely often between neghbors by Lemma 6,we conclude that there exsts a tme GST GST after whch the unrecvweght varables at all nodes are zero: Defnton 1 GST) The tme GST s a tme after whch for all N GST and for all t > GST: unrecvweght t = Propagaton trackng We explan how to track the propagaton of the weghted value from a node as of some tme t > GST. The defnton recursvely defnes two components mantaned at each node k: Theprop component, est t,wt k,prop ), whch s the propagaton of s weghted value at t to k at t, and the agg component, est t k,agg,wt k,agg ), whch s the aggregaton from all nodes but.theprop component s called the component of est t at node k at tme t. Though these defntons depend on and t,wefx and t and omt them, to make the expressons cleaner. Defnton 2 Propagaton trackng) Intally, at t, at all nodes k =, agg s the weghted value est t k,wt k ), and prop s 0, 0). At node, agg s 0, 0) and prop s est t,wt ). For all steps t > t: 1. If the operaton at t s a send at node k, then est t k,agg,wt k,agg ) = estt 1 k,agg,wt 1 k,agg /2) and est t,wt k,prop ) = estt,wt 1 k,prop /2) and the message sent s parttoned: est t 1 k,agg,wt 1 k,agg /2) estt,wt 1 k,prop /2). 2. If the operaton at t s a receve at node k of a message v n,w n ) parttoned to est n,agg,w n,agg ) and est t,w n,prop ) components, then est t k,agg,wt k,agg ) = estt 1 k,agg,wt 1 k,agg ) est n,agg,w n,agg ) and est t,wt k,prop )=estt,wt 1 k,prop ) estt,w n,prop ). It can be readly seen that the agg and prop components partton the weghted value at the node k at all tmes t t: est t k,wt k ) = estt k,agg,wt k,agg ) estt,wt k,prop ). We defne the component rato of node at a node k to be the rato between s prop component n k and the total weght at k: Defnton 3 Component rato) The component rato of est t at node k at t > t s w t k,prop w t k,prop + wt k,agg Bounded rato = wt k,prop wk t We proceed to prove that for any tme t after GST, eventually each node has a component of est t wth a rato that s bounded from below. Denote by M s the set of nodes wth an est t component at tme s > t. Denote by s the sum of weghts at the nodes n M s and n messages sent from nodes n M s wth s < s and not yet receved. Lemma 7 Gven two tmes s and t s.t. s > t > GST, at all nodes n Ms, the estt component rato s at least s ) wms /. Proof We prove by nducton on the steps taken from t. We omt the superscrpt for M herenafter. At tme t the only node wth an est t component s wth a rato of one, and the nvarant holds. Consder the system at tme s, assumng the nvarant holds at s 1. We show that after any of the possble events at s, the nvarant contnues to hold. 1. Send: No effect on the nvarant. The rato at the sender stays the same, and s unchanged. 2. Receve from j M by k M : No effect on the nvarant snce no nodes n M are concerned. 3. Receve from j M by k M : Two thngs change: 1) s = + w k and 2) k becomes a part of M. The frst change decreases the lower bound, therefore the assumpton holds at s for all nodes n M. Denote by α the rato at j when t sent the. message. Accordng to the nducton assumpton, α ) wm /. The new rato at k s mnmal when the weght of the receved message s mnmal.e., ). Therefore, the rato at k, whch s now also n M,sat least

11 LMoSense: lve montorng α > > wk = = nducton hypothess ) wm / ) wm / + w k ) ) wm s / s + w k. w k w k +w k w k >1 > ) w k / +w k + wk w k ) We conclude that the rato at all the nodes n M s s larger than the bound at s. 4. Receve from j M by k M : Denote the weght of the message by w n. Two thngs change: 1) s = + w n and 2) the rato at k. The change of decreases the bound, therefore the assumpton holds at s for all nodes other than k. The relatve weght at k before ) wm / recevng s at least w. Therefore, after M recevng the message, t s at least > ) wm / + w n + w n ) +w n = s ) wm s / We conclude that the rato at all the nodes n M s s larger than the bound at s. 5. Receve from j M by k M : The rato does not decrease below the mnmum between the ratos n j and k, therefore the nvarant s correctness follows drectly from the nducton hypothess. Lemma 8 For any tme t > GST and node, there exsts a tme t > t after whch every node j has an est t component wth rato larger than n ) n/.. Proof Once a node has an est t component, t wll always have an est t component no operaton removes t), and eventually t wll succeed sendng a message to all of ts neghbors Lemma 6). Therefore, due to the connectvty of the network after GST, and accordng to Lemma 7, eventually every node has an est t component. Then we have M GST = N, so = n, and the rato s not smaller than ) n/. GST n Convergence Theorem 1 Lveness) After GST, the estmate at all nodes converges to the read average. Proof We construct a seres of tmes t 0, t 1, t 2,... recursvely, where the ntal tme s t 0 = GST. For every t p 1 we defne t p to be a tme from whch each node k N GST has an est p 1 component wth rato at least ) n/ n for each N GST. Such a t p exsts accordng to Lemma 8. Denote by emax p 1 the largest estmate at a node at tme t p 1,.e., emax p 1 = max {est p 1 }. Assume wthout loss of generalty that the average s zero. If all node estmates are the exact average, then the estmate s zero at all nodes and t does not change. Otherwse, emax p 1 s strctly postve, and there exsts some node j whose estmate s negatve. At t p, a node has a component of est t p 1 j wth weght at least ) n/ n Lemma 8). The weght of the rest of ts components s smaller than n, and ther values are at most emax p 1. Therefore, the estmate of at t p s bounded: est t p < 1 n+ n ) n/ est t p 1 j <0 < ) ) n/ n emax p 1 + t est p 1 j n n n + n ) n/ e p 1 max. Theestmateat s smlarly bounded from below wth respect to the mnmal value at t p 1. The maxmal error absolute dstance from average) at t p s therefore bounded by n n+ n ) n/ the maxmal error at t p 1. We conclude that the maxmal error decreases at least exponentally wth p, and therefore the estmates converge to some value x. Now, the values n sent, receved and cleared are occasonally reset to the est value of ther node sent) or the neghbor s receved and cleared), and snce we are after GST, the weght unrecvweght s zero. Snce these all converge to x, ther weghted sum E converges to the same value,.e., x = E, and we have already shown Lemma 1) that E s eual to the value of the read sum. We conclude that est, converges to the average read value. We note that whle est varables converge, the weghts n the w varables fluctuate contnuously, as nodes lose half ther weght on send and smlarly receve consderable weghts.

12 I. Eyal et al Bounded state varables To conclude, we show that f the rate of dynamsm allows the algorthm to converge between events, all state varables mantaned by the nodes do not grow unboundedly. Theorem 2 Bounded varables) If change and remove Neghbor events occur only when all est varables are n a 2 neghborhood of R, then all state varables are bounded. Proof Consder frst the value component of the aggregates. A change event may sgnfcantly change the estmate value at a node f holds a small weght. Denote the maxmal read value change by d change, the maxmal read value by M and the mnmal read value by m. Snce the mnmal weght at a node s bounded by, the change n bounded by d change /, and the estmate value s bounded n the range [m /, M + /]. Snce the varables receved j), sent j), totaldff j), and unrecvval are aggregates of est s, ther values are also bounded n the same range. As for weghts, the weght n totaldff j) s explctly bounded by havng the sendng node stop sendng f ts weght s too negatve, and hence t s too postve on the opposte sde lne 21). The weght of receved j) s also explctly bounded by resettng and changng epoch f ts weght s too hgh lne 35). Boundng receved j) automatcally bounds cleared j), and sent j ) s bounded explctly lne 21). Fnally, we bound the weght of est. Intally, the sum of weghts n all est varables s n, and ths sum changes on weght send/receve and on removeneghbor. Send and receve cannot ncrease the sum of weghts, snce weght receved was prevously sent. On the other hand, removeneghbor events change est by subtractng totaldff j), possbly ncreasng/decreasng the sum of weghts. Snce the totaldff weghts are bounded by 2 bound, and each lnk could temporarly) ncrement the sum of est weghts n the system f one sde ncreases ts estmate, and the other postpones the decrease to avod negatve weght), we conclude that the sum of est weghts, and hence each est weght, s bounded by n + 2 bound n 2. The unrecvweght varables are smlarly bounded, as they ncrease only on lnk falure by the dfference of weghts transferred on that lnk. Note 1 If we replace the arbtrary neghbor choce of lne 16 wth a random choce from a statc dstrbuton, and take = 0, then after GST, the algorthm operates lke classcal push-sum. Therefore, n ths case the estmates converge n an exponental rate [5,14]. In ths case the sze of the values cannot be bounded, as the weghts can be nfntesmally small. 6 Evaluaton Our formal analyss above shows that LMoSense converges to the correct average durng long enough perods of uescence. In order to evaluate the behavor of LMoSense durng dynamc perods, we have conducted smulatons of several representatve scenaros. Our goal s to asses how fast the algorthm reacts to changes and succeeds to provde accurate nformaton. We compare LMoSense to a perodcally-restartng Push- Sum algorthm. We explan our methodology and metrcs n Sect We frst study how the algorthm copes wth dfferent types of data read changes a gradual creepng change of all values, occurrng, e.g., when temperature s gradually rsng Sect. 6.2), an abrupt value change captured by a step functon Sect. 6.3), and a temporary gltch or mpulse Sect. 6.4). We then study the algorthm s robustness to node and lnk falures Sect. 6.5). 6.1 Methodology We performed the smulatons usng a custom made Python event drven smulaton that smulated the underlyng network and the nodes operaton. Unless specfed otherwse, all smulatons are of a fully connected network of 100 nodes, wth ntal values taken from the standard normal dstrbuton. We have seen that n well connected networks the convergence behavor s smlar to that of a fully connected network. The smulaton proceeds n steps, where n each step, the topology and read values may change accordng to the smulated scenaro, and one node sends a message. Schedulng s unform synchronous,.e., the node performng the acton s chosen unformly at random. Unless specfed otherwse, each scenaro s smulated 1,000 tmes. In all smulatons, we track the algorthm s output and accuracy over tme. In all of our graphs, the X axs represents steps n the executon. We depct the followng three metrcs for each scenaro: a) Base staton. We assume that a base staton collects the estmated read average from some arbtrary node. We show the medan of the values obtaned n the runs at each step. b) ε-inaccuracy. For a chosen ε, we depct the percentage of nodes whose estmate s off by more than ε after each step. The average of the runs s depcted. c) MSE. We depct the average suare dstance between the estmates at all nodes and the read average at each step. The average of all runs s depcted. We compare LMoSense, whch does not need restarts, to a Push-Sum algorthm that restarts at a constant freuency

13 LMoSense: lve montorng a) b) c) Fg. 1 Creepng value change. Every 10 steps, 5 random reads ncrease by We see that LMoSense promptly tracks the creepng change. It provdes accurate estmates to 95 % of the nodes, wth an MSE of about 10 3 throughout the run. In contrast, Perodc Push-Sum s accurate only followng restarts. a Base staton value read medan), b % nodes off by >0.1 average), c MSE average) every 5,000 steps unless specfed otherwse. Ths number s an arbtrary choce, balancng between convergence accuracy and dynamc response. In base staton results, we also show the read average,.e., the value the algorthms are tryng to estmate. 6.2 Slow monotonc ncrease Ths smulaton nvestgates the behavor of the algorthm when the values read by the sensors slowly ncrease. Ths may happen f the sensors are measurng ranfall that s slowly ncreasng. Every 10 steps, a random set of 5 of the nodes read values larger by 0.01 than ther prevous reads. The ntal values are taken from the standard normal dstrbuton. The results are shown n Fg. 1. In Fg. 1a we see that the average s ncreasng at a constant rate, and the LMoSense base staton closely follows. The restartng Push-Sum, however, tres to update ts value only at constant ntervals, unable to follow the read average. The tme t takes for convergence s so long that t never gets close the read average lne. In Fg. 1b we see that after ts ntal convergence, the LMoSense algorthm has most of the nodes mantan a good estmate of the read average wth less than 10 % of the nodes outsde the 0.1 neghborhood. The restartng Push-Sum algorthm, on the other hand, has no nodes n ths neghborhood most of the tme, and most of the nodes n the neghborhood only for short ntervals. Fnally, n Fg. 1c we see that the LMoSense algorthm mantans a small MSE, wth some nose, whereas the restartng Push-Sum algorthm s error uckly converges after restart, untl the creepng change takes over and domnates the MSE causng a steady ncrease untl the next restart. 6.3 Step functon Ths smulaton nvestgates the behavor of the algorthm when the values read by some sensors are shfted. Ths may occur due to a fre outbreak n a lmted area, as close-by temperature nodes suddenly read hgh values. At step 2,500, a random set of 10 nodes read values larger by 10 than ther prevous reads. The ntal values are taken from the standard normal dstrbuton. The results are shown n Fg. 2. Fgure 2a shows how the LMoSense algorthm updates mmedately after the shft, whereas the perodc Push-Sum algorthm updates at ts frst restart only. Fgure 2b shows the rato of erroneous sensors wth error larger than 0.01 uckly droppng rght after the read average change for LMoSense, and at restart for the perodc Push-Sum. Fgure 2c shows the MSE decrease. Both LMoSense and perodc Push-Sum converge at the same rate, but start a dfferent tmes. 6.4 Impulse functon Ths smulaton nvestgates the behavor of the algorthm when the reads of some sensors are shfted for a lmted tme, and then return to ther prevous values. Ths may happen due to sensng errors causng the nodes to read rrelevant data. As an example, one may consder the case of a heavy vehcle drvng by sesmc sensors used to detect earthuakes. The close-by sensors would read hgh sesmc actvty for a short perod of tme. At steps 2,500 and 6,000, a random set of 10 nodes read values larger by 10 than ther prevous reads, and after 100 steps they return to ther values before the shft. The ntal

14 I. Eyal et al. a) b) c) Fg. 2 Response to a step functon. At step 2,500, 10 random reads ncrease by 10. We see that LMoSense mmedately reacts, uckly propagatng the new values. In contrast, Perodc Push-Sum starts ts new convergence only after ts restart. a Base staton value read medan). b % nodes off by >0.01 average), c MSE average) a) b) c) Fg. 3 Response to mpulse. At steps 2,500 and 6,000, 10 random values ncrease by 10 for 100 steps. Both mpulses cause temporary dsturbances n the output of LMoSense. Perodc Push-Sum s oblvous to the frst mpulse, snce t does not react to changes. The restart of Push-Sum occurs durng the second mpulse, causng t to converge to the value measured then. a Base staton value read medan), b % nodes off by >0.01 average), c MSE average) values are taken from the standard normal dstrbuton. The results are shown n Fg. 3. The LMoSense algorthm s reacton s ndependent of the mpulse tme a short perod of nose rases the estmate at the base staton as the mpulse value propagates from the sensors that read the mpulse. Then, once the mpulse s canceled, ths value decreases. The estmate wth respect to the read average s shown n Fg. 3a, and the rato of correct sensors s n Fg. 3b. The mpulse essentally restarts the MSE convergence, as shown n Fg. 3c After an mpulse ends, the error returns to ts startng pont and starts convergence anew. The response of the perodc Push-Sum depends on the tme of mpulse. If the mpulse occurs between restarts as n step 2,500), the algorthm s completely oblvous to t. All three Fg. 3a, b show that apart from the mpulse tme, convergence contnues as f t never happened. If, however, a restart occurs durng the mpulse as n step 6, 000), then the mpulse s sampled and the algorthm converges to ths new value. Ths convergence s smlar to the reacton to the step functon of Sect. 6.3, only n ths case t promptly becomes stale as the mpulse ends. Fgure 3a shows the error uckly propagatng to the base staton. Snce the algorthm has the estmates converge to the read average durng mpulse, the rato of naccurate nodes s 1.0 once the mpulse ends, and the MSE stablzes at a large value as all nodes converge to the wrong estmate.

15 LMoSense: lve montorng a) b) c) Fg. 4 Falure robustness. In a dsc graph topology, the rado range of 10 nodes decays n step 3,000, resultng n about 7 lost lnks n the system. Then, n step 5,000, a node crashes. Each falure causes a temporary dsturbance n the output of LMoSense. Perodc Push-Sum s oblvous to the lnk falure. It recovers from the node falure only after the next restart. a Base staton value read medan), b % nodes off by >0.01 average), c MSE average) 6.5 Robustness To nvestgate the effect of lnk and node falures, we construct the followng scenaro. The sensors are spread n the unt suare, and they have a transmsson range of 0.7 dstance unts. The neghbors of a sensor are the sensors n ts range. The system s run for 3,000 steps, at whch pont, due to battery decay, the transmsson range of 10 sensors decreases by Due to ths decay, about 7 lnks fal, and respectve nodes employ ther removeneghbor functons. We see the effect of ths lnk removal n Fg. 4. In Fg. 4a the effect can hardly be seen, but a temporary decrease of the accurate nodes can be seen n Fg. 4b, and n Fg. 4c we see the MSE rsng sharply. The falure of lnks does not effect the perodc Push-Sum algorthm, whch contnues to converge. In step 5,000, a node fals, removng ts read value from the read average. Upon node falure, all of ts neghbors call ther removeneghbor functons. Fgure 4a shows the extreme nose at the base staton caused by the falure, and n Fg. 4b we see the rato of naccurate nodes rsng sharply before convergng agan. We see n Fg. 4c that the node removal effectvely reures the MSE convergence to restart. However, Perodc Push-Sum has no mechansm for reactng to the change untl ts next restart. Snce the average changes, untl that tme, the percentage of naccurate nodes sharply rses to 1.0, and the MSE reaches a statc value, as the estmates at the nodes converge to the wrong average. Snce n every run a dfferent node crashes, and the medan of the removed value s 0, the node crash does not effect the medan perodc Push-Sum value at the base staton n Fg. 4a. 7 Concluson We have presented LMoSense, a fault-tolerant lve montorng algorthm for dynamc sensor networks. Ths s the frst asynchronous robust average aggregaton algorthm to accommodate dynamc nputs. LMoSense dynamcally tracks and aggregates a large collecton of ever-changng sensor reads. It overcomes message loss, node falures and recoveres, and dynamc network topology changes. The man focus of ths work has been the formal analyss of LMoSense s correctness, namely showng t converges to the read average once the system stablzes. For completeness, we have also demonstrated the behavor of LMoSense n representatve dynamc scenaros, showng ts fast convergence rate. Acknowledgments The authors thank an anonymous revewer for mportant comments on an earler verson of ths work. References 1. Almeda., P.S., Bauero., C., Farach-Colton., M., Jesus., P., Mostero, M.A.: Fault-tolerant aggregaton: flow updatng meets mass dstrbuton. In: OPODIS 2011) 2. Asada, G., Dong, M., Ln, T.S., Newberg, F., Potte, G., Kaser, W.J., Marcy, H.O.: Wreless ntegrated network sensors: low power systems on a chp. In: ESSCIRC 1998) 3. Brk., Y., Kedar., I., Lss, L., Schuster, A.: Effcent dynamc aggregaton. In: DISC 2006) 4. Boyd, S.P., Ghosh, A., Prabhakar, B., Shah, D.: Gossp algorthms: desgn, analyss and applcatons. In: INFOCOM 2005) 5. Boyd, S.P., Ghosh, A., Prabhakar, B., Shah, D.: Randomzed gossp algorthms. IEEE Trans. Inf. Theory 526), )

16 I. Eyal et al. 6. Chen, J.-Y., Pandurangan, G.: Robust computaton of aggregates n wreless sensor networks: dstrbuted randomzed algorthms and analyss. IEEE Trans. Parallel Dstrb. Syst. 179), ) 7. Eyal, I., Kedar, I., Rom, R.: LMoSense: lve montorng n dynamc sensor networks. In: 7th Internatonal Symposum on Algorthms for Sensor Systems, Wreless Ad Hoc Networks and Autonomous Moble Enttes ALGOSENSOR 11) 2011) 8. Fagnan, Fabo, Zamper, Sandro: Randomzed consensus algorthms over large scale networks. IEEE J. Sel. Areas Commun. 264), ) 9. Flajolet, P., Ngel Martn, G.: Probablstc countng algorthms for data base applcatons. J. Comput. Syst. Sc. 312), ) 10. Jan, N., Mahajan, P., Kt, D., Yalagandula, P., Dahln, M., Zhang, Y.: A new consstency metrc for scalable montorng. In: OSDI, Network mprecson 2008) 11. Jelasty, M., Montresor, A.: Epdemc-style proactve aggregaton n large overlay networks. In: Dstrbuted Computng Systems, Proceedngs. 24th Internatonal Conference on, pp IEEE 2004) 12. Jelasty, M., Montresor, A., Babaoglu. Ö.: Gossp-based aggregaton n large dynamc networks. ACM Trans. Comput. Syst. TOCS) 233), ) 13. Jesus, P., Bauero, C., Almeda, P.S.: Fault-tolerant aggregaton for dynamc networks. In: SRDS 2010) 14. Kempe, D., Dobra, A., Gehrke, J.: Gossp-based computaton of aggregate nformaton. In: FOCS 2003) 15. Madden, S., Frankln, M.J., Hellersten, J.M., Hong, W.: A tny aggregaton servce for ad-hoc sensor networks. In: OSDI, Tag 2002) 16. Mosk-Aoyama, D., Shah, D.: Computng separable functons va gossp. In: PODC 2006) 17. Nath, S., Gbbons, P.B., Seshan, S., Anderson, Z.R.: Synopss dffuson for robust aggregaton n sensor networks. In: SenSys 2004) 18. Tanenbaum, A.S.: Computer Networks. Prentce Hall, New Jersey 2003) 19. Warneke, B., Last, M., Lebowtz, B., Pster, K.S.J.: Smart dust: communcatng wth a cubc-mllmeter computer. Computer 341), ) 20. Wuhb, Fetah, Dam, Mads, Stadler, Rolf, Clem, Alexander: Robust montorng of network-wde aggregates through gosspng. IEEE Trans. Netw. Serv. Manag. 62), )