Schedulability Analysis under Graph Routing in WirelessHART Networks

Transcription

1 Scedulability Analysis under Grap Routing in WirelessHART Networks Abusayeed Saifulla, Dolvara Gunatilaka, Paras Tiwari, Mo Sa, Cenyang Lu, Bo Li Cengjie Wu, and Yixin Cen Department of Computer Science, Missouri University of Science and Tecnology Department of Computer Science, State University of New York at Bingamton Cyber-Pysical Systems Laboratory, Wasington University in St. Louis Abstract Wireless sensor-actuator networks are gaining ground as te communication infrastructure for process monitoring and control. Industrial applications demand a ig degree of reliability and real-time guarantees in communication. Because wireless communication is susceptible to transmission failures in industrial environments, industrial wireless standards suc as WirelessHART adopt reliable grap routing to andle transmission failures troug retransmissions and route diversity. Wile tese mecanisms are critical for reliable communication, tey introduce substantial callenges in analyzing te scedulability of real-time flows. Tis paper presents te first worst-case endto-end delay analysis for periodic real-time flows under reliable grap routing. Te proposed analysis can be used to quickly assess te scedulability of real-time flows wit stringent requirements on bot reliability and latency. We ave evaluated our scedulability analysis against experimental results on a wireless testbed of 69 nodes as well as simulations. Bot experimental results and simulations sow tat our delay bounds are safe and enable effective scedulability tests under reliable grap routing. I. INTRODUCTION Wireless sensor-actuator networks (WSANs are gaining ground as te communication infrastructure for industrial process monitoring and control systems. To support monitoring and control, a WSAN periodically delivers data from sensors to a controller and ten delivers its control input data to te actuators troug te multi-op mes network. Wireless control in process industries demands a ig degree of reliability and real-time guarantees in communication [1]. Failures in wireless transmissions are prevalent in industrial environments due to cannel noise, power failure, pysical obstacle, multi-pat fading, and interference from co-existing wireless systems. As a widely adopted industrial wireless standard, WirelessHART [2] introduces te reliable grap routing approac to andle transmission failures troug retransmissions and route diversity. Reliable grap routing [2] employs te following mecanisms to recover from transmission failures. A routing grap is constructed as a directed list of pats between two devices, tereby providing redundant routes for real-time flows between sensors and actuators. For transmission between a receiver and a sender, a time slot can be eiter dedicated (i.e., a time slot wen at most one transmission is sceduled to a receiver or sared (i.e., a time slot wen multiple nodes may contend to send to a common receiver. For eac flow, te network andles transmission failures by allocating a dedicated time slot for eac node on a pat from te source, followed by allocating a second dedicated slot on te same pat for a retransmission, and ten by allocating a tird sared slot on a separate pat for anoter retransmission [2]. Wile effective in acieving reliable communication, tis fault-tolerant mecanism introduces significant callenges in worst-case delay analysis for real-time flows in a WSAN. A WirelessHART network is managed by a centralized network manager responsible for computing te routes and scedules for all field devices. As wireless conditions can cange quickly in industrial environments, it must quickly assess te scedulability of real-time flows. If some flows cannot be guaranteed to meet teir deadlines, te network manager can adapt to te overload troug online admission control and reconfiguration. Fast online scedulability analysis is terefore important for WirelessHART networks to adapt to dynamic environments. In tis paper, we propose te first worst-case end-to-end delay analysis for periodic real-time flows under reliable grap routing. Specifically, we consider periodic real-time flows wose transmissions are sceduled based on fixed priority. In a fixed priority sceduling policy, all transmissions of a flow are sceduled based on te fixed priority of te flow. Wile delay analyses for single or independent routes ave been proposed in te literature [3] [6], an efficient delay analysis under reliable grap routing in a WSAN represents a callenging open problem. Since a routing grap for a flow can consist of an exponential (in number of nodes number of routes between its source and destination, determining an effective delay bound for a flow by enumerating all of tese pats is time consuming, making it unsuitable for WSANs subject to frequent canges to link and cannel conditions in industrial environments. We address tis callenge and propose an efficient end-to-end delay analysis witout enumerating all te pats. WirelessHART networks employ a multi-cannel TDMA (Time Division Multiple Access protocol. In a WirelessHART network, a flow may be delayed by iger priority flows due to (1 cannel contention (wen all cannels are assigned to te iger priority flows in a time slot or (2 transmission conflicts (wen a transmission of te lower priority flow and a transmission of a iger priority flow involve a common

2 node. We use an efficient metod based on dept-first searc to determine an upper bound of transmission conflict delay of eac flow witout enumerating all te pats. We ten observe tat, unlike single or independent routes, transmission conflict may increase cannel contention in grap routing. Troug an analysis of te worst-case scenario for cannel contention in te presence of transmission conflict, we determine te worstcase end-to-end delay bounds of te flows. Moreover, we propose a probabilistic end-to-end delay analysis tat provides tigter but probabilistic delay bounds for soft real-time flows tat do not require absolute delay guarantees. We ave evaluated our scedulability analysis against experiments on a WirelessHART stack implemented on a 69- node wireless testbed. We ave also performed trace-driven simulations on real network topologies. Bot experiments and simulations sow tat our delay bounds are safe in practice and te probabilistic delay bounds represent safe upper bounds wit probability Te worst-case and probabilistic bounds can be used in different application scenarios depending on te level of predictability required. Our analysis ence can be used for effective scedulability test and admission control of real-time flows under reliable grap routing. Section II reviews related work. Section III describes te system model and grap routing mecanisms. Section IV provides an overview of fixed priority sceduling for real-time flows in a WSAN under grap routing. Section V presents te delay analysis under reliable grap routing. Section VI presents te probabilistic delay analysis. Section VII presents te experimental results. Section VIII concludes te paper. II. RELATED WORK Real-time sceduling for wireless networks as been explored in many early [7] and recent works [8] [17]. However, tese works do not focus on efficient worst-case delay analysis in te network. Oter works [4] [6], [17], [18] ave researced delay analysis in wireless sensor networks. Tese works focus on data collection troug a routing tree [4], [5] and/or do not consider multiple cannels [5], [6]. In contrast, we consider a WSAN based on multiple cannels and reliable grap routing of WirelessHART. Besides, our analysis is targeted for realtime flows between sensors and actuators for process control purposes, and is not limited to data collection towards a sink. Real-time sceduling for WSANs based on WirelessHART as received considerable attention in recent works [1], [3], [19] [23]. Te works presented in [22] and [23] address grap routing algoritm and localization, respectively, in WirelessHART networks. None of tese concerns delay analysis. Our earlier work proposed delay analysis [3], [19]. As a first step in establising a delay analysis for WSANs, tis earlier effort is based on single-route routing instead of reliable grap routing, wic are important for reliable communication in process control applications. We ave also studied priority assignment policies in [20] and rate selection algoritms in [21] for real-time flows. Our work in [1], [24] also considered dynamic priority sceduling. However, none of our earlier work considers delay analysis under reliable grap routing. Tis paper presents te first delay analysis for WSANs under reliable grap routing. Since industrial applications impose stringent requirements on bot real-time performance and reliability in as environments wit frequent transmission failures, te delay analysis represents an important contribution to real-time sceduling for real-world WSANs. Efficient delay analysis is particularly useful for online admission control and adaptation (e.g., wen network route, topology, or cannel condition cange so tat te network manager can quickly reassess te scedulability of te flows. A. Network Model III. SYSTEM MODEL Because of te world-wide adoption of WirelessHART in process monitoring and control, we consider a WSAN based on te WirelessHART standard [2]. WirelessHART forms a multi-op mes network consisting of a Gateway, a set of field devices, and several access points. A centralized network manager and te controllers are connected to te Gateway. Te network manager is responsible for managing te entire network suc as routing and transmission sceduling. Te field devices are wirelessly networked sensors and actuators. Access points are wired to te Gateway to provide redundant pats between te wireless network and te Gateway. Te sensor devices periodically deliver sample data to te controllers (troug te access points wired to te Gateway, and te control messages are ten delivered to te actuators. Te network manager creates te routes and scedules of transmissions. To acieve ig reliability, WirelessHART employs a number of mecanisms to andle transmission failures. Transmissions are sceduled based on a multi-cannel TDMA protocol. Eac time slot is of fixed lengt (10 ms, and eac transmission needs one time slot. A transmission and its acknowledgement (ACK are sceduled in te same slot using te same cannel. For transmission between a receiver and its sender, a time slot can be eiter dedicated or sared for te link between te sender and te receiver, and te link is called a dedicated link or a sared link, respectively. In a time slot, wen a link is used as a dedicated link, only one sender is allowed to transmit to te receiver. In a time slot, a sared link associated wit a receiver indicates tat multiple senders can attempt to send to te common receiver in tat slot. Te network uses te cannels defined in IEEE , and adopts cannel opping in every time slot. Any excessively noisy cannel is blacklisted not to be used. Eac receiver uses a distinct cannel for reception in any time slot. As a result, tere are at most m successful transmissions in a time slot, were m is te total number of cannels. Tis design decision prevents potential interference between concurrent transmissions in a dedicated slot and trades network trougput for a iger degree of predictability and reliability tat is essential for industrial applications. WirelessHART supports two types of routing approaces: source routing and grap routing. Source routing provides a single route for eac flow. Te delay analysis for source routing as been addressed in te literature [3]. Te focus 2

3 of tis paper is to develop new delay analysis for grap routing tat acieves a iger degree of reliability by providing multiple pats for eac flow. In grap routing, a routing grap is a directed list of pats tat connect two devices. Packets from all sensor nodes are routed to te Gateway troug te uplink grap. For every actuator, tere is a downlink grap from te Gateway troug wic te Gateway delivers control messages. Te end-to-end communication between a source (sensor and destination (actuator pair appens in two pases. In te sensing pase, on one pat from te source to te Gateway in te uplink grap, te sceduler allocates a dedicated slot for eac device starting from te source, followed by allocating a second dedicated slot on te same pat to andle a retransmission. Te links on tis pat are dedicated links. Ten, to offset failure of bot transmissions along a primary link, te sceduler allocates a tird sared slot on a separate pat to andle anoter retry. Te links on tese pats are sared links. Ten, in te control pase, using te same way, te dedicated links and sared links are sceduled in te downlink grap of te destination. Eac node is equipped wit a alf-duplex omnidirectional radio transceiver tat cannot bot transmit and receive at te same time and can receive from at most one sender at a time. Two or more transmissions tat involve a common node are conflicting, and cannot be sceduled in te same dedicated slot. However, for te case of sared slot, te transmissions aving te same receiver can be sceduled in te same slot. Te senders tat attempt to transmit in a sared slot contend for te cannel using a CSMA/CA sceme. B. Flow Model A periodic end-to-end communication between a source (sensor and a destination (actuator is called a flow. We consider tere are n real-time flows denoted by F 1, F 2,, F n in te network. Te source and te destination of flow F i are denoted by s i and d i, respectively. Te subgrap of te uplink grap tat s i uses to deliver sensor data to te Gateway is denoted by UG i. Te downlink grap for d i is denoted by DG i. Te grap consisting of UG i and DG i is te routing grap of F i, and is denoted by G i. Te period and te deadline of flow F i are denoted by T i and D i, respectively. Time slots are used as time units. We assume D i T i, i. Eac flow F i, 1 i n, as a fixed priority. Transmissions of a flow are sceduled based on its priority. In practice, flows may be prioritized based on deadlines, rates, or criticality. We assume tat te priorities are already assigned using any algoritm, and tat F 1, F 2,, F n are ordered by priorities. Flow F as iger priority tan flow F i if and only if < i. IV. FIXED PRIORITY SCHEDULING In tis section, we provide an overview of te fixed priority transmission sceduling algoritm under reliable grap routing for wic our delay analysis is developed. Due to its simplicity, fixed-priority sceduling is a commonly adopted policy in practice for real-time CPU sceduling, Control-Area Networks, and also for WirelessHART networks. In a fixed priority sceduling policy, eac flow as a fixed priority, and its transmissions are sceduled based on tis priority. Te scedule is created by resolving te transmission conflicts and considering te limited number of cannels. Te complete scedule is split into superframes. A superframe is a series of time slots tat repeat at a constant rate and represents te communication pattern of a set of flows. We first describe ow transmissions are sceduled using grap routing to account for failures. Figure 1(a sows UG (te subgrap of te uplink grap used by F for flow F. In te figure, te dedicated links used by F in te sensing pase are sown in solid lines wile te dotted lines indicate te sared links used by F. Considering tat F is not delayed by any oter flow, te time slots in wic a link is activated are sown beside te links (starting from slot 1. Te first (starting from te source node s dedicated link s u is sceduled first at slot 1. Ten to andle te transmission failure of slot 1, time slot 2 is also allocated for tis link. Ten, te next dedicated link u v is allocated time slots 3 and 4. Similarly, te next dedicated link v a is allocated time slots 5 and 6. Tus, if te first transmission (sceduled on slot 5 along v a succeeds (given at least one transmission along s u and at least one transmission along u v succeeded, ten te packet will reac te access point a in 5 time slots. If te first transmission (sceduled on slot 5 along v a fails but te second one (sceduled on slot 6 along tat link succeeds, ten te packet will reac te access point a in 6 time slots. For every link starting from te source, to andle failure of bot transmissions along te link, te sceduler again allocates a tird sared slot on a separate pat to andle anoter retry. Tere can be situations wen te second transmission on a dedicated link, say s u, succeeds but te ACK gets lost. As a result, s retransmits te packet along te sared link s y on te tird slot (as s is unaware of te successful transmission on te dedicated link wile te packet at u is transmitted troug te subsequent links. Tus a packet can be duplicated and delivered troug multiple routes. We call tis problem ACK-lost problem. To andle ACK-lost problem, we avoid conflicts among te duplicated packets wile sceduling on a routing grap, except te case tat transmissions along te sared links aving te same receiver are allowed to scedule in te same slot. Tus, te links on pats s y z w a are sceduled on slots 3, 4, 5, and 7. Ten te links on pat u x a are sceduled on slots 5 and 7. Ten te links on pat v w a are sceduled on slots 8 and 9. Tus te packet can take at most 9 slots to reac te access point (along s u v w a. Under fixed priority sceduling, te transmissions of te flows are sceduled in te following way. Starting from te igest priority flow F 1, te following procedure is repeated for every flow F i in decreasing order of priority. For current priority flow F i, te network manager scedules its dedicated links and sared links on UG i in its sensing pase on earliest available time slots and on available cannels. It ten scedules te dedicated links and sared links on DG i in te control pase following te same way. A time slot is available if no 3

4 5 w 7, 9 z 8 4 sen L = 8 access point a 7 y 5, 6 v 3, 4 3 u x 5 1, 2 s sared link dedicated link access point a 7 (a UG for F (b UG i and UG Fig. 1. Routing in te sensing pase of F i and F (te numbers beside eac link indicate te time slots allocated to te link. conflicting transmission is already sceduled in tat slot except te case tat transmissions along te sared links aving te same receiver are allowed to scedule in te same slot. Tus a packet is sceduled on multiple pats along te routing grap. Wen tere is no ACK-lost problem, a packet is delivered troug one pat in te routing grap. Oterwise, a packet can be duplicated and tus delivered troug multiple pats. Note tat we do not propose any new algoritm for real-time transmission sceduling or any new fault tolerance mecanism for WirelessHART networks. Instead, te key contribution of our work is an efficient analysis for deriving te worst case delay bounds in a WSAN under grap routing, wic is applicable for any existing fixed-priority sceduling policy for real-time flows in WSANs. Te delay bound provided by our analysis is applicable only to te packets tat are successfully delivered to te destination using existing grap routing mecanisms in WirelessHART. V. DELAY ANALYSIS UNDER RELIABLE GRAPH ROUTING We first formulate te problem of worst-case delay analysis for real-time flows in a WSAN. We ten present te delay analysis for any given fixed priority sceduling policy. A. Problem Formulation For eac flow F i, te sensor (s i periodically generates data at a period of T i wic as to be delivered to te Gateway (troug an access point in te sensing pase, and ten te control message as to be delivered to te actuator (d i in te control pase. Te total communication delay in two pases is called an end-to-end delay of F i. Te flows are called scedulable under a given fixed priority sceduling algoritm A, if A is able to scedule te transmissions were no deadline will be missed. A scedulability test S is sufficient if any set of flows deemed scedulable by S is indeed scedulable. To determine te scedulability of a set of flows, it is sufficient to sow tat, for every flow, an upper bound of its worst case end-to-end delay is no greater tan its deadline. Our objective is to determine an upper bound R i of te end-to-end delay of eac flow F i. Te end-to-end delay analysis will determine te flows to be scedulable if R i D i, i. Note tat we derive upper bounds of communication delays in te network, and do not consider te time needed by te controller. Note tat creating a complete scedule for all flows requires an exponential time since te scedule as to be created up 5 w 7, 9 z s i 8 4 y 5, 6 v 3, 4 3 u x 5 1, 2 s to te yper-period of te flows. Specially, wen te periods are not armonic, te yper-period can be extremely long making it computationally very expensive to determine te scedulability and te delays by creating a complete scedule. In contrast, te purpose of our analysis is to determine te scedulability of te flows very quickly in pseudo polynomial time witout te need to create a complete scedule. Efficient delay analysis is particularly useful for online admission control and adaptation (e.g., wen network route, topology, or cannel condition cange so tat te network manager can quickly reassess te scedulability of te flows and adjust workload or flow parameters (e.g., rates, priorities accordingly. It ence is igly desirable in process monitoring and control applications tat require real-time guarantees since various network dynamics affect te scedulability of te flows frequently requiring to reassess teir scedulability. In transmission sceduling, a lower priority flow may be delayed by iger priority flows due to (a transmission conflicts (wen a transmission of te lower priority flow and a transmission of a iger priority flow involve a common node and (b cannel contention (wen all cannels are assigned to te transmissions of iger priority flows in a time slot. For eac case, we first separately analyze ow reliable grap routing in WSANs affect it. We ten incorporate eac component of te delays into one analysis tat provides an upper bound of a flow s end-to-end delay under grap routing. B. Transmission Conflict Delay under Grap Routing First we analyze te delay tat a flow can experience due to transmission conflicts only under grap routing. Wenever two transmissions conflict, te one tat belongs to te lower priority flow needs to be delayed. Te term delay used in tis subsection will refer to only transmission conflict delay. First we determine te conflict delay tat one iger priority flow F may cause on a lower priority flow F i. Under multipat grap routing, a transmission of F along a link l and a transmission of F i along a link l i may be conflicting in 4 ways as follows wen tese two links involve a common node: 1 Type 1: l is a dedicated link and l i is a sared or dedicated link. 2 Type 2: l is a sared link and l i is a dedicated link. 3 Type 3: l is a sared link and l i is a sared link, and te receiver nodes of te two links are different. 4 Type 4: l is a sared link and l i is a sared link, and te receiver nodes of te two links are te same. In tis case, te transmission of F i is not delayed. In te first 3 cases te transmission of F i is delayed wile for Type 4 conflict it will not be delayed. Terefore, te total delay caused by F on F i depends on ow teir dedicated and sared links intersect in te routing graps. Now we will first determine an upperbound of te conflict delay tat one instance of a iger priority flow F may cause on F i. In te next discussion we limit our attention only to F and F i. Note tat we measure delay in terms of number of slots. In te routing grap G i (consisting of UG i and DG i of flow F i wic involves N i nodes, tere can be O(Ni 2 directed end- 4

5 to-end pats from its source s i to destination d i (calculated as te number of pats in UG i in te sensing pase times te number of pats in DG i in te control pase. If every node in te routing grap as to make te first two tries along a dedicated link and ten to make a tird retransmission along a sared link, ten tis number of pats can be 2 Ni. Tus if t denotes te time complexity (wic is pseudo polynomial for determining delay for a single pat route, te complexity for a flow F i becomes O(t 2 Ni. On te oter and, our metod as time complexity of O(t flow F i (as will be sown later. Among tese end-to-end pats, te one tat experiences te maximum conflict delay from F is called te bottleneck pat wit respect to F. Te conflict delay caused by F along F i s bottleneck pat represents te upper bound of te conflict delay tat F may cause on F i. Let i be an upper bound of conflict delay tat one instance of F may cause along te bottleneck pat of F i. We determine i in an efficient way witout requiring to find te bottleneck pat or witout enumerating all end-to-end pats in G i as described below. Algoritm 1: Finding conflict delay on F i caused by F Procedure FindConflict(UG i, r /* r is a node in UG i * / for eac node u in UG i do status(u:=undiscovered; λ i (u := 0; end DFSearc(r; /* start searc at node r */ return λ i (r; end Procedure / * λ i in subtree rooted at r */ Procedure DFSearc(r status(r:=discovered; /* node r is now discovered */ for eac v cildren(r, UG i do if status(v=undiscovered; ten DFSearc(v; end λ i (r := max{λ i (v v cildren(r, UG i}; x(r := new conflict delay on F i by F observed at node r; λ i (r := λ i (r + x(r; end Procedure Let us call te bottleneck pat (wit respect to F in UG i te bottleneck sensing pat of F i. Let an upper bound of conflict delay caused by F on F i s bottleneck sensing pat be λ,sen i. A value of λ,sen i can be efficiently calculated witout enumerating all pats in UG i as explained below. Let us consider a particular pat p in UG i. Te total number of transmissions of (one instance of F tat may ave Type 1, 2, or 3 conflict on p represents a value of conflict delay along p caused by one instance of F. To illustrate tis, in Figure 1(b, wit flow F, we also sow UG i for flow F i. Te figure sows links s i z, z v, and v a as dedicated links in UG i wile te corresponding sared links are s i y, z w, and v w, respectively. In Figure 1(b, F as 9 transmissions tat may cause delay along p = s i z w a of F i. (Note tat tis is te delay along p considering links s i z, z v, z w, and w a of F i. Link z v is considered because z w is sceduled only after sceduling z v. Now te pat in UG i wose delay (calculated using te above metod is maximum is te bottleneck sensing pat, and its delay represents λ,sen i. Suc a value of λ,sen i is determined quickly by exploring eac link on UG i once based on a deptfirst searc on UG i. Te metod is sown as Algoritm 1, and λ,sen i is determined by calling λ,sen i = FindConflict(UG i, s i ; In Algoritm 1, we use cildren(u, UG i to denote te set of nodes to wic node u transmits in UG i for flow F i. (For example, in Figure 1(a, node s as cildren u and y. Te searc starts at node s i. In tis metod, wen te searc backtracks at a node u, we use λ i (u to denote te maximum conflict delay along a pat among all te pats in te subtree (induced by dept first searc rooted at u. Te value of λ i (u is calculated by taking te maximum of te values from u s cildren and ten by adding te new conflict delay tat we observe at node u, wen te searc finises node u. Note tat we do not need to execute Algoritm 1 for every distinct F to determine λ,sen i. Instead, we need to execute Algoritm 1 only once for all < i to determine λ,sen i our approac igly efficient. Similarly, let λ,con i for flow F i, making be te conflict delay along te bottleneck control pat. Te value of λ,con i is determined using Algoritm 1 on DG i starting te searc at an access point a, i.e., by calling λ,con i = FindConflict(DG i, a; Tus, our metod as time complexity of O( G i + t = O(t (were t denotes te time complexity for determining delay for a single pat route and is pseudo polynomial for a flow F i were O( G i is te time complexity of Algoritm 1 (considering its execution on bot UG i and DG i, wit G i indicating te total number of links and nodes in G i. Lemma 1 provides a bound of i. Lemma 1: For a iger priority flow F and a lower priority flow F i, i λ,sen i + λ,con i. Proof: Since te control pase of F i starts after its sensing pase is complete, te bottleneck pat between s i and d i consists of its bottleneck sensing pat and te bottleneck control pat. Hence, λ,sen i +λ,con i is an upper bound of conflict delay caused by one instance of F along F i s bottleneck pat. Note tat i is an upper bound of delay tat one instance of F can cause along F i s bottleneck pat. Now we will upperbound te total delay caused by all instances of F. In considering te delay caused by multiple instances, we observe tat at te time wen a transmission on a directed pat p in G i conflicts wit some transmission of F, te preceding transmissions on p are already sceduled. Tese already sceduled transmissions on p are no more subject to delay by te subsequent instances of F. For example, in Figure 1(b let us consider te pat s i y z v w a in UG i of F i. If some instance of F conflicts and causes delay on F i s transmission along v w, te next instance of F must not delay F i s transmissions along links s i y, y z, and z v on tis pat since tese are already sceduled. Tus only te transmissions tat are not yet sceduled along pat p will be considered for conflict delay by te subsequent instances of F. Tese observations lead to Lemma 2, and ten to Teorem 3 to upperbound te total delay (due to transmission conflict caused on F i by all instances of F. 5

6 Lemma 2: Let us consider any two instances of a iger priority flow F suc tat eac causes conflict delay on a directed pat p in G i of a lower priority flow F i in a time interval. Ten, tere is at most one common transmission on p tat can be delayed by bot instances. Proof: Let tese two instances of F be denoted by F,1 and F,2, were F,1 is released before F,2. Suppose to te contrary, bot of tese instances cause delay on two transmissions, say τ j and τ r, on directed pat p of F i. Witout loss of generality, we assume tat τ j precedes τ r on p. F,1 causes delay on τ r because τ r is ready to be sceduled. Tis implies tat τ j as already been sceduled. Hence, F,2 wic releases after F,1 cannot cause any delay on τ j, tereby contradicting our assumption. By Lemma 2, for any two instances of F, any directed pat in G i of F i as at most one transmission on wic bot instances can cause delay. Let te link on G i tat may ave maximum conflict delay of Type 1, 2, or 3 wit F be called te bottleneck link of F i (wit respect to F. Tat is, a transmission of F i along tis link may face te igest conflict wit F. Let δi denote te maximum conflict delay along te bottleneck link. (For example, considering only UG i in Figure 1(b, we can see tat δi = 7, since a link of F i can ave conflict wit at most 7 transmissions of F. Here, z v is F i s bottleneck link. In te worst case, te transmission along te bottleneck link of F i (wit respect to F can be delayed by multiple instances of F. Hence, te value of δi plays a major role in determining te worst case delay caused by F on F i as sown in Teorem 3. Teorem 3: In a time interval of t slots, te worst case conflict delay caused by a iger priority flow F on a lower priority flow F i is upper bounded by ( t i + 1.δi + min (δi, t mod Proof: Tere are at most t instances of F in a time interval of t slots. We consider a particular directed pat p in G i of F i. Let te set of transmissions of F wic cause conflict delay along p be denoted by Γ. Wen one instance F,1 of F causes conflict delay on p, a subset Γ 1 of Γ causes delay on p. Now consider a second instance F,2 of F. For F,2, anoter subset Γ 2 of Γ causes delay on p. Wen all subsets Γ 1, Γ 2,, Γ t T are mutually disjoint, by te definition of i, te conflict delay caused by Γ on p is at most i. Hence, te total conflict delay caused by all Γ 1, Γ 2,, Γ t T in tis case is at most i. Tat is, te total conflict delay on p caused by F is at most i. Now let us consider te case wen te subsets Γ 1, Γ 2,, Γ t T are not mutually disjoint, i.e., tere is at least one pair Γ j, Γ k suc tat Γ j Γ k, were 1 j, k t. Let te total delay caused by all instances of F on p in suc case be i + Z i, i.e., te delay is iger tan i by Zi time slots. Te additional delay (beyond i appens because te transmissions tat are common between Γ i and Γ j cause bot instances of F to create delay along p. By Lemma 2, for any two instances of F, p as at most one transmission on wic bot instances can cause delay. If tere is no transmission of p tat is delayed by bot te k-t instance and te (k+1-t instance of F, ten no transmission of p is delayed by bot te k-t instance and te q-t instance of F, for any q > (k + 1, were 1 k < t. Tus, Zi is maximum wen for eac pair of consecutive instances (say, te k-t instance and k + 1-t instance, for eac k, 1 k < t of F, tere is a transmission of p tat is delayed by bot instances. Hence, at most t 1 instances contribute to tis additional delay Zi, eac instance causing some additional delay on a transmission. Since one instance of F can cause delay on a transmission of p at most by δi slots, Zi ( t 1δi. Since te last instance may finis after te considered time window of t slots, te delay caused by it is at most min(δi, t mod slots. Taking tis into consideration, Zi ( t 1δi +min(δ i, t mod. Tus, te total delay caused on p by all instances of F is at most t i + Zi i + ( 1.δi + min(δi, t mod Since te above bound is true for any pat in G i (of F i, it is true for te bottleneck pat in G i. Since te conflict delay along te bottleneck pat represents te conflict delay caused on F i by F, te teorem follows. From Teorem 3, now an upper bound of te total delay tat flow F i can experience from all iger priority flows due to transmission conflicts during a time interval of t slots is calculated as follows. ( ( t i + <i 1.δi + min ( δ i, t mod (1 C. Cannel Contention Delay under Grap Routing In tis section, we analyze te cannel contention delay caused by one iger priority flow F to a lower priority one F i under reliable grap routing. First we analyze te delay witout considering cannel opping. Later, we will analyze te effect of cannel opping. Let E and S denote te total number of dedicated links and total number of sared links of flow F, respectively. Since every dedicated link is sceduled on 2 dedicated slots, tere are 2E + S assignments of cannels for flow F. Note tat a packet is sceduled on multiple pats in its routing grap for fault tolerance. Wile a natural approac to analyzing cannel contention delay of a flow under tis scenario is to consider it as a parallel task, we observe tat te sceduling on routing graps experiences only a little parallelism making it more closer to sequential task sceduling due to te following two problems. ACK-lost problem. Assuming no packet duplication, we could scedule te link w a for delivery troug pats s y z w a on slot 6, ignoring te fact tat link v a is already sceduled on slot 6 because te packet will be delivered troug one pat only (Figure 1(a. But, in presence of ACK-lost problem, to avoid conflict among te 6

7 duplicate packets (of te same packet, we cannot scedule link w a on slot 6. Tus v a and w a are sceduled sequentially, on slot 6 and slot 7, respectively. Impact of transmission conflict on cannel contention delay. Cannel contention delay and transmission conflict delay are often correlated. Specifically, cannel contention delay can increase wen a flow experiences transmission conflict delay. Let us consider links z w and u x (in G tat can be sceduled on slot 5 wen tere are no oter iger priority flow (Figure 1. In te presence of iger priority flows, if any of transmissions z w and u x in F is delayed, for example by 1 slot, due to transmission conflict wit a iger priority flow, wile te oter can appen at slot 5, ten tese two transmissions ave to be sceduled sequentially (instead of sceduling in parallel. Terefore, even toug sceduling of F as some parallelism, in te worst case in presence of transmission conflict, it can cause cannel contention delay on its lower priority flows like a flow tat appens like a sequential task wit execution requirements of 2E +S slots. Based on te above observations, te analysis for upper bounding te cannel contention delay reduces to tat for a set of flows were eac flow F i as te worst-case time requirement of e i slots troug a single pat route, were e i = 2E i + S i. Hence, we leverage our result in [3] wose analysis was given for flows wit single-pat routes to find te cannel contention delay caused by F on F i. Using tat result, in any time interval of x slots, tere are at most m 1 iger priority flows eac flow F among wic can cause at most Ii (x delay ( on F i as expressed below x Ii e (x, e i = min x e i + 1, e + e + ( ( min e 1, max (x e mod ( R, 0 were R is te worst-case end-to-end delay of F. Te delay caused by eac oter iger priority flow F on F i is at most ( x Ji (x, e i = min x e i +1, e +min ( x mod, e Tus, considering a total of m cannels, an upper bound Ω i (x of te cannel contention delay caused by all iger priority flows on F i in any time interval of x slots is derived as follows. Ω i (x, e i = 1 m ( Z i (x, e i + <i J i (x, e i (2 wit Z i (x, e i being te sum of te min(i 1, m 1 largest values of te differences Ii (x, e i Ji (x, e i among te iger priority flows F, < i. Effect of Cannel Hopping. To every transmission, te sceduler assigns a cannel offset between 0 and m 1 instead of an actual cannel, were m is te total number of cannels. All devices in te network maintain an identical list of available cannels. At any time slot t, a cannel offset c (i.e., 1, 2,, m 1 maps to a cannel tat is different from te cannel used in slot t 1 as follows. cannel = (c + t mod m (3 Bot te sender and te receiver of te corresponding transmission link switces to te new cannel. As can be seen from Equation 3, at every time slot any 2 different cannel offsets always map to 2 different cannels. Te sceduler assigns at most one cannel offset to a link at any time wic maps to different pysical cannels in different time slots, keeping te total number of available cannels at m always, and sceduling eac link on at most one cannel at any time. Hence, cannel opping does not ave effect on cannel contention delay. D. End-to-End Delay Bound Now bot types of delays are incorporated togeter to develop an upper bound of te end-to-end delay of every flow. Tis is done for every flow in decreasing order of priority starting wit te igest priority flow. Teorem 4 provides an upper bound R i of end-to-end delay for every flow F i. Considering no delay from iger priority flows, let te worst-case time requirement of F in te sensing pase be denoted by L sen. For example, in Figure 1(a, Lsen = 9 slots (as described in Section IV. A similar sceduling is followed in te control pase also. Similarly, considering no delay from iger priority flows, let te worst-case time requirement of F in te control pase be denoted by L con. Tus, considering no delay from iger priority flows, te time requirement troug a critical pat denoted by L i, of flow F i is L i = L sen i + L con i (4 Teorem 4: Let x i be te minimum value of x L i tat solves Equation 5 using a fixed-point algoritm. x = Ω i (x, e i + L i (5 Ten te end-to-end delay bound R i of flow F i is te minimum value of t x i tat solves Equation 6 using a fixed-point algoritm. t = x i + ( ( t i + 1.δi + min (δ i, t mod <i (6 Proof: According to Equation 2, x i is calculated considering R (i.e., te end-to-end delay bound of F considering bot cannel contention delay and conflict delay of eac iger priority flow F. According to Equation 2, Ω i (x, L i is te cannel contention delay caused by all iger priority flows on F i in any time interval of x slots. Hence x i is te bound of te end-to-end delay of F i wen it suffers only from cannel contention delay caused by iger priority flows (and no conflict delay. Equation 1 provides te bound of transmission conflict delay of F i. Hence, adding tis value to x i must be an upper bound of F i s end-to-end delay under bot cannel contention and transmission conflict. Tus we can determine R i for every flow F i in decreasing order of priority starting wit te igest priority flow using Teorem 4. In solving Equations 5 and 6, if x or t exceeds D i, ten F i is decided to be unscedulable. Time complexity. Since bot x in Equation 5 and t in Equation 6 can reac a value of at most D i using a fixed 7

8 point algoritm, te worst-case time required to determine R i can be O(D i. Given n flows wit a maximum deadline of D max, te required time for te delay analysis of all te flows is n O(D max wic implies a pseudo polynomial time complexity. Note tat tis time is usually muc less tan tat required for computing all superframes, as te latter would take te least common multiple of all periods wic can be extremely large wen te periods are non-armonic. Moreover, as discussed in Subsection V-B, our delay analysis does not require te enumeration of all te possible pats troug a routing grap. Terefore, te proposed analysis is an efficient approac for determining te scedulability of real-time flows in process monitoring and control applications. VI. A PROBABILISTIC END-TO-END DELAY ANALYSIS Grap routing provides a very conservative approac to sceduling transmissions in a WirelessHART network. In te sceduling used in te previous sections, tere is a syncronization at te access points in te sense tat te sceduling in te downlink grap of a flow (te control pase is started after all links in its uplink subgrap are sceduled. However, tere is ig probability tat a packet will be delivered troug te dedicated pat only because eac link on te pat is dedicated and sceduled twice. Terefore, wenever te gateway receives a sensor packet troug te dedicated link, te corresponding control message can be calculated and delivered troug te downlink grap s dedicated link in te next available slot avoiding syncronization at te access points. Te corresponding retry on te sared slot can be sceduled only after all links on te uplink subgrap of te flow are sceduled. Te advantage of suc a sceduling policy is tat te actual end-to-end delay in most cases will be substantially sorter since a packet follows te dedicated links in most cases. Under tis sceduling, we can determine a probabilistic delay bound tat is tigter tan te bound derived in te last section but represents a bound wit ig probability. Tese bounds rely on statistical independence among te links. Considering te dedicated route as E i links, and p k as te probability of a successful transmission along link k, te probability of being successful upon 2 transmissions troug link k is 1 (1 p k 2. Terefore, te probability tat a packet will be delivered troug te dedicated links is E i ( 1 (1 p k 2 (7 k=1 Let, in G i, te pat consisting of all dedicated links be called dedicated pat. Let i denote te total number of transmissions of (one instance of F tat sare a node on te dedicated pat of F i. Similarly, let δ i denote te maximum conflict delay caused by one instance of F on te bottleneck link on F i s dedicated pat (i.e., a link on F i s dedicated pat can sare a node wit at most δ i transmissions of F. Corollary 1 now follows from Teorem 4. Corollary 1: Let x i be te minimum value of x 2E i tat solves Equation 8 using a fixed-point algoritm. x = Ω i (x, 2E i + 2E i (8 Fig. 2. Testbed topology (access points are blue colored Ten te minimum value of t x i tat solves Equation 9 is te worst-case end-to-end delay bound of flow F i wit a E i probability of at least (1 (1 p k 2. t = x i + <i ( i + k=1 ( t ( 1.δ i +min δ i, t mod (9 Proof: By Equation 2, Ω i (x, 2E i represents te cannel contention delay on te dedicated pat of F i. Following Teorem 4, te minimum value of t x i tat solves Equation 9 is te worst case delay bound for te dedicated route. Te proof follows since a packet as (1 Ei (1 p k 2 probability of k=1 being delivered troug te dedicated route. A. Testbed Experiment VII. EXPERIMENT 1 Implementation: We evaluate our delay analysis on an indoor wireless testbed deployed in two buildings at Wasington University [25]. Te testbed consists of 69 TelosB motes, eac equipped wit Cipcon CC2420 radios compliant wit te IEEE standard. We implement a WirelessHART protocol stack on TinyOS [26] and TelosB. Our protocol stack consists of a multi-cannel TDMA MAC protocol wit cannel opping and a routing layer supporting bot source routing and grap routing. Te uplink and downlink graps are generated using te grap routing algoritms presented in [22]. Time is divided into 10 ms slots and clocks are syncronized across te entire network using te Flooding Time Syncronization Protocol (FTSP [27]. Te details of our protocol stack implementation are available in [28]. 2 Experimental Setup: Te senders sceduled in a sared slot follow CSMA/CA mecanism for transmission witin tat slot of te TDMA scedule. We setup te parameters of te CSMA/CA mecanism in a sared slot as follows to ensure tat a node tat acquires te cannel as sufficient time to complete te transmission witin tat slot (a slot consists of 10ms. Tere are 2 backoff intervals and 3 backoff values: initialbackoff, minimumbackoff, and congestionbackoff. In te beginning of a sared slot, a node first makes an initial backoff to avoid capturing a cannel from te oter nodes wo also are sceduled in tat sared slot. Te first backoff interval is random in te range [minimumbackoff, initialbackoff]. Once tis backoff interval elapses, a node performs CCA. If te 8

9 Worst case delay (ms Delivery ratio Flow ID (ordered by priority Experiment Analytical (a Delivery ratio Flow ID (ordered by priority Fig. 3. (b Delay Delay and reliability on testbed cannel is free it transmits. If te cannel is busy, a node will pick anoter backoff period randomly from te range [minimumbackoff, congestionbackoff]. After te second backoff interval elapses, it will immediately transmit if te cannel is free (cecked upon CCA. If te cannel is sensed busy, te node immediately stops, and no furter attempt is made to transmit in tat slot. We set initialbackoff = 2240 µs, minimumbackoff = 320 µs, and congestionbackoff = 960 µs. We use IEEE cannels 15, 16, 19, and 20 in our experiments. For eac link in te testbed, we measured its packet reception ratio (PRR by counting te number of received packets among 250 packets transmitted on te link. Following common practice of industrial deployment, we only consider links wit PRR iger tan 90% on every cannel to determine te testbed topology. Figure 2 is a topology of te testbed sowing te node positions on te two buildings floor plan. We use two nodes (colored in blue in te figure as access points, wic are pysically connected to a root server (Gateway. Te Network Manager runs on tis root server. Te oter motes work as field devices. We experiment by generating 30 flows on our testbed. Te period of eac flow is randomly selected from te range of [10 2 6, ]ms. Te relative deadline of eac flow equals to its period. All flows are scedulable based on our delay analyses. Priorities of te flows are assigned based on te Deadline Monotonic (DM policy. DM assigns priorities to flows according to teir relative deadlines; te flow wit te sortest deadline being assigned te igest priority. 3 Results: We run our experiments long enoug so tat eac superframe is run for at least 20 cycles. We evaluate our proposed approaces in terms of reliability and delay. We use delivery ratio to measure reliability. Te delivery ratio of a flow is defined as te fraction of te packets generated by te flow tat are successfully delivered to destination. Since tere exists no prior work on delay analysis under reliable grap routing, we compare our analytical delay bounds wit te maximum delay observed in experiments on te testbed. Figure 3 sows our results. Figure 3(a sows te delivery ratios of all 30 flows. Tere are 2 flows eac wit a delivery Fig. 4. Acceptance ratio Simulation Analytical Total number of flows Acceptance ratio based on te worst case delay analysis ratio of 0.90, and 4 flows eac wit a delivery ratio of 0.95, wile every oter flow as a delivery ratio of 1. Tis result demonstrates te effectiveness of grap routing in acieving reliable communication over WSANs. Figure 3(b plots te maximum end-to-end delay observed in our experiments and te end-to-end delay bounds derived troug our delay analysis. As te figure sows, our analytical delay bounds are no less tan te experimental maximum delays, demonstrating tat our delay analysis provides safe upper bounds of te actual delays. Te ratio of te analytical delay bound to te maximum delay observed in experiments is at most Note tat te experiments may not ave experienced te worst case scenarios and ence te maximum delays observed in experiments may not represent te actual worst case delays. B. Simulations For a more extensive evaluation, we now use te same testbed topology and evaluate te results in simulations. We generate flows by randomly selecting sources and destinations, and simulate teir scedules in tese topologies. Two nodes in te topology are selected as access points. Te uplink and downlink graps are generated using te same grap routing algoritms as te one we used in testbed experiment. Te periods of te flows are randomly generated in te range [10 2 5, ]ms. Te deadlines are equal to periods. Priorities of te flows are assigned based on te DM policy. In all cases, we use 12 cannels for sceduling. We evaluate our analysis in terms of Acceptance ratio defined as te fraction of te total number of test cases tat are deemed scedulable. 1 Worst Case Delay Analysis: Since tere exists no prior work on delay analysis under reliable grap routing, we analyze te effectiveness of our analysis by simulating te complete scedule of transmissions of all flows released witin te yper-period. In all figures in tis subsection, Simulation indicates te fraction of test cases tat ave no deadline misses in te simulations, and represents conservative upper bounds of acceptance ratios because we did not simulate all possible arrival patterns of te flows; Analytical indicates te acceptance ratio based on our delay analysis. Figure 4 sows te acceptance ratios for 1000 test cases under varying number of flows. For 20 flows, 986 test cases are scedulable troug simulations wile our analysis as determined 818 cases as scedulable sowing an acceptance ratio of Te ratios decrease wit te increase in te number of flows. Te gap between te analytical acceptance ratio and tat based on simulations stems from te pessimism of our analysis wic provides safe upper bounds. Next we evaluate te probabilistic delay analysis in reducing pessimism. 9

10 Delivery ratio tru dedicated route Expected Observed Flow ID Acceptance ratio Fig. 5. (a Delivery ratio troug dedicated links Simulation Analytical Total number of flows (b Acceptance ratio Performance of probabilistic delay analysis 2 Probabilistic Delay Analysis: Again te testbed topology is used in our simulations for te probabilistic delay analysis, and te measured PRR of eac link is used as te probability of a successful transmission along te link. For a test case of 30 flows, we simulate te scedule in 1000 runs were a link s failure or success is determined probabilistically based on its PRR. Figure 5(a sows te fraction (labeled Observed of te packets tat are delivered troug dedicated routes. In te figure Expected indicates te expected fraction of te packets to be delivered troug dedicated routes wic is te analytical probability calculated based on te measured PRR using Equation 7. Te results sow tat more tan 90% of te packets are delivered troug teir dedicated routes, and tis value is close to wat te analytical probability in Equation 7 indicates. Tus te probabilistic bound delay bound corresponds to a 90 percentile delay bound. Figure 5(b sows te acceptance ratios under te probabilistic delay bound. It sows te probabilistic analysis can accept at least 80% of te test cases tat are scedulable according to simulations for up to 35 flows. Te results suggest tat our probabilistic analysis can effectively reduce te pessimism of analytical delay bounds, and ence represents an effective alternative for delay analysis of soft real-time flows for wic probabilistic delay bounds are sufficient. For example, tis analysis may be used for non-critical applications. VIII. CONCLUSION Industrial wireless sensor-actuator networks must support reliable and real-time communication in as environments. Industrial wireless standards suc as WirelessHART adopt a reliable grap routing approac to andle transmission failures troug retransmissions and route diversity. Tese mecanisms introduce substantial callenges in analyzing te scedulability of real-time flows. We ave presented te first worst-case delay analysis under reliable grap routing. We ave also proposed a probabilistic delay analysis tat provides delay bounds wit ig probability. Experiments based on a wireless testbed of 69 nodes and simulations sow tat our analytical delay bounds are safe, and can be used as an effective scedulabity test for real-time flows under reliable grap routing. ACKNOWLEDGMENT Tis work was supported by NSF troug grant CNS (NeTS. REFERENCES [1] A. Saifulla, Y. Xu, C. Lu, and Y. Cen, Real-time sceduling for WirelessHART networks, in RTSS 10. [2] WirelessHART, 2007, ttp:// [3] A. Saifulla, Y. Xu, C. Lu, and Y. Cen, End-to-end delay analysis for fixed priority sceduling in WirelessHART networks, in RTAS 11. [4] J. B. Scmitt and U. Roedig, Sensor network calculus - A framework for worst case analysis, in DCOSS 05. [5] O. Cipara, C. Lu, and G.-C. Roman, Real-time query sceduling for wireless sensor networks, in RTSS 07. [6] T. F. Abdelzaer, S. Prab, and R. Kiran, On real-time capacity limits of multiop wireless sensor networks, in RTSS 04. [7] J. Stankovic, T. Abdelzaer, C. Lu, L. Sa, and J. Hou, Real-time communication and coordination in embedded sensor networks, Proceedings of te IEEE, vol. 91, no. 7, pp , [8] H. Li, P. Senoy, and K. Ramamritam, Sceduling messages wit deadlines in multi-op real-time sensor networks, in RTAS 05. [9] X. Wang, X. Wang, X. Fu, G. Xing, and N. Ja, Flow-based real-time communication in multicannel wireless sensor networks, in EWSN 09. [10] V. Kanodia, C. Li, A. Sabarwal, B. Sadegi, and E. Knigtly, Distributed multi-op sceduling and medium access wit delay and trougput constraints, in MobiCom 01. [11] C. Lu, B. M. Blum, T. F. Abdelzaer, J. A. Stankovic, and T. He, RAP: A real-time communication arcitecture for large-scale wireless sensor networks, in RTAS 02. [12] K. Karenos and V. Kalogeraki, Real-time traffic management in sensor networks, in RTSS 06. [13] T. He, B. M. Blum, Q. Cao, J. A. Stankovic, S. H. Son, and T. F. Abdelzaer, Robust and timely communication over igly dynamic sensor networks, Real-Time Syst., vol. 37, no. 3, [14] T. W. Carley, M. A. Ba, R. Barua, and D. B. Stewart, Contentionfree periodic message sceduler medium access control in wireless sensor/actuator networks, in RTSS 03. [15] K. Liu, N. Abu-Gazale, and K.-D. Kang, JiTS: Just-in-time sceduling for real-time sensor data dissemination, in PERCOM 06. [16] Y. Gu, T. He, M. Lin, and J. Xu, Spatiotemporal delay control for low-duty-cycle sensor networks, in RTSS 09. [17] N. Pereira, B. Andersson, E. Tovar, and A. Rowe, Static-priority sceduling over wireless networks wit multiple broadcast domains, in RTSS 07. [18] A. Saifulla, S. Sankar, J. Liu, C. Lu, B. Priyanta, and R. Candra, CapNet: A real-time wireless management network for data center power capping, in RTSS 14, [19] A. Saifulla, Y. Xu, C. Lu, and Y. Cen, End-to-end communication delay analysis in industrial wireless networks, IEEE Transactions on Computers, vol. 64, no. 5, pp , [20], Priority assignment for real-time flows in WirelessHART networks, in ECRTS 11. [21] A. Saifulla, C. Wu, P. Tiwari, Y. Xu, Y. Fu, C. Lu, and Y. Cen, Near optimal rate selection for wireless control systems, in RTAS 12. [22] S. Han, X. Zu, and A. K. Mok, Reliable and real-time communication in industrial wireless mes networks, in RTAS 11. [23] X. Zu, P.-C. Huang, S. Han, A. Mok, D. Cen, and M. Nixon, RoamingHART: A collaborative localization system on WirelessHART, in RTAS 12. [24] C. Wu, M. Sa, D. Gunatalika, A. Saifulla, C. Lu, and Y. Cen, Analysis of EDF sceduling for wireless sensor-actuator networks, in IWQoS 14. [25] Wireless sensor network testbed, ttp://mobilab.wustl.edu/testbed. [26] TinyOS Community Forum, ttp:// [27] M. Maróti, B. Kusy, G. Simon, and A. Lédeczi, Te flooding time syncronization protocol, in SenSys 04. [28] M. Sa, D. Gunatilaka, C. Wu, and C. Lu, Implementation and experimentation of industrial wireless sensor-actuator network protocols, in EWSN