D6.1.31: Resilient smart grid demandresponse communications utilising hybrid sensor-lte network.

Transcription

1 - 1 - D6.1.31: Resilient smart grid demandresponse communications utilising hybrid sensor-lte Revision History Edition Date Status Editor v created J. Markkula V draft J. Markkula, B. Gebremedhin V Revision J. Haapola V Revision J. Markkula V Final J. Haapola Abstract This report describes the details of the /LTE hybrid network (with multihop (MH) routing), the scenarios, and the results of the simulations for SGEM FP4. The purpose of the developed MH routing method is to manage a situation that a LTE base station becomes inaccessible due to some reason (e.g. equipment or software failure). The smart grid traffic is MH routed through interfaces of the clusterheads to the nearest LTE base station that operates correctly. Different values of the SG traffics are generated and the simulations are performed with distinct parameters and setups to obtain information about the optimal solutions and the performance limits.

2 - 2 - Table of Contents D6.1.31: Resilient smart grid demand-response communications utilising hybrid sensor-lte... 1 Revision History... 1 Abstract... 1 Table of Contents Preface Scope Introduction Simulator Details Simulator Functionality Purpose of the MH Routing Method MAC Super Frame Structure MH Routing Method Routing Control Messages Multipurpose Frame Hello Message BST Lost BST Reconnect Other IEEE MAC Frame Formats Enhanced Beacon Enhanced Acknowledgement Data Frame Simulation Scenario and Setup Simulation Topology Simulation Parameters Simulation Scenarios Superframe Duration Results Load and Packet Delivery Ratio Results Delay Results of the SG Case Delay Results of the SG Case Delay Results of the SG Case Conclusions References... 29

3 - 3-1 Preface This report was done as a part of the Finnish national research project "Smart Grid and Energy Market" SGEM. It was funded by Tekes the Finnish Funding Agency for Technology and Innovation and the project partners. 2 Scope In a hybrid sensor-lte network, automatic meter reading (AMI) meters are equipped with an IEEE interface and they are connected through hybrid cluster heads with the LTE In the previous report [1] and in the conference paper [2], the hybrid sensor-lte network simulations were performed with demand-response traffic. The scope of this deliverable is to manage a situation that a LTE base station (BST) becomes inaccessible to communications. The aim is to study if it is feasible to connect clusterheads (CLHs) that have no connection to any LTE BST to CLHs that have a connection to a LTE BST. A multihop (MH) routing method that utilises the IEEE interface of CLHs to deliver SG traffic to a CLH within operational LTE BST coverage is developed. The results from the simulations performed in Opnet Modeler with Wireless Suite and LTE simulation toolboxes are presented. The focus on the simulations is on technology enabling communications rather than the contents of the data being transferred. 3 Introduction This deliverable describes the simulation setups in /LTE hybrid network with MH routing. A suburban environment is generalised into clusters that are connected through hybrid CLHs with one of the other LTE BSTs. One BST shuts down for a part of the simulation run causing the smart grid traffic to be MH routed through a CLH that has a connection to another BST that operates correctly. The connection from each CLH to the nearest CLH that is connected to other LTE BSTs is obtained in a proactive manner due to frequently broadcasted routing control messages. The simulation parameters are almost the same and the propagation models are exactly the same as in the previous reports [1], [3] and the conference papers [2], [4] but are shortly described also in this report. The main purpose of the simulation scenarios is to observe the functionality and the performance of the proposed MH routing method to obtain sufficient reliability and delay for different smart grid traffics with distinct parameters and setups. The simulation scenarios addressed for SG traffic include three cases with distinct reporting frequencies of RTUs: SG cases 1-2 present the situations where RTU reporting period is decreased for the exception situation when some LTE BST is malfunctioning and SG case 3 presents automatic demand response (ADR) with low-frequency control (tarif updates). All SG cases were simulated by applying one or two of the available frequency channels for MH communications and two different superframe (SF) lengths. The rest of the deliverable is organised as follows. Section 4 explains simulator details including description of the simulator functionality and formats for the messages and for the frames. Section

4 - 4-5 provides the simulation topology, the simulation parameters, the simulation scenarios, and SF duration estimation. Simulation results are presented in Section 6. The conclusions are then drawn in Section 7. 4 Simulator Details Hybrid sensor-lte network simulator is previously presented in [1] and [2]. The new feature, multihop routing functionality for the simulator is presented in the following subsections. 4.1 Simulator Functionality The purpose and the functionality of the Opnet simulator is described next. The document focuses on recently developed and implemented MH routing functionality of the hybrid sensor-lte simulator Purpose of the MH Routing Method The main reason for the development of the MH routing functionality is to support the situations when LTE BST is malfunctioning or shut down, e.g. during a blackout. Thus, the aim is be able to MH route (in UL and DL directions) SG traffic to the nearest hybrid CLH that is connected to correctly operating LTE BST. WSN interface of the each hybrid CLH forms a MH ad hoc network that is applied in exception situations to guarantee the continuation of the SG communication even if in some area the LTE network coverage is lost MAC Super Frame Structure MAC superframe (SF) is divided as a function of time to contention period (CAP) and contention free period (CFP) as presented in the Figure 4-1. Both time periods apply unslotted CSMA-CA MAC technology. CAP is applied for beacon and data traffic delivering between WSN interface of the hybrid CLH and cluster members (WSN RTUs). CFP (initial) is applied for MH routing control messages (hello, BST lost). CFP (BST lost) is applied for delivering MH routed data frames between CLHs. If a CLH is connected to a LTE BST, CFP (BST lost) period does not exist and the CAP period extended to cover that SF portion. Beacon Guard time Guard time CAP, freq 1 Traffic between RTUs and CLH CFP (BST lost), freg 2 MH routed data Figure 4-1: MAC superframe structure. CFP (initial), freq2 Hello and BST lost

5 - 5 - CAP and CFP apply separate frequency channels. Frequency 1 (freq1) is applied for inside a cluster communication during the CAP. RTUs operate only on freq1 and are able to apply CFP as a sleeping period. Frequency 2 (freq2) is applied only by the CLHs during the CFP. One or more MH ad hoc networks can be formed by the CLHs depending on how many frequency channels are occupied for CFP. Guard time that is selected as a turnaround time from transmission mode to receiving mode and vice versa (0.192 ms) is occupied for CLHs to frequency channel change and to compensate inaccuracy of CLH clock times. No data is delivered during the guard time. A packet is only transmitted in current SF period (CAP, CFP (BST lost) or CFP (initial)) if there is enough time for transmission and possible ACK reception. CSMA-CA begins from the initial stage on the start of the each SF period, i.e. number of retransmissions and back off values are initialized. There is also a small random time ( s) that corresponds to CSMA-CA backoff period length with minimum backoff exponent (3), at the start of the each SF period to prevent simultaneous packet transmissions and collisions MH Routing Method When the hybrid CLHs are connected to an LTE BST, most of the SF duration is occupied by CAP. WSN RTUs communicate with their CLH that operates as a relay to the LTE BST that has a wired connection to SG server. The CFP (initial) period occupies a small part of the SF. During the CFP (initial), hello message are broadcasted within a certain time interval (15 s). Each CLH receives a hello messages from its CLH neighbours. Due to these hello messages, each CLH is able to select a neighbour CLH that can be applied as a next hop destination in UL direction (towards different LTE BST) if connection to own LTE BST is lost. Uplink routing table Uplink route from each CLH to the nearest CLH that has a connection to different LTE BST (DC CLH) is formed based on received hello messages. CLH stores to its uplink (UL) routing table information about its every neighbour CLH as presented in the Table 4-1. Each time a CLH generates a hello message when connected to LTE BST, it selects from its UL routing table a neighbour CLH that is connected to DC CLH or has a smallest number of hops to DC CLH, and from whom the last hello message is received at least once during three hello intervals. If there are more than one DC CLH neighbours or more than one CLH with the shortest number of hops to DC CLH, the one with the smallest number of selections will be chosen (the highest select ratio or select ratio other value in hello packet field). To avoid frequent changes in next hop UL CLH selection, select ratio or select ratio other has to be more than one unit higher than the previous value that the next hop UL selection will be changed. Thus, the aim is to minimise the number of hops, while trying to share the routing burden among the CLHs as equally as possible. An example of UL routing table is presented in the Table 4-1. CLH z has four neighbour CLHs. All of them are connected to the same LTE BST (5) and none are DC CLHs. Based on hello rec time, hello message is recently received from all the neighbour CLHs (< 3 * hello interval). Every neighbour CLH has already received a hello message from CLH (z) because communication link is informed to be two-way symmetric (sym neigh = true). CFP length is CFP (initial) (12.5% of SF) because all CLHs are connected to LTE BST. Ref time informs a clock time of each CLH neighbour and it can be applied in network synchronisation if connection to LTE BST is lost. Ref

6 - 6 - time values of the table have a bit inaccuracy because of propagation delay. If locations of the CLHs are known, the propagation delay could be subtracted. Neighbour CLH (b) and CLH (c) have the same distance to DC CLH but different amount of selections. If the maximum amount of selections is 6 (x), CLH (b) is selected one time and CLH (c) twice (ratio =1-y/x). In case that CLH (z) has previously selected CLH (c) as next hop UL, it will keep its selection because the difference between select ratios is only one unit (1/x). Otherwise, it selects CLH (b) because it has a higher select ratio. Table 4-1: An example of UL routing table. num of hops select ratio select ratio other BST id CFP lenght next hop UL sym neigh hello rec ref time Neighbor CLH (a) % CLH b true s s Neighbor CLH (b) % CLH e true s s Neighbor CLH (c) % CLH f true s s Neighbor CLH (d) % CLH c true s s Downlink routing table Downlink (DL) routing table (from BST to CLH) is formed when connection to a LTE BST is lost. Downlink routing table is reverse to uplink routing table. When hybrid CLH is not able to connect to any LTE BST, it generates a BST lost message with artificial delay (0 3 s). The BST lost message is illustrated in the 3 b 5 b 1 B 4 B 1 B frame type frame length BST connected before CFP length CLH addr Figure 4-5 and it is MH routed based on uplink routing table to nearest DC CLH that relays the message through a functioning LTE BST to the SG server. When a CLH receives a BST lost message, it stores its original source (dest CLH) and its one-hop source (next hop to dest CLH) to its DL routing table. The Table 4-2 presents a DL routing table of CLH (e) that has one-hop distance to DC CLH. CLH (b) has two hops and CLH (a) has three hops to DC CLH. After the BST lost message is generated, a hello message is broadcasted to inform neighbour CLHs about the disconnection from the malfunctioning LTE BST. Table 4-2: An example of DL routing table. dest CLH CLH (b) CLH (a) next hop to dest CLH CLH (b) CLH (b) If a hybrid CLH has not received any traffic from SG server within one minute after generating a BST lost message, a new BST lost message will be generated. Each time a CLH generates a BST lost message, it also generates a hello message to inform its neighbour CLHs about BST

7 - 7 - shutdown and CFP (BST lost) period adding to SF. Also, the DC CLH that is applied as a relay between LTE BST and CLH without LTE connection has to decrease the CAP period length and insert CFP (BST lost) period to SF. Next hop CLH selection in UL direction (next hop UL) is not changed after the first successful transmission of BST lost message because unnecessary changing of routes could harm the MH traffic end-to-end delivery. The current version of the simulator does include the functionality for the occasions that a routing path has to be updated because some CLH shuts down or is not willing to MH route the traffic. When a CLH reconnects to a LTE BST, it generates a BST reconnect message that is delivered directly through the LTE network to the SG server that is informed to route the future data directly to the CLH without MH routing in WSN After the BST reconnect message is generated, a hello message is generated after up to 3 s artificial random delay and broadcasted to inform neighbour CLHs about the reconnection to the LTE BST. When a CLH knows that all of its neighbour CLHs are also connected to LTE BST (from hello message), it can restore the CAP period length and remove CFP (BST lost) period from the SF. 4.2 Routing Control Messages Multipurpose Frame Routing control messages are delivered in wireless sensor network (WSN) to form MH routes between hybrid cluster heads (CLH) that are connected to a distinct LTE BST. IEEE e multipurpose frames presented in [5] are applied to deliver these routing control messages. Structure of the multipurpose frame is presented in the Figure 4-2 and its frame control field is presented in the Figure 4-3 [5]. Frame control size is 2 B because PAN id is present. Sequence number size is 1 B. Destination PAN identifier size is 2 B. Destination address size is 2 B. Source PAN identifier size is 1 B. Source address size is 1 B. Auxiliary security header and information elements are not included (size is 0 B). FCS size is 2 B. In total, multipurpose frame size is 11 B without payload field that contains a specific MH routing message. Figure 4-2: Multipurpose frame format. Figure 4-3: Format of the frame control field (multipurpose frame).

8 Hello Message The Hello message is presented in the Figure 4-4 and it is broadcasted to one hop distance by all hybrid CLHs repeating a certain generation interval. The Hello message is included in the payload field of the multipurpose frame. The Hello message contains the following information fields. Frame type informs a network layer about Hello message reception. Frame length is included for variable number of neighbor addresses. Number of hops to nearest hybrid CLH that is connected to another LTE BST is stored to num of hops message field. Relay selection ratio (1-y/x) is calculated as the number of CLHs that have selected this CLH as next hop relay in uplink (UL) direction (y) and maximum number of selections that the CLH allows (x), (select ratio). The relay selection ratio (1-y/x) of the nodes that are connected to other LTE BSTs, (select ratio other), i.e., if the ratio is less than 1 the CLH is selected as a relay to a distinct LTE BST. BST id contains id of the LTE BST to which the CLH is currently connected to, or if not connected the value is -1. Contention free period (CFP) length informs about the current length of the CFP in a super fame (SF). The CFP length is increased when a CLH has to MH route data frames. Address of the CLH that is selected as a next hop relay in UL direction is stored to next hop UL field. Reference time is applied to distribute clock time of a CLH for time synchronization purposes in case that a connection to a LTE BST is lost, (ref time). Address of each neighbour CLH is stored to one hop neighs field. The maximum number of the CLH neighbours is assumed to be 8 due to a simulation topology that is presented later. The total size of the hello message is 14 B + 1 B for each neighbour. 3 b 5 b 1 B 1 B 1 B 1 B 4 B 1 B 4 B (0-8) * 1 B frame type frame length num of hops select ratio select ratio other BST id CFP lengh One next hop UL ref time hop neighs Figure 4-4: Hello message BST Lost BST lost message (7 B) is presented in the 3 b 5 b 1 B 4 B 1 B frame type frame length BST connected before CFP length CLH addr Figure 4-5 and it is generated when a connection to all LTE BSTs is lost. The BST lost message is included in the payload field of the multipurpose frame. BST lost is MH routed through a route formed due to hello messages to nearest CLH that is connected to another operational LTE BST. The connected LTE BST forwards the BST lost message to SG server. The BST lost message causes the route in DL direction to be formed (reverse route of UL route). Frame type informs a network layer about BST lost message reception. Frame length is not currently necessary but it is included to fill a byte with the frame length field. BST connected before informs the address of

9 - 9 - the LTE BST that is shutdown. Current CFP length is stored in the message. CLH addr contains an address of the hybrid CLH that has generated the BST lost message. 3 b 5 b 1 B 4 B 1 B frame type frame length BST connected before CFP length CLH addr Figure 4-5: BST lost message BST Reconnect A BST reconnect message (1 B) is delivered directly through an LTE network to the SG server when a CLH is reconnected to a LTE BST. The BST reconnect message contains information about reconnection to the BST. Thus, the SG server is able route data directly to the CLH without MH routing. 4.3 Other IEEE MAC Frame Formats Some selections in the IEEE e medium access control (MAC) frame fields presented in [5] are made and the utilised enhanced beacon has some additional information stored in its payload field Enhanced Beacon The IEEE e enhanced beacon frame format is presented in the Figure 4-6 and it is generated by every hybrid CLH at the start of the each SF. Distributed synchronous multi-channel extension (DSME) option of the enhanced beacon supports flexible super frame specification and applying of multiple frequency channels. WSN RTUs that belong to a cluster receive beacons from their hybrid CLH. Information about contention access period (CAP) length is included to beacon payload (4 B). The WSN RTUs in a cluster are allowed to transmit and receive data only during a CAP period of the SF. If the CAP length is decreased, RTUs are conscious that their CLH has lost a connection to the LTE BST. Frame control field that is applied in most of the IEEE e frames (2 B) is presented in the Figure 4-7. Sequence number is included (1 B), addressing field reserves 3 B, auxiliary security header is omitted (0 B), IE header contains DSME PAN descriptor (16 B) and FCS is 2 B. IE fields of the DSME PAN descriptor are presented in the Figure 4-8. Pending address is omitted (0 B). Beacon bitmap is presented in the Figure 4-9 and its size is 5 B because SD bitmap value is 1 to support only one SF per each beacon generation. Channel hopping and group ACK are omitted (0 B). Enhanced beacon size is 24 B without and 28 B with the CAP length payload information field.

10 Figure 4-6: Enhanced beacon frame format. Figure 4-7: Format of the frame control field for beacon, data, acknowledgement and MAC command frames. Figure 4-8: Format of DSME PAN descriptor IE. Figure 4-9: Format of the beacon bitmap fields Enhanced Acknowledgement Enhanced acknowledgement (8 B) is presented in the Figure 4-10 and it is applied to confirm successful packet reception and, if not received, the need of a retransmission at the MAC layer. Acknowledgement is applied for all WSN packet transmissions except for Hello transmissions. Frame control field (2B) is presented earlier in the Figure 4-7. Sequence number (1 B) is included. Destination PAN identifier (2 B) and destination address (1 B) are included. Source PAN identifier, source address, aux. security header and information elements are omitted (0 B). FCS size is 2 B.

11 Figure 4-10: Enhanced acknowledgement frame format Data Frame Data frame format is presented in the Figure 4-11 (11 B). Frame control size is 2 B. Sequence number (1 B) is included. Addressing field size is 6 B. Aux. security header and information elements are omitted (0 B). SG payload message (100 B) is stored in data payload field. FCS size is 2 B. Data frame size without payload is 11 B and with the SG message the size is 111 B. Figure 4-11: Data frame format. 5 Simulation Scenario and Setup This section presents simulation topology, parameters, scenarios, and the SF duration estimation. 5.1 Simulation Topology The simulation topology, presented in Figure 5-1, is generalisation of a suburban environment, where the gaps between clusters represent discontinuations in houses, like roads, streams, parks, etc. The suburban environment is equally divided between LTE BST 1 and LTE BST 2 coverage areas. The clusters themselves represent municipal planning of groups of buildings with less order in positioning (random placement of RTUs/UEs). The terrain of the suburban region is quite flat and it is divided into 30 clusters, each containing 25 and, in total 750, houses/apartments with AMI units, each equipped with RTU that is wirelessly connected with the hybrid CLH of the cluster that has a wireless connection with the LTE enodeb. When both LTE BSTs are operating correctly,

12 each hybrid CLH operates as a gateway between two distinct wireless networks with different technologies. If the other LTE BST shuts down, the traffic of the other half of the area is multihop routed to the nearest CLH that has a connection to an operational LTE BST. Operational radio frequency (RF) channels for the clusters were selected in such a way that the distance between the clusters applying the same channel was maximized resulting in no overlap between clusters using the same channel. More information about evaluating the simulation topology is presented in the previous reports [1] and [3]. 790 m LTE BST Cluster 1 25 RTUs 1 CLH LTE EPC, Server m m m LTE BST 2 Figure 5-1: Suburban scenario topology with two LTE BSTs. 10 m 5.2 Simulation Parameters Key parameters for the LTE part of the hybrid sensor-lte network are selected based on [6] and [7], and presented on the left side of the Table 5-1. Single input single output (SISO) antenna configuration is applied. Link adaptation and channel dependent scheduling mode signifies that also RTUs/UEs will take measurements on various sub-bands and calculate separate modulation and coding scheme (MCS) indexes for each sub-band. The BST will try to match the RTUs/UEs to their preferred sub-bands, perform link adaptation, and create wideband MCS index. Packet data convergence protocol (PDCP) was disabled. Thus, internet protocol (IP related headers (IP, UDP, TCP, RTP) were not compressed. [8]

13 Table 5-1: Key parameters for the simulation scenarios. LTE parameter UE/CLH enb parameter RTU/CLH Band width 10 MHz (UL) 10 MHz (DL) Channel band width 2 MHz (16 channels) Base frequency 1800 MHz (BST 1) 1990 MHz (BST 1) 2404, 2409,..., 2469 MHz (inside CL) Base frequency 1810 MHz (BST 2) 2000 MHz (BST 2) 2474 and 2479 MHz (MH) Transmission power 0.2 W 39.8 W Transmission power 71 mw (inside CL), 6.5 mw (MH) Cyclic prefix type 7 symbols per slot Data rate 250 kbps Tx antenna gain -2 dbi 16.5 dbi Error correction 1 % Receiver sensitivity dbm dbm Receiver sensitivity -95 dbm Antenna height 1.5 m (UE), 10 m (CLH) 30 m Antenna height 1.5 m (RTU), 10 m (CLH) Scheduling mode Link adaption and channel dependent SF order (SF lenght) 3 ( ms), 4 ( ms) scheduling Maximum backoff number 4 PDCP compression Disabled Minimum backoff exponent 3 (inside CL), 6 (MH) Pathloss Suburban macrocell with terrain type C Number of retransmissions 3 Pathloss Erceg with terrain type C, pathloss from obstacles -6dB * (0,1,2) (RTU) Key parameters for the part are presented on the right side of the Table 5-1. Base frequencies for the clusters (CL) were selected orthogonally (maximum distance between the same frequencies). Transmission power for inside a CL was selected to enable transmissions between any two nodes inside a CL. Simulations were repeated with one or two frequency channels applied for multihop (MH) communication. Transmission power for MH communication was selected to obtain connection between any two neighbour CLHs. CLH antenna was assumed to be placed on a rooftop height. SF order defines the length of the SF. Simulations were repeated with two distinct SF lengths that were selected to be the most optimal values due to analytical and simulation study. Minimum backoff exponent was selected based on simulation study to optimal value for MH payload data transmissions in CFP (BST lost) period to improve packet delivery ratio (PDR). The rest of the parameters were selected according to [9]. The used path-loss model type in LTE network, suburban macrocell, is based on COST 231 Hata path-loss model [10]. Terrain type C corresponds to mostly flat terrain with light tree densities. The applied path-loss model in level is Erceg that is suitable for communication inside a cluster and between the cluster heads [11]. Path loss from obstacles signifies that there are from 0 to 2, randomly selected, walls between each RTU and the hybrid CLH. Each wall attenuates the signal by 6 db [12], [13]. Hybrid CLHs are assumed to be placed on a rooftop height (e.g. TV antenna pole). Thus, hybrid CLHs do not experience path loss caused by walls on a propagation path to LTE BST or to other CLH. 5.3 Simulation Scenarios Traffic is generated in the simulation scenarios as presented in the Table 5-2. SG cases 1-3 were simulated with applying one or two frequency channels for MH communication and two different SF lengths (two distinct SF order (SO) values, 3 and 4) as presented in the Table 5-1. The reporting frequency of RTUs in UL direction is specified according to SG cases 1-3 (60, 30 and 4 s). DL traffic is transmitted by the server to RTUs that generate UL traffic. Server reporting period is the same in all the scenarios (5 min). 20 simulation runs applying different setups that affect the traffic distributions and node placements are averaged per each simulation. The length of each simulation run is 45 min.

14 The SG case 3 presents an automated demand response (ADR) application where the ADR advanced metering infrastructure (AMI) units only report of the accumulated energy usage at 4 second intervals. The ADR server provides spot pricing updates to the AMI units at five-minute intervals that are transmitted throughout the five-minute period to the AMI units in order to balance out the network load. SG cases 1-2 present the situations where RTU reporting period is decreased when one LTE BST is malfunctioning. Table 5-2: Traffic parameters for the simulation scenarios. Smart grid traffic per RTU (UL) and Server (DL) Start time Generation interval Payload data SG case 1 (UL) random [ ] s 60 s SG case 2 (UL) random [ ] s 30 s SG case 3 (UL) random [ ] s 4 s 100 B SG case 1, 2 and 3 (DL) random [ ] s per each RTU 5 min LTE traffic component QoS class identifier, ARP GBR MNR Smart grid traffic 9 (non-gbr), 9-4 (ACK) QoS classes for LTE traffics are defined at the end of the Table 5-2. QoS class identifier number 9 signifies that there is no guaranteed bit-rate (non-gbr) value for the transmitted data [14]. Smart grid (SG) traffic has maximum number of retransmissions (MNR) set to 4 in the LTE In the WSN, SG traffic was allowed up to 3 retransmissions. The Figure 5-2 presents the operation during a simulation run. During the first 15 minutes both BSTs are functioning correctly and the traffic is delivered directly between CLHs (1-30) and the nearest LTE BST (BST 1 and BST 2). Between 15 and 35 minutes, BST 1 is malfunctioning. Thus, CLHs 1-15 are connected to LTE network through CLHs and BST 2. When a single frequency channel is applied in multihop communication, the traffic is routed, for example, as the arrows present (1-3 additional hops) on the left side of the figure. On the right side of the figure, each CLH applies one of two MH frequencies to minimize the interference between the distinct paths using the same channel. The last 10 minutes (35-45 minutes) BST 1 is functioning correctly again and all the CLHs communicate directly with the nearest LTE BST.

15 BST 1 shut down (15-35 min) LTE BST 1 One MH frequency Not connected to any BST (15-35 min) Hop 3 LTE BST 1 Two MH frequencies Hop 2 CLH MH freq Hop BST 2 working correctly LTE BST 2 Connected to BST LTE BST 2 Figure 5-2: Suburban scenario and operation during a simulation run when applying one or two frequency channels in multihop communication Superframe Duration The two most suitable SF durations are selected based on analytical study. Superframe order (SO) 3 corresponds to SF duration of ms and SO 4 corresponds to duration of ms. These selections are also verified by simulation study. SF duration is optimised for the case that there are in maximum three MHs in communication when connection to LTE BST is lost. The SF structure was presented earlier in the Figure 4-1. CFP (initial) is applied for the routing control messages. CFP (initial) duration is selected to be sufficient for BST lost message delivering through three hops, i.e. CLH is able to inform about disconnection from LTE BST during the current SF. The estimated time for BST lost message delivering through three hops is calculated as ( d + d d ) t + BST _ lost = d rand + 3, BO1 BST _ lost ack where d is the maximum value of the random time at the start of the period (2.24 ms), rand d is BO1 the maximum time reserved for CSMA-CA backkoff (2.24 ms) with minimum backoff exponent that is (3), d is the time spent for one BST lost message transmission (0.608 ms), and BST _ lost d is ack

16 ack wait duration (0.864 ms). In total the estimated duration for three BST lost message transmissions is ms. When a connection to LTE BST is lost, half of the SF is selected to be reserved for the CAP and the half for the CFP. It is rational to select approximately the same time for the cluster internal communication as for the MH communication because the amount of the data is approximately the same. CFP (BST lost) period is applied for multihop routed payload data transmissions. The CFP (BST lost) duration is selected to be sufficient for payload data message delivering through three MH, i.e., CLH is able to deliver a data packet to server during the current SF with a high probability. Time spent for three payload data packet transmissions (three hops) is calculated as data = drand + BO2 ( d + d d ) t 3 +, data ack where d is the maximum time reserved for CSMA-CA backkoff (20.16 ms) with minimum BO2 backoff exponent that is (6), d is the time spent for payload data message transmission (3.744 data ms), d is ack wait duration (0.864 ms). In total the estimated duration for three data message ack transmissions is ms. An IEEE SF consists of 16 SF slots as presented in the Figure 5-3. With SF duration of ms (SO 3), 2 slots have to be reserved for CFP (initial). With SF duration of ms (SO 4), CFP (initial) duration is 1 slot. Thus, CFP (initial) duration is ms with the both SF durations. CFP (initial) duration is a bit higher than the estimated duration of BST lost message transmissions through three hops ( ms). Guard time Guard time CAP CFP (initial) SO 3 (14 slot) SO 4 (15 slot) SO 3 (2 slot) SO 4 (1 slot) Figure 5-3: Super frame structure when connected to LTE BST. For CAP (intra-cluster communication), there are reserved 8 SF slots when connection to LTE BST is lost as presented in Figure 5-4. In case of the SO 3, 6 SF slots is left for CFP (BST lost) (46.08 ms). In case of SO 4, 7 SF slots is left for CFP (BST lost) ( ms). With these SF durations, CFP (BST lost) lengths are the closest values when compared to estimated duration of data message transmission through three hops ( ms).

17 Guard time Guard time CAP CFP (BST lost) CFP (initial) 8 slot SO 3 (6 slot) SO 4 (7 slot) SO 3 (2 slot) SO 4 (1 slot) Figure 5-4: Superframe structure when not connected to LTE BST. The Table 5-3 presents the delay results of multihop UL and DL traffics when 2 MH frequencies and four different SF durations (SO 2-5) were applied with SG case 1 traffic values. It can be seen that in most cases, SO 3 and SO 4 have the lowest delays. Applying SO 3 produces the lowest delays, but SO 4 was also selected for further simulations because the results could vary with distinct traffic values. Table 5-3: Minimum, maximum, average delays of SG case 2 MH traffics with SO 2-5. Number of hops in MH routing Multi-hop UL traffic (BST lost), 2 MH. freq. Multi-hop DL traffic (BST lost), 2 MH freq. 1 hop 2 hop 3 hop SO , 1.909, , 2.891, , 3.396, SO , 2.061, , 2.700, , 2.764, SO , 2.325, , 2.431, , 3.053, SO , 2.261, , 0.964, , 0.966, SO , 0.719, , 0.893, , 1.200, SO , 0.381, , 0.498, , 0.617, SO , 0.501, , 0.601, , 0.571, SO , 0.562, , 0.964, , 0.966, 0.325

18 Results The section presents the results of the OPNET network simulations. 6.1 Load and Packet Delivery Ratio Results The Figure 6-1 presents the average loads for the SG traffic as a function of time. The volume of the SG traffic changes due to message generation interval of the RTUs (SG case 1, 2, and 3). In total, UL traffic is generated by 750 RTUs. Multihop UL traffic is generated by the 375 RTUs, half of the network, that are not currently connected to LTE BST (15-35 min). The generated multihop traffic is delivered between clusterheads through 1-3 additional hops that increase the network burden significantly. DL traffic is generated by the server to corresponding RTUs with 5 min generation intervals in all the cases. The highest amount of SG traffic is generated in SG case 3 (19 kb/s, UL + DL), (9.5 kb/s, Multihop UL + Multihop DL). 100 UL traffic (750 RTUs) Multi-hop UL traffic (375 RTUs) DL traffic (750 RTUs) Multi-hop DL traffic (375 RTUs) Average load [kb/s] ,1 1 min (UL), 5 min (DL) SG case 1 30 s (UL), 5 min (DL) SG case 2 4 s (UL), 5 min (DL) SG case 3 Generation interval per RTU Figure 6-1: Average loads of SG traffic components. The packet delivery ratios (PDR) and the peak loads, over multiple instantiations of the topology are presented in the Table 6-1. Traffic (BST lost) contains all the SG traffic during the simulation time, multihop traffic (BST lost) contains the traffic that was MH routed, and traffic (BST connected) contains the traffic that was delivered when the both BSTs were functioning correctly. PDRs were

19 above the QoS requirement for SG traffic (>99%) in all the cases in BST connected situation. PDR values that are below the QoS requirement are presented in boldface font. When two distinct frequency channels were allocated for multihop communication (2 MH freq), applying SF length of SO3 produced PDRs above the QoS requirement in all the cases. When applying SO4, case 3 traffic caused PDRs under the QoS requirement in most cases. When a single frequency channel was allocated for multihop communication (1 MH freq), PDRs were above the QoS requirement with SO3 in cases 1 and 2 and with SO4 in case 1. The peak load values were dependent on amount of the generated traffic (case 1-3) and amount of generating nodes (750 or 375). Table 6-1: (Packet delivery ratios in percentages with two SF durations) and [peak loads]. Traffic SG case 1 (SO3), (SO4) SG case 2 (SO3), (SO4) SG case 3 (SO3), (SO4) UL traffic (BST lost), 2 MH freq. (99.8), (99.8), [2.6 kb/s] (99.7), (99.7), [4.1 kb/s] (99.5), (99.2), [22.2 kb/s] DL traffic (BST lost), 2 MH freq. (99.2), (99.4), [0.9 kb/s] (99.1), (99.2), [0.9 kb/s] (99.3), (98.2), [0.9 kb/s] Multi-hop UL traffic (BST lost), 2 MH. freq. (99.6), (99.5), [1.7 kb/s] (99.6), (99.7), [2.3 kb/s] (99.0), (97.9), [12.5 kb/s] Multi-hop DL traffic (BST lost), 2 MH freq. (99.1), (99.3), [0.4 kb/s] (99.1), (99.2), [0.4 kb/s] (99.2), (95.8), [0.4 kb/s] UL traffic (BST lost), 1 MH freq. (99.7), (99.7), [2.6 kb/s] (99.6), (99.5), [4.1 kb/s] (85.8), (88.7), [22.2 kb/s] DL traffic (BST lost), 1 MH freg. (99.0), (99.0), [0.9 kb/s] (99.0), (98.7), [0.9 kb/s] (85.1), (83.6), [0.9 kb/s] Multi-hop UL traffic (BST lost), 1 MH freq. (99.6), (99.2), [1.7 kb/s] (99.5), (98.6), [2.3 kb/s] (43.2), (55.3), [12.5 kb/s] Multi-hop DL traffic (BST lost), 1 MH freq. (99.0), (99.0), [0.4 kb/s] (99.0), (98.2), [0.4 kb/s] (42.1), (36.7), [0.4 kb/s] UL traffic (BST connected) (99.9), (99.9), [2.6 kb/s] (99.9), (99.9), [4.1 kb/s] (99.8), (99.8), [22.2 kb/s] DL traffic (BST connected) (99.9), (99.9), [0.8 kb/s] (99.9), (99.9), [0.8 kb/s] (99.8), (99.8), [0.8 kb/s] 6.2 Delay Results of the SG Case 1 The Table 6-2 presents minimum, maximum and average values of the network delays in seconds when the traffic was generated according to SG case 1 (1 min UL, 5 min DL) as presented in the Table 5-2. When the both LTE BSTs were functioning correctly and the traffic was delivered directly through each CLH without MH routing, the lowest delays were produced (approximately 20 ms in UL and 10 ms in DL). For MH traffic (BST lost), applying the shorter SF duration (SO3) produced few tens of milliseconds lower delays than applying SO4. Each hop increased network delays with few tens of milliseconds. The maximum average delay value was relatively low, approximately 210 ms. For example, message delivering through 3 hops contains transmissions between RTU, CLH, CLH, CLH, CLH, BST, evolved packet core (EPC) and server.

20 Table 6-2: Minimum, maximum, and average values of the network delays in seconds. Number of hops in MH routing Multi-hop UL traffic (BST lost), 2 MH. freq. Multi-hop DL traffic (BST lost), 2 MH freq. Multi-hop UL traffic (BST lost), 1 MH freq. Multi-hop DL traffic (BST lost), 1 MH freq. UL traffic (BST connected), no hops DL traffic (BST connected), no hops 1 hop 2 hop 3 hop SO , 2.308, , 2.663, , 2.916, SO , 2.462, , 2.567, , 2.967, SO , 0.382, , 0.496, , 0.542, SO , 0.495, , 0.498, , 0.546, SO , 1.515, , 2.392, , 3.263, SO , 2.808, , 2.029, , 3.094, SO , 0.642, , 0.759, , 0.650, SO , 0.522, , 0.641, , 0.765, SO3 SO4 SO3 SO , 0.074, , 0.074, , 0.059, , 0.065, The average network delays of UL traffics in SG case 1 with one or two MH frequency channels and two SF durations (SO3, SO4) are presented in the Figure 6-2. The shorter SF duration (SO3) produced lower delays than applying SO4 when the traffic was MH routed. It could be assumed that a packet is queued less time between the SF periods (CAP, CFP (BST lost)) with SO3. On the other hand, BST connected traffic has a bit lower delays with SO4 than with SO3 that applied a larger part of the SF for routing control messages (CFP (initial)) when compared to the total SF length. Applying two distinct frequency channels in MH communication decreased the delays when compared to applying a single frequency channel. When the amount of the hops increased, the difference between the delays of SO3 and SO4 decreased.

21 - 21-0,25 Average network delay [s] 0,2 0,15 0,1 0,05 1 MH freg, SO4 2 MH freg, SO4 1 MH freg, SO3 0 2 MH freg, SO3 BST connected 1 multi-hop 2 multi-hop 3 multi-hop Figure 6-2: Average network delays of UL traffic (SG case 1) with one or two MH frequency channels and two SF durations (SO3, SO4). The average network delays of DL traffic in SG case 1 with one or two MH frequency channels and two SF durations (SO3, SO4) are presented in the Figure 6-3. The shorter SF duration (SO3) produced lower delays than applying SO4 for MH routed traffic. On the other hand, applying SO4 seemed to increase the delays less between the hops 1-3. Applying two distinct frequency channels in MH communication decreased the delays when compared to applying singe frequency channel for MH communication. The delay is a bit lower in DL direction than in UL direction because the amount of the traffic is lower and LTE network produces lower delays in DL direction.

22 - 22-0,25 Average network delay [s] 0,2 0,15 0,1 0,05 1 MH freg, SO4 2 MH freg, SO4 1 MH freg, SO3 0 2 MH freg, SO3 BST connected 1 multi-hop 2 multi-hop 3 multi-hop Figure 6-3: Average network delays of DL traffic (SG case 1) with one or two MH frequency channels and two SF durations (SO3, SO4). 6.3 Delay Results of the SG Case 2 The Table 6-3 presents minimum, maximum, and average values of the network delay in seconds when the traffic was generated according to SG case 2 (30 s UL, 5 min DL) as presented in the Table 5-2. When the both LTE BSTs were functioning correctly (BST connected), the lowest delays were produced (approximately 20 ms in UL and 10 ms in DL). For MH traffic (BST lost), applying the shorter SF duration (SO3) produced few tens of milliseconds lower delays than applying SO4. Each hop increased the network delays with few tens of milliseconds. The maximum average delay value was relatively low, approximately 230 ms. The delays were a few milliseconds higher in SG case 2 than in SG case 1 when 2 MH frequency channels were applied. When a single MH channel was applied the delay difference was few tens of milliseconds in most cases. Adding the amount of the traffic increased more the delay when all the MH traffic was delivered in a single channel.

23 Table 6-3: Minimum, maximum and average values of the network delays in seconds. Number of hops in MH routing Multi-hop UL traffic (BST lost), 2 MH. freq. Multi-hop DL traffic (BST lost), 2 MH freq. Multi-hop UL traffic (BST lost), 1 MH freq. Multi-hop DL traffic (BST lost), 1 MH freq. UL traffic (BST connected), no hops DL traffic (BST connected), no hops 1 hop 2 hop 3 hop SO , 2.061, , 2.700, , 2.764, SO , 2.325, , 2.431, , 3.053, SO , 0.381, , 0.498, , 0.617, SO , 0.501, , 0.601, , 0.571, SO , 2.893, , 2.744, , 3.357, SO , 2.482, , 2.415, , 3.076, SO , 0.841, , 0.907, , 1.131, SO , 0.692, , 0.747, , 0.963, SO3 SO4 SO3 SO , 0.086, , 0.077, , 0.082, , 0.063, The average network delays of SG case 2 UL traffics with one or two MH frequency channels and two SF durations (SO3, SO4) are presented in the Figure 6-4. The shorter SF duration (SO3) produced the lower delays than applying SO4 for MH traffic, except when the traffic was delivered to three-hop distance the delay was smaller with SO4 applying two MH channels than with SO3 applying a single MH channel. 0,25 Average network delay [s] 0,2 0,15 0,1 0,05 1 MH freg, SO4 2 MH freg, SO4 1 MH freg, SO3 0 2 MH freg, SO3 BST connected 1 multi-hop 2 multi-hop 3 multi-hop Figure 6-4: Average network delays of UL traffic (SG case 2) with one or two MH frequency channels and two SF durations (SO3, SO4).

24 The average network delays of DL traffic in SG case 2 with one or two MH frequency channels and two SF durations (SO3, SO4) are presented in the Figure 6-5. The shorter SF duration (SO3) produced lower delays than applying SO4, except when the MH traffic was delivered to three-hop distance the delay was approximately the same with SO3 and SO4 when a single MH channel was applied. Obviously, distance of three hops produced a delay that was smaller with SO4 applying two MH channels than with SO3 applying a single MH channel. 0,25 Average network delay [s] 0,2 0,15 0,1 0,05 1 MH freg, SO4 2 MH freg, SO4 1 MH freg, SO3 0 2 MH freg, SO3 BST connected 1 multi-hop 2 multi-hop 3 multi-hop Figure 6-5: Average network delays of DL traffic (SG case 2) with one or two MH frequency channels and two SF durations (SO3, SO4). 6.4 Delay Results of the SG Case 3 The Table 6-4 presents minimum, maximum, and average values of the network delay in seconds when the traffic was generated according to SG case 3 (4 s UL, 5 min DL) as presented in the Table 5-2. When the both LTE BSTs were functioning correctly (BST connected), the average delays were the same as with the previous traffic values (SG case 1 and 2). For MH traffic (BST lost), applying the shorter SF duration (SO3) produced higher delays than applying SO4. When a single MH frequency channel was applied, the delays were excessively high and the PDRs presented in the Table 6-1 were clearly below the QoS requirement of 99%. When 2 MH channels were applied, the delays were acceptable for SG purpose (maximum average delay was 1.23 s) but all PDRs were above the QoS requirement only with SO3 as presented in the Table 6-1.

25 Table 6-4: Minimum, maximum and average values of network delays in seconds. Number of hops in MH routing Multi-hop UL traffic (BST lost), 2 MH. freq. Multi-hop DL traffic (BST lost), 2 MH freq. Multi-hop UL traffic (BST lost), 1 MH freq. Multi-hop DL traffic (BST lost), 1 MH freq. UL traffic (BST connected), no hops DL traffic (BST connected), no hops 1 hop 2 hop 3 hop SO , 4.334, , 5.688, , 5.849, SO , 2.555, , 3.027, , 4.313, SO , 0.746, , 4.013, , 6.106, SO , 0.570, , 1.031, , 1.784, SO , , , , , , SO , , , , , , SO , 2.127, , , , , SO , 0.567, , , , , SO3 SO4 SO3 SO , 0.115, , 0.112, , 0.097, , 0.096, The average network delays of SG case 3 UL traffic with one or two MH frequency channels and two SF durations (SO3, SO4) are presented in the Figure 6-6. With the same set of MH channels (1 or 2), the shorter SF duration (SO3) produced the higher delays than applying SO4. Applying two MH frequency channels instead of one induced much lower network delays MH freg, SO3 1 MH freg, SO4 Average network delay [s] MH freg, SO3 2 MH freg, SO4 0,1 0,01 BST connected 1 multi-hop 2 multi-hop 3 multi-hop Figure 6-6: Average network delays of UL traffic (SG case 3) with one or two MH frequency channels and two SF durations (SO3, SO4).

26 The average network delays of DL traffic in SG case 3 with one or two MH frequency channels and two SF durations (SO3, SO4) are presented in the Figure 6-7. With the same set of MH channels, the shorter SF duration (SO3) produced higher delays than applying SO4 in most cases. It has to be mentioned that exclusively applying SO3 and 2 MH frequency channels produced PDRs above the QoS requirement of 99% in all the cases MH freg, SO3 Average network delay [s] ,1 0,01 1 MH freg, SO4 2 MH freg, SO3 2 MH freg, SO4 0,001 BST connected 1 multi-hop 2 multi-hop 3 multi-hop Figure 6-7: Average network delays of DL traffic (SG case 3) with one or two MH frequency channels and two SF durations (SO3, SO4). 7 Conclusions In this deliverable, the multihop routing method is developed for the previously evaluated hybrid sensor-lte network using Opnet simulator. To manage a situation that an LTE BST fails, clusterheads MH route the traffic to the nearest CLH that has a connection to a BST that operates correctly. The scenarios were addressed from a communications point of view, where the contents of the data payloads and their processing are out of scope for this work. The objects of interest were the behaviour of the MH routing and its ability to deliver the different amount of the traffic. The simulation scenarios addressed for SG traffic include three cases with distinct reporting frequencies of RTUs: SG cases 1-2 present the situations where RTU reporting period is decreased for the exception situation when a LTE BST is malfunctioning and SG case 3 presents automatic demand response (ADR) with low-frequency control (tarif updates). All the cases were