Network Management of Critical Wireless Systems

Transcription

1 Network Management of Critical Wireless Systems Brian Cunningham Applications Engineer Eaton/Cooper Bussmann, Wireless Business Unit* Port Coquitlam, BC V3C 6G Keywords radio, mesh, wireless, network, management, systems, monitoring, spread spectrum, telemetry, frequency hopping, transmitter, receiver, transceiver Abstract With increasing reliance on wireless networks for process control, the question has arisen of how can these be monitored and maintenance proactively scheduled. The benefits of wireless communications have been proven through hundreds of thousands of applications in water/wastewater, oil and gas, and manufacturing, to name a few. Traditionally, once the network was installed, it was forgotten until a problem occurred. Wireless manufacturers recommendations of regularly scheduled maintenance traditionally involved significant labor and hours of driving only to discover no faults, or trends too small to be indicative. As IT and other technical professionals have been using software tools to monitor copper networks for years, now software is available to do this automatically and conveniently for wireless networks. This paper will discuss the importance of network management on critical wireless systems that are relied on by large industrial facilities to supply drinking water to families, electricity to households and fuels for heating and transportation Reorder Page 1

2 1.0 Wireless Systems for Industrial Applications Before a discussion of network management can begin, the reasons why the network is there in the first place need to be covered. As in the title, this paper will focus on wireless systems for specific reasons. Wireless systems are now in use by industrial users for process control and monitoring. Ranging from Ethernet links for video surveillance and PLC connections, to WirelessHART for sending 4-20mA signals, to controlling pumps and monitoring flows, wireless connections offer significant savings in many applications. The savings over wired connections have outweighed the reliability and security concerns, in the many business cases that are presented to management by engineering departments. Particularly for long distance applications, such as controlling water pumping stations, the only hardwired alternatives are leased phone lines. Industrial facilities using 4-20mA signals often use buried conduit to transport signals digging a trench when soil remediation is required such as in older refineries has become so expensive that every alternative is explored first. If the distance exceeds ½ mile, wireless is often the only cost effective approach. Rotating or moving machinery is a natural for wireless applications. Wireless connectivity can be deployed on a variety of frequency spectrums. Fixed frequency 5 watt radios can transmit 80+km and are common in the oil patch for monitoring and controlling pump jacks. Spread spectrum radios allow shorter links, but without the regulatory hassles, making them ideal for in-plant applications. Cellular modems offer the unique capability of access to a site that has cellular coverage, from anywhere in the world with an internet connection. Wireless connectivity options will vary depending on the application, the terrain, the distance and the regulatory requirements of the jurisdiction. 2.0 Traditional Wireless Network Maintenance/Monitoring A short discussion of how things were done in the past will explain the pain and expense that motivated the quest for a better way of doing things. Routine maintenance of wireless systems involved cumbersome site visits. With fixed frequency radios up to 80km away, this involves long distance driving and many unproductive hours. Some networks are as large as 2000 radios using dozens of repeaters. Often the roads connecting sites do not take the most direct path meaning to drive to a site with an 80km radio link can involve 100+km of mileage on the vehicle. Multiply this by 2000 sites and you have a very motivated organization looking for efficiencies. Once on site, the most common parameter checked is the Received Signal Strength Indicator (RSSI). By measuring this on a regular basis, a trend can be obtained which will indicate a point in time of failure. Foliage growth, antenna and co-axial cable corrosion, insufficiently waterproofed connectors, electronic component degradation and 0813 Reorder Page 2

3 drift can all lead to reduced RF performance. If this continues, intermittent operation will be followed by longer and longer outages. Figure 1 Technician at Remote SCADA Site One issue with snapshot measurements of the RSSI taken once or twice a year, is that signals can bounce around a fair bit making a trend hard to spot. This is particularly true with frequency hopping spread spectrum radios; as the frequency changes, multipath can occur reducing the RSSI, which may recover as the radio transmits on a new frequency. Moving machinery can also cause this to occur, as reflective paths come and go. The author has also witnessed this in high wind conditions blowing a tree branch in and out of the radio path. Background noise level is another important parameter that can change over time due to increased proliferation of wireless devices. A rule of thumb is the RSSI should be 10dB stronger than the background noise level. Therefore if the background noise level rises too high, the receiver will be unable to lock onto the lead-in tone of the transmitting radio. Again, this type of signal degradation can occur over a long period of time. Other maintenance functions on a wireless system include simple visual inspection of the hardware. Any mechanical damage caused by storms or infiltration of water into electrical cabinets or antenna connectors should also be corrected. This type of maintenance can be done by non-technical personnel and does not require a laptop with specialized software. Another parameter that should be monitored, that would enable a technician to bring the right tools to the right site, is traffic out the serial or Ethernet port of the radio. In order for the radio to be functioning correctly, the attached devices need to be sending polling requests/replies into the radio. If communications port activity can be monitored independent of the RF port activity, the nature of a failure can be determined with far less troubleshooting Reorder Page 3

4 3.0 Mesh Network Complications Mesh networks with their capabilities to change RF links at any time depending on the network conditions add additional complications. Are a large number of remote radios relying on one or two repeaters? How much traffic is going through those repeaters? Is the traffic reaching the maximum throughput of those repeaters? A site visit to monitor this only gives a snapshot in time how did it perform last week or last month? As these questions illustrate, network traffic is an important parameter to monitor. Consider the scenario where 2 repeaters are available, but one has a stronger signal, and therefore is the chosen one with the other as a backup. As network traffic increases, it may exceed the bandwidth of the chosen repeater. You may wish to force some remote sites to go through the weaker radio path to even out the traffic or even consider a 3 rd dedicated repeater. This is a common scenario when deploying a network of 100+ radios. When initially installed, typically a few remote sites are commissioned every week. Every addition can impact the performance of the network and therefore continuous network performance monitoring of the entire system is an essential task to keep the network stable. Figure 2 Network Topology Showing Repeater Paths If 3 or 4 repeater paths are available, which one is being used right now? Critical information for troubleshooting involves finding out which radio has a problem to narrow down the source. Otherwise driving to remote sites is required to log into each radio. If presented with a network topology map, showing which nodes are up and which are down, the network can be easily visualized and performance understood Reorder Page 4

5 4.0 Network Management Using Software the Requirements A dedicated computer with access to the internet would be a basic requirement. The computer must be running 24 hours in order to monitor the network and sent out /text notifications if alarms occur. Remote access needs to be available so an operator can log in from anywhere and decide if immediate action needs to be taken. Ideally, multiple operators should be able to log in simultaneously, with different levels of authority some for monitoring only, others with full control access, able to change wireless settings on remote radios. This of course requires that an effective security system be in place to prevent intruders from accessing the system. A Network Management System (NMS) will periodically request status updates from the wireless network. This information flow can have some impact on the bandwidth available for regular SCADA traffic. Therefore the monitoring data needs to be minimized and prioritized so not to interfere with the payload data. Allowing time based reporting, as well as on demand to spot check a potential problem, would be required to make the NMS compatible and complimentary to the wireless network. Additionally, a large network may have a mix of wireless and wired equipment installed and may be from different manufacturers. An agnostic NMS will take all of this into account Visualization is a critical element of an NMS to provide immediate and accurate information to the user at any point in time so that device or link issues can be located and action can be taken. Detailed information like RSSI, SNR, Tx/Rx throughput, channel utilization etc. are critical parameters for analyzing problems without traveling to a remote site. Especially early warning thresholds can inform you of potential issues ahead of time and issues can be resolved before any serious degradation of the network can occur. The number of faults or alarms - if recorded can sometimes be traced to external environmental factors such as extreme weather, or passing rail cars both can obstruct wireless signals. Statistics of uptime and com status can allow contractors to sign off on a new installation Reorder Page 5

6 Figure 3 Statistics of Wireless Hosts Figure 4 Graphical Analysis of Host Performance 0813 Reorder Page 6

7 Can you remotely change wireless settings to modify performance? Can you view the wireless nodes on a map to get an idea of distances and clusters? Perhaps you need to turn down the transmit power of one radio to minimize interference to a neighboring system, or add a fixed routing path to a mesh network? If you can see the network on a real world map like Google Earth, visualization of the issues becomes much simpler. If you can change a parameter from the control room, then observe the results, hours/days of labor and mileage can be saved. Figure 5 NMS Showing Wireless Locations on a Google Map The performance of the network or of individual devices can easily be documented using an integrated graph generating and reporting tool that allows the user to print or export such data for external use Reorder Page 7

8 Figure 6 NMS showing all State Transitions Figure 7 Availability Summary of a Remote Site 0813 Reorder Page 8

9 5.0 Network Management Using Software the Business Case If all industrial wireless users were to employ network management software, the author estimates technical support inquiries would drop by 20%. By trending the supply voltage of a solar powered remote site, a technician could be dispatched long before a remote site failed. In the solar world, these failures commonly occur in the middle of the night, typically in the dead of winter. Remote monitoring could ensure a technician is only sent when needed, for common minor maintenance such as cleaning the solar panels or replacing the battery. Consider the following case study: A large biotech company that requires monitoring of fridge, freezer and incubator temperatures. These chambers contain research that leads to cutting edge new drugs helping cure cancer, improve recovery times and treat a host of other illnesses. Often the samples in these chambers are worth hundreds of thousands of dollars. Now consider this the chambers are on castor wheels and frequently moved from one room to another, and sometimes from one building to another. The company has over 70 buildings on this particular campus. There are over 2000 chambers on campus. It is an ideal application for a mesh network, since the repeater paths will need to change. It is also an ideal application for NMS, since the network layout, data traffic levels, RSSI, etc. may all vary with the chambers movements. In this installation, remote monitoring is essential to keep track of the physical location of the chambers. After commissioning, the NMS determined that traffic levels at some key points were too high, and on rare occasion, an entire building would lose communications. This led to the installation of fixed repeaters. These repeaters, left with the choice of many dozens of paths, sometimes chose paths with higher traffic levels on chambers that were subject to frequent movement. This led to fixed routing paths being created to alleviate congestion. By trending the RSSI, facilities staff could see if a chamber was at risk of losing communications. An abrupt drop in signal strength often meant the magnetic mount antenna had been knocked over onto its side. A slow decline would be indicative of an antenna connector or cable issue. Overall, alarm levels were set to alert facilities staff well in advance of failure. When a failure or temperature alarm occurred, protocol dictated the researcher was notified and if necessary, required to drive on site to transfer the samples to another chamber. By providing early warning notifications, the user was able to make an informed decision about the severity of the problem and often avoided an unnecessary trip to the facility. It was essential to provide immediate status updates of network link failures, which if not repaired immediately, could result in the loss of R&D samples and the value that research represented Reorder Page 9

10 6.0 Conclusion Wireless communications can offer significant cost savings over hardwiring, and in some cases such as moving machinery, are the only choice. Historically, maintenance of these systems involved site visits to gather a snapshot of today s performance. Mileage and labor costs add up to make the decision to do preventative maintenance a business decision weighed against the odds of failure and the loss of productivity or material. Mesh networks further complicate manual network monitoring as the repeater paths may change at any moment. A network management system needs to be capable of alarming, trending, visualization as well as the ability to remotely change wireless parameters upon discovery of an anomaly. The rewards of utilizing an NMS system include higher reliability, greater product quality, increased throughput, workforce mobility, as well as more measurements at a lower cost. Technical Support: United States: Australia: ELPRO-Support@cooperindustries.com ELPRO-US-Support@cooperindustries.com 0813 Reorder Page 10

11 Note: Eaton acquired Cooper Industries plc in November Reorder Page 11