High Performance, Secure VPN Servers for Remote Utility, Industrial Automation Systems: Water Pumping Station Security Case Study Industrial Network Security: New Threats The convergence of IT and industrial automation networks has created great opportunities, but with this comes increased security threats from hackers, worms, and viruses. Clearly, remote utility network administrators must rethink their approach to network security. Ethernet networks have proliferated across many of our workplaces today; that includes utilities such as pumping stations, electrical substations, and oil pumping wells. Initial implementation of Ethernet networks at pumping stations disregarded security measures since most of these networks did not have external network access (i.e. connection to the public internet). However, this safety is in fact illusory. However safe that it may seem, it turns out to be just the opposite. Studies have now shown that most attacks (83%*) occur from within the intranet, and not from external internet connections. Further, PLCs and RTUs distributed within the network are not designed to support traditional firewall and anti-virus software protection such as would be used in an IT network. It can easily be the case that employees or equipment vendors using their company laptops outside the workplace network contract various worms, viruses and other malicious malware threats. Those same laptops will be re-connected to the corporate network and propagate those threats, without even needing to encounter and breach network firewalls. Similar types of attack include thumb drives, malicious emails, or other peripherals (smartphones, tablets, etc.) that are physically connected to the local LAN. In a recent high-profile example, in 2010 a particular SCADA system used worldwide was targeted by a specially developed SCADA worm known as Stuxnet. The worm was able to subvert windows-based automation systems, and particularly the associated PLCs that it was designed to attack. Incidents like this highlight the huge importance of security, which has now suddenly become a critical necessity for industrial automation networks. * Network Security: Managing the Risk and Opportunity, AT&T Survey and White Paper (2007) Security for Remote Access Even though allowing remote access to industrial networks introduces vulnerability, it would not be feasible to simply shut down or cut off these networks. Remote utilities dispersed over wide geographic areas, such as pumping stations, are usually numerous and for cost considerations must be managed from central locations. To do otherwise is simply not feasible, so new security measures must be implemented. Administrators can protect against some of the security vulnerabilities by implementing the following: VPNs: Virtual Private Networks that allow secure remote access to a network over internal and external networks including the internet.
Firewalls: To isolate the automation network from the business network and ultimately external networks. LAN security: To prevent unauthorized access to the network and nodes in the first place. Pumping Station Network Overview Throughout the world there are countless pumping stations that handle water movement, generally from one reservoir to another. Pumping stations include wells that extract freshwater drinking supplies from the ground, sewage lift stations that move collected wastewater to sewage treatment plants, and extensive land drainage systems that maintain reclaimed land that is below sea level. Pumping stations are usually a complex collection of distributed devices that can include sterilization equipment, ground and elevated storage tanks, well and booster pumps. Most of these systems are vital within any populated area, thus cyber-terrorists targeting such operations are an obvious concern that must be addressed. Protection of the data acquisition and control systems therefore cannot be overlooked, as attacking these resources could easily cripple a community. For example, pumping stations have traditionally used various SCADA control protocols intended for private network use. Adopting the use of Ethernet networks to be able to remotely monitor and control stations leaves those same SCADA protocols vulnerable to attack. This is simply because there is a complete lack of authentication and encryption capability in private network SCADA systems, leaving them very insecure. Figure 1 illustrates traditional water pumping station network. Without proper security, the Local Control Units (LCUs) in the local pumping control system is vulnerable to attack. Figure 1: A traditional water pumping station network, without security
Security Challenges in Automated Pumping Stations Remote Access: With the wide-geographical placements of pumping stations comes the need for remote access. The approach to remote access must be both secure and economically feasible. When using Ethernet systems, particularly when utilizing existing intranet/internet networks, data transmission must be highly encrypted to prevent malicious attackers from intercepting packets transmitted. Hackers can use those packets to interpret the network topology and command structure to eventually control the system, so preventing access to the transmission is critical. VPNs can be implemented bi-directionally between the pumping stations field sites and the control centre. VPNs utilised must support encryption standards that cannot easily be hacked, encryption such as triple DES (Data Encryption Standard) and AES (Advanced Encryption Standard) with large key sizes that can generally only be broken using brute force. Although there are published attack methods for these encryption systems, they involve extreme methods that require a huge resource, and can therefore be considered beyond practical feasibility. Video Surveillance: Typically, industrial automated networks using Ethernet are sensitive to delay issues and because of this the security measures that are implemented into the network cannot introduce performance diminishing delay into the system. Functions such as VPN or firewall services must provide the minimal transition delay when inspecting packets or encrypting and encapsulating packets for VPN transfer. Therefore any system utilized must provide enough processing power to adequately perform security functions without any substantial loss in the network performance. Otherwise the system selected may be so underengineered as to disrupt the normal application requirements. Video surveillance requires that the network delay is kept to a minimum. Video packets are usually streamed using UDP so the delivery needs to be unaffected by security measures and the packet processing incurred by it. Video surveillance data needs to be transmitted securely so VPNs need to be employed. Using a device with software encryption cannot meet the encryption demands required by a high bandwidth video stream. Therefore it is essential that hardware encryption be employed to ensure that delay sensitive transmission of video is sent smoothly over secure VPN tunnels to centrally located CCTV recording equipment. In order to securely support high bandwidth required for video, it becomes relatively clear that a separate stand-alone solution, i.e. a stand-alone device, is required. Utilizing existing network infrastructure may not have adequate processing capability to handle the additional security functions. Also, being able to maintain the deterministic system behaviour is essential when any security device is added to the network. Moreover, the device introduced must not prevent critical access or stop any mission-critical packets, inadvertently resulting in system failure. In some circumstances that failure could be catastrophic. WAN Redundancy: Critical resources such as pumping stations that are being controlled and monitored remotely needs highly reliable connectivity. However, it could be risky to design a solution without backup or redundant network connectivity over what is known in general terms as the Wide Area Network or WAN (a network linking broad geographical areas). In order to support that redundancy any device that acts as the control and monitoring gateway to critical
remote pumping stations needs to support dual connectivity. Having two WAN links reduces to a minimum the likelihood that network connectivity is lost between the control centre LAN and the pumping station LAN. Operations in Harsh Environments: Pumping stations are normally unmanned locations that do not provide controlled environmental housing for the control and network equipment. Therefore it is necessary that the security hardware installed is robust enough to withstand large temperature and humidity fluctuations without performance degradation or failure. The hardware needs to be hardened to avoid the expense of engineers being dispatched or even more serious damage being caused by the pumping station failing. Figure 2: A water pumping station network, with security components in green. IPSec VPN Server and Client for Remote Access: When a system has multiple geographically sites, such as dispersed remote pumping stations, operators need to be able to remotely access the pumping stations for both monitoring and control purposes. Today, remote access often means using the public internet to gain access from the control room. The gateway that acts as a firewall and authenticator to the network must support Virtual Private Networks or VPN tunnels that act as virtual encrypted pipes to ferry control and monitor IP packets securely back and forth between the pumping station and control centres. Having remote access not only saves travel time and costs but it can reduce system downtime. Although there are multiple VPN technologies, IPSec is the secure VPN protocol predominantly deployed and would need to be supported by the pumping station gateway to support the multiple VPN clients that an operator may choose. IPSec essentially sets up a secure channel
over (possibly multiple) networks, which can be either: private, public or a combination of the two. It provides authentication, confidentiality of the party requesting the VPN tunnel and integrity in packet transfer; this is so that the payload transferred (control and monitoring data) is protected using strong encryption methods. Figure 3: VPN Solutions maintain security and provide remote access. LAN Security, Port Access, 802.1x: The first line of defence for any network or intelligent device is to prevent unauthorized access to the system. Because of their remote nature, pumping station networks are particularly prone to unauthorized access. Monitoring of direct equipment access is not always feasible and can be susceptible to attack over the public internet used for VPN access. Certain protocols such as RADIUS and TACACS+ provide credential authentication mechanisms that can make it difficult for attackers to gain direct network or device access by using the public internet to try and probe the system. With RADIUS the transmission of the user password is encrypted and with TACACS+ all the key authentication parameters are also encrypted. The network devices deployed should support further authentication measures to prevent a user from easy connection, for example, a laptop s NIC directly to an open Ethernet port of the installed network equipment. 802.1x is a port-based authentication method used to validate devices that try to gain access to the protected network. The devices must provide authentication credentials such as username and password or a security certificate to gain access with which 802.1x can then forward the credentials to a RADIUS server for validation. If unsuccessful where an attacker is unable to provide valid credentials then the attempted access to the open ports is stopped by blocking packet transfer to and from the port. Firewall between PLC/RTU Controller and External Traffic: The PLC and RTUs deployed to control pumping stations are highly susceptible to attack by various methods since these devices have never had the capability to support firewall and virus prevention software. Therefore, should a user gain access, attacking these devices and breaching the pumping station operations is relatively simple. The nature of PLC and RTU design prevents them from supporting overly complex software so that they are extremely reliable at the task for which they are intended. However, that leaves them vulnerable to external attack where a hacker can utilize simple techniques such as sending malformed packets, creating insecure HTTP and SMNP services that cannot be closed down, or sending valid commands such as, a firmware upgrade command that should not be sent.
A network planner needs to include a robust inspection firewall between the network s control devices and the external connectivity. A firewall inspects or eavesdrops all incoming and outgoing packets and based on its preconfigured rules of allowable and disallowable packet content, it either passes or drops packets. The firewall further needs to be able to guard against malicious attacks without mitigating the network performance. To obtain that level of performance a network planner needs to include network access devices that sit at the edge of the network and have a hardware/software combination that can provide the necessary gateway performance to protect the network with minimal latency. Since automation networks commonly employ various Fieldbus protocols the firewall chosen needs to be able to restrict communications to only the associated port. Having a firewall with industrial Fieldbus settings means an automation engineer can easily implement the restriction without the need for a complex procedure. Figure 4: Firewall policies inspect traffic to maintain security Use DMZs for Public or Shared Servers: DMZ, or demilitarized zone, is often employed in IT solutions but also serves as a strong defence against attack in automation networks. For maintenance or remote monitoring, some of the data servers or HTTP servers will need to be accessed from public networks or the internet by multiple operators. To maintain security, we should isolate these shared servers and control/scada servers into different networks. This way, general users can only access the shared servers, and are not given access to the control network. Industrial-grade Devices: As mentioned earlier a security device targeted for a pumping station needs to be hardened since usually unmanned pumping stations do not provide
environmental control beyond a secure enclosure. Therefore the hardware needs to be designed to accommodate operation in wide temperature ranges. If a cheaper IT enterprise unit is selected, its likelihood of failing becomes very high since these devices are only designed for narrow indoor controlled temperature ranges. Failure of such a device is more than just the cost of lost man hours required in replacement, this could mean pumping station failure which may present far greater costs. Any security device deployed would require a relatively robust housing targeted for the harsh conditions that a pumping station may encounter. The components need to be contained in a metal enclosure that will not suffer from temperature issues or unexpected stresses from mechanical impact. Along with a durable and strong case the device should also support dual power input to give the operator an option of providing a second emergency power solution during primary power failures. Conformal Coating: In line with operating temperature ranges the device selected also requires protection from humidity. Constant changes in exterior humidity conditions can easily cause condensation and possibly damage the hardware resulting in operational failure. It is imperative that the device electronics are protected using modern conformal coating methods. The thin plastic film applied protects the hardware from contaminants and further acts to prevent corrosion in harsh environments. With the Right Tools, Remote Access and Security Can Go Together Utilizing an access device with IPSec VPN server mode means that engineers who need access to the pumping station devices can securely tunnel from multiple remote locations. Without a secure gateway installed access from remote locations over the public internet can easily be hacked using simple methods. Multiple video-surveillance cameras at each pumping station necessitate selecting a security gateway with hardware encryption to provide enough IPSec tunnel performance that will maintain smooth and secure video streams without affecting transmission of critical control and monitoring protocol packets. Any gateway s firewall needs to support configurable inspection of ingress packets to the pumping station network to provide a line of defence against not only external network attacks but by internally connected company devices infected from outside sources. Also, access to the gateway and other devices throughout the network should support modern (RADIUS or TACACS+) secure user authentication for remote attack attempts. For local physical access where a non-authorized person attempts to directly plug-in to the network, 802.1x port security should be employed. Finally due to the remote locations, a pumping station gateway needs to be durable for the harsh environment it may face and have redundant systems in case the power and networks it relies on fails. Durable means not only designed for wide temperature ranges but also rugged device design that includes rigid metal encasing with IP protection and special conformal coatings. Redundancy means the device needs both secondary power and WAN capabilities to maintain service when primary systems fail.