Experience with Unidirectional Security Gateways Protecting Industrial Control Systems Lior Frenkel 1, Danny Berko 1, Andrew Ginter 2 1Waterfall Security Solutions Ltd., Tel-Aviv, Israel 2Waterfall Security Solutions Ltd., Calgary, Canada andrew. ginter @ waterfall-security.com Presented at CRITIS 2012 7 th International Conference on Critical Information Infrastructures Security Abstract. A number of misconceptions exist in both common usage and in the literature, regarding the capabilities of unidirectional security gateways or "data diodes." The gateways provide strong protections for the safety and reliability of control systems, protections against attacks originating on external networks. However, the technology is widely perceived to suffer from data integrity concerns, perceived to be incompatible with applications and infrastructures using bi-directional communications protocols, and perceived as incompatible with remote access and central management requirements. Experience deploying this equipment at hundreds of sites proves all of these perceptions to be incorrect. This article explains how server replication is used routinely to replace firewalls with unidirectional communications solutions in conventional systems integration scenarios, describes the three most common central/remote administration paradigms in use in industry, and debunks data integrity myths. Keywords: Cyber-security, critical infrastructure, industrial control systems, SCADA security, unidirectional security gateways, data diodes, hardware-enforced unidirectional communications, server replication, data integrity, central management, remote control, one-way communications. 1 Introduction A majority of industrial control systems (ICSs), colloquially known as SCADA systems, in many industrial sectors are connected either directly or indirectly to business networks. Business applications essential to profitability use live control system data to track raw materials and finished goods inventories, to plan maintenance, to order replacement parts, to project future needs and performance, and to analyze and optimize the performance of both physical processes and the business. adfa, p. 1, 2012. Springer-Verlag Berlin Heidelberg 2012
However, these connections between business and control system networks present security risks. These risks include disgruntled employees, contractors and other insiders, conventional viruses, and worms, as well as recent nationstate attacks such as Advanced Persistent Threats carrying out industrial espionage through the use of manual remote control tools and autonomous sabotage-focused attacks, including the Stuxnet worm. In addition, even simple errors by business network administrators or misconfigurations of business networks can threaten the correct operation of control systems and control networks. Conventional protections for control system networks include firewalls, patching, and anti-virus systems. Increasingly, unidirectional security gateways are being deployed to replace one or more layers of firewalls between control system components and business networks. The unidirectional hardware permits business-critical information to be shared from control system networks to business networks, users and applications, without introducing any risk to the safety or reliability of control system networks. The hardware permits information to flow out of control system networks into business networks, without permitting any communication whatsoever back into control system networks. A variety of hardware configurations can be used for this purpose. The most common currently deployed to protect industrial networks uses at some point in the hardware architecture a laser, a receiver and fibre-optic cable. Protected Operations Network External Business Network Historian Server TX Agent Hardware-Enforced One-Way Communications RX Agent Replica Server TX Gateway RX Gateway Fig. 1. Historian Server Replication with Unidirectional Gateways For example, a common hardware configuration is the Waterfall Security Solutions [1] configuration in Figure (1). A Waterfall Unidirectional Gateway consists of two network appliances, a TX gateway appliance in the protected control system network, and an RX gateway appliance in the business network. The TX appliance includes a conventional copper network connection and a laser, and the RX appliance includes a conventional network connection and a photocell. A fiber-optic cable connects the two appliances. The TX appliance is able to send information to the RX appliance, but the RX appliance chipset contains no laser which could send a message back to the TX appliance. Similarly, the TX appliance chipset contains no photocell, and so the TX appliance could not receive a signal from the optical cable, even if one were somehow sent from the RX appliance.
Hardware-enforced unidirectional gateways, no matter the vendor, provide strong protections for the safety and reliability of control systems, protections against attacks originating on external networks. However, the technology is widely perceived to suffer from data integrity concerns, perceived to be incompatible with applications and infrastructures using bi-directional communications protocols, and perceived as incompatible with remote access and central management requirements. Experience gained by deploying this equipment at hundreds of sites proves all of these perceptions to be incorrect. This article explains how server replication is used routinely to replace firewalls with unidirectional communications solutions in conventional control systems integration scenarios, describes the three most common central/remote administration paradigms in use in industry, and debunks data integrity myths. 2 Server Replication Literature describing the application of unidirectional communications components to industrial control systems describes basic TCP and UDP communications, and mentions SMTP, file transfer and proposed SCADA-targeted offerings as well [2], [3]. In contrast, in our experience, the most common deployment mode for unidirectional gateways is to replicate servers. The data which is to be shared with business applications generally resides in either one or more servers, or in one or more devices, on the control system network. The unidirectional solution queries the servers on the protected network for the data to be shared, pushes that data out to the business network over the unidirectional medium, and populates a replica server on the business network with that data. The unidirectional gateways keep the business replicas synchronized with the control system servers in real time. 2.1 Historian Replication Consider the example in Figure (1), namely process historian server replication [1]. After an initial offline synchronization effort, where the database of the historian server is copied to the replica server, the unidirectional solution begins real-time synchronization. Transmit (TX) Agent software on the control system network queries the process historian, asking for all data since the manual synchronization, and all new data, as that data arrives in the historian. These are standard queries supported by all commercial historian products. On the business network, Receive (RX) Agent software populates the replica historian. The RX Agent registers with the replica as a standard device data aggregator 1, reporting to the replica that data received via the unidirectional medium was recently 1 Device data aggregators accumulate device data directly from devices or from other aggregators. Various commercial solutions may identify the aggregator function as interface node or collector functions.
reported from original source devices, just as the original aggregators would have reported this same data to the production historian. Business users and business applications access the replica server(s). In most cases, very faithful replicas can be maintained. Faithful replicas often have the same IP addresses as the original servers, and are identical to those servers in almost every way. Business users often think they are still connected to the original servers. As a result, seamless replacement of firewalls with unidirectional gateways is possible, without the commonly-expected compatibility problems due to widespread use of two-way communications protocols. The unidirectional solution uses two-way protocols to gather data on the source network from the original servers, and to publish data on the business network to replica servers. What passes on the unidirectional medium is often a proprietary, unidirectional protocol, the exact nature of which is irrelevant to users of replica servers. 2.2 Industrial Protocols This same approach can be used to publish Modbus, DNP3 and other data to the business network, data which at first glance appears to be accessible only via query/response type two-way protocols. Take for example the OPC-DA protocol. The protocol is complex and intensely bi-directional, layered on top of DCOM, which rides on DCE, which most commonly uses some form of IP deep in the protocol stack. The unidirectional gateways do not somehow emulate the OPC protocol across a one-way medium. Instead, just as in the historian replication scenario, the gateways replicate OPC servers [1]. OPC is an open specification, and so anyone can write an OPC client, and anyone can write an OPC server. The TX Agent in Figure (2) is a true OPC client, and that client is configured to use the true OPC protocol to query production OPC servers for the data which is to be shared with business users and applications. The TX Agent sends that data across the unidirectional medium, using a proprietary one-way protocol, to the RX agent. The RX Agent is a true OPC server. That server holds the received data until an OPC client on the business network requests the data. Again, OPC clients on the business network interact exclusively with the OPC-DA server replica. This same approach can be applied to emulate Modbus slave devices and DNP3 slave devices, devices which in TCP terminology act as TCP servers. Control Network Business Network OPC TX Agent OPC Client OPC Data Unidirectional OPC Data RX Agent OPC Server OPC OPC Servers OPC Clients
Fig. 2. OPC-DA Server Replication with Unidirectional Gateways 3 Data Integrity While the literature accurately addresses unidirectional communications data integrity concerns, the question of data integrity continues to arise in almost all customer engagements. Data integrity issues are addressed by all vendors of unidirectional communications components, generally at many levels. A sampling of data integrity protection mechanisms includes: High availability: most practitioners familiar with bi-directional communications know that sequence numbers, acknowledgements, timeouts and requests for retransmission can be used to recovery automatically from certain transient hardware failures communications media bit errors, and even temporary disconnections of such media. These mechanisms cannot be used to recover from other kinds of errors, such as permanent failures of transmitting or receiving computers. Many unidirectional communications vendors support true high-availability configurations, able to tolerate any single point of failure, including transient or permanent failures of one or more communications media. Throughput tuning: Since unidirectional receivers are incapable of sending flow control signals to the sending side, it is essential that any such installation be tuned to prevent data loss due to the receiver s inability to process data as fast as the data is sent. Solutions include sizing receive servers and CPUs appropriately, providing hardware and software buffering in the receive server(s) sufficient to handle data bursts without loss of data, and having the sender limit the speed of transmissions, or of the unidirectional medium, to values which installation-specific tuning has shown are tolerable in the receiver. Communications technologies: The above mechanisms can be augmented by a variety of additional measures and techniques, depending on installation requirements and vendor capabilities. For example, in the vast majority of industrial deployments, the transmitting and receiving hardware are in the same room. As a result, high-quality communications components can reduce the bit error rate to practically zero. Some vendors transmit information across the unidirectional medium using error-correcting codes, which can tolerate a certain number of transient bit errors, rather than use only errordetecting codes. Some transmitters can be configured to transmit information multiply, in different orders, and with different delays between transmission, again to reduce the likelihood of data loss, but with the obvious impacts on throughput and channel utilization. In practice, the end users of unidirectional communications technologies which this paper s authors interact with, report that by far the single most common cause of persistent data is scheduled downtime of one or more components in
the communications solution. As a rule, recovering from data gaps introduced by such outages is possible, but is application-specific. For example, in the historian server replication example, one simply dumps the source historian s database for the affected time period and re-loads it on the replica s server. This can be done manually, using technologies provided by the historian vendor, or can sometimes be done automatically, using technologies provided by the unidirectional communications vendor. The result of combining all of these factors is that, depending on which of these mechanisms is available to address data integrity risks, unidirectional communications can be as reliable, or more reliable, than bi-directional data communications deployments. 4 Remote Control and Central Management Many factors drive the requirement for remote control and central management of ICS computers, networks and devices. For example, control system vendor personnel may need either routine or emergency remote access to control system equipment, and both aging workforce issues and cost-control imperatives are driving a trend towards centralized corporate monitoring and management of control systems and networking components. Practitioners often assume that any remote or central management is impossible, because of course unidirectional communications equipment is specifically designed to frustrate such activities. In practice though, there are four common paradigms for remote management and remote access, and each is described below. 4.1 Remote Monitoring The most straightforward of the four scenarios to address is the need for remote monitoring. Corporate network operations centers (NOCs) and security operations centers (SOCs) monitor networks and hosts with a variety of software tools. UDP-based communications such as syslog and SNMP traps can simply be forwarded through unidirectional gateways from protected networks into SOCs and NOCs on business networks. More complex protocols, such as the SNMP query/response protocol can be dealt with using server replication. The unidirectional solution queries protected network components for all of the data which NOC and SOC solutions require, and then emulates the SNMP servers (devices) to the NOC and SOC software using the bi-directional SNMP query/response protocol. At a deeper level, some unidirectional vendors have established partnerships with a variety of NOC and SOC vendors [7]. These partnerships result in software which replicates data aggregation servers, such as McAfee NitroView Security Information and Event Management (SIEM) servers from protected networks to business networks. These replications permit the NOC/SOC servers to gather data directly from monitored equipment on protected control system networks, and then replicate those servers to the business network so that the enterprise
versions of these servers can query the replicas and aggregate the data in those servers into a single database and a single view for enterprise NOCs and SOCs. Similar approaches can be used for control system and equipment vendors. For example, turbine vendors often require that their monitoring centers have continuous access to equipment under support contracts, since regular monitoring, early intervention and comparatively small adjustments can often prevent costly, catastrophic failures which require the replacement of entire turbines. Where unidirectional vendors and turbine vendors support such solutions, replication of essential data sources permits turbine vendor support centers to continue to monitor turbine activity without interruption, as if they were monitoring the original, protected servers and devices [6]. 4.2 Remote Screen View While the ability to carry out conventional NOC, SOC and equipment monitoring via server replication through unidirectional gateways is perhaps not so surprising in hindsight, what about remote control. In the turbine vendor example, what happens when the vendor detects a temperature or vibration anomaly and needs to adjust the equipment to prevent eventual catastrophic failure? The answer depends on the unidirectional communications vendor s capabilities. One such capability is Remote Screen View [6]. Remote Screen View relies on software which captures and transmits screen images over the unidirectional medium, in a manner analogous to remote desktop, VNC or even video monitoring tools. The screen images are made available to business network users via a server of some sort, for example a password-protected web server. Remote administrators can access the screen image / video feeds to see what is occurring on monitored equipment on the protected control system network, but of course cannot directly influence the monitored equipment in any way. Instead, they communicate with personnel with access to the protected equipment, usually by telephone. In the turbine management scenario for instance, the vendor s monitoring applications may alert the vendor s personnel to a condition requiring adjustment. The vendor s support personnel call personnel at the unidirectionally-protected site and ask for assistance. Site personnel verify the caller s identity and route the call to an authorized equipment administrator. That administrator logs into the appropriate equipment, often an engineering workstation, is guided by turbine vendor support personnel to the appropriate applications and dialogs needed to diagnose the problem interactively if necessary, and adjust the turbine to correct the problem. The turbine vendor sees this interaction as supervising site personnel in correct resolution of a problem. The site personnel see the interaction as supervising vendor personnel in their adjustment of the site s equipment. Each perception is legitimate, and each set of needs is being met.
Firewall 4.3 Unsupervised, Occasional Remote Management When there are no qualified personnel at a site, as might be the case in centrallymanaged sites, there may still be a need for occasional remote management. A variety of ad-hoc solutions support this need, and a commercial solution exists in the form of Waterfall Security Solutions Secure Manual Uplink product [7]. Whether ad-hoc or off-the-shelf, the solution lies in temporarily connecting protected control system networks to business networks for remote management. Figure (3) illustrates the Waterfall solution as applied to the turbine management problem. Industrial Network TX Agent Hardware-Enforced One-Way Communications RX Agent Business Network Secure Manual Uplink Fig. 3. Secure Manual Uplink The Waterfall solution consists of a network appliance with at least two conventional copper connectors, and a physical key. When the key is turned, the device electrically connects the input and output copper connections and so connects the business network to the industrial network for a pre-programmed period of time. After the time expires, or in the event of an unanticipated failure, such as a power failure, the device once more automatically disconnects the two networks. The mechanism provides temporary remote control for remote vendors or for central SOC, NOC or other support personnel to the protected ICS network. In practice though, the control system and business networks are never directly connected. Instead, as illustrated in Figure (3), the business network is generally connected to a control system firewall, and other kinds of security technologies such as VPNs and remote access servers often intervene. In this way, the unidirectional communications provide absolute protection from external attacks 99% of the time and the rest of the time, the connection is as secure as conventional security technologies can make it. This kind of solution though, still requires the intervention of staff at the control system site. In the turbine management scenario for example, the vendor must still contact site personnel, identify themselves, and request that the remote access solution be activated, for example by turning the key.
4.4 Operations Wide Area Network (WAN) A small, growing number of end users are eliminating most or all control system operations personnel from their industrial sites, and are managing and operating those facilities entirely by remote control. In these scenarios, a permanent remote control connection to unidirectionally-protected networks is required. The solution we see deployed most commonly is the concept of an operations WAN, as illustrated in Figure (4). Head Office / Central Management Ops Corp Conventional Firewalls Ops Corp Ops Corp Ops Corp Site 1 Site 2 Site 3 Fig. 4. Operations Wide Area Network The operations WAN in the diagram is a set of operations / control system networks, connected via conventional wide-area network technologies, and internally segmented with firewalls and possibly other conventional networking technologies. Ideally, the WAN connections between individual sites are via leased lines, rather than tunneling those connections through business networks or public networks with Virtual Private Networks (VPNs) or their equivalents. For maximum security, the only connection between the operations networks and the corporate / business WAN is via one or more unidirectional gateways. In the illustration, each site has such a connection, and this method of deployment is commonplace in our experience. What this means for central administrations staff and in fact any personnel who routinely need access both to the operations network and the business network is that the offices, workstations or desks used routinely by these personnel must support two computers: one directly connected to the operations WAN, and one connected to the business WAN. The two computers allow operations personnel to interact with each network as needed. Care must be taken to prevent the accidental interconnections of these networks, and technology to detect and alert on such interconnections is strongly indicated as well. In a sense though, such precautions are not out of the ordinary, even for conventional network segregation. When networks are segregated by firewalls, care must still be
taken to avoid accidental network interconnections, the deployment of rogue wireless access points, and so on. Information flows routinely from the operations WAN to the business WAN over one or more unidirectional gateways. Information flows less routinely back into such networks, often through removable media, and sometimes through more exotic mechanisms. Again, care must be taken with removable media. Application whitelisting and removable device control systems are starting to be deployed to address risks due to removable media, at least on equipment, such as dual-computers in the offices of central operations staff where the use of such media is generally authorized. 5 Summary Hardware-enforced unidirectional communications solutions are being deployed routinely in a number of industries, most prominently both nuclear generators and conventional generators in the North American power grid, where recent regulations and guidance [4], [5], recognize that the technology provides stronger security than firewalls are able to. In large part, this adoption is due to off-theshelf server replication solutions for industrial servers and devices make seamless replacements of conventional firewalls possible in a wide variety of circumstances. In addition, data integrity concerns in all of these scenarios have wellunderstood solutions. A variety of remote monitoring, remote support and central management solutions support modern management disciplines without reducing the security value proposition of hardware-enforced unidirectional communications. In short, widely-held concerns regarding impediments to the deployment of this strong security technology have proven unfounded, and unidirectional gateways are being deployed in substantial numbers in a wide variety of industrial contexts. References 1. Waterfall Security Solutions: Introduction to Waterfall Unidirectional Security Gateways: True Unidirectionality, True Security. (2011) 2. Piètre-Cambacédès, L., Sitbon, P.: An Analysis of Two New Directions in Control System Perimeter Security. In: Proceedings of the SCADA Security Scientific Symposium, Scn. 4. Digital Bond Press, Sunrise (2009). 3. Okhravi, H., Sheldon, F.T.: Data Diodes in Support of Trustworthy Cyber Infrastructure. In: CSIIRW '10 Proceedings of the Sixth Annual Workshop on Cyber Security and Information Intelligence Research, Art. #23. ACM, New York, 2010. 4. U.S. Nuclear Regulatory Commission: Regulatory Guide 5.71 Cyber Security Programs for Nuclear Facilities. (2010). 5. Moon, M., Agnew, V., Engelby, B.: Compliance Application Notice 0024: CIP-002 R3 Routable Protocols and Data Diode Devices
6. Frenkel, L.: Advanced Protection for Advanced Threats: Securing Turbine Management Connections. Waterfall Security Solutions, Tel Aviv (2011). 7. McAfee: McAfee Security Innovations Alliance Partner Directory. (2012)