Improving Water and Wastewater SCADA Cyber Security

Improving Water and Wastewater SCADA Cyber Security Standards Certification Education & Training Publishing Conferences & Exhibits Speakers: Bill Phillips and Norman Anderson 2012 ISA Water & Wastewater and Automatic Controls Symposium August 7-9, 2012 Orlando, Florida, USA

Presenter Bill Phillips, PE: Bill specializes in delivery of secure and reliable process control and SCADA network and communications systems, cyber security vulnerability assessment, and facility automation and information system planning and implementation. Bill has over 30 years of process control and SCADA system experience and has focused on control system network and communications cyber security for the last decade. Bill has a BSEE from Clemson University. Aug 7-9, 2012 Orlando, Florida, USA 2

Presenter Norman Anderson, PE: Norman has over 5 years experience in the design and commissioning of Process Control Systems for the Water Sector. Norman has provided secure and reliable PLC, SCADA, and Network hardware and software architecture designs and provided control system automation solutions for a range of facilities. Norman has an M.S. in EE from Iowa State University and an M.S. in Physics from the University of Florida. Aug 7-9, 2012 Orlando, Florida, USA 3

Presentation Outline Need to secure control systems Continuing increase in Cyber Attacks Notable Cyber Attacks Available Guidance and Resources Standards Design Guides Assessment/Design/Implementation/Operation Determining Risk factors and mitigation techniques Our Experience and Examples Summary Aug 7-9, 2012 Orlando, Florida, USA 4

General Increase In Cyber Attacks Number of Vulnerabilities 9,000 8,000 7,000 6,000 5,000 4,000 3,000 2,000 1,000 0 CERT Cataloged Vulnerabilities 1995-2007 Year General Trend of increase in incidents and vulnerabilities. CERT stopped incident monitoring in 2003. *Source: CERT Statistics http://www.cert.org/stats/#vul-year Incidents Reported 160000 140000 120000 100000 80000 60000 40000 20000 Vulnerabilities Aug 7-9, 2012 Orlando, Florida, USA Year 5 0 CERT Reported Incidents 1988-2003 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 Incidents

Reported Incidents by Infrastructure Sector Water/Wastewater is #4 on the list and has twice the incident rate of most commercial facilities. *Source: Summarized by Infrastructure Sector (RISI, 2010) Aug 7-9, 2012 Orlando, Florida, USA 6

Industrial Security Incident Attack Points of Entry Many attacks are through local business networks and via remote access. These are two common connections to industrial networks to allow for machines having email and internet access to connect to SCADA networks and to allow remote vendors to connect to SCADA networks for maintenance. *Source: Summarized by Points of Entry (RISI, 2010) Aug 7-9, 2012 Orlando, Florida, USA 7

Financial Impacts Approximately 23% of the industrial security incidents resulted in damages greater than one million dollars per incident. *Source: Reported in the U.S. (RISI, 2010) Aug 7-9, 2012 Orlando, Florida, USA 8

Media Coverage Pump destroyed at water plant Springfield, IL o Believed to be due to cyberattack (not confirmed by DHS). o Story covered by news media such as the Washington Post, Fox News, CNN, and MSNBC o Even though unconfirmed, the utility was in the national spotlight for weeks Texas SCADA system hacked and screenshots of HMI released o Response to DHS downplay of IL incident o Again carried by major news media o Used a virtual network connection with the internet with simple password to access network Aug 7-9, 2012 Orlando, Florida, USA 9

More Infamous Attacks Maroochy Shire Sewage Treatment Plant in Queensland, Australia. o Attack resulted in approximately 212,000 gallons of raw sewage to spill out into local parks, rivers, and a nearby hotel. o The attack was perpetrated by a disgruntled insider and former Contractor, Vitek Boden, that previously installed the radio-controlled SCADA equipment for the plant. o During the attack period, Boden used a laptop computer and stolen radio on at least 46 occasions to issue unauthorized radio commands to the SCADA System (Abrams and Weiss, 2008) Aug 7-9, 2012 Orlando, Florida, USA 10

More Infamous Attacks, Continued Stuxnet High sophisticated WORM to target Siemens PLCs Used to destroy centrifuges used for uranium enrichment Deployed using USB flash media devices (thumb drives) o No external connections does not equal safety Showed the weaknesses of Industrial Control Systems Native Code Code with virus Duqu (Stuxnet Variant) Discovered by Symantec and appears to be a variant of Stuxnet Not intended to destroy industrial control systems but to steal information from them Aug 7-9, 2012 Orlando, Florida, USA 11

Common Vulnerabilities Denial of Service (DoS): Attempt to make computer network unavailable Would slow or shutdown the communications SCADA network Mitigation techniques include Firewalls, ACLs, Intrusion Prevention Systems SQL Injection Attacks SQL databases using vulnerabilities in websites Can steal database information or destroy data Mitigation techniques include effective patch management, Intrusion Prevention Systems DCOM Most notable are RPC DCOM and Blaster attacks Can take control of computer and install programs, view, delete, etc. Mitigation includes use of intrusion detection, packet filtering, and network segmentation, and port blocking Aug 7-9, 2012 Orlando, Florida, USA 12

Example Control System Attack Animation Explains Control System Attack By Remote Attacker Aug 7-9, 2012 Orlando, Florida, USA 13

Importance of Security Why Security is Important at a Water or Wastewater Facility: Critical Infrastructure and Public Safety o Critical resources o Downtime can affect life safety Operational Reliability and Availability o Attacks can lead to significant downtime Financial Impacts o Loss of revenue for utility and its customers o Mitigation and legal costs Media Attention o Loss of public confidence ostaff intimidation Aug 7-9, 2012 Orlando, Florida, USA 14

Available Guidance AWWA Roadmap to Secure Control Systems in the Water Sector published in 2008 o Goal is in 10 years to have no loss in critical function due to cyber attack o Develops a roadmap with goals at the 1, 3, and 10 year marks. Currently in year 4 (mid-term) of program ANSI/ISA-99.02.01-2009 Security for Industrial Automation and Control Systems: Establishing an Industrial Automation and Control Systems Security Program o Builds upon global standards ISO/IEC 17799 and ISO/IEC 27001 and addresses the difference needed for industrial security o Defines procedures for implementing and assessing secure industrial control systems Aug 7-9, 2012 Orlando, Florida, USA 15

Available Guidance, Continued NIST SP 800-82 o Final Version Published: http://csrc.nist.gov/publications/nistpubs/800-82/sp800-82-final.pdf o Goal is to provide a guideline for critical infrastructures to secure their control systems with the idea to maintain systems online and operating unlike traditional IT systems. NERC Critical Infrastructure Protection (CIP) o Numbers CIP-002-3 through CIP-009-4 (18 standards) related to Cyber security implementation plans o Covers implementation of management controls as well as operating procedures for personnel Aug 7-9, 2012 Orlando, Florida, USA 16

Available Guidance, Continued Cisco/Rockwell Automation Converged Plantwide Ethernet (CPwE) Design and Implementation Guide oprovides design and implementation guidelines for industrial control systems based on the manufacturing industry ogoal is to provide less downtime, higher security, and optimization of Industrial Ethernet networks o Guide provides real network architecture examples, security methods, and implementation methods Aug 7-9, 2012 Orlando, Florida, USA 17

Securing Networks Securing networks requires proper planning to ensure successful implementation. There are four basic stages of planning and implementation for network security: 1. Assessment Determine Risks and Mitigation techniques Risk impact versus cost of mitigation 2. Design Develop appropriate network architecture and segmentation (NOTE : Taylor to selected HMI suite TCP/UDP port requirements) Choose necessary hardware and software 3. Implementation Qualified and certified installers and designers 4. Operation and Maintenance Develop operational procedures for staff Maintain network, hardware, and software Aug 7-9, 2012 Orlando, Florida, USA 18

Assessment The Critical Starting Point First step for proper network security Past Assessments were largely based on RAM-W This method was not very specific or comprehensive Limited guidance was available at the time US-CERT Cyber Security Evaluation Tool (CSET) Developed by DHS to assist in protecting key assets with assistance from NIST Available free from the US-CERT website: http://www.uscert.gov/control_systems/satool.html [training from Control System Security Program (CSSP) also provided] Uses 4 major steps and generates a report based on current industry standards Assessment is then used to plan and prioritize mitigation solutions Aug 7-9, 2012 Orlando, Florida, USA 19

Typical Large Utility Control System Network Aug 7-9, 2012 Orlando, Florida, USA 20

Typical Small Utility Control System Network Aug 7-9, 2012 Orlando, Florida, USA 21

Typical Small Remote Systems No matter the size of the network there are still critical systems to protect. Process control networks are inherently different than IT business networks even though many components are similar. Aug 7-9, 2012 Orlando, Florida, USA 22

Wastewater Utility Control System Design Example Includes redundant WAN connections Internet connection for WAN extension to remote facilities & mobile remote access Compact resilient core network Uses VLANs and firewall subinterfaces to tailor network architecture to SCADA HMI applications suite requirements and to securely support business network access Aug 7-9, 2012 Orlando, Florida, USA 23

Network Segmentation Using VLANs Network organization secures and helps maintain networks. Virtual LANs (VLANs) - Useful for SCADA systems because VLANs define broadcast domains that can be widely separated (i.e. not on the same network segment) Can reduce costs, by allowing host on different networks to share layer 2 switches. Use 802.1q VLAN encapsulation protocol Layer 3 device required to route between VLANs, some Layer 2 devices will support VLANs to some extent. VLAN Approach: o VLAN Range: 1-1005 (normal) & 1006-4094 (extended) o Don t Use VLAN 1 (Native VLAN) o Verify VLAN capabilities of network switches & routers o Use logical approach o Incorporate VLAN designations into IP Addresses Aug 7-9, 2012 Orlando, Florida, USA 24

Network Segmentation Using VLANs (Example) Example: VLAN 10 Network Management VLAN 20 SCADA DMZ VLAN 30 SCADA VLAN 40 Security (Video) VLAN 50 Remote User (DMZ) VLAN 100 Public Media WAN (Inter Facility VPNs) VLAN 110 Backup Public Media WAN Extensions: (For shared media) VLAN 60 Business VLAN 70 Business Remote User (DMZ) Aug 7-9, 2012 Orlando, Florida, USA 25

Network IP Addressing Approach: Use 10.0.0.0 private network Class A for primary VLANs Use 192.168.0.0 private Class Cs for routed links Incorporate facility & VLAN numbers into IP addresses Limit broadcast domains to a single facility Primary VLAN Example: 10.VLAN.Facility.Host/X or 10.Facility.VLAN.Host/X X = Subnet Mask bit count X (between 24 &30) based on anticipated host count WAN Example: 192.168.1.Y/X X = Subnet Mask bit count (between 24 & 30) based on number of nodes Y (between 0 & 252) = Network Number Aug 7-9, 2012 Orlando, Florida, USA 26

Example Firewall Configuration Specification Security Levels - Implicit Deny Lower-to-Higher level Interfaces Typically 3-4 for small to medium size firewalls Sub-interfaces can extend that number Stateful Inspection Can drop otherwise legitimate packets that are not part of an active connection Holds in memory variables defining the state of each connection State variables include things like source and destination addresses, port numbers, packet sequence numbers Access Control Lists Used to apply access control rules at interfaces Format: access-list-number {permit deny} protocol source {sourcemask} destination {destination-mask} [eq destination-port] Aug 7-9, 2012 Orlando, Florida, USA 27

Example Firewall Configuration Specification Security Levels Each Interface & Sub-interface Inside 100 (Most trusted) Outside 0 (Least trusted) DMZ 50 Access Control Lists Permit DMZ to-inside SCADA specific traffic such as web server, terminal server and historian traffic. Permit VPN LAN-to-DMZ authenticated remote user traffic such as web server, terminal server and historian traffic. Remote PLC Connections: Consider a Remote PLC DMZ to avoid direct connections between Internet connected PLCs and the SCADA network Consider dual Ethernet DMZ PLC interfaces (i.e. separate VLANs) to increase separation. Aug 7-9, 2012 Orlando, Florida, USA 28

Domain Controller Implementation Use group policies to manage role based access Separate controllers required for each domain Domain Controller and Active Directory Traffic Uses Remote Procedure Calls (RPC) and Distributed Component Object Model (DCOM) which introduce numerous vulnerabilities. Should not be permitted across firewall boundaries (i.e. don t extend the corporate domain into the SCADA DMZ) Exception When a Read-only Domain Controller (drastically reducing port requirements) is used with an IPSec VPN tunnel connection to extend the SCADA domain into the SCADA DMZ Generally worth the trouble to ease implementation & maintenance of role based access & remote access using RADIUS authentication Aug 7-9, 2012 Orlando, Florida, USA 29

Remote VPN Connections VPNs can securely extend WANs using public media & provide secure remote access to mobile staff Remote Facility Connections Using IPSec Site-to-Site VPNs Used to interconnect two or more facility LANs Encrypts entire IP packet including endpoint private IP addresses Provides, confidentiality, data integrity, origin authentication and replay protection Mobile Remote User Connections TLS/SSL VPNs Uses browser interface to connect mobile remote clients to servers Operate at the session level to provide secure client/server connections Uses certificates to authenticate servers & clients. Uses symmetric keys to provide confidentiality and data integrity Aug 7-9, 2012 Orlando, Florida, USA 30

VPN Tunnel with Encryption Aug 7-9, 2012 Orlando, Florida, USA 31

Remote Access VPNs Aug 7-9, 2012 Orlando, Florida, USA 32

Firewalls for Network Security and Routing Aug 7-9, 2012 Orlando, Florida, USA 33

Converged Plantwide Ethernet (CPwE) Design & Implementation Guide (DIG) LAN resilience alternatives shown & performance comparisons. L2&3 QoS settings recommendations DMZ Example tailored to SCADA CPwE DIG Developed by Cisco Systems & Rockwell Automation Provides detailed guidance & includes LAN configuration alternative testing results These figures from the DIG are from the LAN and DMZ design chapters Aug 7-9, 2012 Orlando, Florida, USA 34

Designing and Implementation Roadblocks Conflicts faced by utilities Lack of regulatory driver Many competing needs Losing sleep each time another event makes the news. What to do? Utility staffs are a resourceful bunch and they find a way to address their concerns Some are able to get funding to specifically address cyber security. Others have to be more creative. Utilities often lack resources to self-perform SCADA security assessments & improvements planning, design and implementation, as mentioned before, there is help. Aug 7-9, 2012 Orlando, Florida, USA 35

Designing and Implementation Roadblocks, Continued Our experience Some utilities, usually bigger ones, have adopted appropriate standards and established internal policies, procedures and standards that they apply to each project More commonly, the utility hasn t established comprehensive standards and isn t aware of the vulnerabilities in their existing systems; but would like to make progress as part of each project. Sometimes it s a grass roots or replacement project which means that they are open to a comprehensive solution but do have budgetary constraints. Other times it more like what can we shoehorn into this small incremental project. Aug 7-9, 2012 Orlando, Florida, USA 36

Example - Incremental Implementation IPSec VPN Tunnel 10 BaseTX 100 BaseTX 100 BaseFX 62.5 125mu Layer 3 Switch W/ EIGRP100 2801/2611 W/ EIGRP100 Tx/Fx Layer 2 Switch ASA 5510 WasteWater- SCADA-SERVER PLANT 1 MOSCAD MTU Modbus WASTEWATER- MODBUS-IP- SCADA-VIEW CONVERTER P1Switch Plant1-SCADA-2960TT-1 Fa0/0 PLANT 2 Industrial Park PLC WW-SCADA-SRVR1 WW-SCADA-SRVR2 Plant2-SCADA-2960TC-1 MTU PLC PLC1A HP JetDirect Lift Station 14.2 Plant2-WW-L14_2-2955 PLC1B Switch Westside Blvd SE SERVER PLC Cabinet OIU PLC PLC (Redundant Quantum) PLC Box PLC WELL 15 100BASE-TX/100BASE-FX Media Converter 54mbps Plant1-SCADA-2801-1 Tank 15 Well15-SCADA-3000-1 LS8 PLC LS 8 Modbus Serial 54mbps Esteem Access Point MODBUS-IP- CONVERTER MODBUS-IP- CONVERTER Esteem Access Point Plant3-SCADA-2960TT-1 PLANT 3 Address? Modbus Modbus+ P3 PLC LS13 PLC LS21 PLC 1536Kb/s (CIR) 58/KDFN/103677/TWCS Plant2-SCADA-2801-1 1536Kb/s (CIR) 58/KDFN/103674/TWCS Plant 6 SCADA VIEW 1536Kb/s (CIR) 58/KDFN/103608/TWCS Plant6-SCADA-2801-1 Plant6-SCADA-2960TT-1 Remote Connections Point-to-Point Connection to Well 12 Esteem AP Bridge Esteem AP Bridge Exalt AP Bridge 54mbps Proxima AP Bridge 54mbps Remote VPN User Remote VPN User Remote VPN User Plant3-SCADA-2801-1 1536Kb/s (CIR) 58/KDFN/10367?/TWCS TW Telecom Metro Ethernet (ILAN) Layer-2 (Bridged) 1536Kb/s (CIR) 58/KDFN/103608/TWCS Remote Connection to BPS 12 1536Kb/s (CIR) Exalt AP Bridge Proxima AP Bridge CITY CENTER 58/KDFN/103610/TWCS 1536Kb/s (CIR) CityCenter-SCADA-3000-1 TW Telecom Internet 58/KDFN/103609/TWCS W DMZ Well10CC-SCADA_DMZ-2960TT-1 WWTP1SCADA3 (HIST) 6Mb/s (CIR) 58/KDFN/103600/TWCS WW DMZ SCADA-ASA: 5510s in HA 20Mb/s (CIR) 58/KDFN/103604/TWCS WELL 10 (Control Center) Well10-WATER-2960TC-1 RX3I Well 10A Building Control Panel 54mbps Proxima AP Bridge Esteem AP Bridge Remote Connections W-SCADA-WIN911 WW-SCADA-HIST Well10-WATER-2955-1 WELL 8 Proxima AP Bridge W-SCADA-TS W_SCADA_CLIENT2 Well10CC-SCADA-3560-1 Well 10 Control Center WW-SCADA-WIN911 3621 Redhill Place Well10-WATER-2611-1 W-SCADA-2960TT-1 Dell 2700 OIT BPS WWTP12850 (EMAIL) WW-SCADA-TS WW-SCADA-2960TT-1 W-SCADA-2960TT-1 PLC Well8-SCADA-3000-1 SHARED DMZ [FUTURE] SCADA-DMZ-RODC01 SCADA-DMZ-RODC02 (DC, NTP, Anti-virus, WSUS) WW-SCADA-CLNT W-SCADA-CLIENT1 (WATER VIEW) W-SCADA-DEV-1 W-SCADA-SERVER1 (Z400) RX3I SHARED SCADA SCADA-DCSEC SCADA-DCPRI Well 10 Booster ATF OIT PLC Esteem AP Bridge Remote Connections Aug 7-9, 2012 Orlando, Florida, USA 37

Example Incremental Installation Initial installation can be done using a single Ethernet switch and no remote connections. Remote connections can be added in the future when they can be secured correctly. Design supports adding disaster recovery elements as budgets allow Initial equipment can be upgraded in the future through firmware to add required additional services such as high availability. Aug 7-9, 2012 Orlando, Florida, USA 38

Example - Single Implementation by Phased and Sequenced Construction Aug 7-9, 2012 Orlando, Florida, USA 39

Keys to Successful Implementation (Abbreviated Version of a Long List) Use equipment with a long useful lifetime and low risk of becoming completely obsolete in the short term. Have a budget in mind and idea of the risk/reward of network connected systems and equipment. Be aware that equipment cost is not an indication of work costs. A $1000 router could cost as much to configure as a $15,000 industrial router. Are staff or service contracts in place to maintain and troubleshoot systems? Systems are only as good as the maintenance done. Make sure that good system documentation and training will be delivered with the improvements. Set up a secure backup configuration storage mechanism & keep a copy of all addressing, configurations, settings, and software. Use qualified integrators having the proper certifications where appropriate. Aug 7-9, 2012 Orlando, Florida, USA 40

Defense in Depth A strategy for layering protection mechanisms to reduce the impact of a single mechanism failure In addition to the technical and operational controls that can be applied to SCADA systems, defense in depth requires long term organizational management and operations commitment to security for: Developing security policies, procedures and educational materials that apply directly to SCADA Conducting periodic security awareness, incident response and disaster recovery training Ongoing maintenance and upgrade of SCADA security throughout its lifecycle Restricting physical access to SCADA infrastructure Aug 7-9, 2012 Orlando, Florida, USA 41

User Access Require login credentials with secure passwords and auto logouts Use USB security where ports are available Simple user interface. Do not allow access to the start menu or other nonessential programs Do not allow access to the computer Aug 7-9, 2012 Orlando, Florida, USA 42

Summary The jury is in, the threat is real and utilities need to act Adequate guidance is available to support standards based cyber security improvements The DHS CSET tool and INL assessment support team provide a SCADA focused tool for conducting selfassessments Without a regulatory driver, funding continues to be a problem. Proper planning, implementation, and maintenance is key for a successful system. Systems cannot be installed and forgotten. Utilities are finding a way to make meaningful progress with both funding and solutions. Aug 7-9, 2012 Orlando, Florida, USA 43