Design Document. Team Members: Tony Gedwillo James Parrott David Ryan. Faculty Advisor: Dr. Manimaran Govindarasu

Transcription

1 12/6/2010 SDMAY11-11 CYBER SECURITY OF SCADA SYSTEMS TEST BED Design Document Team Members: Tony Gedwillo James Parrott David Ryan Faculty Advisor: Dr. Manimaran Govindarasu Design Document Tony Gedwillo James Parrott David Ryan

2 Table of Contents List of Figures... 3 Executive Summary... 4 Acknowledgement... 4 Problem Statement... 4 General Problem Statement... 4 General Solution Approach... 5 Operating Environment... 6 Intended Users and Uses... 6 Intended Users... 6 Intended Uses... 6 Assumptions and Limitations... 6 Assumptions List... 6 Limitations List... 6 Expected End Product and Other Deliverables... 6 Approach Used... 7 Design objectives... 7 Functional Requirements... 7 Virtualization... 7 Power System Simulation and Integration... 7 Cyber Security Assessment... 8 Design Constraints... 8 Technical approach considerations and results... 9 Virtualization Approach... 9 Power System Simulation and Integration Approach Cyber Attack/Security Approach Testing approach considerations Virtualization Testing Power System Simulation and Integration Testing Cyber Security Testing Recommendations regarding project continuation or modification Detailed Design SDMAY

3 Virtualization: Overview Power Flow Simulation and Integration Cyber Security Vulnerability Assessment Project Team Information Faculty Advisor Information Team Information Closing Summary SDMAY

4 List of Figures Figure 1: Design Cycle Diagram... 5 Figure 2: Sample Nessus Workstation Report Figure 3: Sample Nessus Vulnerability List Figure 4: System Diagram Figure 5: One-Line Diagram from PowerFactory Figure 6: Using Spectrum Power TG to close a relay Figure 7: Conceptualization of our testbed's software communicaiton SDMAY

5 Executive Summary Supervisory Control and Data Acquisition (SCADA) systems are the nervous systems for the body of our country s infrastructure. This body includes many systems that are vital to the function of our society: power, water, natural gas, oil, and road traffic systems among many others. However, the nervous systems (SCADA systems) that control our infrastructure are currently vulnerable to cyber-attack. Since the mid-1990 s, security experts have become increasingly concerned about the threat of malicious cyber-attacks on the vital supervisory control and data acquisition (SCADA) systems used to monitor and manage our energy systems. Most SCADA system designs did not anticipate the security threats posed by today s reliance on common software and operating systems, public telecommunication networks, and the Internet. With the critical infrastructure of the SCADA systems and the security threats on these systems, it is important to research ways to correct potential security vulnerabilities. A SCADA test bed will be used for this research. This project will expand on the initial test bed created last year and make it more suitable for real-life scenarios and cyber security attacks. The previous senior design team created the initial SCADA test bed. This test bed included 2 Control Centers, 2 RTUs, 2 Relays, 3 SCALANCEs for encrypted communication, a web server, a DTS, and a light board for demonstrating when a relay trips or is closed. The previous team also tested basic cyberattacks against the system. They were able to demonstrate a basic man-in-the-middle attack that would disrupt commands sent by the control center. The initial test bed was a great start and this year s senior design team will improve on the test bed. The goals of this year s senior design team are to expand the test bed to more nodes, integrate power flow analysis and test more advanced attacks. The basic approach for these goals is to use virtualization software to expand the test bed s nodes, use power flow software for the analysis and use advanced vulnerability assessment tools for testing cyber-attacks. This approach will create a more thorough test bed that is similar to real-world systems, allow for power flow analysis and create cyber-attacks that will show vulnerabilities of the system. Acknowledgement Technical expertise of the test bed has been provided by Iowa State University graduate students Adam Hahn, Aditya Ashok and Siddharth Sridhar. DigSilent expertise has been provided by Iowa State University graduate student Jie Yan. Problem Statement General Problem Statement Our goal is to improve the cyber security of SCADA systems by making our own SCADA test bed, where we can simulate power systems and the communication protocols they use, and attempt cyber-attacks on our systems. Through this process, we can test vulnerabilities of commercial SCADA protection products report their vulnerabilities. We can also demonstrate the effects a SCADA cyber-attack can SDMAY

6 have on a power system. We will be improving the test bed created by the previous year s team. We will be expanding the test bed s number of nodes, adding power flow analysis, and creating more advanced cyber-attacks. SCADA System with Poor Security Improvement Cycle SCADA System with Improved Security System Configuration and Improvement Attack Scenario Vulnerability Assessment Figure 1: Design Cycle Diagram General Solution Approach The three main tasks, as described in our problem statement, are to expand the test bed by having more nodes, add power flow analysis functionality and create and test more advanced cyber-attacks. In order to expand the test bed, we will use virtualization to create more nodes without the need for hardware for each node. This will include virtualization of the relay and RTU. To add power flow analysis to the test bed, we will use software that can connect to the test bed and provide analysis along with providing real world scenarios for the test bed. With regards to the cyber-attacks, we will use vulnerability testing tools to scan for vulnerabilities and then try attacks against the vulnerabilities. SDMAY

7 Operating Environment The operating environment for the test bed is a lab in Coover Hall. The conditions in the lab are normal operating conditions for the test bed equipment. Intended Users and Uses Intended Users The primary users of this system will be graduate and undergraduate students in computer engineering or electrical engineering who are researching the cyber security of SCADA systems. Other users of this system might be researchers or companies interested in learning more about the test bed and its functionality. Intended Uses The primary uses of this system will be the creating and testing of cyber-attacks and researching the effects that a cyber-attack could have on a SCADA system, especially in regards to power flow. Another use of this system might be showing people the basics of how a SCADA system works. Assumptions and Limitations Assumptions List All test equipment will function correctly The test bed is similar to a real-world SCADA system o 15 substations in the test bed will be enough to create real-world scenarios A pfsense firewall solution will be able to function like a SCALANCE device. The test bed will demonstrated to those interested in SCADA systems and cyber-security. Industry might be interested in vulnerabilities found through the test bed. The test bed will be used in the next years for continuation of cyber-security attacks on a SCADA system. Limitations List We have two semesters to complete the project Only 120V will be used by the relays instead of higher voltages in the real-world such as 330KV. Only 2 physical relays will be used due to financial limitations Expected End Product and Other Deliverables At the end of the project period we expect to have a test bed that can be used both for demonstrations and for development of cyber security attacks. This test bed will have over 15 nodes, mostly virtual, with some physical. It will also have the ability to have power flow analysis so it can be used to track the effects a cyber-attack has had on the system. We will also have created cyber-attacks that can be used on the system and demonstrate vulnerabilities. SDMAY

8 Approach Used Design objectives Create a SCADA Testbed that can be used to simulate cyber attacks o This testbed will allow us to mimic real-world power systems and demonstrate the effects of a cyber-attack on a SCADA system. Develop a method to plan, execute, and analyze cyber-attacks on our system o We want to be methodical in our approach to testing our finished system. It is important that we have a consistent system that we can use to report our findings. Functional Requirements Virtualization Create a virtualized platform that allows network stack inspection. o Creating a virtualized platform will be the basis of adding more substations to the current test bed. Since we are limited on financial resources, we are unable to purchase more SIPROTEC Relays and SCALANCE devices. We need a virtualized platform that will allow virtual substations that can connect to the physical test bed. We also need this platform to have the ability of network stack inspection in order for us to test cyberattack scenarios. Create virtualized images for RTUs, Control Center, firewalls and Relays o In order to fully virtualize a substation, we will need to create virtual images for each segment of the substation. Creating a virtualized image for the RTU should be somewhat basic since it is a software application that runs on Windows. Creating a virtualized relay will be more difficult since it will require finding a relay simulator that can communicate with the RTU. We can use an open source firewall solution to simulate the SCALANCE firewalls. Virtualized system should be scalable to provide more realistic scenarios. o We want this system to be scalable to upwards of 30, if not more, substations. To be able to do this, we will first need to purchase and install a physical virtual host server with properly allocated physical resources. The substations should be deployed from the server. Power System Simulation and Integration Integrate DIgSILENT PowerFactory with SCADA test bed o DIgSILENT PowerFactory has the power flow simulation capabilities that we need for our system. We can set breakers and other components on a PowerFactory schematic to correspond to data points stored on our SICAM terminals. We will link PowerFactory and our SICAM RTU s together via OPC protocol. Power Simulation should represent real world scenarios SDMAY

9 o We want to integration between the Power Flow Simulation of PowerFactory and the test bed to be able to represent real world scenarios. This will make the test bed more realistic and applicable to the world s SCADA systems. Cyber Security Assessment Produce report detailing security vulnerabilities of the system o The report will detail each vulnerability found during the assessment, what the possible impact an attack would be if carried out using a particular vulnerability, as well as possible countermeasures to mitigate the effect of each attack. Shall implement attacks discovered during the vulnerability assessment o We will think of scenarios where an attacker could use a particular vulnerability to attack the system, try to implement that attack, and attempt to get the attack to work on a consistent basis. Design Constraints We have a few minor requirements that we have deemed non-functional : Minimal configuration on virtual image deployment o We want our system to be easy to set up and analyze. We don t want to have to configure each of our virtual images individually. Images should have backups to prevent loss o We are currently using one external hard drive to accomplish this task, but we are looking into other solutions. Attack scenarios can be demonstrated without requiring detailed information on attack functionality o The simpler we make our system to operate, the easier it will be to demonstrate it to the Senior Design Review Board and others who wish to see a demonstration. We will document how to perform each attack, and if possible, create shell scripts or batch files to automate the attack. Assessment shall function as comprehensive documentation on the security state of the system o This assessment will attempt to be as comprehensive as possible during the information gathering phase, and will thoroughly document any progress made or failures encountered. This will help any future project teams build upon it the work accomplished this year, and hopefully let them avoid repeating any work that has already been accomplished. All test equipment should function correctly Power system should be represented in a manner that is easy to understand o This will help observers quickly and easily understand the implications of a cybersecurity attack. We are considering using a projector to project our system s one-line diagram onto a wall. However, we would prefer to create an easy to understand display other than a one-line diagram to represent our system. This could be a simple program that we create that reads data points off our OPC server and represents SDMAY

10 them in an aesthetically pleasing and easily understandable manner. This display would make our SCADA system very easy to conceptualize, and it will make our system look more attractive and functional to observers. Technical approach considerations and results Virtualization Approach Software Options for a Virtual Hypervisor o VmWare Server Advantages o o o Can get a free license Can have multiple virtual machines on 1 computer Disadvantages Minimal functionality It runs on top of an operating system so the resources used by the operating system will hinder its performance VmWare ESX Advantages Is the operating system for the computer, minimal resource usage and overhead. Can get a free license from the university Can have multiple virtual machines on 1 computer Already familiar with this software Software is easily installed on non-server class hardware Disadvantages License only lasts 1 year. Citrix XenServer Advantages Is the operating system for the computer, minimal resource usage and overhead. Can have multiple virtual machines on 1 computer Disadvantages No free license available, would need to pay for one. Not as familiar with this software. Microsoft HypverV Advantages Can get a free license from the university Can have multiple virtual machines on 1 computer Is the operating system for the computer Disadvantages Not familiar with this software. SDMAY

11 Software Selection for a Virtual Hypervisor We chose to use VmWare ESX as our virtualization hypervisor. A team member was familiar with the software and has used it before. The university also gives us a 1 year license to the software so there was no need to spend money on the software. It was also easy to install on a PC even though it usually recommends server-class hardware be used. This software also allows for virtual machine templates to be used so it would be easier for use to deploy multiple substations. Software Options for a Software Relay Simulator o Delphin-Informatika IEC Simulator o o Advantages Was developed with use for SICAM PAS and Siemens Relays Connected and worked with SICAM PAS Disadvantages Only 30 day trial, expensive to purchase Trial did not include full functionality Based out of Russia, little amount of support. SISCO AX-S4 MMS Advantages Free educational license Provides a network stack for communication Disadvantages More complex than the other solutions SystemCORP IEC61850 DLL Advantages Free Disadvantages Poor documentation Did not connect well to our system. No Support Software Selection for a Software Relay Simulator We chose to use the SISCO AX-S4 MMS as the software for simulating relays. At first we thought the Delphin-Informatika IEC Simulator would be our selection. It worked well with our system and was developed for the same hardware and software that we are using. The draw backs to the Delphin- Informatika simulator is that the trial only lasted 30 days with basic functionality and that the full license would be too expensive. We did some more research and found the SISCO simulator. The SISCO AX-S4 MMS provides much functionality as a simulator and SISCO provides a free educational license. Even though the SISCO product is more complex and will take longer to learn, it was the best option. SDMAY

12 Power System Simulation and Integration Approach Software Options o Siemens Spectrum Power TG DTS (Dispatcher Training Simulation) Advantages o Software already installed in our lab Software designed to interact with the our system Disadvantages Poor documentation Hard to set up Technical support period had expired DIgSILENT PowerFactory Advantages Has OPC communication capabilities Easy to use Extensive documentation Many people in ECpE department use this software Disadvantages Requires advanced license Software Selection We chose to use DIgSILENT PowerFactory for our power system simulation. It was becoming apparent that we required technical support from Siemens if we were going to use Spectrum Power TG DTS. The manuals were not helpful, and they did not contain the information we needed. This support costs around $20,000 per year a price clearly out of our budget. We found that there was a graduate student here at ISU doing something very similar to our project. He was using an OPC server to control breakers in DIgSILENT PowerFactory. Since this was exactly what we wanted to do, and we knew it could be implemented, we decided to go with that. The use of PowerFactory s OPC capabilities requires an advanced license that costs around $2,000. Since this was way less than the Siemens support cost, that was only going to last a year anyway, we decided it would be better to obtain a license that the whole department could use. Cyber Attack/Security Approach Software Options o Nessus Security Scanner Advantages Remote Vulnerability Scanning Combined the Document Running Services and Document wellknown software vulnerabilities phases into one scan Free License available Disadvantages SDMAY

13 o Is limited by the plugins that have been created Various Open Source Tools Advantages Usually free Disadvantages Not necessarily well documented or supported Software Selection The first piece of software used in performing the vulnerability assessment will be Nessus Security Scanner from Tenable Security. Nessus remotely scans computers for vulnerabilities, both client-side and server side, through tests that are specified via the software s plugin architecture. Nessus generates a report for each computer which contains a list of any vulnerabilities it discovered during the scan, each categorized by port number and severity level, as well as reports generated by the test plugin itself. These reports can be viewed directly on the Nessus Server via a web interface, or exported as an HTML file. Figure 2: Sample Nessus Workstation Report SDMAY

14 Figure 3: Sample Nessus Vulnerability List It is difficult to predict what software will be used to implement the attacks, as the appropriate software will vary depending on the type of vulnerability. Most, if not all tools will be free and open source, though we will not exclude commercial software if it will prove useful. An excellent compilation of common security tools is the Linux distribution called Backtrack 4, which is available for free from its website. Testing approach considerations Virtualization Testing How and where will testing be performed? Testing will be performed in the SCADA lab. We will need to verify the virtual server is running and communications are working. Exactly what will be tested? Communications between virtual RTUs and virtual relays Communications between virtual RTUs and physical command center How will testing accuracy be determined? We will check the RTU operations screen and if it shows that both virtual relay and command center are connected than it is working correctly What information will be recorded on the forms that will be used to record test results? We will record what virtual RTUs and virtual relays are not working and record any errors associated with them. SDMAY

15 Who will be doing testing and how will it be verified? Most likely James Parrott will complete tests. Graduate students will also help in the testing. Power System Simulation and Integration Testing How and where will testing be performed? Testing will be performed in our SCADA lab. We will need to verify that our SCADA testbed is interacting with and controlling our power flow software. Exactly what will be tested? We will need to test each component on our power flow simulation that is linked to our OPC server and controlled by our SCADA system. These components will mainly be relays. How will testing accuracy be determined? Our testing will be very objective, since the components that we are testing virtualized relays only exist in two states: on and off. Our operator will be sitting at our control terminal, and he will toggle the status of a relay. If the change is reflected on our PowerFactory display, and the power flow solution is adjusted accordingly, we know that the tested component is functional. What information will be recorded on the forms that will be used to record test results? Date/Time, name of component tested, location on OPC server, test failed/successful, comments Who will be doing testing and how will it be verified? Most likely Tony Gedwillo will be performing these tests. Our cooperating grad students will help to verify these results by attempting to operate the system. Cyber Security Testing How and where will testing be performed? o In the lab, on the physical substations. Exactly what will be tested? o We will test the overall security configuration of the system and attempt to implement any promising vulnerabilities that are discovered. How will testing accuracy be determined? SDMAY

16 o If an attack works properly, then it was accurate to call examine that vulnerability What information will be recorded on the forms that will be used to record test results? The configuration of each device, as well as whether particular attacks were effective. Who will be doing testing and how will it be verified? o David Ryan will be doing this section of testing in cooperation with Adam Hahn. Recommendations regarding project continuation or modification At this point, we recommend that we continue the project as planned. It appears that we will be able to satisfy our functional requirements in the allotted time. We will be able to virtualize RTU s and relays, connect our power flow software to the testbed via OPC protocol, and execute cyber-attacks on the system. There is no reason to abandon the project, since there was a large initial investment in the equipment used in the lab and we have the time and ability to complete the project as planned. Detailed Design Virtualization: Overview This part of the project requires us to install a virtualized hypervisor, install virtual RTUs and virtual relays on the server and have them connect to the current test bed. As stated in the software selections, we will be using VmWare ESX for the virtual hypervisor and SISCO AX-S4 MMS as the relay simulator. Below is a figure the shows what our test bed with virtualized substations will look like. SDMAY

17 Figure 4: System Diagram Power Flow Simulation and Integration Relevant software and equipment o DIgSILENT PowerFactory This is the software we will use to simulate our power system and solve its power flow. The substations (busses), generators, loads, and relays that we want to reflect real world scenarios will be modeled through this software. These components will be represented on a one line diagram (See Figure 1). The relays modeled in this software will be controlled by our SCADA system via OPC connectivity. This software will function as our OPC client. With this software, we can show the effects of a cyber-attack on a power system. SDMAY

18 Figure 5: One-Line Diagram from PowerFactory o Siemens Spectrum Power TG This software will be used to manually control the statuses of the relays in our system. Here, we can manipulate our power system. This software functions as a Human Machine Interface, or an HMI. SDMAY

19 Figure 6: Using Spectrum Power TG to close a relay o Siemens SICAM PAS Our virtualized RTU s will use SICAM PAS software. This software will provide the OPC server needed to facilitate communications between Spectrum Power TG and PowerFactory. After connections are established between SICAM, PowerFactory, and Spectrum Power TG, SICAM software will mainly be a background system. During an attack simulation, users will not directly use SICAM software, and observers will not be aware of its operation. It simply serves as a communications point. SDMAY

20 Figure 7: Conceptualization of our testbed's software communicaiton Cyber Security Vulnerability Assessment This will be a white-box vulnerability assessment. We have complete access to a fully operational test bed with no danger of causing any harm if we disrupt normal operations. This provides an excellent opportunity to research and test any vulnerabilities that might disrupt normal operations in a functional real-world system. This assessment will concentrate on the assessing the physical substations because they have a wellestablished that will likely change very little in the near future. Any work assessing the physical substations should carry over into the Virtualization and Power Flow Simulation portions of this project. The virtualization component will attempt to emulate the physical substations, and the power-flow simulation should interact the same way with physical or virtual substations. The testing procedure is as follows: SDMAY

21 Validate the System The initial step will be to do a network survey to validate the network, and eliminate any incorrect assumptions from being made due to incorrect or outdated documentation. A reference spreadsheet will be created to record all available information about each device. We will then physically verify that all Ethernet connections are going to the proper place according to the network map. Last, we will record the host names and IP addresses of all machines in the lab, as well any software applications that are installed on each machine. Document Running Services The next step will be to find out how many ports were exposed to the local network, and what services were running on each port. This step will be accomplished Nessus Security Scanner. Nessus will scan through each possible TCP and UDP ports on each computer or hardware device, detecting whether or not each port responds when queried with traffic. If the service isn t directly identifiable to the port scanner, software named Active Ports can be used to discover which executable opens which port. This information will then be recorded to use as a reference guide, in case we ever need to readily identify a particular port number or service. Document Well-Known Software Vulnerabilities During the port scan, it also runs numerous tests on each port to determine if each port is susceptible to a particular vulnerability of any severity level. The client side software scan requires a credentialed scan using Nessus s SMB logon capabilities. When Nessus is provided with the local Windows account credentials, the software is able to check the patch levels of all software on the computer, including Windows itself. Information about the OS patch level will be added to the reference spreadsheet. Search for Implementation Vulnerabilities The final step will be to search for vulnerabilities that are undocumented or specific to our lab implementation. This includes investigating the Siemens software because Nessus does not have any tests to evaluate its security level, as well as searching for any weaknesses in communication or authentication protocols used by any devices or software in the lab. Attack Implementation To evaluate the results of the vulnerability assessment, we will attempt to implement any promising vulnerabilities that are discovered. We will also attempt to make repeating these attacks as simple as possible by documenting the steps on how to perform the attack, and if possible, create shell scripts or batch files to run the attack commands. Produce Report We will produce a report detailing the existing vulnerabilities of the system, the possible impact if an attack were carried out using a particular vulnerability, as well as possible countermeasures to mitigate the effectiveness of a given attack. SDMAY

22 Project Team Information Faculty Advisor Information Dr. Manimaran Govindarasu 3227 Coover Ames, IA Phone: Fax: Team Information James Parrott Computer Engineering 2132 Sunset Ames, IA Phone: David Ryan Computer Engineering 2304 Wallace Rambo Ames, IA Phone: Tony Gedwillo Electrical Engineering 6212 Frederiksen Ct Ames, IA Phone: Closing Summary The goal of our SCADA test bed is to mimic real world SCADA systems and to discover and document vulnerabilities that industrial SCADA systems may have. If industrial SCADA systems are compromised, money and lives can be lost, especially for large scale SCADA systems like electrical power transmission systems. We will use virtualized relays and substations (RTU s) along with control system software and power flow simulation software to model a SCADA system. Once this system is set up, we can complete vulnerability assessments, conduct attack scenarios, and document the effects on our power system and the failures of our security measures. Our hope is that we can provide the power industry, along with any industry that utilizes SCADA systems, with reports on SCADA system vulnerabilities, so that preventative measures can be taken. SDMAY