Truong Le UTSI International Corporation Page 1 ENTELEC 2002 SCADA SYSTEM PERIODIC MAINTENANCE By Truong Le UTSI International Corporation INTRODUCTION Proper maintenance of SCADA equipment and software is the best road to a good night s sleep for SCADA support staff. Neglect of a SCADA system will lead to many kinds of headaches. New problems are much easier to diagnose and resolve when the SCADA system has been running smoothly with a minimum of error messages being generated and adequate resources available to handle abnormal situations. The life cycle costs of a SCADA system are minimized by keeping the system continually well maintained. It takes a small amount of time out of the day to perform maintenance tasks, but the benefits greatly outweigh the costs. Unfortunately, there is just not enough emphasis on the importance of a sound maintenance procedure. SCADA administrators are often occupied with normal daily administrative tasks, as well as emergency requests from internal and external groups, which preempt maintenance tasks. Once these tasks have been neglected for a continuous period, problems begin to mount, the performance of the SCADA system starts to decrease, and the reliability of the system will suffer. SCADA systems are too often treated like poorly maintained cars; being driven into the ground while neglecting to change the air filter, the oil, etc. Eventually, the car will break down and require major repairs. The ramifications and magnitude of a problem are amplified and multiplied with the SCADA system. Serious problems with reliability and performance will likely occur if the proper care and feeding of the SCADA system is ignored. Periodic Maintenance Procedure: What do I need to do? Many SCADA administrators are sometime unaware of all of the different tasks that are necessary in keeping a SCADA system functioning at peak performance. The primary purpose for periodic maintenance is to clear the system of minor problems before they lead to a major disaster in the future. The tasks that should be performed during the routine maintenance effort include the following: Monitor users, Monitor system processes, Monitor daily CPU usage,
Truong Le UTSI International Corporation Page 2 Examine log files, Correct OS errors, Correct SCADA-generated errors, Backup critical data, Verify routine fail-over, Verify that the backup system switches correctly before any unanticipated system crash, Verify data shadowing is working correctly between primary and backup systems, Verify that the backup databases are consistent with the primary databases, Locate bottlenecks in productivity and alert personnel to trouble spots, and Identify bugs. TO PERFORM OR NOT TO PERFORM PREVENTIVE MAINTENANCE We all understand that, to gain a competitive edge in a cost-driven environment, the fundamental benefit in performing a preventive maintenance procedure on the SCADA system is to help keep costs down in the long run. However, some well-intentioned companies choose to perform maintenance on their SCADA system, and some do not. We will look into the logic for each position. This approach will help ensure that we do not have undue bias in our recommendations. We will suggest implementation changes to benefit the SCADA users after all the considerations are reviewed. Reasons To Perform or Not To Perform SCADA System Maintenance To Perform Prevent unscheduled downtime or unbudgeted costs Identify and understand problems and bugs Not To Perform Assumption that modern computer software does not need maintenance SCADA systems have alarms to notify staff Purchase of software maintenance contract IT staff maintains the SCADA system New government regulations Maintenance costs too much Let us review each of these reasons in greater detail. First, we will begin with the reasons companies choose to perform periodic maintenance tasks on their SCADA system.
Truong Le UTSI International Corporation Page 3 REASONS SOME COMPANIES CHOOSE TO PERFORM SCADA MAINTENANCE Prevent Unscheduled Downtime or Unbudgeted Costs Unscheduled downtime of a SCADA system can be costly and force a shutdown of the pipeline. It also disrupts the daily SCADA operation and workflow of the SCADA system support staffs. In addition, since the SCADA system provides the essential data for others in the company, it could disturb and involve multi-level departments and functions with the operating company. Thus, it generates unnecessary stress within the SCADA support staff to resolve the problem. Furthermore, the safety and exposure of the general public could be compromised. During periodic maintenance, a problem can often be identified early and mitigated by using a well-developed corrective action plan. Furthermore, periodic maintenance helps reduce the downtime of the SCADA system. During the maintenance process, the staff can identify potential problems and allow SCADA team members to plan appropriate actions. SCADA system downtime is never a desirable option but, when it is necessary, the backup system should perform identically to the primary. Thus, as part of the preventive maintenance, it is very critical that the SCADA support staff periodically test the backup switching capability to ensure the function is operating correctly. Identify and Understand Problems and Bugs As part of the maintenance process, a SCADA administrator can identify and correct problems or bugs. The administrator might observe an abnormal condition in the SCADA system and can prevent a major crisis before it ever happens. These issues should be identified and corrected to improve the current state of the system. Cleaning up small, simple problems will generally improve the performance and reliability of the SCADA system, as well as allow the administrator to use debug and diagnostic tools on a regular basis and to keep proficient with them. New Government Regulations Due to the extreme potential health, safety, environmental, and financial consequences of pipeline incidences, it is critical to know the current state of the SCADA system. With government regulators mandating new requirements for system integrity and performance levels, companies should be well prepared for abnormal events during the operation of a pipeline. This is another opportune time to show the government that the company is proactive in providing a safe pipeline and preventing any major problems to the public. In the following paragraphs, we discuss the reasons that companies might choose not to perform periodic maintenance operations to their SCADA system and why the logic behind these reasons is flawed.
Truong Le UTSI International Corporation Page 4 REASONS SOME COMPANIES MIGHT CHOOSE NOT TO PERFORM SCADA MAINTENANCE Assumption That Modern Computer Software Does Not Need Maintenance It is easy to understand why some might think that maintenance of a SCADA system is not required. Many of today s computers have built-in functions to correct problems and to diagnose problems. The desktop PC, operating with a Microsoft Windows environment, is an example of a system that many people consider not to require much maintenance. However, the SCADA system, software, and hardware are much more complex then a normal, everyday PC. It has more intricate functionalities and more involved interface capabilities than any laptop. It is usually performing more complex functions at a much faster speed than a PC will or can. Plus, the consequences of failure are much higher. SCADA Systems Have Alarms to Notify Staff Companies assume that the alarming function in a state-of-the-art SCADA system will notify the SCADA administrator when an event has occurred and when action should be taken. It is true that an alarm will be activated when there is a problem, such as a disk full warning, high communication failure rate, data interrupt warning, etc., if configured to do so. However, once any one of these alarm conditions arises, the system is usually already under severe stress and the opportunity for quick and easy solutions to the problem may have already passed. SCADA alarms on the status of the system are usually designed to notify the staff that some part of the SCADA system is in a critical state. Normal SCADA maintenance procedures should keep system parameters from ever getting to a critical state. Purchase of Software Maintenance Contract Some companies have the misconception that, once they purchase a software maintenance agreement with a supplier, the system should run fine without any attention. Yes, it is true that the supplier is obligated to help and correct any problems that occur; however, it is much wiser for companies to correct simple problems before they escalate into big issues that require the attention of the supplier. From a practical standpoint, it is also advisable to try to minimize the need to call the supplier for assistance because resolution of problems referred to the supplier is hardly ever immediate, and the time waiting for problem resolution is likely to be annoying and inconvenient at best. In addition, some types of problems like configuration issues are not generally covered by supplier contracts. IT Staff Maintains the SCADA System Many companies have decided to search for synergy in their tasks to reduce costs. The IT department has taken the responsibilities of managing the complex SCADA system. Unaware of the complexity and complication of the control system software, including its
Truong Le UTSI International Corporation Page 5 operating system and platform, the IT department manages the SCADA system using the same methods as for any other software products for which they are responsible. This is a huge mistake. We all know that a SCADA system is not like any other system. A SCADA system requires proper attention and care to prevent problems from occurring. There is a significant disparity in the mindsets of the IT and SCADA departments regarding what is sufficient reliability. IT departments have been known to think that SCADA personnel were bordering on insane to demand that the system work all the time. One large difference in mindsets is discussing availability figures with each department. In most cases, the SCADA department determines reliability, assuming 24-hour X 7-day operation, with NO SCHEDULED DOWNTIME. Conversely, the IT department often determines their reliability figure by assuming they get several hours of SCHEDULED DOWNTIME each week, which does not count against availability. The IT department also often considers a single point of failure to be acceptable, while the SCADA department does not. There is a significant disparity in needs and requirements of corporate IT and SCADA departments. IT platforms can have scheduled maintenance downtime and perhaps be more forgiving in accepting unscheduled downtime. The SCADA department mindset expects system reliability in 24 x 7 day operation with NO SCHEDULED DOWNTIME. SCADA also carries the burden of responsibility for health and safety in its role as system controller, as well as an engine for procuring financial data. Additionally, a single point of failure may be acceptable in the corporate IT systems, but not in the SCADA environment. These requirements significantly influence the maintenance programs for these systems. Maintenance Costs Too Much There is an obvious cost in performing periodic upkeep of the SCADA system. The SCADA system support group is required to perform additional tasks. However, in the end, by reducing unscheduled downtime, the annual cost of running the SCADA system operation will be greatly reduced. The small cost of performing a periodic maintenance procedure will truly be justified. This cost is minimal compared to the potential savings and mitigated problems. SECONDARY OUTCOME FROM SCADA MAINTENANCE Maintain Historical Performance Information Having historical maintenance records can greatly assist in resolving unexpected issues or provide support to improve the state of the SCADA system. The maintenance process should include the following procedures:
Truong Le UTSI International Corporation Page 6 1. Collect and Track Accurate Records: A good maintenance process should define the types of information the company requires for maintaining the system and monitoring performance trends and needs to collect and track. This data will be very valuable for future use. It can be used to resolve problems and issues that may arise in the future. Conversely, a lack of adequate system information can make it difficult to justify requests for any SCADA needs. A well-documented case study with an ample amount of system data includes system capacity comparison, CPU usage, etc., and would greatly support any cause. 2. Support Equipment, Upkeep, and Replacement Actions: Having good maintenance records can help support a request for system upgrades, new software tools, etc. The staff can justify the need for a new system from the historical data collected during the maintenance process, and can show trends in the decrease of SCADA system performance, as well as the lack of adequate capacity during peak operation of the SCADA system. This situation can cause a system shutdown, which can create disastrous problems. In addition, new software tools can help run a SCADA system more efficiently and effectively. 3. Use in Cost-Benefit Analyses: Data gathered from the maintenance process can be used as a tool to analyze cost benefits of the SCADA system. It is a good opportunity to use the data to justify an upgrade, support staffing needs, software tools, or other requirements. More Precise Knowledge Over Time Over time a periodic maintenance program will increase the knowledge base. The SCADA personnel learn the various SCADA system details and have a better appreciation of the system. The overall reliability of the system will increase. Understanding Where the System Bottlenecks or Problems Exist The SCADA administrator will identify problems and issues in the system. The problems can be located and corrective action can be performed. Thus, the overall state of the system will eventually improve. CONCLUSION Performing some simple periodic maintenance tasks can prevent future costly repairs or disastrous problems from occurring on the SCADA system. It will reduce the amount of time for unscheduled system shutdowns and staff frustration over an unreliable system. We have examined various reasons why companies choose to perform or not to
Truong Le UTSI International Corporation Page 7 perform periodic maintenance. Our conclusion is that periodic SCADA system maintenance is always justified. Thus, one of the main issues is how to schedule these maintenance tasks, along with the daily operational tasks, and not overburden the staff with extra work. We suggest the following: 1. Develop a maintenance process and procedure to help personnel perform and diagnose SCADA problems. The goal is to identify potential problems and to prevent disasters from occurring. 2. Regularly test the automatic failover of the SCADA system to the backup processor to ensure the SCADA system will handle abnormal conditions. Also, operate on the backup system long enough to verify that it is fully functional. 3. Emphasize and demonstrate to the support staff the importance of a maintenance program for a SCADA system by providing training. Training support staffs in all phases of maintenance tasks ensures knowledge of the system, minimizes response time in the event of an emergency, and reduces and authorizes SCADA resources to perform tasks as scheduled. 4. Every few years, have an independent third party familiar with the operation and maintenance of other SCADA systems, audit the state of the SCADA system. This audit process will help ensure that the staff is taking advantage of current procedures and techniques for SCADA system maintenance. It might help justify budgets to management to get an independent opinion from someone without a vested interest. Truong Le, Consultant UTSI International Corporation 1560 West Bay Area Blvd. 2 nd Floor Suite 300 Friendswood, Texas 77546 USA (281) 480-8786, ext. 132 (800) 324-8874, ext. 132 (281) 480-8008, fax tle@utsi.com