Plans for CIP Compliance

Transcription

1 Testing Procedures & Recovery Plans for CIP Compliance DECEMBER 16, 2009 Developed with:

2 Presenters Bart Thielbar, CISA Senior Research hanalyst Sierra Energy Group, a Division of Energy Central Primer on CIP 007, R1, CIP 008 & CIP 009 Kim Morris Director, Architecture and Information Security PNM Resources Testing Procedures & Recovery Plans 2

3 Testing Procedures & Recovery Plans Primer on CIP 007, R1, CIP 008 &C CIP 009 BART THIELBAR, CISA SENIOR RESEARCH ANALYST

4 Disclaimer The information from this webcast is provided for informational purposes only. An entity's adherence to the examples contained within this presentation does not constitute compliance with the NERC Compliance Monitoring and Enforcement Program ("CMEP") requirements, NERC Critical Infrastructure Protection ("CIP") CIP) Reliability Standards, or any other NERC Reliability Standards or rules. While the information included in this material may provide some of the methodology that NERC has elected to use to assess compliance with the requirements of the Reliability Standard, this material should not be treated as a substitute for the Reliability Standard or viewed as additional Reliability Standard requirements. In all cases, the entity should rely on the language contained in the Reliability Standard itself, and not on the language contained in this presentation, to determine compliance with the CIP Reliability Standards. 4

5 Agenda Purpose, Applicable CIP Standards Testing Procedures Incident Reporting Recovery Plans Audit Trail Recommendations 5

6 Why are Testing Procedures and Recovery Plans Important? Managerial visibility and organizational control/confidence Risk Management Proper testing Incident preparation Practice recovery Part of good control framework NERC s Interests = Best Interests of Reliability 6

7 Physical & Electronic Access Controls CIP 007, R1 - Test Procedures R1 Test Procedures Responsible Entity to ensure that new Cyber Assets and modifications to existing CAs do not adversely affect cyber security controls R1.1 Test Procedures R1.2 Document that testing is performed in manner that reflects production environment R1.3 Document test results CIP 008 Incident Reporting and Response Planning R1 Develop and maintain cyber security incident response plan R1.1 Procedures to classify events as reportable Cyber Security Incidents R1.2 Response actions R1.3 Process for reporting to Electricity Sector Information and Analysis Center (ES ISAC) R1.4 Updating plan within 90 days of changes R1.5 Process for reviewing at least annually R1.6 Process for testing at least annually R2 Cyber Security Incident Documentation CIP 009 Recovery Plans for Critical Assets R1 Recovery Plans -- create and annually review recovery plans for CCAs R1.1 Specify actions in response to events R1.2 Define roles and responsibilities of responders R2 Exercises -- recovery plan to be exercised at least annually R3 Change control recovery plan to be updated to reflect ect changes or lessons s learned R4 Backup and Restore include processes and procedures for backup and storage of information required to restore CCAs R5 Testing Backup Media Essential information stored on backup to be tested to ensure availability 7

8 Schedule for Table 3 Entities Requirement Begin Work Substantially Compliant Compliant Auditably Compliant CIP 007, R1 12/31/06 12/31/08 12/31/09 12/31/10 CIP /31/06 12/31/08 12/31/09 12/31/10 CIP /31/06 12/31/08 12/31/09 12/31/10 8

9 Testing Why: Ensure that new cyber assets and/or changes to cyber assets (not just CCA) do not compromise CCA serving Bulk Electric System Significant Change: Security patches, service packs, vendor releases/upgrades (including operating systems, applications, databases, etc.) Environment: Test Environment is very important must reflect the production environment Documentation: Test results must be documented (good, bad or neutral) All consistent with generally accepted best practices for testing procedures 9

10 Why Incident Reporting & Response Planning Matters Reliability of the Bulk Electric System Potential blackouts Criminal element/malicious behavior Relationship with other events that may occur Preserving order/avoiding chaos Evidence preservation Asset preservation Guidance/direction for industry and employees NERC s Interests = Best Interests of Reliability 10

11 Incident Reporting and Response Planning Procedures to classify events as Cyber Security Incidents Response actions including roles and responsibilities i Process for reporting to Electricity ysector Information Sharing and Analysis Center (ES ISAC) Process for updating plan within 90 days of any changes Process for ensuring that Cyber Security Incident Response Plan is reviewed at least annually Process for ensuring the Cyber Incident response plan is tested at least annually. Possibilities: Paper Drill; Operational Exercises; Actual Incident 11

12 Incident Reporting Documentation Documentation relevant to Cyber Security Incidents must be kept for three calendar years Process Reports Evidence (Logs) 12

13 Recovery Plans Actions in response to events Roles and responsibilities Exercise and test the plan at least annually Document changes to plan and communicate to appropriate people within 90 days Include backup and restore processes and procedures necessary to restore CCAs Test the backup media at least annually 13

14 Roles and Responsibilities (Considerations) Management Support Documentation Empowerment Actions Specified to Events Event Duration and Severity 14

15 Testing and Maintaining The Plan Exercised at least annually Paper Drill Operational Exercise Actual Event Updates to reflect lessons learned Communicated to appropriate personnel within 90 days 15

16 Backup and Restore Process, procedures and Instructions for restoration of CCAs Test the restoration, not just the backup Tapes/Media stored offsite and protected from disaster Necessary equipment and software Can be completed offsite Remember/be aware of firmware and OS impacts 16

17 Possible Penalties and Sanctions Up to $1 M per day, per violation Violation Severity (level of non compliance) Violation Risk Factors Mitigating factors may reduce penalties and sanctions Quality of compliance program, self-reporting, voluntary corrective actions, etc. Aggravating factors may increase penalties and sanctions Repeat violations, evasion, inaction, unwarranted intentional violations based on economic choice, etc. May potentially impact reputation, rate cases, etc. 17

18 Audit Trail Considerations Measures associated with requirements measure the documentation Think like a security person and then like an auditor What would YOU want to know if you personally had to manage restoration Documentation, documentation, documentation 18

19 Final Thoughts Testing Procedures and Recovery Plans = Good Security, Governance and drisk kmanagement Asset Integrity Helpful to Personnel Promote the best interests of reliability NERC s Interests = Best Interests of Reliability 19

20 Testing Procedures & Recovery Plans KIM MORRIS DIRECTOR, ARCHITECTURE AND INFORMATION SECURITY

21 Agenda Introductions Applicable CIP Standards Incident Response Disaster Recovery Testing Media Training Questions 21

22 Based in Albuquerque, N.M., PNM Resources is an energy holding company with 2008 consolidated operating revenues from continuing and discontinued operations of $2.5 billion. Through its utilities - PNM and TNMP - and energy subsidiary - First Choice Power - PNM Resources serves electricity to 859,000 homes and businesses in New Mexico and Texas. Current Capacity 2717 MW 22

23 CIP Standards Applicability 23

24 General Terms & Common Definitions Event an action that has occurred on a system Incident an event that escalates to the need to report or recover Recovery restoring a system to full functionality Cyber Security Incident* Any malicious act or suspicious event that: Compromises or was an attempt to compromise the Electronic Security Perimeter or Physical Security Perimeter of a Critical Cyber Asset, or Disrupts, or was an attempt to disrupt the operation of a Critical Cyber Asset Malicious intending to cause harm Suspicious Event an event where the cause is suspected to be of malicious origin * - NERC Glossary Term 24

25 Incident Response Elements 25

26 Cyber Security Incident Response Plan Why Do I Need One? How Do I Respond/Recover?? Internet BPS Reliability Effected Potential Blackouts Chaos / Mayhem ESP ESP Monitoring System Router Firewal l Access Point Access Point Switch Switch Critical Cyber Assets 26 Network Segment Malicious i Insider Network Segment

27 Incident Reporting NERC offers 3 ways to report to ES-ISAC Critical Infrastructure Protection Information Systems (CIPIS) Reliability Coordinator Information System (RCIS) Telephone, Fax or One of the following forms should be used: OE 417 Form Disturbance Reporting form found in NERC Reliability Standard EOP-004 Security Guideline Threat and Incident Reporting Form Free form or your own reporting form as an to [email protected] What does a Report look like? Refer to OE-417: 27

28 OE-417 Form Guidance WHEN TO REPORT When Report to Incident File if Meets Criteria 1 hour 1. Actual physical attack If causes major interruption or major negative impact on critical infrastructure facilities or to operations. 1 hour 1 hour 1 hour 2. Actual cyber or communications attack 3. Complete operational failure of electrical system 4. Electrical System Separation (Islanding) If causes major interruptions of electrical system operations. If isolated or interconnected electrical systems (transmission or distribution) suffer electrical system collapse. If part or parts of a power grid remain(s) operational in an otherwise blacked out area or within the partial failure of an integrated electrical system. 1 hour 5. Uncontrolled loss of firm system load If 300 MW or more for greater than 15 minutes from a single incident. 1 hour 6. Load shedding If 100 MW or more implemented under emergency operational policy. 1 hour 7. Voltage reductions 3 percent or more applied system-wide. 1 hour 8. Public appeal to reduce use of electricity If in emergency condition only to reduce demand. 9. Suspected physical impairment which 6 hours targets any security system or impacts electric power system reliability If any component of any physical security system is vandalized, damaged by an attack, or is suspected to have been altered. 6 hours 10. Suspected cyber computer or communications system impairment If the attempt is believed to have or did happen. 6 hours 11. Loss of electric service If greater than 50,000 customers for 1 hour or more. 6 hours 12. Fuel supply emergencies Fuel inventories or hydro project water storage levels at 50 percent or less of normal, with projected continued downward trend; emergency generation requiring abnormal use of a particular fuel. 28

29 Incident Response Summary Processes must be in place to direct an entity in what to do if a Cyber Security Incident should occur. A Plan should identify the various required elements. The Plan should make sense and proceed in a logical fashion through the course of an incident. Relevant documentation for any reportable incident Logs Investigative Notes 29

30 Disaster Recovery What is a Disaster*? An unanticipated incident or event, including natural catastrophes, technological accidents, or human-caused events, causing widespread destruction, loss, or distress to an organization that may result in significant property damage, multiple injuries, or deaths. What is Disaster Recovery*? Immediate intervention taken by an organization to minimize further losses brought on by a disaster and to begin the process of recovery, including activities iti and programs designed to restore critical business functions and return the organization to an acceptable condition. (*from NERC Security Guidelines for Electric Sector: Continuity of Operations) 30

31 Why Disaster Recovery Planning? Identify and measure company risks Greater probability of recovery Identify Roles and Responsibilities Communications plans Compliance Resources People Technology 31

32 Types Of Events Loss of EMS/SCADA component Server HMI (Human Machine Interface) Database Loss of field equipment Relay Automation ti System RTU Loss of facilities 32

33 Recovery Is Different Than Prevention Prevention attempts to eliminate the need for recovery Recovery means that the preventative measures have failed Need to respond in order to: Rapidly recover failed systems / components Report following reporting guidelines Preserve evidence for post-event analysis and as required by the standards 33

34 Summary Contents of recovery plans for the loss of Critical Cyber Assets Parts availability Assembly directions Data and program restore Minimum tests required after recovery Evidence of drills and exercises Evidence of updates following system or asset change Re-made media Instructions reflecting updated hardware Etc. 34

35 Backup Storage For Restoration Examples of elements included in a good backup storage for restoration plan: Backup information should be stored away from failed system Make sure the backup tapes are not destroyed during a catastrophic failure Should include instructions and directions Include installation media License keys, etc System experts may not be available during restoration 35

36 Media Testing More than backup tape testing Includes tape recovery system Also includes firmware proms Make sure the recover-to system can read the media 36

37 Change Management Testing Content of a good testing process Linkages to Change Management processes Test Plans Test Plan results (signed / dated) Unit Test Integration Test Regression Test Reports where tests failed Cycle through to changed code and re-test 37

38 What to Test Expected functionality resulting from the change Boundary conditions not likely to be found often in real life Interaction of changed component with the rest of the system Regression testing (make sure the fix didn t break something else) 38

39 Disaster Recovery Plan Testing Annual testing or exercise Recovery from an actual incident counts as an exercise Lessons learned analysis following each test or exercise Follow through on any issues encountered Update plans and procedures Include DR testing during Change Management Practice, Practice, Practice 39

40 Test Environments Should mimic real environments Be reasonable copies of real system Should be able to support Integration Testing Allow canned testing scripts of varying conditions Can be used for more than development testing Patch testing Vulnerability Assessment Recovery exercises Configuration change testing 40

41 Testing Methods Document failed test reports and subsequent fixes to be re-tested Document results Sign-off in test plan Maintain records of output results Documented test plan Agreed-to ahead of time Forces test to functionality, not to what was implemented Repeatable at will Unit test Integrated test Exception Testing Automated test scripts 41

42 Training People Tools Recovery Procedures Incident response Communications Reporting 42

43 Summary Plan for unexpected events Test, Test, Test Effective Communications Training Practice, Practice, Practice Documentation, documentation, documentation 43

44 Questions & Answers Contact Information: The magazine for building a smart grid and delivering information-enabled energy. FREE subscriptions available at 44

45 Questions & Answers Contact Information: Your source for IT and smart grid research, analysis, and consulting. Visit 45

46 Questions & Answers Contact Information: Go to where the power industry gathers for news, information, and analysis, visit com 46

47 Questions & Answers Contact Information: 47 Get the inside scoop with Energy Central Professional News Service. Start your FREE trial at

48 Questions & Answers Contact Information: Join the discussion, raise your question, and voice your opinion at 48

49 Questions & Answers Contact Information: The magazine for C-level executives about the business of energy. FREE subscriptions available at 49

50 CIP Compliance Series Webcasts For comprehensive preparation for the implementation, compliance, and auditing phases of the CIP standards program, attend all six. Upgrade and save 10%. Apply your single event purchase to the cost of the entire series. Call or for information. Date Topic 9/23/09 Identifying Critical Assets (On Demand) 10/6/09 Program Governance Issues (On Demand) 10/21/09 Change Management Systems (On Demand) 11/11/09 Personnel Issues & Training (On Demand) 12/2/09 Physical & Electronic Access Controls (On Demand) 12/16/09 Testing Procedures & Recovery Plans (On Demand) 50

51 Thank You for Joining Us For the latest news, articles and blogs, please visit