AMI Cyber Security Incident Response Guidelines

AMI Cyber Security Incident Response Guidelines 1026554

AMI Cyber Security Incident Response Guidelines 1026554 Technical Update, December 2012 EPRI Project Manager G. Rasche ELECTRIC POWER RESEARCH INSTITUTE 3420 Hillview Avenue, Palo Alto, California 94304-1338 PO Box 10412, Palo Alto, California 94303-0813 USA 800.313.3774 650.855.2121 askepri@epri.com www.epri.com

DISCLAIMER OF WARRANTIES AND LIMITATION OF LIABILITIES THIS DOCUMENT WAS PREPARED BY THE ORGANIZATION(S) NAMED BELOW AS AN ACCOUNT OF WORK SPONSORED OR COSPONSORED BY THE ELECTRIC POWER RESEARCH INSTITUTE, INC. (EPRI). NEITHER EPRI, ANY MEMBER OF EPRI, ANY COSPONSOR, THE ORGANIZATION(S) BELOW, NOR ANY PERSON ACTING ON BEHALF OF ANY OF THEM: (A) MAKES ANY WARRANTY OR REPRESENTATION WHATSOEVER, EXPRESS OR IMPLIED, (I) WITH RESPECT TO THE USE OF ANY INFORMATION, APPARATUS, METHOD, PROCESS, OR SIMILAR ITEM DISCLOSED IN THIS DOCUMENT, INCLUDING MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE, OR (II) THAT SUCH USE DOES NOT INFRINGE ON OR INTERFERE WITH PRIVATELY OWNED RIGHTS, INCLUDING ANY PARTY'S INTELLECTUAL PROPERTY, OR (III) THAT THIS DOCUMENT IS SUITABLE TO ANY PARTICULAR USER'S CIRCUMSTANCE; OR (B) ASSUMES RESPONSIBILITY FOR ANY DAMAGES OR OTHER LIABILITY WHATSOEVER (INCLUDING ANY CONSEQUENTIAL DAMAGES, EVEN IF EPRI OR ANY EPRI REPRESENTATIVE HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES) RESULTING FROM YOUR SELECTION OR USE OF THIS DOCUMENT OR ANY INFORMATION, APPARATUS, METHOD, PROCESS, OR SIMILAR ITEM DISCLOSED IN THIS DOCUMENT. REFERENCE HEREIN TO ANY SPECIFIC COMMERCIAL PRODUCT, PROCESS, OR SERVICE BY ITS TRADE NAME, TRADEMARK, MANUFACTURER, OR OTHERWISE, DOES NOT NECESSARILY CONSTITUTE OR IMPLY ITS ENDORSEMENT, RECOMMENDATION, OR FAVORING BY EPRI. THE FOLLOWING ORGANIZATION, UNDER CONTRACT TO EPRI, PREPARED THIS REPORT: Southwest Research Institute (SwRI) This is an EPRI Technical Update report. A Technical Update report is intended as an informal report of continuing research, a meeting, or a topical study. It is not a final EPRI technical report. NOTE For further information about EPRI, call the EPRI Customer Assistance Center at 800.313.3774 or e-mail askepri@epri.com. Electric Power Research Institute, EPRI, and TOGETHER SHAPING THE FUTURE OF ELECTRICITY are registered service marks of the Electric Power Research Institute, Inc. Copyright 2012 Electric Power Research Institute, Inc. All rights reserved.

ACKNOWLEDGMENTS The following organization, under contract to the Electric Power Research Institute (EPRI), prepared this report: Southwest Research Institute (SwRI) Automation and Data Systems Division 6220 Culebra Road, Building 68 San Antonio, TX 78238-5166, USA Principal Investigators T. Do G. Ragsdale W. Arensman This report describes research sponsored by EPRI. EPRI would like to thank the following organization for providing insight on the research problem and feedback on the report: UCAIug SG Security Working Group This publication is a corporate document that should be cited in the literature in the following manner: AMI Cyber Security Incident Response Guidelines. EPRI, Palo Alto, CA: 2012. 1026554. iii

ABSTRACT This document is intended to be used by system and asset owners to assist in the preparation and response to AMI cyber security incidents. This document was developed by conducting interviews with EPRI members, AMI asset owners, and vendors, regarding practices involved in responding to AMI cyber security incidents and mapping the responses to requirements put forth by the Department of Homeland Security (DHS), National Institute of Standards and Technology (NIST), Open Smart Grid (Open-SG) Working Group, and Advanced Metering Infrastructure Security (AMI-SEC) working group. Keywords Cyber Security Incident Response Advanced Metering Infrastructure v

CONTENTS 1 INTRODUCTION... 1-1 1.1 Usage... 1-1 1.2 References... 1-2 2 REFERENCES AND BIBLIOGRAPHIES... 2-1 2.1 List of References... 2-1 2.2 Definition of Terms... 2-1 3 AMI INCIDENT RESPONSE PREPARATION GUIDELINES... 3-1 3.1 Incident Response Organization... 3-1 3.1.1 AMI Entity Identification and Oversight [9 SG.IR-1]... 3-1 3.1.2 Incident Response Program [9 SG.IR-1]... 3-2 3.1.3 AMI Incident Response Development Facility [9 SG.IR-3, SG.IR-4]... 3-3 3.1.4 Incident Response Roles and Responsibilities [9 SG.IR-2]... 3-4 3.2 Incident Response Planning... 3-5 3.2.1 Defining AMI Cyber Incident Response Requirements [3 4.1; 9 SG.IR-1, SG.AT-1]... 3-5 3.2.2 Incident Scenarios Identification [9 SG.IR-1, SG.AT-1]... 3-5 3.2.3 Establishing a Continuity of Operations Plan [1-2.12.2; 9 SG.CP-2]... 3-6 3.2.4 Identifying Roles and Responsibilities for Continuity of Operations [9 SG.AT-6]... 3-7 3.3 Risk and Impact Assessment... 3-9 3.3.1 Identifying Internal Objectives and Functions... 3-9 3.3.2 AMI Incident Classification [3-2]... 3-9 3.3.3 Impact Ranking of AMI Functions [4-2.3]... 3-10 3.3.4 AMI Incident Impact Classification [4-2.3]... 3-11 3.3.5 AMI Incident Impact Analysis [4-2.3]... 3-13 3.4 Incident Response Training... 3-16 3.4.1 Incident Process Training [1-2.7.5; 3-2.12.1; 9 - SG.AT-6]... 3-16 3.4.2 Process Verification Testing [1-2.7.6]... 3-16 4 AMI INCIDENT RESPONSE GUIDELINES... 4-1 4.1 Incident Management Process [9 SG.IR-5]... 4-1 4.1.1 Incident Identification... 4-2 4.1.2 Incident Remediation... 4-6 4.1.3 Incident Review... 4-8 5 CONCLUSION... 5-1 vii

LIST OF FIGURES Figure 3-1 AMI Incident Impact Analysis Flowchart... 3-14 Figure 4-1 AMI Incident Management Process... 4-2 ix

LIST OF TABLES Table 3-1 Scope of Impact... 3-11 Table 3-2 AMI Incident Impact Classification... 3-12 Table 3-3 Likelihood of Occurrence... 3-14 Table 3-4 Risk Rating... 3-15 Table 3-5 AMI Incident Type... 3-15 Table 4-1 Incident Scenario Description... 4-3 Table 4-2 Physical Incident Scenarios... 4-4 Table 4-3 Authentication Incident Scenarios... 4-4 Table 4-4 Communications Incident Scenarios... 4-5 Table 4-5 Software Incident Scenarios... 4-6 Table 4-6 Incident Remediation Strategies... 4-7 xi

1 INTRODUCTION The following document provides cyber security incident response guidelines and recommendations specific to Advanced Metering Infrastructure (AMI) systems. The document assumes that incident response guidelines and practices already exist for enterprise systems. The guidelines and practices described are specific to the unique components comprising the AMI system, hereafter referred to as the AMI logical architecture as defined in [3 4.2]. (See Sections 1.2 and 2.1 for reference descriptions.) AMI logical architectures are different from other energy-related system architectures in several regards. The AMI logical architecture exists within the distribution grid and is closely associated with the utility s customers. As the AMI is concerned with energy distribution and not transmission it does not fall under the scope of North American Electric Reliability Corporation (NERC) Critical Infrastructure Protection (CIP), Independent System Operator (ISO), or other bulk energy controls. Instead, the AMI is more subject to mass media and customer scrutiny. The AMI also differs from other energy-related systems in that major components of the AMI, such as the AMI meter operate outside the fence lines of utilities in areas that are easily physically accessible as compared to traditional electrical distribution grid components (e.g., substations and transformers). In some instances, AMI meters and web portals interact directly with customer-provided appliances and energy management systems using well-understood protocols. This document is dedicated to addressing these differences to the extent that the differences relate to AMI incident response. The document does not address incident response within the AMI-associated business systems that are external to the AMI logical architecture as described in [3 4.2]. It is also assumed that the reader has access to and will become familiar with applicable standards cited in this document. The standards are included by citation only. The reader is encouraged to reference and become familiar with standards applicable to AMI systems. 1.1 Usage This format of this document closely follows that of the National Institute of Standards and Technology (NIST) Special Publication (SP) 800-53, Information Security document [5] and the U.S. Department of Homeland Security, Catalog of Control Systems Security: Recommendations for Standards Developers document [1]. This document adopts the convention of adding a supplemental guidance to each requirement from the Advanced Security Acceleration Project for Smart Grid (ASAP-SG) Security Profile 2.0 document [3]. In this document, the supplemental guidance section contains the identified industry guidelines for implementing the requirement. 1-1

Requirements are listed in the following format: Requirement Title Description Requirement Supplemental Guidance Requirements Enhancement Rationale This document is divided into two major sections: AMI Incident Response Preparation Guidelines and AMI Incident Response Guidelines. The AMI Incident Response Preparation Guidelines section lays the organizational groundwork for developing policies to respond to Advanced Metering Infrastructure (AMI) cyber security incidents. The AMI Incident Response Guidelines section provides guidance on how to best respond to identified AMI cyber security threats. 1.2 References References throughout this document adopt the following format [<Reference #> - <Section #>]. <Reference #> - Refers to the document source listed in the reference section <Section #> - Refers to the specific section within the reference source The absence of a <Section #> within the citation implicitly refers to the document in its entirety. 1-2

2 REFERENCES AND BIBLIOGRAPHIES 2.1 List of References 1. U.S. Department of Homeland Security, Catalog of Control Systems Security: Recommendations for Standards Developers, April 2011. 2. U.S. Department of Commerce. National Institute of Standards and Technology, Computer Security Incident Handling Guide, Special Publication 800-61, Revision 1. 3. The Advanced Security Acceleration Project (ASAP-SG) - Security Profile For Advanced Metering Infrastructure, Version 2.0, June 22, 2010. 4. AMI-SEC (Advanced Metering Infrastructure Security) Task Force - AMI Risk Assessment (draft document), AMI-SEC Risk Assessment v0.9a-20080319.doc. 5. NIST Special Publication (SP) 800-53, Information Security, Revision 3, Updated 5/1/2010, http://csrc.nist.gov/publications/nistpubs/800-53-rev3/sp800-53-rev3- final_updated-errata_05-01-2010.pdf. 6. UCAIUG: AMI SEC-ASAP AMI System Security Requirements V1.01, December 17, 2008. 7. EPRI IECSA Volume II, Final Release, EPRI Use Case Repository, EPRI.com. 8. AMI Network V 3.0.doc, EPRI Use Case Repository, EPRI.com. 9. The Smart Grid Interoperability Panel (SGIP) Cyber Security Working Group (CSWG), NISTIR 7628 Guidelines for Smart Grid Cyber Security: Vol. 1, Smart Grid Cyber Security Strategy, Architecture and High-Level Requirements, August 2010. 2.2 Definition of Terms AMI entity the owner and/or operator of the AMI logical architecture (e.g., cooperatives, municipal agency, or a corporate body). Use case a set of related activities and communications that render value in support of an AMI objective. AMI function a set of use cases supporting the achievement of a common AMI objective (e.g., billing). AMI component a subdivision of the AMI logical architecture primarily involved in one or more AMI use case communications or sequence of activities. AMI components may secondarily participate in non-ami use cases. AMI components are described in [3-4.2]. AMI system synonymous with AMI internal architecture are described in [3-4.2]. AMI-associated components are subdivisions of the AMI logical architecture external to the AMI internal architecture (e.g., Outage Management System, Customer Information System, and 2-1

Distribution Automation System). The AMI associated components are depicted as the AMI logical architecture external perspective in [3-4.2]. AMI logical architecture [3-4.2] In this document when the term AMI logical architecture is used, we are referring to the AMI logical architecture internal perspective. The components which are encompassed within the AMI logical architecture internal perspective are: AMI Meters Field Tools AMI Communications Network Devices AMI Head End Meter Data Management System AMI Network Management System AMI Meter Management System Level of effort is a measure of resources required to achieve an end (e.g., successful attack on an Advanced Metering Infrastructure (AMI) asset. Resources may include combinations of time, skill, money, and access to technology). Loss of Control The inability to exercise AMI control functions (e.g., remote disconnect). Loss of Communications The inability to communicate with the AMI (e.g., send/receive commands from a meter). Loss of Trust/Integrity The inability to verify the security of functions or data within the AMI (e.g., compromise of keys, integrity check failures, billing data compromise). Loss of Privacy The inability to protect the privacy of information (e.g., compromise of keys, loss of encryption functionality). Loss of Attribution The inability to attribute an AMI action (e.g., meter log on) to its actor. Natural Event An event which can be correlated with other situational awareness data (e.g., weather forecast, fires, earthquakes). These events are generally caused by acts of nature. Man-Made Event An event not caused by nature and is generally caused as a result of a manmade event. Malicious Event An event which has been caused with ill-intent. It generally occurs in the absence of natural or unintentional events. These events are identified by correlating the event with other situational awareness data (e.g., meter tampers, meter authentication attempts). Depending on the severity of the event, the AMI entity may need to notify law enforcement (e.g., local, federal).types of events malicious events include: AMI Cyber Attack Vandalism Energy Theft 2-2

Non-Malicious Event An event which has been caused unintentionally or as the result of operator or system design error. These events generally arise as a result of AMI operation, misconfiguration, or design errors. Objective a level of achievement necessary for a desired outcome or state of AMI entity business activity (e.g., customer communication). 2-3

3 AMI INCIDENT RESPONSE PREPARATION GUIDELINES The purpose of Section 3 and its subsections are to define the preparation guidelines for planning the incident response procedures defined in Section 4. The guidelines provide the organizational framework upon which incident response procedures are rationally created, validated, and justified. For this section, a subset of guidelines has been selected which pertain specifically to the unique aspects of the AMI such as: AMI meters operate in geographically distributed and easily physically accessible locations (e.g., homes, apartment buildings, and strip malls). AMI meters may contain remote connect/disconnect capabilities. AMI meter data may be used in organizational demand response programs. AMI systems often contain mesh communication networks allowing for the potential to leverage the network to attack a large number of meters. The guidelines in this section should be considered as a supplement to any existing organizational incident response plan and should be used to augment incident response approaches as applicable to AMI. This section is divided into four subsections: incident response organization, incident response planning, risk and impact assessment, and training. Please note that the requirements referenced in the following sections from the National Institute of Standards and Technology Interagency Report (NISTIR) are not specific to AMI. Supplemental guidance has been provided to tailor these requirements to AMI. 3.1 Incident Response Organization This subsection describes the requirements and provides guidance on how to organize an AMI incident response plan. 3.1.1 AMI Entity Identification and Oversight [9 SG.IR-1] The responsibility and accountability for AMI incidents fall to the AMI entity which has responsibility and authority over the entity s AMI logical architecture. An individual or a group of individuals responsible for the different domains within the AMI entity should be identified and charged with leading the development of the AMI incident response program. It is important for this individual leader, or collective group of leaders, to take responsibility for the planning and implementation of cyber security incident response processes and procedures. 3-1

3.1.1.1 Requirement An individual or a collective group of individuals with authority that encompasses all aspects of the AMI logical architecture should oversee the AMI incident response team and process. AMI entities may subdivide leadership of the AMI into different operational domains. Examples of such operational domains are: Business Processes Responsible for handling the business operations of the AMI logical architecture such as revenue assurance and billing. AMI Operations Responsible for day-to-day operation of the AMI internal architecture such as maintenance and implementation of the meter data management system. Information Technology (IT) Operations Responsible for overall management of the AMI associated components such as servers, backend appliances, and the organization s security information and event management system (SIEM). For each identified operational domain within the organization, a leader should be identified to coordinate the incident response process as it pertains to their domain. 3.1.1.2 Supplemental Guidance The designated leader within the AMI entity directs the formation of the incident response capability and governs the administration of the capability once it is operational. The AMI incident response designated leader has approval authority for incident plans, policies, and procedures [1-2.2, 2.3]. 3.1.1.3 Requirements Enhancement None 3.1.2 Incident Response Program [9 SG.IR-1] The AMI logical architecture realizes use cases, thereby achieving business objectives specific to the automated collection of meter data, provisioning of power to a customer, and ensuring the integrity of the entity-customer relationship. AMI-specific use cases must be realized in a timely, reliable, and secure fashion for the entity s business objectives to be achieved. The purpose of the AMI incident response program is to sustain or restore the AMI use cases despite incidents that would otherwise disrupt operation of the AMI logical architecture. An AMI incident response program addresses the unique cyber security risks with a corresponding capability to continue or resume operations of an AMI system in the event of disruption of AMI use cases and objectives. 3-2

The AMI incident response program entails the preparation, testing, and maintenance of AMIspecific policies, procedures, processes, systems, and skill sets. The program enables the AMI entity to restore the AMI system s operational status after the occurrence of a disruption. Disruptions can come from natural disasters, such as earthquakes, tornados, floods, or from manmade events like riots, terrorism, or vandalism. The ability for the AMI system to function after such an event is directly dependent on implementing a program prior to incidents using the organization s planning process. [1-2.12]. 3.1.2.1 Requirement The AMI entity establishes an AMI incident response program as an identifiable and recognized organizational function. 3.1.2.2 Supplemental Guidance The official establishment and visibility of the AMI incident response program as a formal and unique activity is important to its adoption and support within the AMI entity. 3.1.2.3 Requirements Enhancement None 3.1.3 AMI Incident Response Development Facility [9 SG.IR-3, SG.IR-4] AMI logical architectures are complex internal and associated components satisfying AMI entity goals including reliability, integrity, and functionality. Like other systems, it is desirable to develop and verify incident response programs without risk to an operational AMI logical architecture. Therefore, it is good practice to use an AMI incident response development facility separate from the production AMI system. 3.1.3.1 Requirement A facility exists as a development and test vehicle for the creation, maintenance, and verification of AMI incident response policies, procedures, training, processes, and components. 3.1.3.2 Supplemental Guidance AMI systems have distinct differences from enterprise systems often found within an AMI entity and compose of systems such as: Electric Meters RF, Wireless, Wired, and Cellular Communications Networks Signing and Encryption Appliances Meter Data Management Systems (MDMS) Additionally, many components of the AMI system operate in potentially harsh and physically unsecured environments. The facility should, to the extent possible, replicate the unique aspects of the AMI system and its environment. The facility may consist of functional test beds used in development to test configurations of meters and systems prior to wide-scale deployments. 3-3

3.1.3.3 Requirements Enhancement None 3.1.4 Incident Response Roles and Responsibilities [9 SG.IR-2] The skill sets required to develop, test, and execute an AMI incident response program are many and in many cases unique to the AMI logical architecture. Likewise, the duties to be performed within the program are often unique to the AMI system. For example, the understanding of the AMI meter and its communications networks (e.g., wide-area and neighborhood area) is a skill set different from a typical IP-based enterprise system. The designated roles to respond to a meter intrusion or an AMI network denial-of-service attack are also different as a meter technician may require training in non-ip communication protocols, AMI hardware configuration parameters, AMI diagnostic techniques, and cyber security methods. 3.1.4.1 Requirement The duties, skill sets, and responsibilities of roles within the AMI incident response program are clearly defined and communicated within the AMI entity. 3.1.4.2 Supplemental Guidance Each organization may subdivide its AMI leadership into different functional domains. The AMI incident response team should consist of members from each of these functional domains and should include representatives from domains relating to the AMI-associated components. The following groups should be considered in the formation of a cross-functional team that plans for and responds to AMI incidents: Business (e.g., Legal, Accounting, Human Resources, Public Relations) Charged with responding to the business impacts of the AMI incident. AMI Operations Charged with responding to the operational impacts of the AMI incident. Information Technology (IT) Operations Charged with responding to the impacts of the AMI incident as it relates to the IT system. The cross-functional team may be adjusted depending on the scope of impact of the AMI incident. 3.1.4.3 Requirement Enhancement The AMI framework is an integral part of the larger organizational framework. The organizational framework includes AMI incident response roles with well-defined skills, accountability, and responsibilities. 3-4

3.2 Incident Response Planning This section provides requirements and guidance on how to plan and prepare for an AMI incident. 3.2.1 Defining AMI Cyber Incident Response Requirements [3 4.1; 9 SG.IR-1, SG.AT-1] The AMI system is implemented to satisfy a set of AMI-specific use cases and derive business value. Typical use cases can be found within the Advanced Security Acceleration Project (ASAP-SG) - Security Profile for Advanced Metering Infrastructure, Version 2.0, document [3 4.1]. The document provides traceability of AMI requirements to business use cases and associated business objectives supported by the AMI system. The goal of the AMI incident response program is to satisfy AMI requirements associated with data integrity, reliability, timeliness, privacy, and accessibility when incidents disrupt the realization of AMI use cases. A thorough understanding of AMI requirements is necessary for the proper formulation of policies, procedures, training, and resources comprising the AMI incident response program. 3.2.1.1 Requirement The AMI incident response program satisfies as set of clearly defined and articulated requirements traceable to the AMI system use cases and business objectives. 3.2.1.2 Supplemental Guidance The AMI use cases produce a set of business objectives when successfully achieved or a set of business impacts when disrupted by an incident. AMI incident response requirements should be considered in light of the business objectives and impacts associated with a given incident scenario. 3.2.1.3 Requirement Enhancements None 3.2.2 Incident Scenarios Identification [9 SG.IR-1, SG.AT-1] A proactive stance toward AMI incidents requires the AMI program should anticipate incidents before they are encountered. The incident scenario analysis considers the impacts upon use cases and business objectives caused by benign or malicious attacks on the AMI meters, communication devices, head end, forecasting system, meter management system, network, management system, and other AMI components as defined in [3-4.3]. 3.2.2.1 Requirement The AMI incident response program identifies AMI incident scenarios and the possible causes for those AMI incidents. The incident scenarios should identify forensic evidence required for detection of the AMI incident. The AMI incident response program should anticipate AMI incident scenarios which may not yet occurred. 3-5

3.2.2.2 Supplemental Guidance In order to proactively identify AMI incidents, it is important that the organization develop an overall AMI incident response plan that provides a process for responding to AMI security incidents regarding vulnerabilities which have been identified and have not yet been remediated, as well as vulnerabilities for which the organization has accepted the risk. The AMI incident response plan should include methods for assigning incident impact and assetvalue metrics (e.g., monetary or objective-oriented). The AMI incident plan should include practices for: Reviewing and updating incident response procedures How to respond to newly released vulnerabilities Metrics for determining the impact of newly identified vulnerabilities Methods for determining the type of AMI incident (e.g., malicious, non-malicious, natural) The risk analysis described in Section 3.3.5 should be applied to scenarios after the plausibility of the scenario has been adequately established through examination or actual occurrence. Additionally, the AMI incident response plan should include: Risk mitigation strategies Policies and procedures for responding to incidents based on the impact and type of AMI incident 3.2.2.3 Requirement Enhancement None 3.2.3 Establishing a Continuity of Operations Plan [1-2.12.2; 9 SG.CP-2] A continuity of operations plan ensures the resumptions of services in the event of an AMI incident. 3.2.3.1 Requirement The organization should develop, test, and implement a continuity of operations plan dealing with the overall issue of maintaining or re-establishing operation of the AMI system in case of an undesirable interruption. The plan should address roles, responsibilities, points of contact, and activities associated with restoring system operations after a disruption or failure. Designated officials within the organization review and approve the continuity of operations plan. Individuals responsible for the systems covered by the continuity of operations plan should be trained on the implementation and execution of the continuity of operations plan. 3-6

3.2.3.2 Supplemental Guidance A continuity of operations plan addresses both business continuity planning and recovery of all vital AMI system operations. 3.2.3.3 Requirements Enhancement The continuity of operations plan may include: Verification of communications with all AMI meters, collectors, and relays The restoration of disconnect switches to known states Verification of the integrity of meter firmware Verification of the integrity of meter data The continuity of operations plan should also address: Unintentional AMI cyber security incidents (e.g., operator error, accidents) Natural events (e.g., high velocity winds, flooding, earthquakes) Intentional incidents as a result of: Disgruntled current and past employees Hackers 3.2.3.3.1 Rationale Experience demonstrates that systems fail and mistakes occur. An AMI continuity of operations plan allow for an orderly recovery from such situations. Critical analysis of a system often brings weaknesses to light, thereby giving the AMI incident response program opportunities to design additional safeguards into the AMI system necessary for an orderly recovery. 3.2.4 Identifying Roles and Responsibilities for Continuity of Operations [9 SG.AT-6] The organization s continuity of operations plan should define and communicate the specific roles and responsibilities for each part of the plan in relation to various types of disruptions to the operation of the AMI system. 3.2.4.1 Requirement The organization s continuity of operations plan defines and communicates the specific roles and responsibilities for each part of the plan in relation to various types of AMI incidents. [1 2.12.3]. 3.2.4.2 Supplemental Guidance An AMI incident continuity of operations plan defines the roles and responsibilities of the various employees and contractors in the event of an incident. The plan identifies responsible personnel to lead the recovery and response effort if an incident occurs. The following roles and responsibilities are representative of the organizational domains involved with AMI and, as a result, they should be considered within the continuity of operations plan: 3-7

Incident Coordinator Responsible for organizing, dispatching, and coordinating the incident response team Trained in the incident response policies, procedures, roles, responsibilities, and goals AMI Operations Lead Responsible for the day-to-day operations of AMI system within the organization Assists in identifying key personnel within the AMI operations group to assist in the incident response process Leads efforts to quantify and categorize incident impacts Trained in the AMI functions, use cases, goals, metrics, and incident response procedures IT Operations Lead Responsible for the day-to-day operations of information systems within the organization Assists in identifying key personnel within the IT operations group to assist in the incident response process Assists the AMI Operations lead in the quantification and categorization of incident impacts Trained in information technology and enterprise cyber security Business Operations Lead Responsible for the day-to-day business operations within the organization Assists in contacting the affected business entities within the organization Assists the AMI lead in the quantification and categorization of incident impacts Leads the mapping of the AMI entity's organizational objectives into cyber security metrics Trained in the AMI entity s planning and budgeting Public Relations Lead Responsible for correspondence with the media, regulators, and other external groups Trained in the legal entity s external communications, reporting, and marketing policies and practices Legal Lead Responsible for liaising with law enforcement and regulatory agencies regarding planning and responding to AMI cyber security incidents Trained in judicial evidentiary data collection and custodial control requirements, external reporting procedures and policies, and regulation governing cyber security incidents These generic categories were compiled from the interviews conducted with EPRI members, AMI asset owners, and vendors regarding the practices involved for responding to AMI cyber security incidents. Each organization may have different titles and categories for these roles and should make their best determination in mapping the provided categories to their organization. 3-8

3.2.4.3 Requirements Enhancement None 3.2.4.3.1 Rationale Specific roles and responsibilities within continuity plans help alleviate confusion in the event of a significant incident. This clarification can ease the process and better focus the organization when recovering from a disruption. 3.3 Risk and Impact Assessment This section provides requirements and guidance on how to perform a risk and impact assessment with the goal of assisting the AMI entity in prioritizing incident response efforts. 3.3.1 Identifying Internal Objectives and Functions AMI entity objectives should be considered as part of the risk and impact assessment. Methods and use cases for those objectives should identify the tangible and intangible benefits to the AMI entity. 3.3.1.1 Requirements The organization identifies AMI entity objectives and the use cases implemented to achieve those goals. 3.3.1.2 Supplemental Guidance An AMI entity should identify internal objectives and functions that would be affected by an AMI incident and should consider those objectives when assigning a risk and impact rating to the identified incidents. Examples of AMI entity internal objectives that may be affected by AMI incidents are: Real-Time Pricing (RTP) Top Level [7 D19] Remote Connect/Disconnect [6 2.1.1] Demand Response Utility Commanded Load Control [7 D12] AMI Network (Moving Data Elements from the AMI Head-End to Smart Meter and from the Smart Meter to the AMI Head-End) [8] Firmware Updates [9 2.3.17] 3.3.1.3 Requirements Enhancement AMI entity objectives and use cases are impacted by AMI incidents in varying degrees. Use cases affected by AMI cyber-security incidents are identified and associated with organizational objectives. 3.3.2 AMI Incident Classification [3-2] The AMI program should define common methods, vernacular, and criteria for describing a given AMI incident. Incidents may be classified according to common incident characteristics (e.g., failure mechanisms, mitigation methods, resource requirements, impact assessment ranking). Categorizing incidents facilitates better planning, organization, and uniformity within the incident response program. 3-9

3.3.2.1 Requirement AMI incidents are subdivided into categories and classified according scope of impact, likelihood of occurrence, and incident type. 3.3.2.2 Supplemental Guidance The following criteria should be used to describe AMI incidents: Scope of Impact: A classification based on the number of systems affected by the AMI incident. Scope of impact ratings can be used to determine an incident s severity and impact on the operation of the AMI logical architecture. Likelihood of Occurrence: A classification based on the current threat environment of how likely the AMI incident is to occur. Likelihood of occurrence ratings can be used to reprioritize responses to AMI incidents. Type: A classification based on the intent of the perpetrator of the AMI incident (e.g., natural, manmade, malicious, or non-malicious). 3.3.2.3 Requirements Enhancement None 3.3.3 Impact Ranking of AMI Functions [4-2.3] The AMI program should assess the level of AMI entity concern associated with the loss of an AMI function. 3.3.3.1 Requirement AMI incident impact rankings are assigned to assets, functions, or use cases according to the consequence and severity of an incident that disrupts portions or all of the functional capability. 3.3.3.2 Supplemental Guidance AMI incident impact rankings should be developed through threat assessment exercises. The threat assessment exercise should enumerate all components of the AMI logical architecture, enumerate the components interactions with external systems, assess the potential vulnerabilities within each component and component interface, and evaluate the impact (e.g., organizational and operational) of the vulnerability being exercised. Example impact ranking definitions are provided below in Table 3-1. 3-10

Table 3-1 Scope of Impact Scope of Impact 1 Low 2 Medium 3 High The incident has a low impact limited to a single unit (e.g., Electric Meter). Measures should be taken to address the threat related to the single unit. The incident has a medium impact limited to units in a single grouping of AMI logical architecture components (e.g., a meter cell). Measures should be taken to identify the root cause of the threat and remediate the threat across all affected systems. The incident has a high impact and the effects span the entire AMI logical architecture (e.g., meter, routers, head end). Measures should be taken to immediately contain the threat and a crossfunctional team of AMI operators, IT system operators, AMI vendor, and business representatives should be formed to remediate the threat. The scope of impact ratings listed here is provided as an example. Scope of impact for vulnerabilities should be determined on a case-by-case basis depending on the function and number of devices impacted. 3.3.3.3 Requirements Enhancement None 3.3.4 AMI Incident Impact Classification [4-2.3] The AMI incident response program should classify AMI incidents based upon the impact of the AMI incident to AMI entity objectives. 3.3.4.1 Requirement Incidents are assigned an impact classification according to the AMI entity objective impacted by an incident. 3.3.4.2 Supplemental Guidance Table 3-2 provides an example mapping of AMI incidents to impact classifications based upon incident s scope of impact. It is recommended that the reader formulate their own classification of AMI incidents and tailor it based on the unique aspects of their AMI deployment (e.g., usage of C12.18 keys) and their organization s AMI objectives. 3-11

Table 3-2 AMI Incident Impact Classification ID AMI Incident Classification Description 1 Theft of Energy An attacker attempts to steal energy by modifying the meter and/or the metering data. 2 Theft of C12.18 Logon Credentials 3 Bypass of Security Controls 4 Failed Communications Integrity An attacker has stolen the C12.18 login credentials from a single meter. An attacker has bypassed security controls on the meter. An integrity check failure has occurred on the meter. Impact Classification Low The attack is associated with a single occurrence of energy theft and has negligible impact. Medium The attack is associated with multiple occurrences of energy theft and has a direct monetary impact. High The attack is associated with a largescale theft of energy and can have both monetary and functional impacts Low The C12.18 credentials are different for each meter and have a minimal impact. Medium The C12.18 credentials are by multiple meters but are limited due to the physical requirement for exploitation. High The C12.18 credentials are in use by multiple meters and also have a high scope of impact due to the ability to perform the attack wirelessly or remotely. Low The bypass of security credentials only affects a single device at a time and has a limited impact. Medium The bypass of security controls affects multiple devices but has limited impact in terms of control. High The bypass of security controls affects multiple devices and allows full bypass of security controls on the affected devices. Low The integrity check failure occurs in a limited number of instances and affects a limited number of meters. Medium The integrity check failure occurs on a large scale and may affect the quality of data being received from the meters. High The integrity check failure occurs across all data being received from the meters. Thresholds on number of meters affected should be defined by each organization to rank the impact of the incident. 3-12

Table 3-2 (continued) AMI Incident Impact Classification ID AMI Incident Classification Description 5 Power Outage An AMI incident results in a power outage. 6 Software Error An AMI incident results in a software error occurring on the meter. Impact Classification Low A limited number of customers are affected (e.g., single household or business). Medium Multiple customers are affected but the effects are limited and isolated to a specific geographical area (e.g., neighborhood). High A large-scale power outage occurs affecting a significant portion of the AMI deployment (e.g., city or state). High impact outages may also include life and safety critical services (e.g., hospitals and emergency first responders). Low The software error occurs in a limited number of devices and has a low impact (e.g., causes a reset or reboot). Medium The software error occurs in multiple meters and has a high impact (e.g., toggling of disconnect switch, denial of service) High The software error occurs a significant portion of the AMI deployment (e.g., city or state) and has a high impact (e.g., toggling of disconnect of switch). 3.3.4.3 Requirements Enhancement None 3.3.5 AMI Incident Impact Analysis [4-2.3] Define the methods for assessing the value of the AMI function to an AMI entity in terms that can also be used to quantify incident impact if part or all of the AMI logical architecture are rendered ineffective. 3.3.5.1 Requirement The AMI incident impact metrics assigned to the AMI incidents quantify incident response risk exposure. 3.3.5.2 Supplemental Guidance The AMI entity should identify the AMI functions and use cases to be included in the incident impact analysis. The following methodology shown in Figure 3-1 may be adopted in assigning risk rankings to AMI assets. 3-13

Figure 3-1 AMI Incident Impact Analysis Flowchart First, an impact ranking is assigned to AMI assets using a methodology such as that defined in the Table 3-1. Next, an exercise is performed where AMI incidents are defined and categorized. AMI incidents are assigned a scope of impact ranking based on the number AMI assets they affect. Finally, a likelihood of occurrence rating is assigned to the AMI incident based on historical evidence of similar types of incidents. An example of likelihood of occurrence ratings to be used is provided below in Table 3-3. The likelihood of occurrence table provided is a simplified version based off of the AMI-SEC likelihood interpretation policy [4 2.6.1]. Table 3-3 Likelihood of Occurrence Likelihood of Occurrence 1 Low Not expected to occur, or may occur in exceptional circumstances. 2 Medium Could occur at some time. 3 High Will probably occur in most circumstances. Once the AMI incidents have been defined, categorized, and assigned an impact and likelihood of occurrence rating, a risk rating can be determined by using a matrix such as shown in Table 3-4. 3-14

Table 3-4 Risk Rating Risk Rating Likelihood of Occurrence Scope of Impact Low Medium High 1 2 3 1 Low Low Medium Medium 2 Medium Low Medium High 3 High Medium High High The risk ratings help to guide planning, budgeting, and resource allocation processes to address the issues of highest concern and impact. Additional metrics may be included in the risk rating in the event of an incident response such as the type of AMI incident. The following type classification in Table 3-5 has been provided as an example: Table 3-5 AMI Incident Type Type 1 Malicious 2 Non- Malicious 3 Natural The incident has been determined to have been caused by a malicious actor. Additional measures may be taken to contain the threat until the threat has been remediated. The incident has been determined to have been caused by a non-malicious actor such as an operational error. Depending on the severity of the error, measures may be taken to contain the incident impact. The incident has been determined to have been caused by a natural event such as an earthquake or storm. Normal operational procedures should be utilized to contain and remediate the incident. Such ratings are dynamic and may not be able to be determined prior to an incident occurring; however, it may be used in adjusting the incident response in the course of investigating an AMI incident. 3.3.5.3 Requirements Enhancement None 3-15

3.4 Incident Response Training This section provides requirements and guidance on how to develop and assess incident response training. 3.4.1 Incident Process Training [1-2.7.5; 3-2.12.1; 9 - SG.AT-6] The AMI incident response program should identify the types of training and materials required to prepare the AMI entity, its personnel, and those affected by AMI incidents. 3.4.1.1 Requirement The program includes training on the creation and implementation of the AMI incident response plans for employees, contractors, and stakeholders. The program provides refresher training annually. The training covers employees, contractors, and stakeholders in the implementation of the continuity of operations plan. 3.4.1.2 Supplemental Guidance Training needs to be provided to individuals in the AMI community so that all understand the content, purpose, and implementation of the incident response plans. Different levels of incident response training may be prepared for personnel with different levels of AMI responsibility. Refer to 3.2.4 for the different AMI incident response roles and responsibilities and recommended training. 3.4.1.3 Requirements Enhancement None 3.4.2 Process Verification Testing [1-2.7.6] Regular testing helps to determine the effectiveness of the AMI incident response plans. 3.4.2.1 Requirement The program regularly tests AMI incident response plans to validate the efficacy of the program. 3.4.2.2 Supplemental Guidance Following the preparation of the AMI incident response plans, a schedule is developed to review and test each of the plans to ensure that it continues to meet its AMI objectives. 3.4.2.3 Requirements Enhancement None 3-16

4 AMI INCIDENT RESPONSE GUIDELINES This section focuses on guidelines for responding to AMI incidents. This section introduces the AMI incident management process and provides specific guidance regarding incident detection, remediation, and review. 4.1 Incident Management Process [9 SG.IR-5] A phased approach can greatly streamline the incident notification and mobilization process. An example of one such approach is provided. The organization may tailor the approach to best fit their organization. 1) Incident Identification a. Incident is identified through an automated process such as an SIEM or reported manually through another source. b. Evidence such as alarms, events, and situational awareness data is collected and preserved. c. Potential incident scenarios are classified and identified based on available evidence and the incident response teams are notified. 2) Remediation a. If a remediation does not exist, the incident response team will develop a remediation strategy which may involve a root cause analysis of the incident. b. The applicable remediation strategy is applied based on the identified scenario. 3) Review a. If necessary, regression testing is performed in order to verify that the vulnerability or exploit has been remediated. b. A review of the incident is performed in order to determine if the incident response strategy was effective and whether additional training for incident responders is necessary or if the incident detection rule sets need to be fine-tuned. Figure 4-1 provides a detailed view of the AMI incident management process. 4-1

Figure 4-1 AMI Incident Management Process The following sections provide guidance for the incident management process. 4.1.1 Incident Identification Incident identification is the first step in the incident management process. Identification of AMI incidents involves the collection of forensic evidence (e.g., alarms, events, and situational awareness data) in order to determine the incident scenario. Identification of AMI incidents and the incident scenario allows for the AMI entity to determine the appropriate next steps, such as developing a remediation strategy or accepting the risk of a similar incident occurring. 4.1.1.1 Security Information and Event Management Systems [9 SG.IR-6] Security Information and Event Management (SIEM) systems allow an organization to organize and correlate live situational awareness data from multiple systems, including non-ami systems, to detect security events or incidents. 4.1.1.1.1 Guidance Organizations should develop AMI-specific rule sets for their SIEM systems. SIEM rule sets may include situational awareness data to assist in the detection of AMI incidents and incident scenarios. The SIEM should be configured to collect alarms and events from all components of the AMI logical architecture. Once the rule sets have been developed and the SIEM s have been configured, the organization should fine tune the rule sets in such a way that they do not result in an unacceptable number of false positive AMI incident detections. 4-2

The fine tuning of SIEM rule sets is a time-consuming process as it may take some time to set the appropriate alarming thresholds and integrate the AMI system into the overall organizational process. Additionally, periodic software updates, hardware changes, and software changes may result in the need to adjust existing SIEM rules. The AMI entity should be cognizant of these needs and treat fine tuning of SIEM rule sets as a necessary continuous process. 4.1.1.2 Data Mining Data mining allows an organization to analyze archives of past security logs to identify intrusions. 4.1.1.2.1 Guidance Organizations should archive, for an organizationally defined period of time, AMI-related SIEM alarms, events, and situational awareness data. Data mining solutions may then be used to allow for a forensic investigation to detect signs of current or past intrusions and fine-tune AMI SIEM rule sets to detect intrusions and reduce the number of false positive AMI incident detections. 4.1.1.3 Log Message Integrity Integrity checks on log messages can be used to determine if messages have been tampered with or modified from acquisition to analysis, transmission, and storage. Additionally, integrity checks should be used to allow for preservation of forensic evidence for legal purposes. 4.1.1.3.1 Guidance Organizations should utilize integrity checks such as cryptographic signatures to detect tamper or modification of log messages during transmission. Modification to data in transit may indicate a cyber-security intrusion. Organizations should also employ integrity checks to preserve SIEM data for forensic and legal purposes. 4.1.1.4 Incident Scenarios [9 SG.IR-8] This section provides guidance on how to identify specific AMI incidents. The following fields in Table 4-1 are used to describe each of the incident scenarios listed in the sections below. After the incident scenario has been determined, the incident classification field in Table 4-6 is used to determine the appropriate remediation strategy to employ. Table 4-1 Incident Scenario Description ID Incident Scenario Alarms and Events Situational Awareness Data Incident Classification Identifier for the incident scenario Description of the incident scenario Alarms and events associated with the incident scenario Evidence to support that the incident has occurred Classification of the incident to allow the incident response team to determine an appropriate remediation strategy to employ 4.1.1.4.1 Physical The following incident scenarios shown in Table 4-2 occur as a result of a physical action being taken upon the meter. 4-3