Gap Assessment for ASME-ITI/ AWWA J Standard and Leading Vulnerability Assessment Tools

Transcription

1 Gap Assessment for ASME-ITI/ AWWA J Standard and Leading Vulnerability Assessment Tools Web Report #4358 Subject Area: Management and Customer Relations

2

3 Gap Assessment for ASME-ITI/ AWWA J Standard and Leading Vulnerability Assessment Tools

4 About the Water Research Foundation The Water Research Foundation (formerly Awwa Research Foundation or AwwaRF) is a member-supported, international, 501(c)3 nonprofit organization that sponsors research to enable water utilities, public health agencies, and other professionals to provide safe and affordable drinking water to consumers. The Foundation s mission is to advance the science of water to improve the quality of life. To achieve this mission, the Foundation sponsors studies on all aspects of drinking water, including resources, treatment, distribution, and health effects. Funding for research is provided primarily by subscription payments from close to 1,000 water utilities, consulting firms, and manufacturers in North America and abroad. Additional funding comes from collaborative partnerships with other national and international organizations and the U.S. federal government, allowing for resources to be leveraged, expertise to be shared, and broad-based knowledge to be developed and disseminated. From its headquarters in Denver, Colorado, the Foundation s staff directs and supports the efforts of more than 800 volunteers who serve on the board of trustees and various committees. These volunteers represent many facets of the water industry, and contribute their expertise to select and monitor research studies that benefit the entire drinking water community. The results of research are disseminated through a number of channels, including reports, the Web site, Webcasts, conferences, and periodicals. For its subscribers, the Foundation serves as a cooperative program in which water suppliers unite to pool their resources. By applying Foundation research findings, these water suppliers can save substantial costs and stay on the leading edge of drinking water science and technology. Since its inception, the Foundation has supplied the water community with more than $460 million in applied research value. More information about the Foundation and how to become a subscriber is available on the Web at

5 Gap Assessment for ASME-ITI/ AWWA J Standard and Leading Vulnerability Assessment Tools Prepared by: Shannon D. Spence and Corinne M. Tuozzoli Malcolm Pirnie, the Water Division of ARCADIS 44 South Broadway, 15th Floor, White Plains, NY Jointly sponsored by: Water Research Foundation 6666 West Quincy Avenue, Denver, CO and Association of Metropolitan Water Agencies 1620 I Street NW Suite 500, Washington, DC Published by:

6 DISCLAIMER This study was funded by the Water Research Foundation (Foundation) and the Association of Metropolitan Water Agencies (AMWA). The Foundation and AMWA assume no responsibility for the content of the research study reported in this publication or for the opinions or statements of fact expressed in the report. The mention of trade names for commercial products does not represent or imply the approval or endorsement of the Foundation or AMWA. This report is presented solely for informational purposes. Copyright 2011 by Water Research Foundation ALL RIGHTS RESERVED. No part of this publication may be copied, reproduced or otherwise utilized without permission.

7 CONTENTS TABLES... vii FOREWORD... ix ACKNOWLEDGMENTS... xi EECUTIVE SUMMARY... xiii Objectives... xiii Background... xiii GAP Analysis... xiii Addressing the Gaps and Quantifying Effort... xiv Recommendations... xiv CHAPTER 1: FEATURE DEFINITION... 1 Introduction... 1 Method... 2 CHAPTER 2: GAP ANALYSIS... 5 Introduction... 5 Method... 5 CHAPTER 3: RECOMMENDATIONS Introduction Method Conclusion APPENDI A: J FEATURES APPENDI B: J GAP ANALYSES APPENDI C: J RECOMMENDATIONS v

8

9 TABLES 2.1 Comparison of J Standard and ARAM-W Steps SEMS Summary Table VSAT Summary Table ARAM-W TM Summary Table...28 A.1 J Standard Features Matrix...32 B.1 J Standard Gap Analysis Matrix - SEMS...42 B.2 J Standard Gap Analysis Matrix - VSAT...52 B.3 J Standard Gap Analysis Matrix ARAM-W...61 C.1 J Recommendations - SEMS...70 C.2 J Recommendations - VSAT...80 C.3 J Recommendations ARAM-W...86 vii

10

11 FOREWORD The Water Research Foundation (Foundation) is a nonprofit corporation that is dedicated to the implementation of a research effort to help utilities respond to regulatory requirements and traditional high-priority concerns of the industry. The research agenda is developed through a process of consultation with subscribers and drinking water professionals. Under the umbrella of a Strategic Research Plan, the Research Advisory Council prioritizes the suggested projects based upon current and future needs, applicability, and past work; the recommendations are forwarded to the Board of Trustees for final selection. The Foundation also sponsors research projects through collaborative programs and various joint research efforts with organizations such as the U.S. Environmental Protection Agency, the U.S. Bureau of Reclamation, and the Association of California Water Agencies. This publication is a result of one of these sponsored studies, and it is hoped that its findings will be applied in communities throughout the world. The following report serves not only as a means of communicating the results of the water industry's centralized research program but also as a tool to enlist the further support of the nonmember utilities and individuals. Projects are managed closely from their inception to the final report by the Foundation's staff and large cadre of volunteers who willingly contribute their time and expertise. The Foundation serves a planning and management function and awards contracts to other institutions such as water utilities, universities, and engineering firms. The funding for this research effort comes primarily from the Subscription Program, through which water utilities subscribe to the research program and make an annual payment proportionate to the volume of water they deliver and consultants and manufacturers subscribe based on their annual billings. The program offers a cost-effective and fair method for funding research in the public interest. A broad spectrum of water supply issues is addressed by the Foundation's research agenda: resources, treatment and operations, distribution and storage, water quality and analysis, toxicology, economics, and management. The ultimate purpose of the coordinated effort is to assist water suppliers to provide the highest possible quality of water economically and reliably. The true benefits are realized when the results are implemented at the utility level. The Foundation's trustees are pleased to offer this publication as a contribution toward that end. Roy L. Wolfe, Ph.D. Chair, Board of Trustees Water Research Foundation Robert C. Renner, P.E. Executive Director Water Research Foundation ix

12

13 ACKNOWLEDGMENTS The authors of this report thank the Water Research Foundation (Foundation) for its financial, technical, and administrative assistance in funding and managing this project. Specifically, the authors thank the Foundation Project Manager, Ms. Mary Messec Smith, and the Project Advisory Committee members: Mr. Kevin Gertig, City of Fort Collins Utilities Mr. George Hoke, Fairfax Water Mr. Charles M. Murray, Fairfax Water Mr. John P. Sullivan, P.E., Boston Water and Sewer Commission The authors gratefully acknowledge the support and assistance of Mr. Doug Owen, P.E., Mr. Devesh Sinha, Mr. Ryan Zink and Mr. Joshua Ross in the completion of this project. xi

14

15 EECUTIVE SUMMARY OBJECTIVES The objective of this research project was to conduct a gap analysis between the Joint ASME-ITI/AWWA J Risk Analysis and Management for Critical Asset Protection (RAMCAP ) Standard for Risk and Resilience Management of Water and Wastewater Systems (J Standard), and the three existing water/wastewater vulnerability assessment tools Security and Environmental Management System (SEMS), the Vulnerability Self Assessment Tool (VSAT), and Automated-Risk Assessment Methodology tool for Water Sector (ARAM- W TM ). Specifically, the project sought to achieve the following goals: Identify critical gaps, if any, between the J Standard and the results produced through execution of the SEMS, VSAT, and ARAM-W TM. Propose refinements necessary to achieve compatibility with the Standard, including scope and scale of effort required. BACKGROUND A primary step in improving the security of critical infrastructure is the execution of a vulnerability assessment. To that end, critical infrastructure sectors have developed a number of assessment tools. However, the Department of Homeland Security (DHS) determined that the Federal Government needed to be able to compare risks both within and between sectors. Therefore, in 2007 the DHS developed the Risk Analysis and Management for Critical Asset Protection (RAMCAP) methodology, which provides a standardized framework for measuring risk, both within and across critical infrastructure sectors. The water sector has developed several vulnerability assessment tools including SEMS, VSAT, and ARAM-W, and in the last few years efforts were undertaken to revise these three tools to bring them in alignment with RAMCAP. The effort to automate the Risk Assessment Methodology for Water (RAM-W ) began in 2009, and ARAM-W (the automated version) is expected to be available in mid VSAT was updated in The committee tasked with the development of what became the J Standard began meeting in Members of the committee included government and industry representatives, utilities, consultants, and the developers of both VSAT and ARAM. The public comment period on the J Standard closed on 11/30/09, and it was published in July of GAP ANALYSIS A detailed review found gaps between each of the three software packages and the J Standard. ARAM-W TM met 54 of the 79 features of the Standard (68%); VSAT met 52 of the 79 (66%); and SEMS met 38 of the 79 (48%). None of the software tools calculate resilience the largest single inconsistency with the J Standard, and one that accounts for a number of peripheral inconsistencies in each package. Other significant gaps include the following: xiii

16 xiv Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools SEMS Consequence calculations do not include duration and severity of service denial The threat likelihood is assigned only at the utility level, not at the threat-asset pair level VSAT The software does not include dependency and proximity hazards The software does not calculate risk directly but instead uses a proxy measure, the calculation and manipulation of which is unclear ARAM-W TM The consequences for the loss of critical assets are not evaluated for each specific threat on the asset The software does not include dependency and proximity hazards ADDRESSING THE GAPS AND QUANTIFYING EFFORT The investigators then developed recommendations to address the identified gaps and the associated labor by analyzing to the extent possible the databases used by each package. For the purposes of calculating the amount of development effort that might be required, four categories of software upgrades were identified small (8-16 hours), medium (16-40 hours) and large ( hours). Using these broad ranges it was estimated that it will take approximately 1250 hours of labor to address the gaps in SEMS, 1150 hours for VSAT, and 700 hours for ARAM-W TM. However and very importantly there were also a number of gaps for which the effort to correct could not be quantified. SEMS had seven such gaps, VSAT had five, and ARAM- W TM had eight of these unknown size upgrades. Of these unquantifiable gaps, the largest are as follows: SEMS adding the ability to identify improvement packages that affect multiple threat-asset pairs and have the greatest benefit VSAT adding risk calculations instead of risk-reduction units. ARAM-W TM updating the consequences to be evaluated for the loss of critical assets for each specific threat on the asset Clearly it was very difficult to estimate with any accuracy, the amount of effort needed to bring any one of these software tools into compliance with the J Standard. RECOMMENDATIONS Focusing on addressing the discrepancy between the J Standard and the software tools around the issue of resilience could quickly bring VSAT and/or ARAM-W TM much closer to compliance with the Standard. In addition, clarifying with the software developers the exact labor needed to address the larger unknown upgrades is also a logical next step.

17 CHAPTER 1: FEATURE DEFINITION INTRODUCTION The format of an industry standard is such that the high level requirements are outlined in the body of the standard in a relatively succinct fashion and the details are outlined in the appendices that follow. The Joint ASME-ITI/AWWA J Risk Analysis and Management for Critical Asset Protection (RAMCAP ) Standard for Risk and Resilience Management of Water and Wastewater Systems includes six chapters and eight appendices. A simplified outline of the J Standard s Table of Contents is listed below. Foreword 1 Introduction 1.1 Origin 1.2 Evolution of RAMCAP 1.3 RAMCAP in the Water Sector 1.4 History of the Standard 1.5 ANSI Approval Dates 2 RAMCAP Overview 3 Organization of This Document 4 Comments Committee Roster Risk and Resilience Management of Water and Wastewater Systems 1 Scope 2 Definitions 3 Bibliography 4 Requirements 4.1 Asset 4.2 Threat 4.3 Consequence Analysis 4.4 Vulnerability Analysis 4.5 Threat Analysis 4.6 Risk and Resilience Analysis 4.7 Risk and Resilience Management 5 Process Control 6 Verification Appendices: Appendix A: Guidance on the Use of this Standard Appendix B: Optional Use of RAMCAP Scales for Recording Consequence and Vulnerability Estimates Appendix C: Glossary Appendix D: Expanded Bibliography Appendix E: RAMCAP Reference Threats 1

18 2 Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools Appendix F: Proxy Indicator of Terrorism Threat Likelihood for the Water Sector Appendix G: Integrated Analysis of Natural Hazards Appendix H: Water Sector Utility Resilience Analysis Approach Chapter 1 of the J Standard is short with more detail regarding the required steps provided in Chapter 4, which lays out the mandatory seven-step RAMCAP process, as follows: 1) Asset 2) Threat 3) Consequence Analysis 4) Vulnerability Analysis 5) Threat Analysis 6) Risk/Resilience Analysis 7) Risk/Resilience Management The eight appendices (A-H) include a greater level of detail regarding each step of the J standard and address, to a much greater extent, the intent of the Standard. It appears that the committee that drafted the Standard worked to keep it flexible while still creating an approach that would produce consistent results that could be used for comparison both within and across critical infrastructure sectors. Thus, the Standard also identifies the preferred approach to the execution of each step. Generally (but not always) these are outlined in the appendices. Each software product analyzed as part of this project was measured against not only how well it adhered to the mandatory portions of the Standard, but also how well it met the Standard s intent. METHOD In analyzing the J Standard, the researchers, defined a mandatory feature as one that is in a mandatory section of the Standard is written in the text with the words shall or will. A non-mandatory feature of the Standard is defined as a statement that is either in a nonmandatory section or is written in the text with the words should or may. The Standard also clearly delineates which appendices are mandatory and which are nonmandatory as follows: Mandatory: o Appendix E: RAMCAP Reference Threats Non-Mandatory: o Appendix A: Guidance on the Use of this Standard o Appendix B: Optional Use of RAMCAP Scales for Recording Consequence and Vulnerability Estimates o Appendix C: Glossary o Appendix D: Expanded Bibliography o Appendix F: Proxy Indicator of Terrorism Threat Likelihood for the Water Sector o Appendix G: Integrated Analysis of Natural Hazards

19 Chapter 1: Feature Definition 3 o Appendix H: Water Sector Utility Resilience Analysis Approach In various places in the Standard there are conflicts and/or cross references between mandatory sections; and between mandatory and non-mandatory sections. In the cases where there are conflicts between mandatory sections the authors made a note of the conflict and analyzed the software(s) against the most conservative interpretation of the conflicting statements. In the cases where there are conflicts between mandatory and non-mandatory sections the authors took a two-pronged approach. As already stated, the authors assumed that a feature was mandatory if it met the definition as outlined above (i.e. with the words shall or will ) and is in a mandatory section of the Standard. However, if the authors felt - based on an analysis of the cross references and wording that the intent of the Standard was clearly indicated in a nonmandatory section or feature, they also analyzed the software(s) against that feature. For example, the Standard clearly identifies preferred methods for approaching many of the mandated seven steps in non-mandatory appendices including: Appendix B for the estimation of consequences Appendix E to identify reference threats Appendix F to calculate the Proxy Measure for malevolent threats Appendix G to estimate the risk of the natural hazards Appendix H to calculate the Operational Resilience Index for resilience. Based on this approach the researchers created a master J Standard Features Matrix (Table A.1) that documents each section of the Standard. This matrix was used during the analysis of each software package. The matrix lists each chapter, section and the associated features of the Standard. Some features are found in multiple sections or appendices and are listed as such. You will note that the matrix does not proceed strictly in either numerical or alphabetical order. This is because the body of the Standard refers to the appendices on an ad hoc basis. Therefore, all parts of the Standard are addressed in the matrix but not necessarily in numerical or alphabetical order. The matrix also includes comments on software usability and aesthetics.

20 4 Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools

21 Chapter 2: Gap Analysis 5 CHAPTER 2: GAP ANALYSIS INTRODUCTION The J Standard was analyzed against the three currently available vulnerability assessment tools Security and Environmental Management System (SEMS), the Vulnerability Self Assessment Tool (VSAT) and Automated-Risk Assessment Methodology tool for Water Sector (ARAM-W ). The SEMS RAMCAP Risk Assessment is a part of a larger software suite. This suite includes Drinking Water and Wastewater Compliance, Asset Management, and Security & Emergency Management modules. The RAMCAP Risk Assessment is a part of the Security & Emergency Management module. There are additional advanced features that may also be purchased which integrate the SEMS software with other commonly used software systems such as ArcView GIS, SCADA, billing systems, and LIMS. VSAT and ARAM-W are each standalone tools designed to assess a utility s risk and vulnerability. VSAT was developed in 2002 and updated in 2010 by the National Association of Clean Water Agencies (NACWA), in collaboration with the PA Consulting Group and SCIENTECH, Inc. It was funded by the U.S. Environmental Protection Agency (EPA). In 2009, water sector stakeholders identified a need to automate the Risk Assessment Methodology for Water (RAM-W ) and this development effort was started in January ARAM-W is the automated version of the RAM-W tool and is expected to be available to the public in mid METHOD To execute the gap analysis, the researchers approached each software package from the viewpoint of an end user; an end user was defined as a water or wastewater utility staff member not necessarily familiar with the software package. The research investigators executing the analysis were very familiar with the J Standard but were not familiar with SEMS, VSAT, or ARAM-W. For the analysis, the investigators utilized the sample vulnerability assessment included as part of VSAT with each of the three software packages. The sample assessment was for a small combined water and wastewater utility. The assessment identified 14 critical assets and countermeasures, 6 man-made threats, and 24 natural disaster threats. In addition to using the same sample assessment with all three software packages as part of their analysis, the investigators also made changes to parameters such as countermeasures and types of threats in order to determine if the software was executing its internal calculations in ways that met the intent of the J Standard. Each item listed in the J Standard Features Matrix in Appendix A was compared to the software functionality to determine if the software met both the mandatory and nonmandatory features of the J Standard. The detailed findings were then captured in the J Standard Gap Analysis Matrix (Gap Analysis Matrix) in Appendix B. The Gap Analysis Matrix has been broken down as follows: Table B.1 (SEMS), Table B.2 (VSAT) and Table B.3 (ARAM-W ) for ease of review. An overview of the gaps that were found in each software package is described in the remainder of this chapter. These are presented in the same 5

22 6 Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools order as the seven steps of the J Standard. The steps are as follows, listed with their corresponding paragraphs in the Standard: 1. Asset (4.1) 2. Threat (4.2) 3. Consequence Analysis (4.3) 4. Vulnerability Analysis (4.4) 5. Threat Analysis (4.5) 6. Risk/Resilience Analysis (4.6) 7. Risk/Resilience Management (4.7) SEMS The SEMS Technologies website ( states that the SEMS Technologies Risk Assessment Software (SEMS) is a RAMCAP-consistent program that can be used to help perform a risk assessment of a water or wastewater utility. However, while the software may be RAMCAP consistent (not specifically analyzed as a part of this project), the analysis showed that there are many gaps between the SEMS software and the J Standard. These gaps are as follows: Paragraph 4.1: Asset. The purpose of asset characterization is to identify critical assets to be considered in the subsequent steps Mission. No gap identified. The software meets the requirement of the J Standard to identify the utility s mission Critical Assets. No gap identified. To meet the Standard, the software practitioner must create a list of all of the utility s assets and select those that are critical. The SEMS software provides a dropdown list of predefined assets, e.g., chemical pumps, storage tanks, and valves Supporting Infrastructures Meets J Standard. Improvements suggested. To meet the Standard, the software must allow for critical internal or external supporting infrastructures such as financial records, legal documents, planning documents, mutual aid agreements, etc., to be identified. The SEMS software does not provide any critical internal or external supporting infrastructures in its predefined list, although the user can add these manually Countermeasures and mitigation measures/features No gap identified. The SEMS software meets the requirement to provide predefined countermeasures and to allow the software practitioner to be able to add additional countermeasures and descriptions and/or details Consequence metrics Gap identified. To meet the Standard, the software must allow the practitioner to estimate the worst reasonable consequences for each asset without regard to the threat. These consequence metrics include potential for fatalities, serious injuries, major economic loss to facility or community, loss of public confidence, and inhibiting effective function of national defense or civilian government. The SEMS software does not allow the user to define the worst reasonable consequences without regard to threat Prioritize Assets - Gap identified. To meet the Standard, the software must allow the practitioner to identify assets as critical and to rank them using categories of high, medium

23 Chapter 2: Gap Analysis 7 and low. The SEMS software does this. However, the software does not allow the practitioner to reprioritize assets based on consequences, which is also part of the Standard. Paragraph 4.2: Threat. In the Threat step, once the critical assets have been determined, the user must define the threats that could potentially impact each asset Malevolent Threat No gap identified. The software uses J Standard threats Natural Hazards Threat Gap Identified. To meet the Standard, when selecting the natural hazard threats, the software must allow the practitioner to define the range of magnitudes, from the smallest magnitude that could cause serious harm to the largest reasonable magnitude, of each natural hazard. Although the SEMS software does identify what standards should be used for selecting the magnitudes of some natural hazards to be used as part of the analysis (e.g. FEMA flood maps), it does not provide the actual ranges. The investigator was unable to find direct links to reference materials that would allow the practitioner to determine the likelihood of each magnitude of each natural hazard that might impact the assets. In addition, the software also does not include some natural hazards, such as wildfires and ice storms although these can be added manually. However, the largest apparent gap is that in the SEMS software, the investigator found that changing the magnitude of a natural disaster did not seem to affect the threat probability or overall risk results from the analysis Dependency Hazards Threat Meets J Standard. Improvements suggested. The Standard requires the user to define any dependency threats due to interruptions in utilities, suppliers, employees, customers and transportation; as well as threats due to the close proximity of the utility to dangerous neighboring sites. These threats are not predefined within the SEMS software and must be manually added to the threat list Threat-Asset Pairs No gap identified. The Standard requires the practitioner to be able to assign potential threats to each asset. The SEMS software does this Threat-Asset Pair Ranking Gap identified. To meet the Standard, once all of the potential threats have been assigned to each asset (thus creating threat-asset pairs) the practitioner must then rank them (using professional judgment) in order from the greatest to the least resulting consequences. The SEMS software does not allow for this ranking and instead proceeds with the analysis of all threat-asset pairs as if each were of equal importance Critical Threat-Asset Pairs Gap identified. To meet the Standard, the practitioner must use the ranking developed under paragraph to select the critical threatasset pairs to be further analyzed (or to treat them all as critical). The SEMS software allows the practitioner to select critical assets but not critical threat-asset pairs. Paragraph 4.3: Consequence Analysis. In the Consequence Analysis step, once the critical threat-asset pairs are identified, the worst reasonable consequences that can be caused by the specific threats on the assets are defined Threat Scenario No gap identified. To meet the Standard, the software must allow the practitioner to identify the worst reasonable consequences that can be caused by specific threats on specific assets for each threat-asset pair. The SEMS software does this.

24 8 Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools Estimate Consequences Gap identified. To meet the Standard, the software must provide the practitioner with three options for estimating fatalities, serious injuries, financial loss to the owner/operator, financial loss to the community and duration and severity of service denial for the affected customers of the utility. These include: a single-point estimate, a single indicator (a bin value) and a range. The SEMS software does provide a dropdown list of J Standard ranges for each of the different types of consequences except for duration and severity of service denial for the affected customers of the utility. However, it does not provide the option for the practitioner to enter point value estimates or single indicators for each consequence. As this feature was met by only one out of three requirements in the J Standard the researchers have identified it as a gap Estimate Consequences (other) Gap identified. To meet the Standard, the software must allow the practitioner to add additional consequences, if desired. The investigator was not able to find a way to add additional consequences Document Assumptions No gap identified. To meet the Standard, the software must provide a field for practitioners to document their assumptions and procedures for performing the consequence analysis. The SEMS software provides a location for the practitioner to do this Record Consequence Gap identified. To meet the Standard, the software must provide both a field for consequence ranges and a field for the practitioner to insert a point estimate. The SEMS software provides the preferred J Standard consequence ranges, but it does not provide a field for the practitioner to insert a point estimate. Paragraph 4.4: Vulnerability Analysis. A key component of the J Standard risk assessment is the vulnerability analysis, which determines the likelihood that given a threat or hazard will occur Review No gap identified. To meet the Standard, the software must provide fields to input pertinent details of utility/facility construction, systems and layout. The SEMS software provides a place to do this when identifying the asset Analyze Vulnerability Gap Identified. To meet the Standard, the software must provide a field for the practitioner to analyze the vulnerabilities of each asset in order to estimate the likelihood that, given the occurrence of a threat, the estimated consequences will result. The SEMS software utilizes the vulnerability values that are listed in the J Standard in Appendix B, Table B-5. However, the software does not allow the user to actually determine this value using standard methods; i.e. fault, event or failure tree analysis, path analysis, vulnerability logic diagrams, computer simulation methods, or expert judgment. In the SEMS software, asset vulnerability is analyzed based on the number of countermeasures assigned to the threat-asset pair (as executed under Step 4.1.4) and the ability of the countermeasure to detect, delay, and respond to the threat. The software displays the vulnerability as a J Standard percentage range. This calculation is performed behind the scenes, within the software Document Method Gap identified. To meet the Standard, the software must have a field where practitioners can define their methods of analyzing vulnerability (e.g. those listed in paragraph 4.4.2). The SEMS software does not Record Estimates Gap identified. To meet the Standard, the software must have a field where practitioners can record the vulnerabilities that they have calculated for each

25 Chapter 2: Gap Analysis 9 critical asset. The SEMS software displays the vulnerability as a J percent range, but it does not allow the user to input the vulnerability manually. Paragraph 4.5: Threat Analysis. In the Threat Analysis step the practitioner determines the likelihood (or frequency) that a specific malevolent event, dependency/proximity hazard, or natural hazard will occur to a specific critical asset. Gap identified. To meet the Standard, the software must allow the practitioner to assign the likelihood (or frequency) of each of the selected hazards and threats in relation to each asset. However, the SEMS software selects (from a table) a single threat likelihood and assigns it to the entire utility. The only explanation that the investigators could identify for the values in the table is that they are provided by the DHS. According to the software, this scale takes into account the population served by the utility; the amount of onsite gaseous chlorine storage; the economic impact of the utility; and the number of critical customers served by the utility. The single threat likelihood is a significant difference from the J Standard. See paragraphs through below for further explanation Malevolent Threats - Gap identified. To meet the Standard, the software must include malevolent threat likelihood calculations using proxy measures, best estimates, or conditional assessments. The SEMS software uses only best estimates. As this feature was met by only one out of three methods required by the J Standard the researchers have identified this as a gap Natural Hazards Gap identified. To meet the J Standard, the software must include natural hazards threat likelihood calculations. It must also allow the practitioner to assign the likelihood or frequency of a natural hazard to the asset, based on historical data that may be provided by the software via maps, data, or links to reference materials. The software must then calculate the risk of each natural hazard and sum them to determine the overall risk due to natural hazards. The SEMS software has fields for historic information and magnitudes (text boxes). However, the investigators found that these values do not impact the calculated results of the analysis Dependency and Proximity Hazards Gap identified. To meet the J Standard, the software must include dependency and proximity hazards threat likelihood calculations. It must also include historical data on dependency and proximity hazards to determine the likelihood that the threats will occur to the assets Record Estimates Gap identified. To meet the J Standard, the software must allow the practitioner to determine the likelihood of each specific threat occurring to each specific asset and record this estimate, along with the method and reasoning for the estimate. The investigator did not find a place in the software to input this estimate. Paragraph 4.6: Risk/Resilience Analysis. Once the consequence, vulnerability, and threat likelihood have been determined for each threat-asset pair, the overall risk and resilience of the utility is calculated. Gap identified. The SEMS software does not include any Risk/Resilience Analysis. This is a significant difference from the Standard. See paragraphs through below for further explanation.

26 10 Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools Calculate Risk Gap identified. To meet the Standard, the software must allow the practitioner to calculate the risk associated with each threat-asset pair based on the consequence, vulnerability, and threat likelihood values selected during earlier steps. However, the investigators did not find a place to input a specific value for vulnerability or threat likelihood Calculate Resilience Gap identified. To meet the Standard, the software must determine the overall resilience of the utility, including the duration of service denial and severity of service denial (in gallons per day), in order to determine the resilience of each threatasset pair Operational Resilience Asset Resilience Metric Gap identified. To meet the J Standard, the software must calculate the asset s resilience Owner s Economic Resilience - Gap identified. To meet the Standard, the software must calculate the owner s economic resilience Community Economic Resilience - Gap identified. To meet the Standard, the software must calculate the community s economic resilience Record Risk and Resilience Estimates Gap identified. To meet the Standard, the software must calculate the overall risk to the utility using the J Standard risk equation; the risk to each specific threat-asset pair; and the different types of resilience as defined in paragraphs through The SEMS software gives each threat-asset pair a Risk Score based on a tiered binning system. The software also creates an Overall Risk graph, plotting the consequence versus the vulnerability of each asset. However, the investigators could not find any explanation on how the binning system was developed. In addition, the SEMS software does not allow for resilience calculations. Paragraph 4.7: Risk/Resilience Management. Once the risk of the utility and of each threat-asset pair has been determined, the utility continues the process by deciding whether actions are needed to enhance all-hazards security or resilience or both. Gap identified. The SEMS software does not proceed any further in allowing the practitioner to manage risk and resilience. See paragraphs through below for further explanation Decide Gap identified. To meet the Standard, the software must allow the practitioner to decide what risk and resilience levels are acceptable Define Gap identified. To meet the Standard, the software must allow the practitioner to define new countermeasures and mitigation/resilience options to reduce unacceptable risk to specific threat-asset pairs Estimate Gap identified. To meet the Standard, the software must allow the practitioner to estimate the costs of the new countermeasures and mitigation/resilience options Assess Gap identified. To meet the Standard, the software must allow the practitioner to assess the options by analyzing the facility or asset under the assumption that the option has been implemented Identify Gap identified. To meet the Standard, the software must allow the practitioner to identify those options that have benefits to multiple threat-asset pairs. The SEMS software does not Calculate Gap identified. To meet the Standard, the software must allow the practitioner to calculate the net benefits and benefit-cost ratios of the selected countermeasures.

27 Chapter 2: Gap Analysis Review & Rank Gap identified. To meet the Standard, the software must allow the practitioner to review the selected countermeasures and rank them in order to determine which ones will be most effective in reducing the utility s risk. Summary. In summary, the SEMS software is a user-friendly risk assessment tool that is one module of a larger software package, and that utilizes many of the J Standard tables and definitions for threat and consequences. However, the analysis showed that there are many gaps between the SEMS software and the J Standard, the largest of which are as follows: Paragraph Natural Hazards Threat Changing the magnitude of a natural hazard does not appear to affect the value of the threat probability or overall risk results. Paragraph Estimate Consequences The software does not include duration and severity of service denial for the affected customers of the utility, nor does it provide the practitioner with point value estimates or single indicators for each consequence. Paragraph Analyze Vulnerability - The vulnerability of a threat-asset pair is determined by the software, not the practitioner, and the investigators could not identify a way of changing or verifying the determination of this value. Paragraph 4.5 Threat Analysis - The threat likelihood is assigned only at the utility level, not at the threat-asset pair level, and is determined using a table instead of proxy measures, best estimates, or conditional assessments as per the Standard. Paragraph 4.6 Risk/Resilience Analysis - The SEMS software does not address resilience. Paragraph 4.7 Risk/Resilience Management The SEMS software does not provide risk or resilience management. Vulnerability Self Assessment Tool (VSAT) According to the Environmental Protection Agency s website ( the Vulnerability Self Assessment Tool (VSAT) software is a RAMCAP-consistent risk assessment application for water, wastewater, and combined utilities. However, the analysis executed under this project demonstrated that, although the software may be RAMCAP consistent (a determination outside the scope of this project), it does not meet the RAMCAP J Standard in some key respects. The details of this analysis follow. Paragraph 4.1: Asset. The purpose of asset characterization is to identify critical assets to be considered in the subsequent steps Mission No gap identified. To meet the Standard, the software must allow the practitioner to input the utility s mission. The VSAT software provides a field for this Critical Assets. No gap identified. To meet the Standard, the practitioner must create a list of the utility s assets and select those that are critical. At the start of a VSAT analysis, the software requires the practitioner to define all of the utility s assets. The software contains a list of commonly used assets and also allows custom assets to be defined.

28 12 Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools Supporting Infrastructures No gap identified. To meet the Standard, the software must allow for critical internal or external supporting infrastructures such as financial records, legal documents, planning documents, mutual aid agreements, etc., to be identified. The VSAT software does this Countermeasures and mitigation measures/features No gap identified. To meet the Standard, the software must provide a way for the practitioner to identify and document existing protective countermeasures and mitigation measures/features. VSAT contains a list of countermeasures commonly found at utilities in the United States and their associated costs. In addition, the practitioner can also add custom countermeasures and edit the countermeasures to add additional details Consequence metrics Gap identified. To meet the Standard, the software must allow the practitioner to estimate the worst reasonable consequences for each asset without regard to the threat. These consequence metrics include potential for fatalities, serious injuries, major economic loss to facility or community, loss of public confidence, and inhibiting effective function of national defense or civilian government. VSAT does not allow the user to calculate the worst reasonable consequences resulting from the destruction or loss of an individual asset, regardless of the threat. The software skips this step and only considers the consequences associated with specific threat-asset pairs Prioritize Assets Gap identified. To meet the Standard, the software must allow the practitioner to identify assets as critical and to rank them using categories of high, medium and low. VSAT does not allow the user to rank the critical assets. In VSAT, assets are either critical or not critical. Paragraph 4.2: Threat. Once the critical assets have been determined, the user must define the threats that could potentially impact each asset Malevolent Threat No gap identified. The software uses J Standard threats Natural Hazards Threat Meets J Standard. Improvements suggested. To meet the Standard, when selecting the natural hazard threats, the software must allow the practitioner to define the range of magnitudes, from the smallest magnitude that could cause serious harm to the largest reasonable magnitude, of each natural hazard. Although the VSAT software does adequately meet this requirement for most of the natural hazards, it does not provide the magnitude ranges for wildfires, nor does it include direct links to reference materials to help determine the likelihood of each magnitude Dependency Hazards Threat Meets J Standard. Improvements suggested. The Standard requires the user to define any dependency threats due to interruptions in utilities, suppliers, employees, customers and transportation, as well as threats due to the close proximity of the utility to dangerous neighboring sites. The researchers could not locate these threats in the VSAT software, although threats can be manually added to the threat list Threat-Asset Pairs No gap identified. The J Standard requires the practitioner to assign potential threats to each asset. The VSAT software does this Threat-Asset Pair Ranking Gap identified. To meet the J Standard, once all of the potential threats have been assigned to each asset (thus creating threat-asset pairs) the user must then rank them (using professional judgment) in order from greatest to least resulting

29 Chapter 2: Gap Analysis 13 consequences. The VSAT software does allow the practitioner to create threat-asset pairs, but it does not allow for this ranking Critical Threat-Asset Pairs Meets J Standard. Improvements suggested. To meet the J Standard, the practitioner must use the ranking developed under paragraph to select the critical threat-asset pairs to be further analyzed (or alternately, to treat them all as critical). The VSAT software does not rank the threat-asset pairs and does not allow the user to remove threat-asset pairs anywhere during the analysis. This can result in a very long list of threat-asset pairs for the user to evaluate for the remainder of the analysis. Paragraph 4.3: Consequence Analysis. Once the critical threat-asset pairs are identified, the worst reasonable consequences that can be caused by the specific threats on the assets are defined Threat Scenario No gap identified. To meet the J Standard, the software must allow the practitioner to identify the worst reasonable consequences that can be caused by specific threats on specific assets for each threat-asset pair. The VSAT software does this Estimate Consequences Gap identified. To meet the Standard, the software must provide the practitioner with three options for estimating fatalities, serious injuries, financial loss to the owner/operator, financial loss to the community and duration and severity of service denial for the affected customers of the utility. These include a single-point estimate, a single indicator (a bin value), and a range. The VSAT software utilizes the Water Health Economic Analysis Tool (WHEAT) in addition to single-point estimates, single indicators, and ranges to measure each of the different types of consequences. However, it does not estimate consequences for the affected customers Estimate Consequences (other) Gap identified. To meet the Standard, the software would need to include the ability to add additional consequences, if desired. The VSAT software only provides fields for four consequence types (fatalities, injuries, economic cost of owner, and economic cost to region) Document Assumptions No gap identified. To meet the Standard, the software must provide a field for practitioners to document their assumptions and procedures for performing the consequence analysis. The VSAT software relies on the WHEAT tool to determine the consequences. VSAT and WHEAT outputs are exported to an Excel table where the software s assumptions are documented Record Consequence No gap identified. To meet the Standard, the software must provide both consequence ranges and a field for the practitioner to insert a point estimate. In the VSAT software, the practitioner can enter a point estimate or select from the predefined J Standard bins. Paragraph 4.4: Vulnerability Analysis. A key component of the Standard risk assessment is the vulnerability analysis, which determines the likelihood that a given threat or hazard occurs Review No gap identified. To meet the Standard, the software must provide fields to input pertinent details of utility/facility construction, systems, and layout. VSAT provides a place to do this when identifying the asset.

30 14 Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools Analyze Vulnerability Gap identified. To meet the Standard, the software must provide a field for the practitioner to analyze the vulnerabilities of each asset to estimate the likelihood that, given the occurrence of a threat, the estimated consequences will result by utilizing fault, event or failure trees, path analysis, vulnerability logic diagrams, computer simulation methods, or expert judgment. The VSAT software allows the practitioner to choose the detection level (certain, probable, possible, none), amount of delay (very, strong, limited, no delay), and response speed (fast, variable, slow, none) relative to each threat-asset pair. The software then uses these values to determine the vulnerability of each threat-asset pair. This method is equivalent to expert judgment. As this feature was met by only one out of three selection options identified in the J Standard, the researchers have identified this as a gap Document Method No gap identified. To meet the Standard, the software must have a field where the practitioner can define their methods of analyzing vulnerability (e.g. those listed in paragraph 4.4.2). The VSAT software does this Record Estimates Gap Identified. To meet the Standard, the software must have a field where practitioners can record the vulnerabilities that have been calculated for each critical asset. VSAT determines the likelihood of damage using countermeasure capability (very high, high, moderate, and low). This method does not allow the practitioner to use point values. Also, as stated in 4.4.2, the software only allows for one method of determining the vulnerability of the threat-asset pairs. Paragraph 4.5: Threat Analysis. In the Threat Analysis step, the practitioner determines the likelihood (or frequency) that a specific malevolent event, dependency/proximity hazard, or natural hazard will occur to a specific critical asset Malevolent Threats Gap Identified. To meet the Standard, the software must include malevolent threat likelihood calculations using proxy measures, best estimates, or conditional assessments. The VSAT software allows the practitioner the choice to use Best Estimate or Conditional Assessment at the start of the assessment. If Best Estimate is chosen, then the software only allows the practitioner to determine the threat likelihood of each threat and record them as very high, high, moderate, or low likelihoods (basically a best estimate). It does not allow the practitioner to use proxy measures or conditional assessments. As this feature was met by only two out of three requirements in the J Standard, the researchers have identified this as a gap Natural Hazards Meets J Standard. Improvements suggested. To meet the Standard, the software must include natural hazards threat likelihood calculations. It must also allow the practitioner to assign the likelihood or frequency of a natural hazard to the asset, based on historical data that may be provided by the software via maps, data, or links to reference materials. The software must then calculate the risk of each natural hazard and sum them to determine the overall risk due to natural hazards. VSAT allows the practitioner to assign the likelihood (or frequency) of a natural hazard to the asset based on historical data for predefined natural hazards of floods, tornados, hurricanes, earthquakes, fire, snowstorm, and windstorm. Although it is missing historical data for wildfires, ice storms, snow storms, and other natural hazards, it does allow the user to manually input this information. The software does not calculate the overall risk due to natural hazards Dependency and Proximity Hazards Gaps identified. To meet the Standard, the software must include dependency and proximity hazards threat likelihood calculations. It must

31 Chapter 2: Gap Analysis 15 also include historical data on dependency and proximity hazards to determine the likelihood that the threats will occur to the assets. The investigators were not able to find a field to input these values Record Estimates No gaps identified. To meet the J Standard, the software must allow the practitioner to determine the likelihood of each specific threat occurring to each specific asset and record this estimate, along with the method and reasoning for the estimate. The VSAT software does this. Paragraph 4.6: Risk/Resilience Analysis. Once the consequence, vulnerability, and threat likelihood have been determined for each threat-asset pair, the overall risk and resilience of the utility is calculated. Gaps identified. The VSAT software does not include Resilience Analysis. This is a significant departure from the Standard. See paragraphs through below for further explanation Calculate Risk Gaps identified. To meet the Standard, the software must allow the practitioner to calculate the risk associated with each threat-asset pair based on the consequence, vulnerability, and threat likelihood values selected during earlier steps. The VSAT software does not calculate the overall risk to the utility, nor does it calculate the risk to each threat-asset pair using the J Standard formula (Risk = Consequences x Vulnerability x Threat Likelihood). Instead of calculating risk, VSAT calculates, displays, and generates data and written reports of each threat-asset pair s consequences, vulnerability, and threat likelihood Calculate Resilience Gap identified. To meet the Standard, the software must determine the overall resilience of the utility, including the duration of service denial and severity of service denial (in gallons per day), in order to determine the resilience of each threatasset pair Operational Resilience Asset Resilience Metric Gap identified. To meet the Standard, the software must calculate the asset s resilience Owner s Economic Resilience - Gap identified. To meet the Standard, the software must calculate the owner s economic resilience Community Economic Resilience - Gap identified. To meet the Standard, the software must calculate the community s economic resilience Record Risk and Resilience Estimates Gap identified. To meet the Standard, the software must calculate the overall risk to the utility using the J Standard risk equation; the risk to each specific threat-asset pair; and the different types of resilience as defined in paragraphs through The VSAT software does not calculate or record the overall or specific risk or resiliencies. Paragraph 4.7: Risk/Resilience Management. Once the risk of the utility and of each threat-asset pair has been determined, the utility continues with the process by deciding whether actions are needed to enhance all-hazards security or resilience or both Decide Gap identified. To meet the J Standard, the software must allow the practitioner to decide what risk and resilience levels are acceptable. The VSAT software does not.

32 16 Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools Define No gap identified. To meet the J Standard, the software must allow the practitioner to define new countermeasures and mitigation/resilience options to reduce unacceptable risk to specific threat-asset pairs. The VSAT software enables practitioners to make improvements to their security by adding new packages of possible countermeasures and then performing an improvement analysis of the threat-asset pairs. Practitioners are also able to create their own countermeasure packages Estimate No gap identified. To meet the J Standard, the software must allow the practitioner to estimate the costs of the new countermeasures and mitigation/resilience options. VSAT does this Assess Gap identified. To meet the J Standard, the software must allow the practitioner to assess the options by analyzing the facility or asset under the assumption that a specific countermeasure or mitigation/resilience option has been implemented. Once the analysis has been performed for each threat-asset pair with the countermeasures in place, VSAT does not assess the improved risk levels. Instead, it again displays the consequence, vulnerability, and threat likelihood ratings for each threat-asset pair and compares the improved ratings to the baseline ratings. An additional issue is that the software also calculates risk reduction units. The investigators felt that the explanation as to how these units are calculated is confusing. In addition, these units can only be used for comparison purposes inside VSAT, as risk reduction units are not risk but a proxy approach for comparing packages of countermeasures. An overall improved risk and resilience and the individual resilience of each threat-asset pair is never calculated for the improved assessment Identify Meets Standard. Improvements suggested. To meet the J Standard, the software must allow the practitioner to identify those options that have benefits to multiple threat-asset pairs. Although the practitioner can create multiple packages of countermeasures to compare and then determine which ones benefit the utility the most, the practitioner cannot compare the reduced risk values but only the new consequence, vulnerability, and threat likelihood ratings Calculate Gap identified. To meet the Standard, the software must allow the practitioner to calculate the net benefits and benefit-cost ratios of the selected countermeasures. VSAT calculates the annualized and capital costs of each countermeasure, as well as packages of countermeasures and compares them to the risk reduction units of the other countermeasures and packages. The software does not calculate the net benefits and benefit-cost ratios Review & Rank Meets Standard. Improvements suggested. To meet the Standard, the software must allow the practitioner to review and rank the selected countermeasures in order to determine which ones will be most effective in reducing the utility s risk. Although VSAT provides a risk reduction unit for each package, the means by which it calculates the risk reduction unit is difficult to understand. The only explanation that the investigators found was in the VSAT Methodology Guide which states, The risk reduction is quantified with a series of dimensionless numbers, which can be placed in context only when reflecting a movement between cells. Summary. In summary, the VSAT software is a user-friendly risk assessment tool that calculates consequence, vulnerability, and threat likelihood; integrates the WHEAT tool; and utilizes many of the J Standard tables and definitions for threat.

33 Chapter 2: Gap Analysis 17 However, the analysis showed that there are deviations between the VSAT software and the J Standard, the largest of which are as follows: Paragraph Analyze Vulnerability - Lack of availability to use fault, event or failure trees, path analysis, vulnerability logic diagrams, or computer simulation methods to determine vulnerability. VSAT only uses a form of expert judgment. Paragraph Malevolent Threats The software only allows the practitioner to determine the threat likelihood of each threat and record them as very high, high, moderate, or low, which is basically a best estimate. The software does not allow the practitioner to use proxy measures or conditional assessments. Paragraph Dependency and Proximity Hazards The software does not include dependency or proximity hazards. Paragraph Calculate Risk The software does not calculate risk directly but instead uses a proxy measure, the calculation and manipulation of which is unclear. Paragraph Calculate Resilience the software does not address the issue of resiliency. Automated Risk Assessment Methodology for Water and Wastewater Utilities (ARAM-W ) As stated in the ARAM-W User s Manual, the Automated Risk Assessment Methodology for Water and Wastewater Utilities (ARAM-W ) software is a RAMCAP (2007) compliant risk assessment application for water, wastewater, and combined utilities. While the scope of this project did not include confirming whether the software is compliant with RAMCAP (2007), the analysis showed ARAM-W did not meet all of the features of J Standard. The ARAM-W software follows the Risk Assessment Methodology for Water and Wastewater Utilities (RAM-W ) steps, which in turn are roughly equivalent to the J Standard as outlined in Table 2.1 below. Table 2.1: Comparison of J Standard and ARAM-W Steps J Standard ARAM-W Paragraph 4.1 Asset Section 1 & 2. Planning & Facility Paragraph 4.2 Threat Section 4. Threat Assessment Paragraph 4.3 Consequence Analysis Section 3. Consequence Assessment Paragraph 4.4 Vulnerability Analysis Paragraph 4.5 Threat Analysis Section 5. Vulnerability Analysis Paragraph 4.6 Risk/Resilience Analysis Section 6. Risk Analysis Paragraph 4.7 Risk/Resilience Management Section 7. Risk Management & Reduction However, while the J Standard states that the steps can be performed out of order, trying to ensure that the Standard is being followed while using ARAM-W can be quite confusing as either ARAM-W or the Standard need to be executed out of order. There are also instances where the ARAM-W steps account for multiple parts of the J Standard. For consistency, the analysis below follows the order of J Standard steps.

34 18 Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools Paragraph 4.1: Asset. The purpose of asset characterization is to identify critical assets to be considered in the subsequent steps Mission. No gap identified. The software meets the requirement of the J Standard to identify the utility s mission. The ARAM-W software requires the practitioner to input the utility s missions and rank them Critical Assets. No gap identified. To meet the Standard, the software practitioner must create a list of all of the utility s assets and select those that are critical. The software requires that the practitioner select the undesired events that may impact the facility and associate them with specific assets Supporting Infrastructures No gap identified. To meet the Standard, the software must allow for critical internal or external supporting infrastructures such as financial records, legal documents, planning documents, mutual aid agreements, etc., to be identified. The ARAM-W software meets the Standard by requiring the practitioner to select infrastructure categories when defining the asset type Countermeasures and mitigation measures/features No gap identified. The ARAM-W software meets the requirement by having the practitioner include countermeasures, descriptions, and/or details during the creation of the Adversary Sequence Diagrams Consequence metrics Meets J Standard. Improvements suggested. To meet the Standard, the software must allow the practitioner to estimate the worst reasonable consequences for each asset without regard to the threat. These consequence metrics include potential for fatalities, serious injuries, major economic loss to facility or community, impacts to the environment, loss of public confidence, and inhibiting effective function of national defense or civilian government. The ARAM-W software meets the Standard by providing a Consequence Matrix for the practitioner to define the consequence metrics for the utility. This allows the practitioner to select most of the consequences identified in the J Standard and to add additional ones. To meet the full intent of the J Standard, the software could also include impacts to the environment as a standard consequence Prioritize Assets - No gap identified. To meet the Standard, the software must allow the practitioner to identify assets as critical and to rank them using categories of high, medium and low. The ARAM-W software meets the Standard by having the practitioner rank the facilities against each other. The facilities are then paired with the missions. The Mission Score is multiplied by the Facility Score to determine the Total Mission Score of each facility. These Total Mission Scores are then used to rank the facilities. The practitioners can then select the facilities they consider critical and continue the assessment with just those facilities. Next, using a predefined fault tree for water/wastewater treatment, collection or distribution, source, storage, or a generic water utility, the practitioner can use or alter predefined fault trees and select the most reasonable undesired events that will have serious consequences to their facilities. The development of the facility-specific fault tree shows the practitioner the possible paths that could cause the undesired events and allows the practitioner to drill down to the asset level. Those assets are then used for the remainder of the assessment. Paragraph 4.2: Threat. Once the critical assets have been determined, the practitioner must define the threats that could potentially impact each critical asset.

35 Chapter 2: Gap Analysis Malevolent Threat No gap identified. The software uses J Standard threats Natural Hazards Threat Meets J Standard. Improvements suggested. To meet the Standard, when selecting the natural hazard threats the software must allow the practitioner to define the range of magnitudes, from the smallest magnitude that could cause serious harm to the largest reasonable magnitude, of each natural hazard. When selecting natural hazards, the ARAM-W software allows the practitioner to select from four predefined natural hazard threats (earthquakes, hurricanes, tornadoes, and floods). Although these are the only natural hazard threats required to meet the J Standard, to meet the full intent of the Standard the software could also include wild fires, ice storms, and the option to add additional natural threats, if desired Dependency Hazards Threat Meets J Standard. Improvements suggested. The Standard requires the practitioner to define any dependency threats due to interruptions in utilities, suppliers, employees, customers, and transportation, as well as threats due to the close proximity of the utility to dangerous neighboring sites. Although the ARAM-W software allows practitioners to manually add these threats to their lists of malevolent threats, the J Standard reference table is not provided and might be helpful Threat-Asset Pairs No gap identified. The Standard requires the practitioner to be able to assign potential threats to each asset. The software meets the Standard by having the practitioner assign threats to each undesired event/asset location pair, creating undesired event/asset location/threat pairs (triplets) Threat-Asset Pair Ranking Gap identified. To meet the Standard, once all of the potential threats have been assigned to each asset (thus creating threat-asset pairs), the practitioner must then rank them (using professional judgment) in order from the worst to the least resulting consequences. The ARAM-W software does not allow the practitioner to rate or rank the pairs prior to determining vulnerability or threat likelihoods Critical Threat-Asset Pairs Meets J Standard. Improvements suggested. To meet the Standard, the practitioner must use the ranking developed under paragraph to select the critical threat-asset pairs to be further analyzed (or alternately to treat them all as critical). The ARAM-W software does not allow the practitioner to choose critical threat-asset pairs and does not allow the practitioner to remove pairs from the list. Allowing the practitioner to remove threat-asset pairs would be a helpful improvement for practitioners with large numbers of threat-asset pairs that are not critical. Paragraph 4.3: Consequence Analysis. Once the critical threat-asset pairs are identified, the worst reasonable consequences that can be caused by the specific threats on the assets are defined Threat Scenario Gap identified. To meet the Standard, the software must allow the practitioner to identify the worst reasonable consequences that can be caused by specific threats on specific assets for each threat-asset pair. Although the consequences for the loss of critical assets and the resulting undesired event occurring were previously determined, these are not evaluated for each specific threat on the asset Estimate Consequences Gap identified. To meet the Standard, the software must provide the practitioner with three options for estimating fatalities, serious injuries, financial loss to the owner/operator, financial loss to the community, and duration and severity of service

36 20 Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools denial for the affected customers of the utility. These include a single-point estimate, a single indicator (a bin value) and a range. The ARAM-W software partially meets the Standard. The software references the RAMCAP Consequence Table that lists units of consequence for consequences. However, ARAM-W groups the values into more granular units (very low, low, medium, high and very high) than the J Standard does. Thus, this portion of the J Standard is not fully met because the software does not use the same bin ranges Estimate Consequences (other) No gap identified. To meet the Standard, the software must allow the practitioner to add additional consequences, if desired. The ARAM- W software meets the Standard by allowing practitioners to add any additional consequence metrics that they feel apply to the loss of a critical asset caused by an undesired event Document Assumptions No gap identified. To meet the Standard, the software must provide a field for practitioners to document their assumptions and procedures for performing the consequence analysis. The software meets the Standard by allowing practitioners to document their assumptions in a text field Record Consequence No gap identified. To meet the Standard the software must provide both a field for consequence ranges and a field for the practitioner to insert a point estimate. The software meets the Standard by allowing the practitioner to select consequence ranges and by also allowing the practitioner to enter a consequence point estimate. Paragraph 4.4: Vulnerability Analysis. A key component of the J Standard risk assessment is the vulnerability analysis, which determines the likelihood that a given malevolent threat or natural hazard threat occurs Review No gap identified. To meet the Standard, the software must provide fields to input pertinent details of utility/facility construction, systems, and layout. The software meets the Standard by having a Site Survey portion to enter and save data worksheets Analyze Vulnerability Meets J Standard. Improvements suggested. To meet the Standard, the software must provide a field for the practitioner to analyze the vulnerabilities of each asset to estimate the likelihood that, given the occurrence of a threat, the estimated consequences will result. The practitioner does this by utilizing fault, event or failure trees, path analysis, vulnerability logic diagrams, computer simulation methods, or expert judgment. The ARAM-W software meets the Standard by allowing the practitioner to create Adversary Sequence Diagrams (ASD) for each of the undesired event/asset location/threat pairs. The software could better meet the Standard by providing a means to use other methods to analyze vulnerability Document Method No gap identified. To meet the Standard, the software must have a field where practitioners can define their methods of analyzing vulnerability (i.e. those listed in paragraph 4.4.2). The ARAM-W software meets the Standard by allowing the practitioner to type in a justification for the vulnerability estimate to document the method used Record Estimates No gap identified. To meet the Standard, the software must have a field where the practitioner can record the vulnerabilities calculated for each critical asset. The ARAM-W software meets the Standard by allowing the practitioner to use either the ASD-based analysis or a User Estimate to estimate vulnerabilities.

37 Chapter 2: Gap Analysis 21 Paragraph 4.5: Threat Analysis. In the Threat Analysis step the practitioner determines the likelihood (or frequency) that a specific malevolent event, dependency/proximity hazard, or natural hazard will occur to a specific critical asset Malevolent Threats - Meets J Standard. Improvements suggested. To meet the Standard, the software must include malevolent threat likelihood calculations using proxy measures, best estimates, or conditional assessments. The ARAM-W software meets the Standard minimum by allowing the practitioner to determine the threat likelihood using conditional, expert judgment, or a questionnaire method. For the conditional method, the likelihood value is automatically considered to be 100% based on the assumption that the attack will in fact occur. For the expert judgment method, the practitioner can input a likelihood of high, medium, or low from a drop-down menu. For the questionnaire method, the likelihood value is determined based on the responses to approximately 10 questions. The questions align with the threat factors outlined in the RAMCAP Framework document for estimating the likelihood of attack, and include initial considerations of capability, history, current interest, current surveillance, documented threats, potential consequences, ideology, and ease of attack. The software could better meet the intent of the J Standard by also including a proxy indicator method for determining the likelihood of a malevolent threat Natural Hazards Meets J Standard. Improvements suggested. To meet the Standard, the software must include natural hazards threat likelihood calculations. It must also allow the practitioner to assign the likelihood or frequency of a natural hazard to the asset, based on historical data that may be provided by the software via maps, data, or links to reference materials. The software must then calculate the risk of each natural hazard and sum them to determine the overall risk due to natural hazards. The ARAM-W software allows the practitioner to assign the frequency to natural hazards, but it does not sum them to determine an overall risk due to natural hazards. Also, although the software does allow the practitioner to define the likelihood of the four mandatory natural hazard threats, it would better meet the intent of the Standard if it also allowed the practitioner to assign and define other natural hazard threats, such as wildfires and ice storms, and their probabilities Dependency and Proximity Hazards Gap identified. To meet the Standard, the software must include dependency and proximity hazards threat likelihood calculations. It must also include historical data on dependency and proximity hazards to determine the likelihood that the threats will occur to the assets. The investigators were unable to locate a field to identify dependency and proximity hazards threat likelihoods Record Estimates Gap identified. To meet the Standard, the software must allow the practitioner to determine the likelihood of each specific threat occurring to each specific asset and record this estimate, along with the method and reasoning for the estimate. The ARAM- W software partially meets the Standard by allowing the practitioner to determine the likelihood of the malevolent and natural threats, but it does not seem to allow the practitioner to address dependency and proximity hazards. Paragraph 4.6: Risk/Resilience Analysis. Once the consequence, vulnerability, and threat likelihood have been determined for each threat-asset pair, the overall risk and resilience of the utility is calculated.

38 22 Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools Calculate Risk Meets J Standard. Improvements suggested. To meet the Standard, the software must allow the practitioner to calculate the risk associated with each threat-asset pair based on the consequence, vulnerability, and threat likelihood values selected during earlier steps. The ARAM-W software meets the Standard by calculating risk for each individual undesired event/asset location/threat pair and displays the results as no risk, low, medium, high, or very high. Unfortunately, the software does not display the results of consequence, vulnerability, threat likelihood, and risk as numerical quantities but as values of very low, low, medium, high and very high. Thus, checking the risk calculations is not possible Calculate Resilience Gap identified. To meet the Standard, the software must determine the overall resilience of the utility, including the duration of service denial and severity of service denial (in gallons per day), in order to determine the resilience of each threatasset pair. The ARAM-W software does not execute any resilience calculations as outlined in the following paragraphs Asset Resilience Metric Gap identified. To meet the Standard, the software must calculate the asset s resilience Owner s Economic Resilience - Gap identified. To meet the J Standard, the software must calculate the owner s economic resilience Community Economic Resilience - Gap identified. To meet the J Standard, the software must calculate the community s economic resilience Record Risk and Resilience Estimates Gap identified. To meet the J Standard, the software must calculate the overall risk to the utility using the J Standard risk equation; the risk to each specific threat-asset pair; and the different types of resilience as defined in paragraphs through Paragraph 4.7: Risk/Resilience Management. Once the risk of the utility and of each threat-asset pair has been determined, the utility continues with the process by deciding whether mitigation actions are needed to enhance all-hazards security or resilience or both. As the ARAM-W software does not address resilience, most of the sub-features under Paragraph 4.7 are not met Decide Gap identified. To meet the J Standard, the software must allow the practitioner to decide what risk and resilience levels are acceptable. The ARAM-W software partially meets the Standard in that it includes a field to determine if the risk for each undesired event/asset location/threat pair is acceptable. However, the software does not include resilience in this decision Define Gap identified. To meet the J Standard, the software must allow the practitioner to define new countermeasures and mitigation/resilience options to reduce unacceptable risk to specific threat-asset pairs. The ARAM-W software enables the practitioner to create potential upgrade packages to mitigate or reduce the risk to facilities, but it does not have options to increase resiliency Estimate No gap identified. To meet the J Standard, the software must allow the practitioner to estimate the costs of the new countermeasures and mitigation/resilience options. The ARAM-W software allows the practitioner to enter cost estimates and impacts on operations, schedule and public opinion for each upgrade package created Assess Gap identified. To meet the J Standard, the software must allow the practitioner to assess the options by analyzing the facility or asset under the assumption that

39 Chapter 2: Gap Analysis 23 the option has been implemented. The ARAM-W software calculates the new risk, vulnerability, threat potential and consequence of each threat-asset pair for each upgrade package, but it does not include resiliency in the assessment Identify Gap identified. To meet the J Standard, the software must allow the practitioner to identify those options that have benefits to multiple threat-asset pairs. The ARAM-W software displays a report of the baseline data, including risk, vulnerability, potential/frequency, and consequence for each threat-asset pair. The software also displays the upgrade packages that apply to the highest risk threat-asset pair, but it does not determine which improvements are beneficial to multiple threat-asset pairs Calculate Gap identified. To meet the J Standard, the software must allow the practitioner to calculate the net benefits and benefit-cost ratios of the selected countermeasures. The ARAM-W software does not do this Review & Rank Gap identified. To meet the J Standard, the software must allow the practitioner to review the selected countermeasures and rank them in order to determine which ones will be most effective in reducing the utility s risk. The ARAM-W software does not do this. Summary. In summary, the ARAM-W software is a user-friendly risk assessment tool that utilizes many of the J Standard tables and definitions for threats and consequences. It also has multiple points in the process where existing information can be saved to create a comprehensive assessment of the utility. However, the fact that the software does not follow the order of the Standard makes utilizing the software and the Standard together a challenge. The analysis indicated that there are some inconsistencies between the ARAM-W software and the J Standard, the largest of which are as follows: Paragraph Threat Scenario The consequences for the loss of critical assets are not evaluated for each specific threat on the asset. Paragraph Dependency and Proximity Hazards The software does not include dependency or proximity hazards. Paragraph Calculate Resilience The software does not address the issue of resiliency.

40 24 Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools

41 Chapter 3: Recommendations 25 CHAPTER 3: RECOMMENDATIONS INTRODUCTION The recommendations included in this chapter were developed based on the identified discrepancies between the J Standard and the software packages as outlined in Chapter 2. Recommendations to address these discrepancies were developed for the three software packages (SEMS, VSAT and ARAM-W ) and were determined by analyzing to the extent possible the databases used by each package. In addition, a range of the number of hours needed to implement the recommendations was estimated. These estimates were made assuming that they would be implemented by experienced IT personnel. METHOD In order to develop recommendations to address the identified gaps in the three software packages, the researchers first attempted to discover and understand each package s underlying database structure. This was done in order to determine how easily those structures could be extended or modified to close each of the gaps found during the assessment. While SEMS and VSAT use standard database formats, and the researchers were able to view their contents, ARAM-W uses a proprietary, encrypted format that halted any further inspection on the part of the investigators. This prevented detailed exploration of the capabilities of the software. Thus, a more accurate estimate as to how changes could be made to SEMS and VSAT was possible. The gaps found between the software and the requirements of the J Standard were placed into four broad categories for the purposes of calculating the amount of development effort that might be required to bring the software packages in line with the Standard. The four categories of software upgrades were small, medium, large, and unknown. A general description of each follows: Small upgrades were generally minor issues, often a modification of the user interface in some way. Because these changes are simple to implement, it was estimated that they should take one to two days to resolve. Medium upgrades were either improvements to existing functionality or the addition of missing features; in either case, it was estimated that it would require 16 to 24 hours to correct. Large upgrades might require entire pieces of new functionality. However, these fixes would also require time-intensive modifications that may take one to three weeks of work to complete. Unknown upgrades were those where there was some impediment in the investigator s analysis; thus obtaining an accurate estimate of effort needed to correct the gap was not possible. Challenges to further analysis by the investigators included: 1. inability to determine database structure and thus the ease or difficulty of the changes required, 2. unknown calculation details, and 3. inability to determine how the software operates when given incorrect input or when it encounters extreme and likely incorrect results. 25

42 26 Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools Additionally, upgrades of unknown size were further divided, where possible, based on assumptions regarding the developers use of standard programming and software design practices. This allowed many of the upgrades of unknown effort to be estimated with further precision small, medium, and large. While there is no way to ensure that these effort descriptors are correct without further examination of the software structure and code, they represent an educated guess as to the amount of effort required, assuming that the developers followed industry best practices. Based on the amount of effort estimated for each recommendation, an estimated number of hours was determined to bring each lacking feature into compliance with the Standard. SEMS In general, SEMS has more gaps to address than either VSAT or ARAM-W and would most likely require more time and resources to bring it into compliance with the J Standard. The required total number of upgrades (41) means that SEMS meets approximately 48% of the 79 Standard features (Table 3.1). Table 3.1: SEMS Summary Table Size of Upgrade Number of Required Upgrades Range of Estimated Hours Estimated Hours of Required Upgrades Small Medium Large Unknown (likely small) Unknown (likely medium) Unknown (likely large) Unknown 7 (unknown) TOTAL 41 1,240 - Summary of large upgrades: 1. Providing the ability to calculate the resilience of assets. The most cost-effective upgrade (although it is a large change) might be changing SEMS to allow for repeated analysis runs with different sets of countermeasures without starting the analysis from the beginning. In the investigator s opinion this would greatly improve usability. VSAT VSAT has fewer issues than SEMS, and because of the availability of the database structure, the number of upgrades of indeterminate effort is the lowest of the three software

43 Chapter 3: Recommendations 27 packages. In addition only 27 upgrades are needed to meet the 79 Standard features; or in other words, VSAT is 66% compliant (Table 3.2). However, the estimated number of changes that will likely be large should also be taken into consideration. In particular, attention should be given to the fact that VSAT does not absolutely quantify the amount of risk to which critical assets are exposed. Table 3.2: VSAT Summary Table Size of Upgrade Number of Required Upgrades Range of Estimated Hours Estimated Hours of Required Upgrades Small Medium Large Unknown (likely small) Unknown (likely medium) Unknown (likely large) Unknown 5 (unknown) TOTAL 27 1,152 - Summary of large upgrades: 1. Calculating resiliency. In the opinion of the investigators, the most cost-effective upgrade would be the calculation of the net benefits of each package of countermeasures. If this relatively simple step were done, quite a few other gaps would be resolved (in particular, the calculation of the costbenefit ratio of each package of countermeasures and the determination of the most effective countermeasures). ARAM-W ARAM-W has a larger proportion of issues of unknown size, due to the fact that its database structure was not available to the investigators. However, this is offset by the fact that it has the least number of overall gaps compared to the other software packages. In particular, only 25 upgrades are needed to meet the 79 Standard features; or in other words, ARAM-W is 68% compliant (Table 3.3). Of the 8 recommended upgrades where the amount of effort to resolve them is unknown, half can be resolved with the implementation of resilience calculations. In general, it appears that ARAM-W meets the greatest part of the Standard and thus would require the least amount of corrective actions to completely satisfy the requirements of the Standard.

44 28 Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools Table 3.3: ARAM-W TM Summary Table Size of Upgrade Number of Required Upgrades Range of Estimated Hours Estimated Hours of Required Upgrades Small Medium Large Unknown (likely small) Unknown (likely medium) Unknown (likely large) Unknown 8 (unknown) TOTAL Summary of large upgrades: Usability 1. Incorporating the ability to account for dependency threats to the assets, as well as including the proxy method of assessing the risk level to each facility. 2. Calculating resiliency. The most cost-effective single upgrade might be the addition of resilience calculations. It should be noted that while the primary focus of the analysis was the functionality of the three software packages, other important considerations such as the software interface and process flow could also be important when selecting a software package. The user interfaces of the different software packages are quite different. The SEMS RAMCAP assessment is just one component of the larger SEMS software system. Thus, its menu-based navigation requires the user to constantly return to the main screen and review multiple past screens to proceed with the assessment. The investigators were not able to find any workaround for this. On the other hand, the VSAT navigation is tab-based, which makes navigation between components and steps in the analysis much more logical and intuitive. In contrast, ARAM-W s navigation is a bit awkward, with entire tabs of the interface hidden from the practitioner until a single check box is selected. The flow through the different software packages also varies. SEMS and VSAT encourage a forward flow of information from section to section. Each begins by asking for facility information, defining assets, countermeasures, and potential threats. At this point, their flows diverge: while SEMS performs a one-time analysis, VSAT allows for repeated iterations of risk level analysis, followed by options for improvement. In contrast, ARAM-W starts the analysis procedure by identifying harmful consequences and then works backwards from there to

45 Chapter 3: Recommendations 29 determine the risk level. While both approaches have the potential to meet the Standard, the forward flow of SEMS and VSAT are more intuitive and follow the flow of the Standard more closely. CONCLUSION Of the three software packages tested for compliance with the J Standard, ARAM- W TM appears to require the fewest upgrades, as it currently meets 68% of the features. VSAT also addresses a high number of the Standard s features (66%), while SEMS addresses the fewest (48%). None of the software tools calculate resilience the largest single inconsistency with the J Standard, and one that accounts for a number of peripheral inconsistencies in each package. While the project investigators did their best to estimate the amount of effort needed to upgrade each software package s features to meet the Standard, the large number of unknowns made this difficult. Focusing on addressing the discrepancy between the J Standard and the software tools around the issue of resilience could quickly bring VSAT and/or ARAM-W TM much closer to compliance with the Standard. In addition, clarifying with the software developers the exact labor needed to address the larger unknown upgrades is also a logical next step.

46 30 Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools

47 Appendix A: J Features 31 APPENDI A: J FEATURES 31

48 Table A.1 - J Standard Features Matrix Feature Met Reference Section Title Mandatory Features Non-mandatory Features Notes Yes No Some Explanation 1 Scope all-hazards risk and resilience analysis of vulnerabilities to man-made threats, natural hazards, and dependencies and proximity to hazardous sites Definitions 2.1 asset critical asset 2.2 consequence 2.3 consequence mitigation 2.4 countermeasure detect deter devalue delay respond 2.5 dependency 2.6 dependency hazard 2.7 event tree analysis 2.8 failure mode 2.9 fault tree analysis 2.10 frequency 2.11 hazard 2.12 incident 2.13 initiating event 2.14 likelihood 2.15 preparedness 2.16 probability 2.17 proximity hazard 2.18 response 2.19 reference threat 2.2 resilience 2.21 risk 2.22 risk analysis 2.23 risk management 2.24 scenario 2.25 system 2.26 threat 2.27 threat likelihood 2.28 vulnerability vulnerability assessment/vulnerability 2.29 analysis 2.30 vulnerability estimate scope must encompass the same requirements definitions must be the same when provided 32

49 Feature Met Reference Section Title Mandatory Features Non-mandatory Features Notes Yes No Some Explanation 2.31 worst reasonable case 3 Bibliography - 4 Requirements Asset - identify the mission or critical functions to determine which assets perform or support the multiple fields for mission or Mission mission critical functions 4.1.2, A.4.1 Critical Assets identify critical assets multiple fields for critical assets Supporting Infrastructures identify critical internal and external supporting infrastructure multiple fields for critical internal or external supporting infrastructure Countermeasures and mitigation measures/features identify and document existing protective countermeasures and mitigation measures/features estimate worst reasonable consequences for each asset without regard to the threat Consequence metrics 4.1.5(.1) Consequence metrics potential for fatalities 4.1.5(.2) Consequence metrics serious injuries major economic loss to facility 4.1.5(.3) Consequence metrics or community 4.1.5(.4) Consequence metrics impacts to the environment 4.1.5(.5) Consequence metrics loss of public confidence inhibiting effective function of national defense or civilian 4.1.5(.6) Consequence metrics government Prioritize assets prioritize critical assets using estimated consequences refer to definition for items to list multiple fields for all existing protective countermeasures and mitigation measures/features see below Can group these into Human, $ and other (i.e.. environmental) Can use Hi, Med, Lo, V Hi, etc. field for each consequence field for each critical asset to assign rank or reorder them in order of consequence man-made hazards or accidents, natural hazards, dependency hazards; identify general and - 4.2, A.4.2 Threat specific threat scenarios to serve as reference threats Appx E Reference Threats listed as shall under body preferred method , Appx E Malevolent Threat Air attack field to describe the type of threat to be considered 4.2.1, Appx E Malevolent Threat Land attack field to describe the type of threat to be considered 4.2.1, Appx E Malevolent Threat Water attack field to describe the type of threat to be considered 4.2.1, Appx E Malevolent Threat various magnitudes of attack elements field to describe the type of threat to be considered 33

50 Feature Met Reference Section Title Mandatory Features Non-mandatory Features Notes Yes No Some Explanation Malevolent Threat field to describe the type of threat 4.2.1, Appx E Weapons types to be considered Malevolent Threat field to describe the type of threat 4.2.1, Appx E equipment to be considered Malevolent Threat field to describe the type of threat 4.2.1, Appx E tools to be considered Malevolent Threat field to describe the type of threat 4.2.1, Appx E explosives to be considered Malevolent Threat field to describe the type of threat 4.2.1, Appx E tactics to be considered Malevolent Threat field to describe the type of threat 4.2.1, Appx E means of delivery/transport to be considered Malevolent Threat field to describe the type of threat 4.2.1, Appx E number of adversaries to be considered Malevolent Threat field to describe the type of threat 4.2.1, Appx E insiders to be considered Malevolent Threat field to describe the type of threat 4.2.1, Appx E outsiders to be considered Natural Hazards Threat 4.2.2, Appx E & G hurricanes Natural Hazards Threat 4.2.2, Appx E & G floods Natural Hazards Threat software must define the range of 4.2.2, Appx E & G tornadoes magnitudes from the smallest that Natural Hazards Threat would cause serious harm to the 4.2.2, Appx E & G earthquakes largest reasonable case Natural Hazards Threat 4.2.2, Appx E & G wildfires Natural Hazards Threat 4.2.2, Appx E & G ice storms Natural Hazards Threat should give the user the option of 4.2.2, Appx E & G other including other threats Dependency Hazards Threat field to describe the type of threat 4.2.3, Appx G utilities to be considered Dependency Hazards Threat field to describe the type of threat 4.2.3, Appx G suppliers to be considered Dependency Hazards Threat field to describe the type of threat 4.2.3, Appx G employees to be considered Dependency Hazards Threat field to describe the type of threat 4.2.3, Appx G customers to be considered Dependency Hazards Threat field to describe the type of threat 4.2.3, Appx G transportation to be considered Dependency Hazards Threat field to describe the type of threat 4.2.3, Appx G proximity to be considered should give the user the option of Dependency Hazards Threat including other threats - not 4.2.3, Appx G other included in comparisons 34

51 Feature Met Reference Section Title Mandatory Features Non-mandatory Features Notes Yes No Some Explanation & 4.2.5? Threat-Asset Pairs Critical Threat-Asset Pairs evaluate and rank threat-asset pairs select critical threat-asset pairs to be used going forward or use all pairs can use multiple approaches to evaluate and rank pairs; matrix using small, med., large or scales 1-10, etc. - software to have matrix or another method of ranking pairs check box or field to identify selected critical threat-asset pairs 4.3, A.4.3 Consequence Analysis 4.3.1, Appx B Threat Scenario 4.3.2, Appx B 4.3.2, Appx B Estimate Consequences - loss of life to either employees or the general public Estimate Consequences - serious injury to either employees or the general public identifies the worst reasonable consequences that can be caused by the specific threats on the assets as identified in 4.1 apply worst reasonable case assumptions for each threat scenario measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in predefined ranges represented by the RAMCAP "bins" (Appx B) measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in predefined ranges represented by the RAMCAP "bins" (Appx B) optional single indicator - dollar equivalence of fatalities and serious injuries in excess of insurance optional single indicator - dollar equivalence of fatalities and serious injuries in excess of insurance field for single point estimate or bin number field for single point estimate or bin number , Appx B Estimate Consequences - Financial loses to owner/operator measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in predefined ranges represented by the RAMCAP "bins" (Appx B) field for single point estimate or bin number 35

52 Feature Met Reference Section Title Mandatory Features Non-mandatory Features Notes Yes No Some Explanation 4.3.2, Appx B Estimate Consequences - service denial for the affected customers measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in predefined ranges represented by the RAMCAP "bins" (Appx B) field for single value 4.3.2, Appx B Estimate Consequences - economic losses to society and the general public Estimate Consequences - other Document assumptions measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in predefined ranges represented by the RAMCAP "bins" (Appx B) if degradation in public confidence, environmental quality, ability of civilian or military agencies to function, etc. room for descriptive analysis must be provided document specific assumptions and procedures used for performing the consequence analysis, the worst reasonable case assumptions and the results of the consequence analysis optional single indicator - value of a statistical life additional consequences - sociopolitical impacts, natural security impacts, lost strategic capability to cause harm or output, detrimental effects on brand value, public confidence, psychological impacts, and environmental degradation field for single point estimate or bin number should give the user the option of including other consequences should give users a space to save this information possibly link files, the documentation could include maps and calculations - not required Record consequence record the consequence values using point estimates or ranges ranges in appx B field for ranges or point estimates 4.4, A.4.4 Vulnerability Analysis Review review pertinent details of the facility construction, systems and layout; identify vulnerabilities or weaknesses in the protection system fields to input pertinent details of construction, systems and layout 36

53 Feature Met Reference Section Title Mandatory Features Non-mandatory Features Notes Yes No Some Explanation 4.4.2, A.4.4 Analyze Vulnerability Document method analyze vulnerability of each critical asset to estimate the likelihood that, given the occurrence of a threat, the consequences result document method and results of the vulnerability analysis 4.4.4, Appx B Record Estimates Record the estimates from likelihood of malevolent event, dependency/proximity hazard or 4.5, A.4.5 Threat Likelihood Analysis natural hazard use proxy measure, best estimate, or conditional assessment to determine may use fault or event tree analysis, path analysis, vulnerability logic diagrams, computer simulation methods, or expert judgment rules-ofthumb use point estimates or RAMCAP scales; if bins are used, the midpoint is used for the calculation Proxy Measure (Appx F) is optional and preferred field to document method used field for point estimates, bin number or ranges 4.5.1, Appx F Malevolent Threats field for threat estimate may have a field for number of F.3.1 Proxy Indicator - Node 1 Number of U.S. attacks per year attacks Metro Region (RMS metro area F.3.2 Proxy Indicator - Node 2 classes) may have a field for likelihood Target Type (RMS target type F.3.3 Proxy Indicator - Node 3 analysis) may have a field for likelihood F.3.4 Proxy Indicator - Node 4 Proportion: Regional Number may have a field for likelihood may have a field for ratio of F.3.5 Proxy Indicator - Node 5 This Facility capacity to metro area may have a field for likelihood F.3.6 Proxy Indicator - Node 6 This Threat-Asset Pair (product of VxCxDetection) calculated by multiplying each F.3.7 Proxy Indicator - Node 7 Overall Proxy Likelihood proxy indicator 4.5.2, G.2 Natural Hazards earthquakes software must have a field to enter the risk of earthquakes for each magnitude from historical records , G.3 Natural Hazards hurricanes 4.5.2, G.4 Natural Hazards tornadoes 4.5.2, G.5 Natural Hazards floods Appx G is optional and provides data to estimate the risk of each natural hazard - risk calculated by CxVxT; would be nice to have look-up maps/tables for each natural hazard software must have a field to enter the risk of hurricanes for each magnitude from historical records software must have a field to enter the risk of tornadoes from historical records software must have a field to enter the risk of floods for each magnitude from historical records 37

54 Feature Met Reference Section Title Mandatory Features Non-mandatory Features Notes Yes No Some Explanation software may have a field to enter the risk of ice storms, extreme cold ice storms, extreme cold weather, wildfires, avalanche, weather, wildfires, avalanche, tsunami, landslide, mud slide from 4.5.2, Appx G Natural Hazards tsunami, landslide, mud slide historical records 4.5.2, Appx G Natural Hazards other 4.5.2, Appx G Natural Hazards Risk Dependency and Proximity Hazards Record Estimates 4.6, A.4.6 Risk and Resilience Analysis Calculate Risk 4.6.2, Appx H Calculate the current level of resilience Operational Resilience Asset Resilience Metric use local historical records for frequency, severity and duration of service denials record the method used for making the estimates and the estimates themselves as single point values or ranges estimates the owner's risk and resilience and the community's resilience relative to each threatasset pair for each threat-asset pair calculate risk: CxVxT=R use either threat-asset pair resilience metric or holistic approach in Appx H Duration x severity x vulnerability x threat likelihood = asset resilience metric use midpoint of ranges from Appx B Appx H is non-mandatory software may have a field to enter the risk of other natural hazards must have field for total natural hazard risk software must have a field for likelihood should have room for methods to be saved - software must calculate risk using the numbers input previously for C, V and T for each threat-asset pair Operational Resilience Index (ORI) calculated by choosing values from Table H-1 then calculating ORI by multiplying software must calculate asset the indicator value by the weight resilience metric using duration and and adding all values (should severity from 4.3, vulnerability have fields for the value and from 4.4 and threat likelihood from weight or by pick box) Owner's Economic Resilience Community Economic Resilience lost revenue due to the threatasset pair (asset resilience x unit price) lost economic activity to the community served by the utility Financial Resilience Index (FRI) calculated by choosing values from Table H-2 then calculating FRI by multiplying the indicator value by the weight and adding all values (should have fields for the value and weight or by pick box) software must calculate asset resilience x unit price (have field for unit price of asset) software must have field for lost economic activity to the community (same as 4.3) 38

55 Feature Met Reference Section Title Mandatory Features Non-mandatory Features Notes Yes No Some Explanation , A.4.7 Record Risk and Resilience Estimates Risk and Resilience Management Utility Resilience Index - software should calculate URI using the values of ORI and FRI and the weights given in Table H-1 & H-2 (URI=ORI x w1 + FRI x w2) software must have fields for threatasset pair resilience , A Decide 4.7.2, A Define 4.7.3, A Estimate 4.7.4, A Assess decide what risk and resilience levels are acceptable define countermeasure and mitigation/resilience options for the threat-asset pairs that are not acceptable. Include devalue, deter, detect, delay and response; consequence reductions, resilience enhancements estimate investment and operating costs for each option; include regular maintenance and periodic overhaul; adjust to present value revisit 4.3 through 4.6 to reestimate the risk and resilience levels as if the option was implemented; calculate the estimated benefits of the option identify the options that have benefits that apply to multiple threat-asset pairs field for acceptable risk and resilience level 4.7.5, A Identify 4.7.6, A Calculate calculate the net benefits field for value fields for each threat asset-pair fields for costs for each option above field for the new value of risk a way to highlight or mark the options 4.7.7, A Calculate calculate the benefit-cost ratio field for value rank the most cost effective 4.7.9, A Review & Rank measures to implement a field for rank 39

56 32 Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools

57 Appendix B: J Gap Analyses 33 APPENDI B: J GAP ANALYSES 41

58 Table.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap All-hazards risk and resilience analysis of vulnerabilities to manmade Scope must include all of the same requirements. 1 Scope threats, natural hazards, and dependencies and proximity to hazardous sites. Definitions Is the list the same? If not, how is it different? Is 2.1, Appx C asset the content for each definition the same? critical asset 2.2 consequence 2.3 consequence mitigation 2.4 countermeasure detect deter devalue delay respond 2.5 dependency 2.6 dependency hazard 2.7 event tree analysis 2.8 failure mode 2.9 fault tree analysis 2.10 frequency 2.11 hazard 2.12 incident 2.13 initiating event 2.14 likelihood 2.15 preparedness 2.16 probability 2.17 proximity hazard 2.18 response 2.19 reference threat 2.2 resilience 2.21 risk 2.22 risk analysis 2.23 risk management 2.24 scenario 2.25 system 2.26 threat 2.27 threat likelihood 2.28 vulnerability vulnerability assessment 2.29 /vulnerability analysis 2.30 Definitions vulnerability estimate 2.31 worst reasonable case 3, Appx D Bibliography not included in software analysis 4 Requirements 4.1 Asset The software does calculate risk, however, resilience is not calculated, there is no input for vulnerability, no predefined fields for dependency or proximity hazards, threat likelihoods, or scales for natural hazard magnitudes. No definitions are provided. 1 of 10 42

59 Table B.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Mission identify the mission or critical functions to determine which assets perform or support the mission Software must provide multiple fields for mission or critical functions. Software provides drinking water and wastewater systems missions to choose from. Several generic missions are provided (bubble checks) and an option to write your own (text box) is included , A.4.1 Critical Assets identify critical assets Supporting Infrastructures Countermeasures and mitigation measures/features identify critical internal and external supporting infrastructure identify and document existing protective countermeasures and mitigation measures/features estimate worst reasonable consequences for each asset without regard to the threat refer to definition for types of items Software must provide multiple fields for critical assets. Software must provide multiple fields for critical internal or external supporting infrastructure. Software must provide multiple fields for all existing protective countermeasures and mitigation measures/features. Software must estimate the worst reasonable Consequence metrics consequences for each asset (.1) Consequence metrics potential for fatalities 4.1.5(.2) Consequence metrics serious injuries major economic loss to facility or 4.1.5(.3) Consequence metrics community Can group these into Human, $ and other (i.e (.4) Consequence metrics impacts to the environment environmental) Can use Hi, Very Hi, Med, Lo, etc (.5) Consequence metrics loss of public confidence field for each consequence (.6) Consequence metrics inhibiting effective function of national defense or civilian government Software allows user to add descriptions to predefined assets or create new ones. User can define asset priority (dropdown menu for low, med., or high). Software includes two check boxes for the following options: including the asset in the risk assessment (and reason for opting out) and if it is an emergency asset. There is no separate field for critical infrastructure, but the user has the ability to create their own assets. The software enables the user to create infrastructure as an asset and define it as critical and have it included in the analysis. There is a check list of all the predefined countermeasures. User can add descriptions and detail to these (text box) or create your own. Define the detection (dropdown none, possible, probable, or certain), delay (dropdown none, limited, strong, or very strong), and response (dropdown none, slow, variable, or fast). The software does not allow the user to define the worst reasonable consequences without regard to threat. 2 of 10 43

60 Table B.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Prioritize assets prioritize critical assets using estimated consequences Software must provide a field for each critical asset to assign rank or re-order them in order of consequence. The user can choose if the asset is critical or not and prioritize (using a dropdown menu of high, med., or low) but can not reprioritize based on consequences ( no list of asset and consequences in comparison). 4.2, A.4.2 Threat Appx E 4.2.1, Appx E 4.2.1, Appx E 4.2.1, Appx E Reference Threats Malevolent Threat Malevolent Threat Malevolent Threat man-made hazards or accidents, natural hazards, dependency hazards; identify general and specific threat scenarios to serve as reference threats Within the Standard body Reference Threats are mandatory and Appx E is mandatory Air attack Land attack Water attack Software must provide a field to characterize the kind of hazard/threat. - malevolent threat to be considered and the outcome of which is included in the software calculations. malevolent threat to be considered and the outcome of which is included in the software calculations. malevolent threat to be considered and the outcome of which is included in the software calculations. The software includes a list of most of the J threats though no proximity, dependency and some specific natural hazards are included. Check boxes allow the user to select of all that apply for each asset, creating asset-threat pairs. The user can an add their own threats and add descriptions and details (text boxes) , Appx E Malevolent Threat various magnitudes of attack elements malevolent threat to be considered and the outcome of which is included in the software calculations. Same as J , Appx E Malevolent Threat Weapons types malevolent threat to be considered and the outcome of which is included in the software calculations , Appx E Malevolent Threat equipment malevolent threat to be considered and the outcome of which is included in the software calculations , Appx E Malevolent Threat tools malevolent threat to be considered and the outcome of which is included in the software calculations. 3 of 10 44

61 Table B.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap 4.2.1, Appx E Malevolent Threat explosives malevolent threat to be considered and the outcome of which is included in the software calculations , Appx E Malevolent Threat tactics malevolent threat to be considered and the outcome of which is included in the software calculations , Appx E 4.2.1, Appx E 4.2.1, Appx E 4.2.1, Appx E 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.2, Appx E & G Malevolent Threat Malevolent Threat Malevolent Threat Malevolent Threat Natural Hazards Threat Natural Hazards Threat Natural Hazards Threat Natural Hazards Threat Natural Hazards Threat Natural Hazards Threat Natural Hazards Threat means of delivery/transport number of adversaries insiders outsiders hurricanes floods tornadoes earthquakes wildfires ice storms Undefined other malevolent threat to be considered and the outcome of which is included in the software calculations. malevolent threat to be considered and the outcome of which is included in the software calculations. Non-mandatory in 4.2, mandatory in 4.2.1, shall be analyzed as mandatory. Software must provide a field to describe the type of malevolent threat to be considered. Non-mandatory in 4.2, mandatory in 4.2.1, shall be analyzed as mandatory. Software must provide a field to describe the type of malevolent threat to be considered. Software must define the range of magnitudes from the smallest that would cause serious harm to the largest reasonable case. Software must define the range of magnitudes from the smallest that would cause serious harm to the largest reasonable case. Not listed in body, listed in non-mandatory appendix, analyzed as non-mandatory. Software should give the user the option of including other natural hazards or threats. Same as J The software does not provide ranges or direct links to reference materials to determine magnitudes. However it does say what standards should be used for the magnitudes. Also, changing the magnitude of the natural disaster does not affect the likelihood or overall risk. The software does not include these as predefined but can be added as an "other". No ranges or direct links to reference materials to determine magnitudes , Appx G 4.2.3, Appx G Dependency Hazards Threat Dependency Hazards Threat utilities suppliers dependency threat to be considered. dependency threat to be considered. The software does not include these as predefined fields but they could potentially be added as an "other" , Appx G Dependency Hazards Threat employees dependency threat to be considered. 4 of 10 45

62 Table B.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap 4.2.3, Appx G Dependency Hazards Threat customers dependency threat to be considered , Appx G 4.2.3, Appx G 4.2.3, Appx G Dependency Hazards Threat Dependency Hazards Threat Dependency Hazards Threat transportation proximity Threat-Asset Pairs create threat-asset pairs Evaluate and Rank Threat- Asset Pairs Critical Threat-Asset Pairs 4.3, A.4.3 Consequence Analysis 4.3.1, Appx B Threat Scenario 4.3.2, Appx B 4.3.2, Appx B Estimate Consequences - loss of life to either employees or the general public Estimate Consequences - serious injury to either employees or the general public evaluate and rank threat-asset pairs select critical threat-asset pairs to be used going forward or use all pairs identifies the worst reasonable consequences that can be caused by the specific threats on the assets as identified in 4.1 apply worst reasonable case assumptions for each threat scenario measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) Undefined other dependency threat to be considered. dependency threat to be considered. Software should give the user the option of including other kinds of dependency hazards or threats. Software must allow user to create threat-asset pairs. Software must evaluate and rank threat-asset pairs, can use multiple approaches including a matrix using small, med., large or scales 1-10, etc. Software must provide a check box or field to identify selected critical threat-asset pairs. (Standard does not define critical, this is left up to the user, i.e.. top 10, top 20?) Software must identify the worst reasonable consequence of a threat on assets. Software must assume the worst reasonable case for each threat. Software must provide a field for single point optional single indicator - estimate of consequences or a bin number. When dollar equivalent of fatalities reviewing bins, the values must match. Under 4.3 and serious injuries in excess this is defined as an "or" under it is defined as of insurance an "and" analysis shall include "and" as a more conservative approach. optional single indicator - dollar equivalence of fatalities and serious injuries in excess of insurance Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach. The software does not include these as predefined fields but they could potentially be added as an "other". The software does not rate or rank them based on rough magnitude of consequences prior to determining vulnerability or threat likelihoods. The user can select critical assets but can not select critical threat-asset pairs. The software does not include single point estimates or bin numbers. It uses a dropdown menu of J standard ranges. 5 of 10 46

63 Table B.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap 4.3.2, Appx B 4.3.2, Appx B 4.3.2, Appx B Estimate Consequences - Financial losses to owner/operator Estimate Consequences - service denial for the affected customers Estimate Consequences - economic losses to society and the general public Estimate Consequences - other measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) if degradation in public confidence, environmental quality, ability of civilian or military agencies to function, etc. room for descriptive analysis must be provided optional single indicator - value of a statistical life additional consequences that can be considered - sociopolitical impacts, natural security impacts, lost strategic capability to cause harm or output, detrimental effects on brand value, public confidence, psychological impacts, and environmental degradation Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach. Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach. Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach. Software must provide the ability to describe primary consequence, and should give the user the option of including other consequences. The software does not include single point estimates or bin numbers. It uses a dropdown menu of J standard ranges. The software does not include the consequence of service denial for the affected customers. The software does not inlcude single point estimates or bin numbers. It uses a dropdown menu of J standard ranges The software only provides fields for 4 consequence types (fatalities, injuries, economic cost of owner, and economic cost to region) Document assumptions 4.3.5, Appx B Record consequence 4.4, A.4.4 Vulnerability Analysis document specific assumptions and procedures used for performing the consequence analysis, the worst reasonable case assumptions and the results of the consequence analysis record the consequence values using point estimates or ranges Preferred ranges are in Appx B. Software must give users a space to document the assumptions made in the analysis, and should have the ability to include other documentation such as maps and calculations. Software must provide a field for ranges or point estimates. Document assumptions can be typed into the generated report. The software only uses ranges for the consequence value. 6 of 10 47

64 Table B.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Review review pertinent details of the facility construction, systems and layout; identify vulnerabilities or weaknesses in the protection system Software must provide fields to input pertinent details of construction, systems and layout. The user can add in as general information or security information during the creation of the asset (text boxes) , A.4.4 Analyze Vulnerability Document Method document method and results of the vulnerability analysis 4.4.4, Appx B Record Estimates Record the estimates from , A.4.5 Threat Likelihood Analysis likelihood of malevolent event, dependency/proximity hazard or natural hazard 4.5.1, Appx F Malevolent Threats F.3.1 Proxy Indicator - Node 1 F.3.2 Proxy Indicator - Node 2 F.3.3 Proxy Indicator - Node 3 F.3.4 Proxy Indicator - Node 4 analyze vulnerability of each critical asset to estimate the likelihood that, given the occurrence of a threat, the consequences result use proxy measure, best estimate, or conditional assessment to determine F.3.5 Proxy Indicator - Node 5 This Facility may use fault, event or failure tree analysis, path analysis, vulnerability logic diagrams, computer simulation methods, or expert judgment rules-ofthumb use point estimates or RAMCAP scales; if bins are used, the midpoint is used for the calculation Proxy Measure (Appx F) is optional and preferred Number of U.S. attacks per year Metro Region (RMS metro area classes) Target Type (RMS target type analysis) Proportion: Regional Number F.3.6 Proxy Indicator - Node 6 This Threat-Asset Pair F.3.7 Proxy Indicator - Node 7 Overall Proxy Likelihood Software must provide a field for the vulnerability analysis of each asset, and should use one of the following methods: event-tree analysis, path analysis, vulnerability logic diagrams, computer simulations, or judgment rules-of-thumb. Software must provide a field to document the vulnerability analysis method used. Software must provide a field for point estimates, and the field should allow for utilization of calculations from bins. Software must provide a field for an estimate of threat severity. Software must provide a field for estimate of malevolent threats. Software may have a field for number of attacks. Software may have a field for likelihood. Software may have a field for likelihood. Software may have a field for likelihood. Software may have a field for ratio of capacity to metro area. Software may have a field for likelihood (product of V x C x Detection). Software may have a field calculated by multiplying each proxy indicator. The software does not have fields to calculate or input values for vulnerability. Instead, it is calculated, by the software, based on the response time, delay, and detection (dropdown menus) of every countermeasure for each threat-asset pair. (Adding more countermeasures to a pair decreases the vulnerability of that pair) The software provides a vulnerability scale in the report but no explanation of the process. The software displays the vulnerability as a J percent range and bin but does not allow the user to input the vulnerability manually. The software uses the J tier table based on the facility (population and critical customers). The software does not determine likelihood for each threat. 7 of 10 48

65 Table B.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap 4.5.2, G.2 Natural Hazards earthquakes 4.5.2, G.3 Natural Hazards hurricanes 4.5.2, G.4 Natural Hazards tornadoes 4.5.2, G.5 Natural Hazards floods 4.5.2, Appx G Natural Hazards 4.5.2, Appx G Natural Hazards other 4.5.2, Appx G Natural Hazards Risk Dependency and Proximity Hazards Record Estimates use local historical records for frequency, severity and duration of service denials record the method used for making the estimates and the estimates themselves as single point values or ranges Appx G is optional and provides data to estimate the risk of each natural hazard - risk is calculated by C x V x T; would be nice to have look-up maps/tables for each natural hazard ice storms, extreme cold weather, wildfires, avalanche, tsunami, landslide, mud slide software must have a field to enter the risk of earthquakes for each magnitude from historical records. Software must have a field to enter the risk of hurricanes for each magnitude from historical records. Software must have a field to enter the risk of tornadoes from historical records. Software must have a field to enter the risk of floods for each magnitude from historical records. Software may have a field to enter the risk of ice storms, extreme cold weather, wildfires, avalanche, tsunami, landslide, mud slide from historical records. Software may have a field to enter the risk of other natural hazards. Software must have field for total natural hazard risk. Software must have a field for predicted dependency and proximity likelihood. Software must have room for selected methods to be documented. The software has fields for historic information and magnitudes (text boxes), however, these values do not impact the results of the analysis. The software has fields for historic information and magnitudes (text boxes), however, these values do not impact the results of the analysis. The software does not have a field for total natural hazard risk. This could potentially be done manually by performing an analysis with only natural hazards. The software provides and explains a threat likelihood scale and displays the threat likelihood for each threat-asset pair but does not allow the user to determine these values individually for each pair and explain their reasoning. 4.6, A.4.6 Risk and Resilience Analysis Calculate Risk estimates the owner's risk and resilience and the community's resilience relative to each threat-asset pair for each threat-asset pair calculate risk: C x V x T = R use midpoint of ranges from Appx B Software must estimate the owner's resilience and owner's and community's risk for each threat-asset pair. Software must calculate risk using the numbers input previously for C, V and T for each threat-asset pair. The software calculates the overall risk but not the resilience. It should also be noted that the risk was calculated without the user assigning their own vulnerability and threat likelihood for each pair. The software calculates the risk for each pair but without the user assigning their own vulnerability and threat likelihood for each pair , Appx H Calculate the current level of resilience use either threat-asset pair resilience metric or holistic approach in Appx H Appx H is nonmandatory Some form of a resilience calculation is mandatory, we assume that the preferred approach is to measure the standard in two ways: threat-asset pair and holistic approach but both of the methods described in the standard are nonmandatory. 8 of 10 49

66 Table B.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Operational Resilience Index (ORI) calculated by choosing values from Table H-1 then calculating ORI by multiplying the indicator value by the weight and adding all values (should have fields for the value and weight or by pick box) Operational Resilience Asset Resilience Metric Owner's Economic Resilience Community Economic Resilience Duration x severity x vulnerability x threat likelihood = asset resilience metric lost revenue due to the threat-asset pair (asset resilience x unit price) lost economic activity to the community served by the utility Software may calculate asset resilience metric using duration and severity from 4.3, vulnerability from 4.4 and threat likelihood from 4.5. Financial Resilience Index (FRI) calculated by choosing values from Table H-2 then calculating FRI by multiplying the indicator value by the weight and adding all values (should have fields for the value and weight or by pick box) Software may calculate asset resilience x unit price (have field for unit price of asset). Also software may have field for lost economic activity to the community (same as 4.3) , A.4.7 Record Risk and Resilience Estimates Risk and Resilience Management Utility Resilience Index - software should calculate URI using the values of ORI and FRI and the weights given in Table H-1 & H-2 (URI=ORI x w1 + FRI x w2) Software may have fields for threat-asset pair resilience. Software must have fields for both threat-asset pair resilience and holistic resilience. - The software creates a scatter plot of consequence versus vulnerability and places each threat-asset pair on the plot. It also generates a report in which the risks of all of the pairs are listed in a table and ranked from the greatest risk to the lowest. However, no resilience is calculated or recorded. 9 of 10 50

67 Table B.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap 4.7.1, A Decide 4.7.2, A Define 4.7.3, A Estimate 4.7.4, A Assess 4.7.5, A Identify decide what risk and resilience levels are acceptable define countermeasure and mitigation/resilience options for the threat-asset pairs that are not acceptable. Include devalue, deter, detect, delay and response; consequence reductions, resilience enhancements estimate investment and operating costs for each option; include regular maintenance and periodic overhaul; adjust to present value revisit 4.3 through 4.6 to estimate the risk and resilience levels as if the option was implemented; calculate the estimated benefits of the option identify the options that have benefits that apply to multiple threat-asset pairs 4.7.6, A Calculate calculate the net benefits 4.7.6, A Calculate calculate the benefit-cost ratio rank the most cost effective measures 4.7.7, A Review & Rank to implement * Incomplete can indicate that the feature was partially met or that it could use some improvements for useablility. Software must provide a field for acceptable risk and resilience level. Software must provide fields for countermeasure and mitigation/resilience for each threat asset-pair. Software must provide fields for costs for each option above. Software must provide a field for the new value of risk. Software must provide a way to highlight or mark options. Software must include a calculation of the net benefits. Software must include a calculation of the benefitcost ratio. Software must include a field for ranking cost effective measures. The software does not provide way to set this level or rank the threat-asset pairs. The user is forced to include all pairs for the remaining analysis. The user can define countermeasures to lower the risk, but only by redoing the analysis and adding additional countermeasures. The user can determine the new risk value but only be restarting the analysis with additional countermeasures applied. The software does not calculate cost for each countermeasure or the benefits since the only way to calculate a lower risk is to restart the analysis. 10 of 10 51

68 Table B.2 - J Standard Gap Analysis Matrix - VSAT Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap All-hazards risk and resilience analysis of vulnerabilities to man-made threats, Scope must include all of the same requirements. 1 Scope natural hazards, and dependencies and proximity to hazardous sites. Definitions Is the list the same? If not, how is it different? Is the 2.1, Appx C asset content for each definition the same? critical asset 2.2 consequence 2.3 consequence mitigation 2.4 countermeasure detect deter devalue delay respond 2.5 dependency 2.6 dependency hazard 2.7 event tree analysis 2.8 failure mode 2.9 fault tree analysis 2.10 frequency 2.11 hazard 2.12 incident 2.13 initiating event 2.14 likelihood 2.15 preparedness 2.16 probability 2.17 proximity hazard 2.18 response 2.19 reference threat 2.2 resilience 2.21 risk 2.22 risk analysis 2.23 risk management 2.24 scenario 2.25 system 2.26 threat 2.27 threat likelihood 2.28 vulnerability vulnerability assessment 2.29 /vulnerability analysis 2.30 Definitions vulnerability estimate 2.31 worst reasonable case 3, Appx D Bibliography not included in software analysis 4 Requirements 4.1 Asset Mission identify the mission or critical functions to determine which assets perform or support the mission Software must provide multiple fields for mission or critical functions. The software does not calculate risk or resilience. No definitions are provided. The software provides a field (text box). 1 of 9 52

69 Table B.2 - J Standard Gap Analysis Matrix - VSAT Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap 4.1.2, A.4.1 Critical Assets identify critical assets Supporting Infrastructures Countermeasures and mitigation measures/features identify critical internal and external supporting infrastructure identify and document existing protective countermeasures and mitigation measures/features estimate worst reasonable consequences for each asset without regard to the threat refer to definition for types of items Software must provide multiple fields for critical assets. Software must provide multiple fields for critical internal or external supporting infrastructure. Software must provide multiple fields for all existing protective countermeasures and mitigation measures/features. Software must estimate the worst reasonable Consequence metrics consequences for each asset (.1) Consequence metrics potential for fatalities 4.1.5(.2) Consequence metrics serious injuries major economic loss to facility or 4.1.5(.3) Consequence metrics community Can group these into Human, $ and other (i.e (.4) Consequence metrics impacts to the environment environmental) Can use Hi, Very Hi, Med, Lo, etc (.5) Consequence metrics loss of public confidence field for each consequence (.6) Consequence metrics Prioritize assets 4.2, A.4.2 Threat Appx E 4.2.1, Appx E 4.2.1, Appx E 4.2.1, Appx E 4.2.1, Appx E 4.2.1, Appx E Reference Threats Malevolent Threat Malevolent Threat Malevolent Threat Malevolent Threat Malevolent Threat inhibiting effective function of national defense or civilian government prioritize critical assets using estimated consequences man-made hazards or accidents, natural hazards, dependency hazards; identify general and specific threat scenarios to serve as reference threats within the Standard body Reference Threats are mandatory and Appx E is mandatory Air attack Land attack Water attack various magnitudes of attack elements Weapons types Software must provide a field for each critical asset to assign rank or re-order them in order of consequence. Software must provide a field to characterize the kind of hazard/threat. - malevolent threat to be considered. malevolent threat to be considered. malevolent threat to be considered. malevolent threat to be considered. malevolent threat to be considered. The software contains a list of common assets used at most facilities. The user can also add custom assets (text box) and edit assets to specify details (text box). The software provides critical internal or external supporting infrastructures within the predefined common assets or they may be added manually. The software contains a list of countermeasures found at American utilities and their costs. The user can add custom countermeasures (text box) and edit to specify details (text box). The user can select consequences manually. The software does not allow the user to prioritize assets. The software provides a list of potentially relevant threats for wastewater and water utilities. All threats are placed under their respective threat types. Same as J of 9 53

70 Table B.2 - J Standard Gap Analysis Matrix - VSAT Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Malevolent Threat 4.2.1, Appx E equipment malevolent threat to be considered. Malevolent Threat 4.2.1, Appx E tools malevolent threat to be considered. Malevolent Threat 4.2.1, Appx E explosives malevolent threat to be considered. Malevolent Threat 4.2.1, Appx E tactics malevolent threat to be considered. Malevolent Threat 4.2.1, Appx E means of delivery/transport malevolent threat to be considered. Malevolent Threat 4.2.1, Appx E number of adversaries malevolent threat to be considered. Non-mandatory in 4.2, mandatory in 4.2.1, shall be analyzed as mandatory. Software must provide a Malevolent Threat field to describe the type of malevolent threat to be Same as J , Appx E insiders considered. Non-mandatory in 4.2, mandatory in 4.2.1, shall be analyzed as mandatory. Software must provide a Malevolent Threat field to describe the type of malevolent threat to be 4.2.1, Appx E outsiders considered. Natural Hazards Threat 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.3, Appx G 4.2.3, Appx G 4.2.3, Appx G 4.2.3, Appx G 4.2.3, Appx G Natural Hazards Threat Natural Hazards Threat Natural Hazards Threat Natural Hazards Threat Natural Hazards Threat Natural Hazards Threat Dependency Hazards Threat Dependency Hazards Threat Dependency Hazards Threat Dependency Hazards Threat Dependency Hazards Threat hurricanes floods tornadoes earthquakes wildfires ice storms utilities suppliers employees customers transportation Undefined other Software must define the range of magnitudes from the smallest that would cause serious harm to the largest reasonable case. Software must define the range of magnitudes from the smallest that would cause serious harm to the largest reasonable case. Not listed in body, listed in non-mandatory appendix, analyzed as non-mandatory. Software should give the user the option of including other natural hazards or threats. dependency threat to be considered. dependency threat to be considered. dependency threat to be considered. dependency threat to be considered. dependency threat to be considered. The software includes fires within the library of standard natural hazards but does not provide a range of magnitude. The user can enter "own" risk and check it as a natural hazard (check box). The software includes snow and wind storms within the library. The software does not include these as predefined fields but they could potentially be added as an "other" under the user defined threat section. 3 of 9 54

71 Table B.2 - J Standard Gap Analysis Matrix - VSAT Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap 4.2.3, Appx G 4.2.3, Appx G Dependency Hazards Threat Dependency Hazards Threat proximity Undefined other dependency threat to be considered. Software should give the user the option of including other kinds of dependency hazards or threats. The software does not include these as predefined fields but they could potentially be added as an "other" under the user defined threat section Threat-Asset Pairs create threat-asset pairs Software must allow user to creat threat-asset pairs Evaluate and Rank Threat-Asset Pairs Critical Threat-Asset Pairs 4.3, A.4.3 Consequence Analysis 4.3.1, Appx B Threat Scenario 4.3.2, Appx B 4.3.2, Appx B 4.3.2, Appx B Estimate Consequences - loss of life to either employees or the general public Estimate Consequences - serious injury to either employees or the general public Estimate Consequences - Financial loses to owner/operator evaluate and rank threat-asset pairs select critical threat-asset pairs to be used going forward or use all pairs identifies the worst reasonable consequences that can be caused by the specific threats on the assets as identified in 4.1 apply worst reasonable case assumptions for each threat scenario measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) optional single indicator - dollar equivalent of fatalities and serious injuries in excess of insurance optional single indicator - dollar equivalence of fatalities and serious injuries in excess of insurance Software must evaluate and rank threat-asset pairs, can use multiple approaches including a matrix using small, med., large or scales 1-10, etc. Software must provide a check box or field to identify selected critical threat-asset pairs. (Standard does not define critical, this is left up to the user, i.e.. top 10, top 20?) Software must identify the worst reasonable consequence of a threat on assets. Software must assume the worst reasonable case for each threat. Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach. Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach. Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach. The software does not rate or rank the threatasset pairs based on rough magnitude of consequences prior to determining vulnerability or threat likelihoods. The user cannot remove asset-threat pairs once they have been used in an analysis (even if it's just the baseline analysis). The user can use Water Health Economic Analysis Tool (WHEAT) to determine the consequences of each threat on an asset. The user can use Water Health Economic Analysis Tool (WHEAT) to determine the consequences of each threat on an asset. 4 of 9 55

72 Table B.2 - J Standard Gap Analysis Matrix - VSAT Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap 4.3.2, Appx B 4.3.2, Appx B Estimate Consequences - service denial for the affected customers Estimate Consequences - economic losses to society and the general public Estimate Consequences - other Document assumptions 4.3.5, Appx B Record consequence 4.4, A.4.4 Vulnerability Analysis Review 4.4.2, A.4.4 Analyze Vulnerability measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) if degradation in public confidence, environmental quality, ability of civilian or military agencies to function, etc. room for descriptive analysis must be provided document specific assumptions and procedures used for performing the consequence analysis, the worst reasonable case assumptions and the results of the consequence analysis record the consequence values using point estimates or ranges review pertinent details of the facility construction, systems and layout; identify vulnerabilities or weaknesses in the protection system analyze vulnerability of each critical asset to estimate the likelihood that, given the occurrence of a threat, the consequences result optional single indicator - value of a statistical life additional consequences that can be considered - sociopolitical impacts, natural security impacts, lost strategic capability to cause harm or output, detrimental effects on brand value, public confidence, psychological impacts, and environmental degradation Preferred ranges are in Appx B. may use fault, event or failure tree analysis, path analysis, vulnerability logic diagrams, computer simulation methods, or expert judgment rules-ofthumb Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach. Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach. Software must provide the ability to describe primary consequence, and should give the user the option of including other consequences. Software must give users a space to document the assumptions made in the analysis, and should have the ability to include other documentation such as maps and calculations. Software must provide a field for ranges or point estimates. Software must provide fields to input pertinent details of construction, systems and layout. Software must provide a field for the vulnerability analysis of each asset, and should use one of the following methods: event-tree analysis, path analysis, vulnerability logic diagrams, computer simulations, or judgment rules-of-thumb. The software does not include the consequence of service denial for the affected customers. The user can use Water Health Economic Analysis Tool (WHEAT) to determine the consequences of each threat on an asset. The software only provides fields for 4 consequence types (fatalities, injuries, economic cost of owner, and economic cost to region). The software results display in tables and bar plots. VSAT can output a more detailed table to Excel. The Water Health Economic Analysis Tool (WHEAT) can be downloaded and used to help determine the consequence values and can output the results to an Excel table. The user can enter a specific value (text box) or select a predefined J100 bins with their corresponding ranges (bubble) Note: Bin 0 (0-25) has been broken down into 0A (0), 0B (1-5), and 0C (6-25). The user can add details about facility or assets and their locations (text boxes). The user can choose countermeasures and add details and locations (text box). The software relies on the judgment of the user and does not provide for the other methods. 5 of 9 56

73 Table B.2 - J Standard Gap Analysis Matrix - VSAT Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap document method and results of the Software must provide a field to document the Document method vulnerability analysis vulnerability analysis method used , Appx B Record Estimates Record the estimates from , A.4.5 Threat Likelihood Analysis 4.5.1, Appx F Malevolent Threats F.3.1 Proxy Indicator - Node 1 F.3.2 Proxy Indicator - Node 2 F.3.3 Proxy Indicator - Node 3 F.3.4 Proxy Indicator - Node 4 likelihood of malevolent event, dependency/proximity hazard or natural hazard use proxy measure, best estimate, or conditional assessment to determine F.3.5 Proxy Indicator - Node 5 This Facility use point estimates or RAMCAP scales; if bins are used, the midpoint is used for the calculation Proxy Measure (Appx F) is optional and preferred Number of U.S. attacks per year Metro Region (RMS metro area classes) Target Type (RMS target type analysis) Proportion: Regional Number F.3.6 Proxy Indicator - Node 6 This Threat-Asset Pair F.3.7 Proxy Indicator - Node 7 Overall Proxy Likelihood 4.5.2, G.2 Natural Hazards earthquakes 4.5.2, G.3 Natural Hazards hurricanes Appx G is optional and provides data to estimate the risk of each natural hazard - risk is calculated by C x V x T; would be nice to have look-up maps/tables for each natural hazard Software must provide a field for point estimates, and the field should allow for utilization of calculations from bins. Software must provide a field for an estimate of threat severity. Software must provide a field for estimate of malevolent threats. Software may have a field for number of attacks. Software may have a field for likelihood. Software may have a field for likelihood. Software may have a field for likelihood. Software may have a field for ratio of capacity to metro area. Software may have a field for likelihood (product of V x C x Detection). Software may have a field calculated by multiplying each proxy indicator. software must have a field to enter the risk of earthquakes for each magnitude from historical records Software must have a field to enter the risk of hurricanes for each magnitude from historical records. For each pair, the user has to rate the vulnerability by "bubbling" the detection (certain, probable, possible, none), delay (very, strong, limited, no delay), and response (fast, variable, slow, none). The software then determines the likelihood as a J percentage range (with rounding) and countermeasure capability (very high, high, moderate, low). The user can also enter comments (text box). For each pair, the user has to rate the likelihood by bubbling very high, high, moderate, or low. The user can also enter comments (text box). The software allows for the option to use best estimate or a 100% probability for all pairs (bubble). The software determines the probability of EQ1 through EQ5 magnitude earthquakes (magnitudes based on ranges of peak ground acceleration) by the zip code of the asset. It also determines the 50 year probability of excedance and the annual probability of excedance. The user must then interpret those results to set the likelihood as very high, high, moderate, or low. The software provides maps for return periods for category H1 through H5 on the Saffir- Simpson Scale. The user must then determine the probabilities. The user must then interpret those results to set the likelihood as very high, high, moderate, or low. 6 of 9 57

74 Table B.2 - J Standard Gap Analysis Matrix - VSAT Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap 4.5.2, G.4 Natural Hazards tornadoes 4.5.2, G.5 Natural Hazards floods 4.5.2, Appx G Natural Hazards 4.5.2, Appx G Natural Hazards other 4.5.2, Appx G Natural Hazards Risk Dependency and Proximity Hazards Record Estimates 4.6, A.4.6 Risk and Resilience Analysis use local historical records for frequency, severity and duration of service denials record the method used for making the estimates and the estimates themselves as single point values or ranges estimates the owner's risk and resilience and the community's resilience relative to each threat-asset pair Appx G is optional and provides data to estimate the risk of each natural hazard - risk is calculated by C x V x T; would be nice to have look-up maps/tables for each natural hazard ice storms, extreme cold weather, wildfires, avalanche, tsunami, landslide, mud slide Software must have a field to enter the risk of tornadoes from historical records. Software must have a field to enter the risk of floods for each magnitude from historical records. Software may have a field to enter the risk of ice storms, extreme cold weather, wildfires, avalanche, tsunami, landslide, mud slide from historical records. Software may have a field to enter the risk of other natural hazards. Software must have field for total natural hazard risk. Software must have a field for predicted dependency and proximity likelihood. Software must have room for selected methods to be documented. Software must estimate the owner's resilience and owner's and community's risk for each threat-asset pair. The software determines probability of T1 through T5 magnitude tornadoes (magnitudes based on ranges of wind speeds) by the zip code of the asset. It also determines the number of tornadoes in the past 50 years, the property damage, number of injuries, and number of fatalities. The user must then interpret those results to set the likelihood as very high, high, moderate, or low. The software determines probability of F1 and F2 magnitude floods (user has the option to define their own flood severities and probabilities) by the zip code of the asset. It provides a link to FEMA reference materials. The user must then interpret those results to set the likelihood as very high, high, moderate, or low. The software does not include a field to enter historical data. The software can include the likelihood of the other threats but not based on historical data. The software has a tab for natural threats where all of the asset-threat pairs of natural hazards are displayed, but no total risk is displayed. Also, the added hazards, e.g., windstorms, snowstorms, are not displayed in this tab. The user can include comments about the threat likelihood chosen and displays in the report. The software does not calculate overall risk or resiliency Calculate Risk 4.6.2, Appx H Calculate the current level of resilience for each threat-asset pair calculate risk: C x V x T = R use either threat-asset pair resilience metric or holistic approach in Appx H use midpoint of ranges from Appx B Appx H is nonmandatory Software must calculate risk using the numbers input previously for C, V and T for each threat-asset pair. Some form of a resilience calculation is mandatory, we assume that the preferred approach is to measure the standard in two ways: threat-asset pair and holistic approach but both of the methods described in the standard are nonmandatory. The software includes a list of "knowledge base" information for resiliency within the predefined asset list but does not use them to calculate resiliency. 7 of 9 58

75 Table B.2 - J Standard Gap Analysis Matrix - VSAT Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Operational Resilience Index (ORI) calculated by choosing values from Table H-1 then calculating ORI by multiplying the indicator value by the weight and adding all values (should have fields for the value and weight or by pick box) Operational Resilience Asset Resilience Metric Owner's Economic Resilience Community Economic Resilience , A , A Decide 4.7.2, A Define 4.7.3, A Estimate 4.7.4, A Assess Record Risk and Resilience Estimates Risk and Resilience Management Duration x severity x vulnerability x threat likelihood = asset resilience metric lost revenue due to the threat-asset pair (asset resilience x unit price) lost economic activity to the community served by the utility decide what risk and resilience levels are acceptable define countermeasure and mitigation/resilience options for the threat-asset pairs that are not acceptable. Include devalue, deter, detect, delay and response; consequence reductions, resilience enhancements estimate investment and operating costs for each option; include regular maintenance and periodic overhaul; adjust to present value revisit 4.3 through 4.6 to estimate the risk and resilience levels as if the option was implemented; calculate the estimated benefits of the option Software may calculate asset resilience metric using duration and severity from 4.3, vulnerability from 4.4 and threat likelihood from 4.5. Financial Resilience Index (FRI) calculated by choosing values from Table H-2 then calculating FRI by Utility Resilience Index - software should calculate URI using the values of ORI and FRI and the weights given in Table H-1 & H-2 (URI=ORI x w1 + FRI x w2) Software may have fields for threat-asset pair resilience. Software must have fields for both threat-asset pair resilience and holistic resilience. - Software must provide a field for acceptable risk and resilience level. Software must provide fields for countermeasure and mitigation/resilience for each threat asset-pair. Software must provide fields for costs for each option above. Software must provide a field for the new value of risk. The software does not calculate risk or resilience. The software does not provide a way to set this level or rank the threat-asset pairs and the user is forced to include all pairs for the remaining analysis. The software calculates risk reduction units. The calculation and explination are confusing. The calculation can only be used for comparison purposes. 8 of 9 59

76 Table B.2 - J Standard Gap Analysis Matrix - VSAT Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap 4.7.5, A Identify identify the options that have benefits that apply to multiple threat-asset pairs 4.7.6, A Calculate calculate the net benefits 4.7.6, A Calculate calculate the benefit-cost ratio rank the most cost effective measures 4.7.7, A Review & Rank to implement * Incomplete can indicate that the feature was partially met or that it could be improved upon. Software must provide a way to highlight or mark options. Software must include a calculation of the net benefits. Software must include a calculation of the benefitcost ratio. Software must include a field for ranking cost effective measures. The user can create multiple packages of countermeasures and improvements to compare and determine which one benefits the utility the most. The user can create upgrade packages and compare the annualized cost, capital cost and risk reduction units but cannot calculate net benefit and benefit-cost ratios. 9 of 9 60

77 Table B.3 - J Standard Gap Analysis Matrix - ARAM-W Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap All-hazards risk and resilience analysis of vulnerabilities to manmade threats, natural hazards, and Scope must include all of the same requirements. The software does not calculate resilience. dependencies and proximity to 1 Scope hazardous sites. Definitions Is the list the same? If not, how is it different? Is 2.1, Appx C asset the content for each definition the same? No definitions were included in the software critical asset 2.2 consequence 2.3 consequence mitigation 2.4 countermeasure detect deter devalue delay respond 2.5 dependency 2.6 dependency hazard 2.7 event tree analysis 2.8 failure mode 2.9 fault tree analysis 2.10 frequency 2.11 hazard 2.12 incident 2.13 initiating event 2.14 likelihood 2.15 preparedness 2.16 probability 2.17 proximity hazard 2.18 response 2.19 reference threat 2.2 resilience 2.21 risk 2.22 risk analysis 2.23 risk management 2.24 scenario 2.25 system 2.26 threat 2.27 threat likelihood 2.28 vulnerability vulnerability assessment 2.29 /vulnerability analysis 2.30 vulnerability estimate 2.31 worst reasonable case 3, Appx D Bibliography not included in software analysis 4 Requirements 4.1 Asset 1 of 8 61

78 Table B.3 - J Standard Gap Analysis Matrix - ARAM-W Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap 4.1.5(.4) Consequence metrics impacts to the environment Appx E 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.3, Appx G 4.2.3, Appx G 4.2.3, Appx G 4.2.3, Appx G 4.2.3, Appx G 4.2.3, Appx G 4.2.3, Appx G Reference Threats Natural Hazards Threat Natural Hazards Threat Natural Hazards Threat Dependency Hazards Threat Dependency Hazards Threat Dependency Hazards Threat Dependency Hazards Threat Dependency Hazards Threat Dependency Hazards Threat Dependency Hazards Threat Evaluate & Rank Threat-Asset Pairs within the Standard body Reference Threats are mandatory and Appx E is mandatory wildfires ice storms utilities suppliers employees customers transportation proximity evaluate and rank threat-asset pairs Undefined other Undefined other Can group these into Human, $ and other (i.e.. environmental) Can use Hi, Very Hi, Med, Lo, etc. field for each consequence. - Software must define the range of magnitudes from the smallest that would cause serious harm to the Not listed in body, listed in non-mandatory appendix, analyzed as non-mandatory. Software should give the user the option of including other natural hazards or threats. Software must provide a field to describe the type of dependency threat to be considered. Software must provide a field to describe the type of dependency threat to be considered. Software must provide a field to describe the type of dependency threat to be considered. Software must provide a field to describe the type of dependency threat to be considered. Software must provide a field to describe the type of dependency threat to be considered. Software must provide a field to describe the type of dependency threat to be considered. Software should give the user the option of including other kinds of dependency hazards or threats. Software must evaluate and rank threat-asset pairs, can use multiple approaches including a matrix using small, med., large or scales 1-10, etc. The user must select the undesired events, (i.e. release of chemicals, loss of power, loss of critical pump/valve system), from a predefined event tree. There is a separate tree for each type of utility and users can add additional events. The user can then determine the consequence of each undesired event, which are determined by assuming the loss of the asset which leads to the undesired event, without regard to the threats that may cause the loss of the asset. Although environmental impact is not predefined, the user has the ability to define any new consequences that they feel apply. The software does not include any additional natural threats or allow any other threats to be added. The software does not include dependency threats, the RAMCAP labels for dependency threats, or dependency threats within the RAMCAP reference tables. The software does not rate or rank the undesired event/asset location/threat pairs based on rough magnitude of consequences prior to determining vulnerability or threat likelihoods. 2 of 8 62

79 Table B.3 - J Standard Gap Analysis Matrix - ARAM-W Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Critical Threat-Asset Pairs 4.3, A.4.3 Consequence Analysis 4.3.1, Appx B Threat Scenario 4.3.2, Appx B 4.3.2, Appx B 4.3.2, Appx B 4.3.2, Appx B Estimate Consequences - loss of life to either employees or the general public Estimate Consequences - serious injury to either employees or the general public Estimate Consequences - Financial loses to owner/operator Estimate Consequences - service denial for the affected customers select critical threat-asset pairs to be used going forward or use all pairs identifies the worst reasonable consequences that can be caused by the specific threats on the assets as identified in 4.1 apply worst reasonable case assumptions for each threat scenario measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) Software must provide a check box or field to identify selected critical threat-asset pairs. (Standard does not define critical, this is left up to the user, i.e.. top 10, top 20?) Software must identify the worst reasonable consequence of a threat on assets. Software must assume the worst reasonable case for each threat. Software must provide a field for single point optional single indicator - estimate of consequences or a bin number. When dollar equivalent of fatalities reviewing bins, the values must match. Under 4.3 and serious injuries in this is defined as an "or" under it is defined as excess of insurance an "and" analysis shall include "and" as a more conservative approach. optional single indicator - dollar equivalence of fatalities and serious injuries in excess of insurance Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach The user cannot remove undesired event/asset location/threat pairs once they have been assigned in the baseline analysis. Although the consequences for the loss of critical assets and the resulting undesired event occurring were previously determined, they are not evaluated for each specific threat on the assets. Although the consequences for the loss of critical assets and the resulting undesired event occurring were previously determined, they are not evaluated for each specific threat on the assets. Although the consequences for the loss of critical assets and the resulting undesired t i i l d t i d 3 of 8 63

80 Table B.3 - J Standard Gap Analysis Matrix - ARAM-W Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Estimate Consequences - economic losses to society and 4.3.2, Appx B the general public 4.4, A.4.4 Vulnerability Analysis 4.4.2, A.4.4 Analyze Vulnerability 4.5.1, Appx F Malevolent Threats measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) analyze vulnerability of each critical asset to estimate the likelihood that, given the occurrence of a threat, the consequences result use proxy measure, best estimate, or conditional assessment to determine optional single indicator - value of a statistical life may use fault, event or failure tree analysis, path analysis, vulnerability logic diagrams, computer simulation methods, or expert judgment rules-ofthumb Proxy Measure (Appx F) is optional and preferred Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach Software must provide a field for the vulnerability analysis of each asset, and should use one of the following methods: event-tree analysis, path analysis, vulnerability logic diagrams, computer simulations, or judgment rules-of-thumb. Software must provide a field for estimate of malevolent threats. event occurring were previously determined, they are not evaluated for each specific threat on the assets. The user can determine the vulnerability either by entering a user estimated single value or by using a path analysis with the adversary sequence diagram. The software could better meet the Standard by providing a means to use other methods to analyze vulnerability. For each undesired event/asset location/threat pair, the user can choose to determine the likelihood of attack using conditional, expert judgment, or a questionnaire method. For the conditional method, the likelihood value is automatically considered High (or 100%) based on the assumption that the attack will occur. For the expert judgment method, the user can input a likelihood of high, medium or low (drop down menu). For the questionnaire method, the likelihood value is determined based on the responses to the questions asked. The questions follow the threat factors outlined in the RAMCAP - The Framework document for estimating likelihood of attack and include initial consideration of capability, history, current interest, current surveillance, documented threats, potential consequences, ideology and ease of attack. The user answers the different questions by clicking on the appropriate circle and the threat potential is calculated based on the responses to the questions. F.3.1 Proxy Indicator - Node 1 F.3.2 Proxy Indicator - Node 2 F.3.3 Proxy Indicator - Node 3 Number of U.S. attacks per year Metro Region (RMS metro area classes) Target Type (RMS target type analysis) Software may have a field for number of attacks. Software may have a field for likelihood. Software may have a field for likelihood. 4 of 8 64

81 Table B.3 - J Standard Gap Analysis Matrix - ARAM-W Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Proportion: Regional The software does not include the proxy F.3.4 Proxy Indicator - Node 4 Number Software may have a field for likelihood. method. Software may have a field for ratio of capacity to F.3.5 Proxy Indicator - Node 5 This Facility metro area. Software may have a field for likelihood (product F.3.6 Proxy Indicator - Node 6 This Threat-Asset Pair of V x C x Detection). Software may have a field calculated by F.3.7 Proxy Indicator - Node 7 Overall Proxy Likelihood multiplying each proxy indicator , Appx G Natural Hazards 4.5.2, Appx G Natural Hazards other 4.5.2, Appx G Natural Hazards Risk Dependency and Proximity Hazards Record Estimates 4.6, A.4.6 Risk and Resilience Analysis 4.6.2, Appx H Calculate the current level of resilience use local historical records for frequency, severity and duration of service denials record the method used for making the estimates and the estimates themselves as single point values or ranges estimates the owner's risk and resilience and the community's resilience relative to each threat-asset pair ice storms, extreme cold weather, wildfires, avalanche, tsunami, landslide, mud slide use either threat-asset pair resilience metric or holistic approach in Appx H Appx H is no mandatory Software may have a field to enter the risk of ice storms, extreme cold weather, wildfires, avalanche, tsunami, landslide, mud slide from historical records. Software may have a field to enter the risk of other natural hazards. Software must have field for total natural hazard risk. Software must have a field for predicted dependency and proximity likelihood. Software must have room for selected methods to be documented. Software must estimate the owner's resilience and owner's and community's risk for each threat-asset pair. Some form of a resilience calculation is mandatory, we assume that the preferred approach is to measure the standard in two ways: threat-asset pair and holistic approach but both of the methods described in the standard are no mandatory. The software only includes four natural threats and does not allow for the addition of any others. The software only includes four natural threats and does not allow for the addition of any others. The software does not include total risk of natural hazards. Dependency and proximity threats are not included in the software. The user can include comments about the threat likelihood chosen or information about the sources used and displays the threat potential values as none, low, medium, high, or very high in the report. The software does not calculate resilience. The software does not calculate resilience. 5 of 8 65

82 Table B.3 - J Standard Gap Analysis Matrix - ARAM-W Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Operational Resilience Index (ORI) calculated by choosing values from Table H-1 then calculating ORI by multiplying the indicator value by the weight and adding all values (should have fields for the value and weight or by pick box) The software does not calculate resilience Operational Resilience Asset Resilience Metric Duration x severity x vulnerability x threat likelihood = asset resilience metric Software may calculate asset resilience metric using duration and severity from 4.3, vulnerability from 4.4 and threat likelihood from 4.5. Financial Resilience Index (FRI) calculated by choosing values from Table H-2 then calculating FRI by multiplying the indicator value by the weight and adding all values (should have fields for the value and weight or by pick box) The software does not calculate resilience Owner's Economic Resilience Community Economic Resilience lost revenue due to the threat-asset pair (asset resilience x unit price) lost economic activity to the community served by the utility Software may calculate asset resilience x unit price (have field for unit price of asset). Also software may have field for lost economic activity to the community (same as 4.3). The software does not calculate resilience. 6 of 8 66

83 Table B.3 - J Standard Gap Analysis Matrix - ARAM-W Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Utility Resilience Index - software should calculate URI using the values of ORI and FRI and the weights given in Table H-1 & H-2 (URI=ORI x w1 + FRI x w2) The software displays the overall risk and the risk for each undesired event/asset location/threat pair as either none, low, medium, high, or very high. However, resilience is not calculated or displayed , A , A Decide 4.7.2, A Define 4.7.4, A Assess 4.7.5, A Identify Record Risk and Resilience Estimates Risk and Resilience Management decide what risk and resilience levels are acceptable define countermeasure and mitigation/resilience options for the threat-asset pairs that are not acceptable. Include devalue, deter, detect, delay and response; consequence reductions, resilience enhancements revisit 4.3 through 4.6 to estimate the risk and resilience levels as if the option was implemented; calculate the estimated benefits of the option identify the options that have benefits that apply to multiple threat-asset pairs Software may have fields for threat-asset pair resilience. Software must have fields for both threat-asset pair resilience and holistic resilience. - Software must provide a field for acceptable risk and resilience level. Software must provide fields for countermeasure and mitigation/resilience for each threat asset-pair. Software must provide a field for the new value of risk. Software must provide a way to highlight or mark options. The software includes a field to determine if the risk for each undesired event/asset location/threat pair is acceptable (check box). The software does not include resilience in this decision. The user can create upgrade packages and develop them by redoing the ASD analysis with the inclusion of more countermeasures and other enhancements. The user can also create upgrade packages which only affect the consequences which can be entered. The software does not include resilience enhancement measures. The software calculates the new risk of each pair for each upgrade package. The software displays a report of the baseline data and upgrade packages for the highest risk threatasset location pair. It does not calculate benefits of the options or include resilience. The user can create multiple packages of countermeasures and improvements to compare but the software does not determine which improvements are included in multiple packages , A Calculate calculate the net benefits Software must include a calculation of the net benefits. The software can compare packages but there are no calculations for net benefit , A Calculate calculate the benefit-cost ratio Software must include a calculation of the benefitcost ratio. The software does not calculate net benefit, nor does it compare the benefit to the cost. 7 of 8 67

84 Table B.3 - J Standard Gap Analysis Matrix - ARAM-W Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap 4.7.9, A Review & Rank rank the most cost effective measures to implement * Incomplete can indicate that the feature was partially met or that it could be improved upon. Software must include a field for ranking cost effective measures. The software does not consider the most effective package, only the packages and their affect on each pair, especially the highest risk pair. 8 of 8 68

85 Appendix C: J Recommendations 35 APPENDI C: J RECOMMENDATIONS 69

86 Table C.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix Scope must include all of the same requirements. 1 Scope All-hazards risk and resilience analysis of vulnerabilities to man-made threats, natural hazards, and dependencies and proximity to hazardous sites. Is the list the same? If not, how is it different? Is the 2.1, Appx C asset content for each definition the same? critical asset 2.2 consequence 2.3 consequence mitigation 2.4 countermeasure detect deter devalue delay respond 2.5 dependency 2.6 dependency hazard 2.7 event tree analysis 2.8 failure mode 2.9 fault tree analysis 2.10 frequency 2.11 hazard 2.12 incident 2.13 initiating event 2.14 likelihood 2.15 preparedness 2.16 probability 2.17 proximity hazard 2.18 response 2.19 reference threat 2.2 resilience 2.21 risk 2.22 risk analysis 2.23 risk management 2.24 scenario 2.25 system 2.26 threat 2.27 threat likelihood 2.28 vulnerability vulnerability assessment 2.29 /vulnerability analysis 2.30 vulnerability estimate 2.31 Definitions worst reasonable case Supporting Infrastructures identify critical internal and external supporting infrastructure Software must provide multiple fields for critical internal or external supporting infrastructure. The software does calculate risk; however, resilience is not calculated, there is no input for vulnerability, no predefined fields for dependency or proximity hazards, threat likelihoods, or scales for natural hazard magnitudes. No definitions are provided. There is no separate field for critical infrastructure, but the user has the ability to create their own assets. The software enables the user to create infrastructure as an asset and define it as critical and have it included in the analysis. The software should provide a method to calculate resilience, including the supporting user interface elements to gather the user input required for such calculations. The software must provide the user a reference or documentation section in the program, where definitions of these terms can be determined and evaluated for fitness. The software should provide critical infrastructure within the predefined assets. Large/Unknown fix (likely large). A combination of user interface elements (another section from the main RAMCAP menu) and internal functionality would be required to support calculation of resilience. Medium. Requires the creation of an additional user interface screen, but there is little to no computation required to display these definitions to the user. Medium. The functionality for critical infrastructure exists, but is burdensome for the user to perform. Additional user interface elements would make this estimate worst reasonable Software must estimate the worst reasonable consequences for each asset without consequences for each asset Consequence metrics regard to the threat 4.1.5(.1) Consequence metrics potential for fatalities 4.1.5(.2) Consequence metrics serious injuries major economic loss to facility or 4.1.5(.3) Consequence metrics community 4.1.5(.4) Consequence metrics impacts to the environment Can group these into Human, $ and other (i.e (.5) Consequence metrics loss of public confidence environmental) Can use Hi, Very Hi, Med, Lo, etc. field for each consequence (.6) Consequence metrics inhibiting effective function of national defense or civilian government The software does not allow the user to define the worst reasonable consequences without regard to threat. The software must determine the worst reasonable consequence for each asset prior to determining the threats that could affect the asset. This must be done to help prioritize the assets. Small. Allowing the user to define a baseline set of consequences when describing each asset in the Asset Information page would satisfy this requirement. This requires the addition of more form fields (in particular, dropdown menus) to the user interface of 10

87 Table C.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix Prioritize assets prioritize critical assets using estimated consequences Software must provide a field for each critical asset to assign rank or re-order them in order of consequence. The user can choose if the asset is critical or not and prioritize (using a dropdown menu of high, med., or low) but can not reprioritize based on consequences (no list of asset and consequences in comparison). The software must have a way to rank the assets in order of their importance after the worst reasonable consequences have been determined for each asset. Small. The Asset Information page should have an additional dropdown menu that allows the user to select an asset criticality based on the consequences related to that asset. This is a user interface component that displays previously entered information, so no additional calculation should be required , Appx E Malevolent Threat Air attack malevolent threat to be considered and the outcome of which is included in the software calculations , Appx E Malevolent Threat Land attack malevolent threat to be considered and the outcome of which is included in the software calculations , Appx E 4.2.1, Appx E 4.2.1, Appx E 4.2.1, Appx E 4.2.1, Appx E 4.2.1, Appx E 4.2.1, Appx E 4.2.1, Appx E Malevolent Threat Malevolent Threat Malevolent Threat Malevolent Threat Malevolent Threat Malevolent Threat Malevolent Threat Malevolent Threat Water attack various magnitudes of attack elements Weapons types equipment tools explosives tactics means of delivery/transport malevolent threat to be considered and the outcome of which is included in the software calculations. malevolent threat to be considered and the outcome of which is included in the software calculations. malevolent threat to be considered and the outcome of which is included in the software calculations. malevolent threat to be considered and the outcome of which is included in the software calculations. malevolent threat to be considered and the outcome of which is included in the software calculations. malevolent threat to be considered and the outcome of which is included in the software calculations. malevolent threat to be considered and the outcome of which is included in the software calculations. malevolent threat to be considered and the outcome of which is included in the software calculations. Same as J The software must have a way to select the type of malevolent threat, as well as have internal calculations that take into account that type of threat when determining the outcome. Small/Unknown (likely small). Addition of a user interface element to choose the type of malevolent threat requires the addition of a combo box or other field, but taking into account the impact of that choice in calculations is of unknkown complexity without knowing the internal calculations of SEMS , Appx E 4.2.1, Appx E Malevolent Threat Malevolent Threat number of adversaries insiders malevolent threat to be considered and the outcome of which is included in the software calculations. Non-mandatory in 4.2, mandatory in 4.2.1, shall be analyzed as mandatory. Software must provide a field to describe the type of malevolent threat to be considered. Same as J The software must have a way to select the type of malevolent threat, as well as have internal calculations that take into account that type of threat when determining the outcome of 10

88 Table C.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.3, Appx G 4.2.3, Appx G 4.2.3, Appx G 4.2.3, Appx G 4.2.3, Appx G Natural Hazards Threat Natural Hazards Threat Natural Hazards Threat Natural Hazards Threat Natural Hazards Threat Natural Hazards Threat Natural Hazards Threat Dependency Hazards Threat Dependency Hazards Threat Dependency Hazards Threat Dependency Hazards Threat Dependency Hazards Threat hurricanes floods tornadoes earthquakes wildfires ice storms utilities suppliers employees customers transportation Undefined other Software must define the range of magnitudes from the smallest that would cause serious harm to the largest reasonable case. Software must define the range of magnitudes from the smallest that would cause serious harm to the largest reasonable case. Not listed in body, listed in non-mandatory appendix, analyzed as non-mandatory. Software should give the user the option of including other natural hazards or threats. dependency threat to be considered. dependency threat to be considered. dependency threat to be considered. dependency threat to be considered. dependency threat to be considered. The software does not provide ranges or direct links to reference materials to determine magnitudes. However, it does say what standards should be used for the magnitudes. Also, changing the magnitude of the natural disaster does not affect the likelihood of overall risk. The software does not include these as predefined but can be added as an "other." No ranges or direct links to reference materials exist in the software to help determine magnitudes. The software does not include these as predefined fields but they could potentially be added as an "other" hazard. The software must provide a range of magnitudes that would damage the assets for each natural disaster type. The software should have dependency hazards predefined within the list of potential threats; however, they can still be entered manually within the user defined section. Medium/Unknown (likely small). Missing hazards can be added to the SEMS database (in particular, the Threat table) in order to not be manually entered by the user. Since the standards for the magnitudes are known, the software should be able to provide ranges for the magnitudes when selecting the threat. The software's calculation of risk should be amended to include this magnitude, but without knowing the process an estimate of effort cannot be given. Medium. The software should provide additional choices in the list of potential threats to cover dependency hazards. This can be accomplished through the addition of these threats to the SEMS database (specifically the Threat table) , Appx G Dependency Hazards Threat proximity dependency threat to be considered & 4.2.5? Threat-Asset Pairs evaluate and rank threat-asset pairs Software must evaluate and rank threat-asset pairs, can use multiple approaches including a matrix using small, med., large or scales 1-10, etc. The software does not rate or rank pairs based on a rough magnitude of consequences prior to determining vulnerability or threat likelihoods. The software must rate and rank the pairs based on their rough magnitude of consequences solely based on the possible consequences of the threat on the asset. Unknown (likely medium). The software's calculations of magnitude of consequences are not visible and may require substantial reworking to include other factors of 10

89 Table C.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix Critical Threat-Asset Pairs 4.3.2, Appx B 4.3.2, Appx B 4.3.2, Appx B 4.3.2, Appx B Estimate Consequences - loss of life to either employees or the general public Estimate Consequences - serious injury to either employees or the general public Estimate Consequences - Financial loses to owner/operator Estimate Consequences - service denial for the affected customers select critical threat-asset pairs to be used going forward or use all pairs measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) optional single indicator - dollar equivalent of fatalities and serious injuries in excess of insurance optional single indicator - dollar equivalence of fatalities and serious injuries in excess of insurance Software must provide a check box or field to identify selected critical threat-asset pairs. (Standard does not define critical, this is left up to the user, i.e.. top 10, top 20?) Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach. Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach The user can select critical assets but can not select critical threat-asset pairs. The software does not include single point estimates or bin numbers. It uses a dropdown menu of J standard ranges. The software does not include the consequence of service denial for the affected customers. The software does not give the option of proceeding with an analysis of certain pairs. Must be able to choose the pairs that must be analyzed for their risk. The software could provide a field for a consequence point estimate in the case that an exact number is known or desired. The software must include a field to enter a single point estimate or bin number of the consequence of service denial for the affected customers. Medium/Unknown (likely medium). The addition of the ability to select threat-asset pairs requires the addition of some user interface elements, but revising the analysis to be restricted to certain subsets will require a revision to the underlying logic, the extent of which is unknown. Small. The addition of a user interface element (likely a text field) for the user to enter a value rather than selecting from a predefined set of J standard ranges should be easy to implement. Small/Unknown (likely small). In addition to adding the user interface element for the user to enter an estimate or bin number, the impact that that estimate has on the calculations is unable to be determined , Appx B Estimate Consequences - economic losses to society and the general public measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) optional single indicator - value of a statistical life Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach The software does not inlcude single point estimates or bin numbers. It uses a dropdown menu of J standard ranges The software could provide a field for a consequence point estimate in the case that an exact number is known or desired. Small. The addition of a user interface element (likely text field) for the user to enter a value rather than selecting from a predefined set of J standard ranges should be easy to implement of 10

90 Table C.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix if degradation in public confidence, environmental quality, ability of civilian or military agencies to function, etc. room for descriptive analysis must be provided Software must provide the ability to describe primary consequence, and should give the user the option of including other consequences. The software only provides fields for 4 consequence types (fatalities, injuries, economic cost of owner, and economic cost to region). The software should provide a field for a descriptive analysis Small. Addition of another text area to store a qualitative description of the consequences would require an update to the user interface of the Consequence Assignment page and an additional column in the SEMS database (specifically the Consequence table) for the asset-threat pair Estimate Consequences - other 4.3.5, Appx B Record consequence 4.4.2, A.4.4 Analyze Vulnerability record the consequence values using point estimates or ranges analyze vulnerability of each critical asset to estimate the likelihood that, given the occurrence of a threat, the consequences result additional consequences that can be considered - sociopolitical impacts, natural security impacts, lost strategic capability to cause harm or output, detrimental effects on brand value, public confidence, psychological impacts, and environmental degradation Preferred ranges are in Appx B. may use fault, event or failure tree analysis, path analysis, vulnerability logic diagrams, computer simulation methods, or expert judgment rules-ofthumb Software must provide a field for ranges or point estimates. Software must provide a field for the vulnerability analysis of each asset, and should use one of the following methods: event-tree analysis, path analysis, vulnerability logic diagrams, computer simulations, or judgment rules-of-thumb. The software only uses ranges for the consequence value. The software does not have fields to calculate or input values for vulnerability. Instead, it is calculated based on the response time, delay, and detection (dropdown menus) of every countermeasure for each threat-asset pair. (Adding more countermeasures to a pair decreases the vulnerability of that pair) The software must add a field to record additional consequences for the analysis. The software should additionally provide a field to enter a point estimate of the consequence values. The software must provide a field for the user to assign a vulnerability to each threat-asset pair instead of being calculated automatically based on the countermeasures. The software may use one of the suggested methods for determining the vulnerability Small. Addition of another text area for entering additional consequences should be a minor change to the user interface as well as addition of a row per entry into the SEMS database (specifically the Consequence table). Small. Addition of a text field used to enter this data requires a modification the user interface. Small/Unknown (likely medium). While adding a user interface element to allow for entry of a vulnerability for each threat-asset pair requires just a user interface modification, the changes that that entry would make in the calculation method cannot be determined. Large. The software currently uses a determination system that would likely require major modifications to accommodate another method, such as eventtree analysis Document Method document method and results of the vulnerability analysis Software must provide a field to document the vulnerability analysis method used. The software provides a vulnerability scale in the report but no explanation of the process. The software must document the method used to achieve the results. Small. The software could simply output a description of the method on the same page as the results, which would require displaying static text in a new text field of 10

91 Table C.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix The software must allow the user to assign a vulnerability to each threat-asset pair and then record this value. Small. This requires storage of the user's selection (either from the range of J100 values or the user's manual input) in an additional column in the database, in the table AssetThreat , Appx B 4.5, A.4.5 Threat Likelihood Analysis 4.5.1, Appx F Malevolent Threats Record Estimates Record the estimates from F.3.1 Proxy Indicator - Node 1 F.3.2 Proxy Indicator - Node 2 F.3.3 Proxy Indicator - Node 3 F.3.4 Proxy Indicator - Node 4 likelihood of malevolent event, dependency/proximity hazard or natural hazard use proxy measure, best estimate, or conditional assessment to determine F.3.5 Proxy Indicator - Node 5 This Facility Proxy Measure (Appx F) is optional and preferred Number of U.S. attacks per year Metro Region (RMS metro area classes) Target Type (RMS target type analysis) Proportion: Regional Number F.3.6 Proxy Indicator - Node 6 This Threat-Asset Pair F.3.7 Proxy Indicator - Node 7 Overall Proxy Likelihood use point estimates or RAMCAP scales; if bins are Software must provide a field for point estimates, used, the midpoint is used for and the field should allow for utilization of the calculation calculations from bins. Software must provide a field for an estimate of threat severity. Software must provide a field for estimate of malevolent threats. Software may have a field for number of attacks. Software may have a field for likelihood. Software may have a field for likelihood. Software may have a field for likelihood. Software may have a field for ratio of capacity to metro area. Software may have a field for likelihood (product of V x C x Detection). Software may have a field calculated by multiplying each proxy indicator. The software displays the vulnerability as a J percent range and bin but does not allow the user to input the vulnerability manually. The software uses the J tier table based on the facility (population and critical customers). The software does not determine likelihood for each threat. The software may use J100 scales or point estimates to record the calculated vulnerability. The software must allow the user to assign the likelihood or frequency of all hazards and threats on a specific asset. The software could add proxy measures to help determine the threat likelihood. Small. Adding the ability for the user to enter their own values for vulnerability would only require a user interface change, and the corresponding value would be stored in the aforementioned database column for each threatasset pair. Medium/Unknown (likely medium). Adding the capability for users to assign likelihood of each threat involves the addition of another column to the AssetThreat table in the SEMS database, along with a user interface element to allow users to input and store values in the field. However, how this likelihood will be incorporated into calculations is unknown. Small/Unknown (likely medium). While adding a user interface element to allow for entry of a proxy measures for vulnerability requires just a user interface modification, the changes that that entry would make in the calculation method of overall threat likelihood cannot be determined of 10

92 Table C.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix 4.5.2, G.2 Natural Hazards earthquakes 4.5.2, G.3 Natural Hazards hurricanes 4.5.2, G.4 Natural Hazards tornadoes 4.5.2, G.5 Natural Hazards floods 4.5.2, Appx G Natural Hazards 4.5.2, Appx G Natural Hazards other Appx G is optional and provides data to estimate the risk of each natural hazard - risk is calculated by C x V x T; would be nice to have look-up maps/tables for each natural hazard ice storms, extreme cold weather, wildfires, avalanche, tsunami, landslide, mud slide software must have a field to enter the risk of earthquakes for each magnitude from historical records Software must have a field to enter the risk of hurricanes for each magnitude from historical records. Software must have a field to enter the risk of tornadoes from historical records. Software must have a field to enter the risk of floods for each magnitude from historical records. Software may have a field to enter the risk of ice storms, extreme cold weather, wildfires, avalanche, tsunami, landslide, mud slide from historical records. Software may have a field to enter the risk of other natural hazards. The software has fields for historic information and magnitudes (text boxes), however, these values do not impact the results of the analysis. The software must allow the user to assign the likelihood or frequency of a natural hazard to the facility based on historical data. The software may provide historical data or links to data to help determine the probability that a specific magnitude of natural hazard will occur. Unknown (likely medium). The calculation of the results of the analysis needs to be modified and while the basic formula is known, SEMS' implementation of the calculation is not transparent and so the amount of effort required to implement these changes is unknown , Appx G Natural Hazards Risk Dependency and Proximity Hazards use local historical records for frequency, severity and duration of service denials Software must have field for total natural hazard risk. Software must have a field for predicted dependency and proximity likelihood. The software does not have a field for total natural hazard risk. This could potentially be done manually by performing an analysis with only natural hazards. The software must calculate the risk for each natural hazard and sum them to determine the overall risk due to natural hazards. The software must include historical data on dependency and proximity hazards to determine the likelihood that the threats will occur to the assets. Unknown (likely medium). While it seems like this would be simple enough to implement (a summation of the individual risks), without knowledge of the calculation no assumptions with regards to implementation difficulty can be made. Medium/Unknown (likely medium). The data format of the historical data needs to be consistent for all entries in order to be used for calculations (text fields need to be validated according to these parameters). If this is the case, there must be an update to the calculation to reflect these user inputs Record Estimates record the method used for making the estimates and the estimates themselves as single point values or ranges Software must have room for selected methods to be documented. The software provides and explains a threat likelihood scale and displays the threat likelihood for each threat-asset pair but does not allow the user to determine these values individually for each pair and explain their reasoning. The software must allow the user to determine the likelihood of each threat occurring to an asset and record this estimate, along with the method and reasoning for the estimate. Medium. This information should be added as additional columns to the AssetThreat table in the database, and user interface elements should be added to the Consequence Assignment page to allow the user to enter data to these fields. 4.6, A.4.6 Risk and Resilience Analysis estimates the owner's risk and resilience and the community's resilience relative to each threat-asset pair Software must estimate the owner's resilience and owner's and community's risk for each threat-asset pair. The software calculates the overall risk but not the resilience. It should also be noted that the risk was calculated without the user assigning their own vulnerability and threat likelihood for each pair. The software must use the provided consequence and allow the user to input the vulnerability and threat likelihood for each threat-asset pair to calculate the owner's and community's overall risk. Must include duration of service denial and severity or service denial (gpd) to determine the owner's and community's overall resilience. Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time of 10

93 Table C.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix Calculate Risk for each threat-asset pair calculate risk: C x V x T = R use midpoint of ranges from Appx B Software must calculate risk using the numbers input previously for C, V and T for each threat-asset pair. The software calculates the risk for each pair but without the user assigning their own vulnerability and threat likelihood for each pair. The software must use the provided consequence and allow the user to input the vulnerability and threat likelihood for each threat-asset pair to calculate the risk for each pair. Small/Unknown. Adding user interface fields for the user to choose their own values for vulnerability and threat likelihood is likely a minor undertaking, but there may be more effort required for the updated calculation , Appx H Calculate the current level of resilience use either threat-asset pair resilience metric or holistic approach in Appx H Appx H is nonmandatory Some form of a resilience calculation is mandatory, we assume that the preferred approach is to measure the standard in two ways: threat-asset pair and holistic approach but both of the methods described in the standard are nonmandatory. The software must include duration of service denial and severity of service denial (gpd) to determine the resilience of each threat-asset pair. Unknown. The software contains no current functionality relating to resilience, so incorporating this may require extensive development time Operational Resilience Asset Resilience Metric Duration x severity x vulnerability x threat likelihood = asset resilience metric Operational Resilience Index (ORI) calculated by choosing values from Table H-1 then calculating ORI by multiplying the indicator value by the weight and adding all values (should have fields for the value and weight or by pick box) Software may calculate asset resilience metric using duration and severity from 4.3, vulnerability from 4.4 and threat likelihood from 4.5. The software must calculate the asset resilience. Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time Owner's Economic Resilience lost revenue due to the threat-asset pair (asset resilience x unit price) Financial Resilience Index (FRI) calculated by choosing values from Table H-2 then calculating FRI by multiplying the indicator value by the weight and adding all values (should have fields for the value and weight or by pick box) The software must calculate the owner's economic resilience. Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time. Software may calculate asset resilience x unit price (have field for unit price of asset). Also software may have field for lost economic activity to the community (same as 4.3). The software must calculate the community economic resilience. Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time Community Economic Resilience lost economic activity to the community served by the utility 77 8 of 10

94 Table C.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix Utility Resilience Index - software should calculate URI using the values of ORI and FRI and the weights given in Table H-1 & H-2 (URI=ORI x w1 + FRI x w2) The software creates a scatter plot of consequence versus vulnerability and places each threat-asset pair on the plot. It also generates a report in which the risks of all of the pairs are listed in a table and ranked from the greatest risk to the lowest. However, no resilience is calculated or recorded. The software must calculate and record the resilience of each threat-asset pair Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time Record Risk and Resilience Estimates Software may have fields for threat-asset pair resilience. Software must have fields for both threat-asset pair resilience and holistic resilience , A Decide 4.7.2, A Define decide what risk and resilience levels are acceptable define countermeasure and mitigation/resilience options for the threat-asset pairs that are not acceptable. Include devalue, deter, detect, delay and response; consequence reductions, resilience enhancements Software must provide a field for acceptable risk and resilience level. Software must provide fields for countermeasure and mitigation/resilience for each threat asset-pair. The software does not provide way to set this level or rank the threat-asset pairs. The user is forced to include all pairs for the remaining analysis. The user can define countermeasures to lower the risk, but only by redoing the analysis and adding additional countermeasures. The software must be able to record the desired risk level and eliminate those threatasset pairs which have a risk that falls below this desired level from the remaining analysis. The software must allow users to make improvements to the baseline analysis by adding possible countermeasures for the next part of the analysis. Medium/Unknown (likely small). In order to take into account a user-defined risk and resilience level, a slider or some other user interface component must be added for adjustments. Additionally, there should be some visual representation of which pairs will accordingly be included in the analysis. The user interface additions should be feasible, but there is not enough information to be able to determine how SEMS will determine how that cutoff point is used to filter pairs. Medium. This functionality can be addressed by having an option to re-run the analysis after querying the user for additional countermeasures, without forcing them to manually begin the process again. Such a change would require some user input fields on the results page, as well as a button that would re-run the analysis, so the only functional changes are additions to the user interface of 10

95 Table C.1 - J Standard Gap Analysis Matrix - SEMS Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix 4.7.3, A Estimate estimate investment and operating costs for each option; include regular maintenance and periodic overhaul; adjust to present value Software must provide fields for costs for each option above. The software must allow the user to assign a cost for each countermeasure to determine the annual and capital cost for each option. Small/Unknown (likely small). Allowing the user to enter cost information for each countermeasure can be achieved by adding text fields to the Countermeasure Information page and associating their inputs with those coutnermeasures. Without knowing how the cost calculations are performed, however, it is impossible to determine how much effort it would take to compute annual and capital costs from these inputs , A Assess revisit 4.3 through 4.6 to estimate the risk and resilience levels as if the option was implemented; calculate the estimated benefits of the option Software must provide a field for the new value of risk. The user can determine the new risk value but only by restarting the analysis with additional countermeasures applied. The software must use the provided new consequence, vulnerability, and threat likelihood to calculate the new risk for each threat-asset pair. Medium. This functionality can be addressed by having an option to re-run the analysis after querying the user for new risk and resilience values, without forcing them to manually begin the process again. Such a change would require some user input fields on the results page, as well as a button that would re-run the analysis, so the only functional changes are additions to the user interface , A Identify identify the options that have benefits that apply to multiple threat-asset pairs Software must provide a way to highlight or mark options. The software must show which improvements affect multiple threat-asset pairs and have the greatest benefits. Unknown (likely large). Without knowledge of how the program structures and ranks benefits, it is impossible to determine how difficult it would be to select a subset of the options , A Calculate calculate the net benefits 4.7.6, A Calculate calculate the benefit-cost ratio rank the most cost effective measures 4.7.7, A Review & Rank to implement * Incomplete can indicate that the feature was partially met or that it could be improved upon. Software must include a calculation of the net benefits. Software must include a calculation of the benefitcost ratio. Software must include a field for ranking cost effective measures. The software does not calculate cost for each countermeasure or the benefits since the only way to calculate a lower risk is to restart the analysis. The software must calculate the benefit for each countermeasure improvement, the net benefit, the benefit-cost ratios, and then rank them to determine the most effective measure. Large. Because the SEMS system only calculates for the currently selected countermeasures and risks, redoing the calculation to take into account all of these elements without restarting the analysis will require a fundamental revision to the calculations of 10

96 Table C.2 - J Standard Gap Analysis Matrix - VSAT Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix All-hazards risk and resilience analysis Scope must include all of the same requirements. 1 Scope of vulnerabilities to man-made threats, natural hazards, and dependencies and proximity to hazardous sites. Definitions Is the list the same? If not, how is it different? Is the 2.1, Appx C asset content for each definition the same? critical asset 2.2 consequence 2.3 consequence mitigation 2.4 countermeasure detect deter devalue delay respond 2.5 dependency 2.6 dependency hazard 2.7 event tree analysis 2.8 failure mode 2.9 fault tree analysis 2.10 frequency 2.11 hazard 2.12 incident 2.13 initiating event 2.14 likelihood 2.15 preparedness 2.16 probability 2.17 proximity hazard 2.18 response 2.19 reference threat 2.2 resilience 2.21 risk 2.22 risk analysis 2.23 risk management 2.24 scenario 2.25 system 2.26 threat 2.27 threat likelihood 2.28 vulnerability vulnerability assessment 2.29 /vulnerability analysis 2.30 vulnerability estimate 2.31 worst reasonable case estimate worst reasonable Software must estimate the worst reasonable consequences for each asset without consequences for each asset Consequence metrics regard to the threat 4.1.5(.1) Consequence metrics potential for fatalities 4.1.5(.2) Consequence metrics serious injuries major economic loss to facility or 4.1.5(.3) Consequence metrics community Can group these into Human, $ and other (i.e (.4) Consequence metrics impacts to the environment environmental) Can use Hi, Very Hi, Med, Lo, etc (.5) Consequence metrics loss of public confidence field for each consequence (.6) Consequence metrics inhibiting effective function of national defense or civilian government The software does not calculate risk or resilience. The software does not provide these definitions. The user can select consequences manually. The software should provide a method to calculate resilience, including the supporting user interface elements to gather the user input required for such calculations. The software must provide the user a reference or documentation section in the program, where definitions of these terms can be determined and evaluated for fitness. The software must determine the worst reasonable consequence for each asset prior to determining the threats that could affect the asset. This must be done to help prioritize the assets. Medium/Unknown (likely large) fix. A combination of user interface elements and internal functionality would be required to support calculation of resilience. Medium. Requires the creation of an additional user interface screen, but there is little to no computation required to display these definitions to the user. Medium. In order to configure which consequences apply to each asset before determining which threats apply, additional user interface sections must be added to the Edit Asset page Prioritize assets prioritize critical assets using estimated consequences Software must provide a field for each critical asset to assign rank or re-order them in order of consequence. The software does not allow the user to prioritize assets. The software must have a way to rank the assets in order or their importance. This should be done after the worst reasonable consequences have been determined for each asset. Medium. The Assets tab should have another sub-tab that allows for a ranking of assets by importance. This requires a user interface change and a way to store user selections in the database (in the "assetlist" table) of 6

97 Table C.2 - J Standard Gap Analysis Matrix - VSAT Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix 4.2.2, Appx E & G 4.2.3, Appx G 4.2.3, Appx G 4.2.3, Appx G 4.2.3, Appx G 4.2.3, Appx G 4.2.3, Appx G Natural Hazards Threat Dependency Hazards Threat Dependency Hazards Threat Dependency Hazards Threat Dependency Hazards Threat Dependency Hazards Threat Dependency Hazards Threat wildfires utilities suppliers employees customers transportation proximity Software must define the range of magnitudes from the smallest that would cause serious harm to the largest reasonable case. dependency threat to be considered. dependency threat to be considered. dependency threat to be considered. dependency threat to be considered. dependency threat to be considered. dependency threat to be considered. The software includes fires within the library of standard natural hazards but does not provide a range of magnitude. The software does not include these as predefined fields but they could potentially be added as an "other" under the user defined threat section. The software must provide a range of wildfire magnitudes that would damage the assets. The software should have dependency hazards predefined within the list of potential threats; however, they could still be entered manually within the user-defined section. Medium. The natural hazards editing page (which appears for other natural hazards) should be extended to appear for wildfires. Since this functionality is already in place for other threats, extending it should not require too much modification. Medium. Defining the list of what kinds of dependency threats are possible will require the creation of a new table in the database. Then in order to allow the user to enter a selection from among these choices, user interface elements in the threat editing section should be added to choose a value from the database entries in that table & 4.2.5? Threat-Asset Pairs evaluate and rank threat-asset pairs Critical Threat-Asset Pairs 4.3.2, Appx B Estimate Consequences - service denial for the affected customers select critical threat-asset pairs to be used going forward or use all pairs measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) Software must evaluate and rank threat-asset pairs, can use multiple approaches including a matrix using small, med., large or scales 1-10, etc. Software must provide a check box or field to identify selected critical threat-asset pairs. (Standard does not define critical, this is left up to the user, i.e.. top 10, top 20?) Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach The software does not rate or rank the threatasset pairs based on rough magnitude of consequences prior to determining vulnerability or threat likelihoods. The user cannot remove asset-threat pairs once they have been used in an analysis (even if it's just the baseline analysis). The software does not include the consequence of service denial for the affected customers. The software must rate and rank the pairs based on their rough magnitude of consequences solely based on the possible consequences of the threat on the asset. The software does not give the option of proceeding with an analysis of certain pairs. Must be able to choose the pairs that must be analyzed for their risk. The software must include a field to enter a single point estimate or bin number of the consequence of service denial for the affected customers. Unknown (likely medium). The most logical place in VSAT to be able to display this ranking is in the Threats Assignment and Review page, having an option to sort the entries in that list by a variety of criteria, including consequences. only caused by the threat. However, it is unknown if the system calculates such a value, and if not it must be generated before being displayed here. Medium/Unknown (likely medium). Allowing the user to select a subset of threat-asset pairs to analyze would require a user interface method for selection of a subset, as well as modification of the calculation procedure to take into account only a subset of the threat-asset pairs. This functionality would likely be similar to or the same as the functionality below (where the user can select a level of acceptable risk to filter the threatasset pairs for analysis). Small. In order to allow for the user to enter an estimate of the consequences to users, there must be an additional field in the Edit Assets page, as well as the modification of the assetlist table in the database to store the user input of 6

98 Table C.2 - J Standard Gap Analysis Matrix - VSAT Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix Estimate Consequences - other if degradation in public confidence, environmental quality, ability of civilian or military agencies to function, etc. room for descriptive analysis must be provided additional consequences that can be considered - sociopolitical impacts, natural security impacts, lost strategic capability to cause harm or output, detrimental effects on brand value, public confidence, psychological impacts, and environmental degradation Software must provide the ability to describe primary consequence, and should give the user the option of including other consequences. The software only provides fields for 4 consequence types (fatalities, injuries, economic cost of owner, and economic cost to region). The software should provide a field for a descriptive analysis The software must add a field to record additional consequences for the analysis. Medium. In order to allow for descriptive analysis of an estimated consequence, the software should have an additional user input text field on the Edit Asset page, as well as a column to store that information in the "assetlist" table in the database. Medium. In order to allow for a description of additional consequences to an asset, the software needs to have an additional user input text field on the Edit Asset page, as well as a column to store that information in the "assetlist" table in the database , A.4.4 Analyze Vulnerability analyze vulnerability of each critical asset to estimate the likelihood that, given the occurrence of a threat, the consequences result 4.4.4, Appx B Record Estimates Record the estimates from may use fault, event or failure tree analysis, path analysis, vulnerability logic diagrams, computer simulation methods, or expert judgment rules-ofthumb use point estimates or RAMCAP scales; if bins are used, the midpoint is used for the calculation Software must provide a field for the vulnerability analysis of each asset, and should use one of the following methods: event-tree analysis, path analysis, vulnerability logic diagrams, computer simulations, or judgment rules-of-thumb. Software must provide a field for point estimates, and the field should allow for utilization of calculations from bins. The software relies on the judgment of the user and does not provide for the other methods. For each pair, the user has to rate the vulnerability by "bubbling" the detection (certain, probable, possible, none), delay (very, strong, limited, no delay), and response (fast, variable, slow, none). The software then determines the likelihood as a J percentage range (with rounding) and countermeasure capability (very high, high, moderate, low). The user can also enter comments (text box). The software must implement at least one other method for running a vulnerability of each critical asset. The software must allow for the user to have a greater degree of control over the vulnerability settings for each of these parameters. Unknown (likely large). Since no vulnerability analysis method is currently in place, the analysis of the vulnerability of each critical asset may require significant reworking (of both code and data structure) in order to take into account another analysis method. Medium/Unknown (likely medium). For the user to be able to select from a wider set of values for each of these parameters, different user interface elements should be selected (such as a text field or slider) that allow for a broader range of values. Additionally, these changes could imply that the likelihood calculation, if it is relying upon the few predetermined levels set for each parameter, will require reworking to accept new kinds of values. 4.5, A.4.5 Threat Likelihood Analysis likelihood of malevolent event, dependency/proximity hazard or natural hazard Software must provide a field for an estimate of threat severity. For each pair, the user has to rate the likelihood by bubbling very high, high, moderate, or low. The user can also enter comments (text box). The software should allow the user to have more fine-grained control of the likelihood of a threat for each asset (3 choices is too granular). Medium/Unknown (likely medium). For the user to be able to select from a wider set of values for the threat likelihood, different user interface elements should be selected (such as a text field or slider) that allow for a broader range of values. Additionally, this change could imply that the likelihood calculation, if it is relying upon the few predetermined levels available for each threat-asset pair, will require reworking to accept a broader range of values of 6

99 Table C.2 - J Standard Gap Analysis Matrix - VSAT Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix 4.5.1, Appx F Malevolent Threats use proxy measure, best estimate, or conditional assessment to determine Proxy Measure (Appx F) is optional and preferred Software must provide a field for estimate of malevolent threats. Number of U.S. attacks per F.3.1 Proxy Indicator - Node 1 year Software may have a field for number of attacks. Metro Region (RMS metro F.3.2 Proxy Indicator - Node 2 area classes) Software may have a field for likelihood. Target Type (RMS target type F.3.3 Proxy Indicator - Node 3 analysis) Software may have a field for likelihood. F.3.4 Proxy Indicator - Node 4 Proportion: Regional Number Software may have a field for likelihood. Software may have a field for ratio of capacity to F.3.5 Proxy Indicator - Node 5 This Facility metro area. Software may have a field for likelihood (product of F.3.6 Proxy Indicator - Node 6 This Threat-Asset Pair V x C x Detection). Software may have a field calculated by multiplying F.3.7 Proxy Indicator - Node 7 Overall Proxy Likelihood each proxy indicator , Appx G Natural Hazards 4.5.2, Appx G Natural Hazards other 4.5.2, Appx G Natural Hazards Risk ice storms, extreme cold weather, wildfires, avalanche, tsunami, landslide, mud slide Software may have a field to enter the risk of ice storms, extreme cold weather, wildfires, avalanche, tsunami, landslide, mud slide from historical records. Software may have a field to enter the risk of other natural hazards. Software must have field for total natural hazard risk. The software allows for the option to use best estimate or a 100% probability for all pairs (bubble). The software does not include a field to enter historical data. The software can include the likelihood of the other threats but not based on historical data. The software has a tab for natural threats where all of the asset-threat pairs of natural hazards are displayed, but no total risk is displayed. Also, the added hazards, e.g., windstorms, snowstorms, are not displayed in this tab. The software should provide the user additional ways to estimate the odds of malevolent threats to the asset. The software could add proxy measures to help determine the threat likelihood. The software should include historical data on additional natural hazards to determine the likelihood that the threat will occur to the assets. The software must calculate the risk for each natural hazard and sum them to determine the overall risk due to natural hazards. Medium/Unknown (likely medium). Adding the capability for users to assign likelihood of each threat involves the addition of another column to the "threatasset" table in the VSAT database, along with a user interface element to allow users to input and store values in the field. However, how this likelihood will be incorporated into calculations is unknown. Small/Unknown (likely medium). While adding a user interface element to allow for entry of a proxy measures for vulnerability requires just a slight change to the UI, the changes that that entry would make in the calculation method of overall threat likelihood cannot be determined. Medium/Unknown (likely medium). Additional user interface elements will be necessary in the edit mode of the Natural Threats page, in addition to another column in the "threats" table in the VSAT database. Then, the values entered by the user and stored in this column will need to be taken into account when calculating threat likelihood for particular assets. Unknown (likely small). Because it is not evident from the database structure how the risk is calculated for each natural hazard, it is impossible to deduce how to arrive at a value for the overall risk from natural hazards Dependency and Proximity Hazards use local historical records for frequency, severity and duration of service denials Software must have a field for predicted dependency and proximity likelihood. The software must include historical data on dependency and proximity hazards to determine the likelihood that the threats will occur to the assets. Medium/Unknown (likely medium). The data format of the historical data needs to be consistent for all entries in order for the data entered there to be used for calculations. If this is the case, there must be an update to the calculation to reflect these user inputs. 4.6, A.4.6 Risk and Resilience Analysis estimates the owner's risk and resilience and the community's resilience relative to each threat-asset pair Software must estimate the owner's resilience and owner's and community's risk for each threat-asset pair. The software does not calculate overall risk or resiliency. The software must use the provided consequence, vulnerability, and threat likelihood to calculate the owner's and community's overall risk. It also must include duration of service denial and severity or service denial (gpd) to determine the owner's and community's overall resilience. Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time of 6

100 Table C.2 - J Standard Gap Analysis Matrix - VSAT Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix Calculate Risk for each threat-asset pair calculate risk: C x V x T = R use midpoint of ranges from Appx B Software must calculate risk using the numbers input previously for C, V and T for each threat-asset pair. The software must use the provided consequence, vulnerability, and threat likelihood to calculate the risk for each threat-asset pair. Medium. VSAT has already calculated the three components of the risk calculation: consequence, vulnerability, and threat likelihood. Because these elements are already present, calculating the risk should not be difficult , Appx H Calculate the current level of resilience Operational Resilience Asset Resilience Metric Owner's Economic Resilience Community Economic Resilience use either threat-asset pair resilience metric or holistic approach in Appx H Appx H is nonmandatory Operational Resilience Index (ORI) calculated by choosing values from Table H-1 then calculating ORI by multiplying the indicator value by the weight and adding all values (should have fields for the value and weight or by pick box) Duration x severity x vulnerability x threat likelihood = asset resilience metric lost revenue due to the threat-asset pair (asset resilience x unit price) lost economic activity to the community served by the utility Software may calculate asset resilience metric using duration and severity from 4.3, vulnerability from 4.4 and threat likelihood from 4.5. Financial Resilience Index (FRI) calculated by choosing values from Table H-2 then calculating FRI by multiplying the indicator value by the weight and adding all values (should have fields for the value and weight or by pick box) Software may calculate asset ili it i (h Some form of a resilience calculation is mandatory, we assume that the preferred approach is to measure the standard in two ways: threat-asset pair and holistic approach but both of the methods described in the standard are nonmandatory. The software includes a list of "knowledge base" information for resiliency within the predefined asset list but does not use them to calculate resiliency. The software must include duration of service denial and severity or service denial (gpd) to determine the resilience of each threat-asset pair. The software must calculate the asset resilience. The software must calculate the owner's economic resilience. The software must calculate the community economic resilience. Unknown. The software contains no current functionality relating to resilience, so incorporating this may require extensive development time. Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time. Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time. Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time Record Risk and Resilience Estimates Utility Resilience Index - software should calculate URI using the values of ORI and FRI and the weights given in Table H-1 & H-2 (URI=ORI x w1 + FRI x w2) Software may have fields for threat-asset pair resilience. Software must have fields for both threat-asset pair resilience and holistic resilience. The software does not calculate risk or resilience. The software must calculate and record the risk and resilience estimates for both threat-asset pairs and for the overall utility. Medium/Unknown (likely large). In order to calculate the risk, the software should use its alreadycalculated values for consequence, vulnerability, and threat likelihood and multiply them to find the resulting risk. However, there is no foundation (already calculated values) upon which to begin the calculation of resilience, so it is difficult to gauge how much effort such an implementation would require of 6

101 Table C.2 - J Standard Gap Analysis Matrix - VSAT Feature Met Reference Section No. Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix 4.7.1, A Decide 4.7.4, A Assess 4.7.5, A Identify decide what risk and resilience levels are acceptable revisit 4.3 through 4.6 to estimate the risk and resilience levels as if the option was implemented; calculate the estimated benefits of the option identify the options that have benefits that apply to multiple threat-asset pairs 4.7.6, A Calculate calculate the net benefits 4.7.6, A Calculate calculate the benefit-cost ratio 4.7.7, A Review & Rank rank the most cost effective measures to implement * Incomplete can indicate that the feature was partially met or that it could be improved upon. Software must provide a field for acceptable risk and resilience level. Software must provide a field for the new value of risk. Software must provide a way to highlight or mark options. Software must include a calculation of the net benefits. Software must include a calculation of the benefitcost ratio. Software must include a field for ranking cost effective measures. The software does not provide a way to set this level or rank the threat-asset pairs and the user is forced to include all pairs for the remaining analysis. The software calculates risk reduction units. The calculation and explanation are confusing. The calculation can only be used for comparison purposes. The user can create multiple packages of countermeasures and improvements to compare and determine which one benefits the utility the most. The user can create upgrade packages and compare the annualized cost, capital cost and risk reduction units but cannot calculate net benefit and benefit-cost ratios. The software must be able to record the desired risk level and eliminate those threat-asset pairs which have a risk that falls below this desired level from the remaining analysis. The software must use the provided new consequence, vulnerability, and threat likelihood to calculate the new risk for each threat-asset pair. The software should contain the functionality to highlight which of the countermeasure packages provides the greatest or greater benefits, to point the user towards the countermeasures that would be most cost-effective. Must calculate the benefit for each countermeasure improvement, the net benefit, the benefit-cost ratios, and then rank them to determine the most effective measure. Large. Allowing the user to select a subset of the threat-asset pairs would best be done on the Baseline Summary page, where the user could set an arbitrary acceptable risk level and see which of the threat-asset pairs will be included as unacceptable risks. In order for this to be implemented, a horizontal slider or a dropdown box with different levels of risk should be added to the page and the asset-threat pairs that are included in the analysis will need to be indicated. In order to modify the calculation, the database entries that are selected from the "threatasset" table for the analysis will need to be restricted based on the user input. Medium. VSAT has already calculated the three components of the risk calculation: consequence, vulnerability, and threat likelihood. Because these elements are already present, calculating the risk should not be difficult. Small. Assuming the benefit for each countermeasure package is already calculated and ranked (see below), highlighting the highest ranked packages should be trivial. Medium. Because the different countermeasure upgrade packages already have calculated costs, creating a comparison between these packages based on these calculations should not require too large of a change. Additionally, the user interface should be updated to display these rankings of countermeasure improvement packages of 6

102 Table C.3 - J Standard Gap Analysis Matrix - ARAM-W Feature Met Reference Section Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix 1 Scope Definitions All-hazards risk and resilience analysis of vulnerabilities to manmade threats, natural hazards, and dependencies and proximity to hazardous sites. Scope must include all of the same requirements. The software does not calculate resilience. The software should provide a method to calculate resilience, including the supporting user interface elements to gather the user input required for such calculations. Large/Unknown (likely large) fix. A combination of user interface elements and internal functionality would be required to support calculation of resilience. Is the list the same? If not, how is it different? Is 2.1, Appx C asset the content for each definition the same? critical asset 2.2 consequence 2.3 consequence mitigation 2.4 countermeasure detect deter devalue delay respond 2.5 dependency 2.6 dependency hazard 2.7 event tree analysis 2.8 failure mode 2.9 fault tree analysis 2.10 frequency 2.11 hazard 2.12 incident 2.13 initiating event 2.14 likelihood 2.15 preparedness 2.16 probability 2.17 proximity hazard 2.18 response 2.19 reference threat 2.2 resilience 2.21 risk 2.22 risk analysis 2.23 risk management 2.24 scenario 2.25 system 2.26 threat 2.27 threat likelihood 2.28 vulnerability vulnerability assessment 2.29 /vulnerability analysis 2.30 vulnerability estimate 2.31 worst reasonable case No definitions were included in the software. The software must provide the user a reference or documentation section in the program, where defitions of these terms can be determined and evaluated for fitness. Medium. Requires the creation of an additional user interface screen, but there is little to no computation required to display these definitions to the user. 1 of 7 86

103 Table C.3 - J Standard Gap Analysis Matrix - ARAM-W Feature Met Reference Section Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix Can group these into Human, $ and other (i.e.. environmental) Can use Hi, Very Hi, Med, Lo, etc. field for each consequence. The user must select the undesired events, (i.e. release of chemicals, loss of power, loss of critical pump/valve system), from a predefined event tree. There is a separate tree for each type of utility and users can add additional events. The user can then determine the consequence of each undesired event, which are determined by assuming the loss of the asset which leads to the undesired event, without regard to the threats that may cause the loss of the asset. Although environmental impact is not predefined, the user has the ability to define any new consequences that they feel apply. The software should provide the user a way to select their own set of consequences and edit these consequences to include custom consequences outside of the predefined set. Medium/unknown (likely small). Requires the creation of a user interface for the editing of consequences which will need to be stored in the database (.4) Consequence metrics impacts to the environment 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.2, Appx E & G 4.2.3, Appx G 4.2.3, Appx G Natural Hazards Threat Natural Hazards Threat Natural Hazards Threat Dependency Hazards Threat Dependency Hazards Threat wildfires ice storms utilities suppliers Undefined other Software must define the range of magnitudes from the smallest that would cause serious harm to the largest reasonable case. Not listed in body, listed in non-mandatory appendix, analyzed as non-mandatory. Software should give the user the option of including other natural hazards or threats. Software must provide a field to describe the type of dependency threat to be considered. Software must provide a field to describe the type of dependency threat to be considered. The software does not include any additional natural threats or allow any other threats to be added. Medium/unknown (likely small). The software must allow for the user to add Requires the creation of a user additional natural threats, as well as including interface for the addition of new more default options to select from when natural threats to the system, as choosing a natural hazard. The most logical place well as a way to add these threats for this functionality to be implemented would be to the database. Additionally, on the "Natural Hazard Identification" screen, adding other natural threats to where the user could select from a larger set of the existing default threats options for the hazard type, as well as add more should be a very simple task (adding a new row per threat to threats. the database) , Appx G 4.2.3, Appx G 4.2.3, Appx G Dependency Hazards Threat Dependency Hazards Threat Dependency Hazards Threat employees customers transportation Software must provide a field to describe the type of dependency threat to be considered. Software must provide a field to describe the type of dependency threat to be considered. Software must provide a field to describe the type of dependency threat to be considered. The software does not include dependency threats, the RAMCAP labels for dependency threats, or dependency threats within the RAMCAP reference tables. The software must include another type of threats, dependency threats. Adding this functionality would best be accomplished through the creation of another subsection under the Threat Assessment heading in the left menu on the tab for each facility. In that new section, the software will need to have the ability to select the type of dependency threat, as well as enter additional data about the dependency threat. Large/Unknown (likely medium)/unknown. Requires the creation of a new threat type, along with all of the associated changes required for that change (user interfacecreation of additional screens, database updates, calculation updates) , Appx G Dependency Hazards Threat proximity Software must provide a field to describe the type of dependency threat to be considered , Appx G Dependency Hazards Threat Undefined other Software should give the user the option of including other kinds of dependency hazards or threats. 2 of 7 87

104 Table C.3 - J Standard Gap Analysis Matrix - ARAM-W Feature Met Reference Section Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix Evaluate & Rank Threat-Asset Pairs evaluate and rank threat-asset pairs Software must evaluate and rank threat-asset pairs, can use multiple approaches including a matrix using small, med., large or scales 1-10, etc. The software does not rate or rank the undesired event/asset location/threat pairs based on rough magnitude of consequences prior to determining vulnerability or threat likelihoods. The software should calculate and display the magnitude of the consequences for each asset before running the analysis. The most logical place for this rating/ranking to be displayed is in the Asset list under facility characterization. Small/Unknown (likely small). Displaying the rating on the asset list would require a small modification to the user interface. However, the database structure for ARAM-W is not visible, so it is impossible to determine how difficult this kind of calculation would be Critical Threat-Asset Pairs 4.3, A.4.3 Consequence Analysis 4.3.1, Appx B Threat Scenario 4.3.2, Appx B 4.3.2, Appx B Estimate Consequences - loss of life to either employees or the general public select critical threat-asset pairs to be used going forward or use all pairs identifies the worst reasonable consequences that can be caused by the specific threats on the assets as identified in 4.1 apply worst reasonable case assumptions for each threat scenario measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss Estimate Consequences - indicator; or in pre-defined ranges serious injury to either represented by the RAMCAP "bins" employees or the general public (Appx B) optional single indicator - dollar equivalent of fatalities and serious injuries in excess of insurance optional single indicator - dollar equivalence of fatalities and serious injuries in excess of insurance Software must provide a check box or field to identify selected critical threat-asset pairs. (Standard does not define critical, this is left up to the user, i.e.. top 10, top 20?) Software must identify the worst reasonable consequence of a threat on assets. Software must assume the worst reasonable case for each threat. Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach. Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach The user cannot remove undesired event/asset location/threat pairs once they have been assigned in the baseline analysis. Although the consequences for the loss of critical assets and the resulting undesired event occurring were previously determined, they are not evaluated for each specific threat on the assets. The software should allow for a subset of the Medium/Unknown (likely threat-asset pairs to be chosen from each analysis medium). result page in order to run another analysis. The Allowing the user to select the Analysis Calculation Mode section should allow threat-asset pairs to use in the for the selection of which threat-asset pairs are calculation would require a allowed to continue to be used in the next check box to be added for each analysis. threat-asset pair displayed, that can be toggled by the user if the pair should be included in further analyses. How the calculation would be impacted by this selection is unknown, however. The software should allow for the relationship between the consequences and threats to be explored through the database to determine what kind of a relationship exists between the two. This relationship should be displayed in the Threat Assessments section of the interface. Unknown (likely medium). The structure of the database makes a large impact on the association of the threats and the consequences. Depending on how the existing structure is laid out, this could be either a very simple association (if the relational structure is welldesigned) or could require a reworking of the database layout to accomplish this change , Appx B Estimate Consequences - Financial loses to owner/operator measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach 3 of 7 88

105 Table C.3 - J Standard Gap Analysis Matrix - ARAM-W Feature Met Reference Section Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix 4.3.2, Appx B 4.3.2, Appx B Estimate Consequences - service denial for the affected customers Estimate Consequences - economic losses to society and the general public measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) measured by 2 of the following: natural units reported and considered individually; converted into a single, summary economic value, reported and considered as a single loss indicator; or in pre-defined ranges represented by the RAMCAP "bins" (Appx B) optional single indicator - value of a statistical life Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach Software must provide a field for single point estimate of consequences or a bin number. When reviewing bins, the values must match. Under 4.3 this is defined as an "or" under it is defined as an "and" analysis shall include "and" as a more conservative approach Although the consequences for the loss of critical assets and the resulting undesired event occurring were previously determined, they are not evaluated for each specific threat on the assets. The software should allow for the relationship between the consequences and threats to be explored through the database to determine what kind of a relationship exists between the two. This relationship should be displayed in the Threat Assessments section of the interface. Unknown (likely medium). The structure of the database makes a large impact on the association of the threats and the consequences. Depending on how the existing structure is laid out, this could be either a very simple association (if the relational structure is welldesigned) or could require a reworking of the database layout to accomplish this change , A.4.4 Analyze Vulnerability 4.5.1, Appx F Malevolent Threats analyze vulnerability of each critical asset to estimate the likelihood that, given the occurrence of a threat, the consequences result use proxy measure, best estimate, or conditional assessment to determine may use fault, event or failure tree analysis, path analysis, vulnerability logic diagrams, computer simulation methods, or expert judgment rules-ofthumb Proxy Measure (Appx F) is optional and preferred Software must provide a field for the vulnerability analysis of each asset, and should use one of the following methods: event-tree analysis, path analysis, vulnerability logic diagrams, computer simulations, or judgment rules-of-thumb. Software must provide a field for estimate of malevolent threats. The user can determine the vulnerability either by entering a user estimated single value or by using a path analysis with the adversary sequence diagram. The software could better meet the Standard by providing a means to use other methods to analyze vulnerability. For each undesired event/asset location/threat pair, the user can choose to determine the likelihood of attack using conditional, expert judgment, or a questionnaire method. For the conditional method, the likelihood value is automatically considered High (or 100%) based on the assumption that the attack will occur. For the expert judgment method, the user can input a likelihood of high, medium or low (drop down menu). For the questionnaire method, the likelihood value is determined based on the responses to the questions asked. The questions follow the threat factors outlined in the RAMCAP - The Framework document for estimating likelihood of attack and include initial consideration of capability, history, current interest, current surveillance, documented threats, potential consequences, ideology and ease of attack. The user answers the different questions by clicking on the appropriate circle and the threat potential is calculated based on the responses to the questions. The software should allow for the user to have another means to analyze the vulnerability of critical assets beside a user estimate or a path analysis. The software should allow the user to have a greater degree of control when selecting the threat likelihood. In particular, the "expert judgment" options should be expanded to include more than three different threat levels. Unknown (likely medium). The size of this fix depends entirely upon the other method chosen to analyze vulnerability. Depending on its complexity, this could take a significant amount of time. Small/Unknown (likely small). The user interface would need to be adjusted to include more values (probably 5 or more differing levels of threat), and those levels will need to be taken into account into the calculation. 4 of 7 89

106 Table C.3 - J Standard Gap Analysis Matrix - ARAM-W Feature Met Reference Section Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix F.3.1 Proxy Indicator - Node 1 Number of U.S. attacks per year Software may have a field for number of attacks. F.3.2 Proxy Indicator - Node 2 Metro Region (RMS metro area classes) Software may have a field for likelihood. F.3.3 Proxy Indicator - Node 3 F.3.4 Proxy Indicator - Node 4 Target Type (RMS target type analysis) Proportion: Regional Number Software may have a field for likelihood. Software may have a field for likelihood. The software does not include the proxy method. The software must include another option for assessing the risk level of a facility (the proxy method). F.3.5 Proxy Indicator - Node 5 This Facility F.3.6 Proxy Indicator - Node 6 This Threat-Asset Pair F.3.7 Proxy Indicator - Node 7 Overall Proxy Likelihood Software may have a field for ratio of capacity to metro area. Software may have a field for likelihood (product of V x C x Detection). Software may have a field calculated by multiplying each proxy indicator. Large. Addition of another calculation method for the risk level will require some considerable development time, as well as requiring modifications to the database and user interface to display the result of such a calculation , Appx G Natural Hazards 4.5.2, Appx G Natural Hazards other 4.5.2, Appx G Natural Hazards Risk ice storms, extreme cold weather, wildfires, avalanche, tsunami, landslide, mud slide Software may have a field to enter the risk of ice storms, extreme cold weather, wildfires, avalanche, tsunami, landslide, mud slide from historical records. Software may have a field to enter the risk of other natural hazards. Software must have field for total natural hazard risk. The software only includes four natural threats and does not allow for the addition of any others. The software does not include total risk of natural hazards. The software must allow for a greater range of flexibility when defining the natural threats to the assets. In particular, the software needs the functionality to add more natural hazards. This Medium/unknown (likely functionality needs to exist in order to allow for medium). Requires the creation of a user other threat types (e.g. a tsunami) or to allow for interface for the addition of new threats of the same type but of different natural hazards to the system, as magnitude (i.e. a hurricane of magnitude 2 well as a way to add these should be considered differently than a hurricane hazards to the database. of magnitude 4). These changes should be incorporated into the Natural Hazard Identification section of the interface. The software must display the total risk to the assets from natural hazards. Ideally, this should appear at the bottom of the Natural Hazard Identification page, where the list of natural hazards already resides. Small. Because the individual risk from each natural hazard is known, this change involves adding a text field and displaying a sum of all of the natural hazards in that field Dependency and Proximity Hazards use local historical records for frequency, severity and duration of service denials Software must have a field for predicted dependency and proximity likelihood. Dependency and proximity threats are not included in the software. The software must display the predicted likelihood of a proximity and dependency threat for the assets. The ideal location for these to display would be the Threat Assessment section of the user interface. Small/Unknown (likely medium). The addition of a user interface element to display the results of these likelihood calculations is a minor task, but the calculation of the likelihoods may require more effort Record Estimates record the method used for making the estimates and the estimates themselves as single point values or ranges Software must have room for selected methods to be documented. The user can include comments about the threat likelihood chosen or information about the sources used and displays the threat potential values as none, low, medium, high, or very high in the report. The software should include some form of documentation to let the user know how the questionnaire computes the threat likelihood. Small. The user interface should include a brief description of how the answers to the questionnaire impact the overall computed threat likelihood, which is a minor user interface addition). 5 of 7 90

107 Table C.3 - J Standard Gap Analysis Matrix - ARAM-W Feature Met Reference Section Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix 4.6, A.4.6 Risk and Resilience Analysis estimates the owner's risk and resilience and the community's resilience relative to each threat-asset pair Software must estimate the owner's resilience and owner's and community's risk for each threat-asset pair. The software does not calculate resilience. The software must include duration of service denial and severity of service denial (gpd) to determine the resilience of each threat-asset pair. Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time , Appx H Calculate the current level of resilience Operational Resilience Asset Resilience Metric Owner's Economic Resilience Community Economic Resilience Record Risk and Resilience Estimates use either threat-asset pair resilience metric or holistic approach in Appx H Appx H is no mandatory Operational Resilience Index (ORI) calculated by choosing values from Table H-1 then calculating ORI by Duration x severity x vulnerability x multiplying the indicator threat likelihood = asset resilience value by the weight and metric adding all values (should lost revenue due to the threat-asset pair (asset resilience x unit price) lost economic activity to the community served by the utility Financial Resilience Index (FRI) calculated by choosing values from Table H-2 then calculating FRI by multiplying the indicator value by the weight and adding all values (should Some form of a resilience calculation is mandatory, we assume that the preferred approach is to measure the standard in two ways: threat-asset pair and holistic approach but both of the methods described in the standard are no mandatory. y software should calculate URI using the values of ORI and FRI and the weights given in Table H-1 & H-2 (URI=ORI x w1 + FRI x w2) Software must have fields for both threat-asset pair Software may have fields for resilience and holistic resilience. The software does not calculate resilience. The software does not calculate resilience. The software does not calculate resilience. The software does not calculate resilience. The software displays the overall risk and the risk for each undesired event/asset location/threat pair as either none, low, medium, high, or very high. However, resilience is not calculated or displayed. The software must include duration of service denial and severity of service denial (gpd) to determine the resilience of each threat-asset pair. The software must include duration of service denial and severity of service denial (gpd) to determine the resilience of each threat-asset pair. The software must include duration of service denial and severity of service denial (gpd) to determine the resilience of each threat-asset pair. The software must include duration of service denial and severity of service denial (gpd) to determine the resilience of each threat-asset pair. The software must include duration of service denial and severity of service denial (gpd) to determine the resilience of each threat-asset pair. Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time. Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time. Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time. Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time. Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time , A Decide decide what risk and resilience levels are acceptable Software must provide a field for acceptable risk and resilience level. The software includes a field to determine if the risk for each undesired event/asset location/threat pair is acceptable (check box). The software does not include resilience in this decision. The software must include duration of service denial and severity of service denial (gpd) to determine the resilience of each threat-asset pair. Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time , A Define define countermeasure and mitigation/resilience options for the threat-asset pairs that are not acceptable. Include devalue, deter, detect, delay and response; consequence reductions, resilience enhancements Software must provide fields for countermeasure and mitigation/resilience for each threat asset-pair. The user can create upgrade packages and develop them by redoing the ASD analysis with the inclusion of more countermeasures and other enhancements. The user can also create upgrade packages which only affect the consequences which can be entered. The software does not include resilience enhancement measures. The software must include duration of service denial and severity of service denial (gpd) to determine the resilience of each threat-asset pair. Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time. 6 of 7 91

108 Table C.3 - J Standard Gap Analysis Matrix - ARAM-W Feature Met Reference Section Reference Section Title Mandatory Features Non-mandatory Features Notes/Comments Yes No Incomplete* Gap Recommendations Type/Size of Fix 4.7.4, A Assess 4.7.5, A Identify revisit 4.3 through 4.6 to estimate the risk and resilience levels as if the option was implemented; calculate the estimated benefits of the option identify the options that have benefits that apply to multiple threat-asset pairs Software must provide a field for the new value of risk. Software must provide a way to highlight or mark options. The software calculates the new risk of each pair for each upgrade package. The software displays a report of the baseline data and upgrade packages for the highest risk threat-asset location pair. It does not calculate benefits of the options or include resilience. The user can create multiple packages of countermeasures and improvements to compare but the software does not determine which improvements are included in multiple packages. The software must include duration of service denial and severity of service denial (gpd) to determine the resilience of each threat-asset pair. The software should make a visible distinction for the improvements that have been placed into multiple packages. The preferred way to indicate this would be a highlighting of the improvements that have been included in multiple countermeasure packages. Unknown. The software contains no current functionality relating to resilience, so incorporating this could require extensive development time. Medium. Searching through the packages (in the database) for their included improvements will yield the list of packages that contain the same improvements, and these packages can be displayed with a small marker next to them to highlight that fact , A Calculate calculate the net benefits 4.7.7, A Calculate calculate the benefit-cost ratio Software must include a calculation of the net benefits. Software must include a calculation of the benefitcost ratio. The software can compare packages but there are no calculations for net benefit. The software does not calculate net benefit, nor does it compare the benefit to the cost. The software must calculate the net benefit of each countermeasure package and display it to the user. This information should appear on the Upgrade Packages page as an additional field for each package. The software must calculate the benefit-cost ratio for each countermeasure upgrade package and display it to the user. The most appropriate place for this to occur is in the "Upgrade Packages" page. For each entry in the list of upgrade packages, the software must display its benefits and cost so the user can select the most appropriate upgrade package. Small. The database contains the information pertaining to the benefit for each improvement in the package, so determining the sum and outputting it for the user involves little calculation and addition of a user interface element to display the output. Small. Assuming the above net benefit for each package has been calculated, the cost for each improvement package is known and the ratio is a simple calculation that can then be displayed to the user with a user interface element , A Review & Rank rank the most cost effective measures to implement * Incomplete can indicate that the feature was partially met or that it could be improved upon. Software must include a field for ranking cost effective measures. The software does not consider the most effective package, only the packages and their affect on each pair, especially the highest risk pair. Assuming that the net benefit has been determined (see above), the software should use that information, along with the cost of each upgrade package, to determine the costeffectiveness of each package. This costeffectiveness should be displayed for each package on the "upgrade packages" page. Small. The cost-benefit ratio is equivalent to the cost effectiveness, and so finding the most effective package involves selecting the package with the (already calculated above) highest cost-benefit ratio. 7 of 7 92

109

110 WEB-ONLY /11-RF 6666 West Quincy Avenue, Denver, CO USA P F Gap Assessment for ASME-ITI/AWWA J Standard and Leading Vulnerability Assessment Tools 4358