An approach for evaluating methods for risk and vulnerability assessments

Risk, Reliability and Societal Safety Aven & Vinnem (eds) 2007 Taylor & Francis Group, London, ISBN 978-0-415-44786-7 An approach for evaluating methods for risk and vulnerability assessments E. Ford IRIS, Norway T. Aven University of Stavanger, Norway H. Wiencke & W. Røed Proactima, Norway ABSTRACT: This paper presents and discusses an approach for evaluating methods for risk and vulnerability assessments, covering both accidental events and security issues. The approach provides guidance on the selection of one specific, suitable method for various types of decision situations, reflecting different levels of potential consequences and associated uncertainties. The evaluation addresses the various stages of the risk assessment process, covering planning, execution and use of the risk and vulnerability analyses. Some examples of application of the evaluation approach are outlined. 1 INTRODUCTION A natural consequence of society s development from a border-restricted one with few interacting inhomogeneous systems, to a globalized one with many inhomogeneous systems interacting, is the occurrence of risks related to this interaction. A specific characteristic of this development is the ever-growing dependability upon information and communication technology (ICT). Such dependability gives rise to vulnerability in the systems that interact, and in an infrastructure context, societal vulnerability. This is reflected in risk management in terms of a steadily growing demand for means to mitigate risk and vulnerability, not merely related to safety (as traditionally was the case), but also to security, and only not restricted to certain industries or areas (such as oil/gas or nuclear industry), but to broad parts of society. In recent years many methods have been developed attempting to meet these needs. These methods vary with respect to focus, approach, time, resources required etc. Given a certain context, where the aim is to assess risk and vulnerability, and where we have identified a decision problem, a main challenge is: Which method is suitable for this specific decision problem? This paper is a part of the BAS 5 project, a cooperation between the Norwegian Research Establishment (FFI), University of Stavanger, Gjøvik University College, the Directorate for Civil Protection and Emergency Planning, Statnett, the Norwegian National Security Agency, and the Norwegian University of Science and Technology (NTNU), which aims to develop a methodology and analyze the vulnerabilities in those infrastructures that depend on ICT. A particular focus is on systems and infrastructures critical to the modern society. In Wiencke, et al. (2006), a framework for selection of methodology for risk and vulnerability assessments of infrastructures depending on ICT is presented, outlining the main steps, from specifying a problem definition and determining type of methodology to selecting a specific method. The selection of a specific method must be based on a foundational basis. The evaluation approach presented in this paper is part of such a foundation, and is an integrated part of the framework presented by Wiencke, et al. (2006). For discussions and approaches regarding risk and vulnerability assessments related to ICT, we refer to Wiencke, et al. (2006) and the references therein. The selection of a specific method is the main focus of the present paper. We refer to Aven (2006) for a discussion of the other elements of the process, especially addressing security applications. These references also provide a discussion of the probabilistic foundations of the analyses. We refer to Wiencke et al. (2006) for a discussion of selection of type of methodology. In the present paper we adopt a perspective on risk expressing that risk is the combination of possible consequences and associated uncertainties. Probability is used as a measure of uncertainty. 1375

2 FRAMEWORK The starting point is a holistic risk management process covering three main activities: Establish context Perform risk and vulnerability assessment Risk treatment Each of these main tasks is further divided into smaller tasks, as shown in Figure 1. The figure is based on the risk management process described in the AS/NZS 4360 standard (2004) and in ISO (2002). A framework for the method selection process is presented in Figure 2. Once the problem definition has been established, information has been gathered, and a plan for the subsequent stages has been arranged, a suitable method must be selected, which meets the demands of the decision problem. That is, the selected method must correspond to the objective of Implementation, monitoring and evaluation of futuredevelopment Security Risk Management Process Problem definition,information gathering and planning Selection of methods for risk and vulnerability assessment Identification of potential threats/hazards effecting the ICT system. Perform consequence analysis for identified hazards and threats Perform casual analysis and assess uncertainties for identified hazards Establish the overall risk picture. Risk evaluation. Identify, assess and evaluate measures. Proposed priority of measures based on a holistic evaluation Management review and decision Safety 1)Establish context 2) Risk and vulnerability assessment 3) Risk Treatment Figure 1. Framework for selection of working processes and methodology for risk and vulnerability assessment (Wiencke, et al., 2006). Classification of decisionproblem and selection of type of methodology Simplified risk and vulnerability analysis. Other.. VAM Problem definition, information gathering and planning Tele Risk.. Standard risk and vulnerability analysis. Selection of specific method FRAP FIRM ebios COBRA.. Perform risk and vulnerability analysis Model basedrisk and vulnerability analysis. Figure 2. Framework for selection of method for risk and vulnerability assessment (Wiencke, et al., 2006). CRAMM CORAS Other.. the analysis, the criticality of the ICT system, rules and regulations, etc. The first step in this process is to select the type of methodology. There are three different classes: Simplified risk and vulnerability analysis Standard risk and vulnerability analysis Model based risk and vulnerability analysis These classes differ with respect to degree of formalism and sophistication, and thus with respect to what is required in order to use them. Once a desired type of methodology is selected, it remains to choose the specific method. The process of selecting a specific method is done in part based upon existing in-depth evaluations of different methods. One of the tasks in the BAS 5 project has been conducting such evaluations for a variety of risk and vulnerability methods. When applying the framework then, a portfolio of method evaluations exists for selection support. The following chapter outlines both the process of evaluating a method as well as the process of selecting a method when evaluations are available. 3 THE EVALUATION APPROACH The evaluation approach gives guidelines as to how a method should be evaluated, and how evaluations can be used to make a well-informed selection of a specific method. In order for the process of evaluating risk analysis methods to function as a structured process, some common criteria must be defined. In the BAS 5 project, these criteria are related to: 1. Methodology 2. Experience and competence needed 3. Required resources The methodology criterion addresses the theoretical foundation of the approach, how it is structured in terms of which steps or activities it contains, and so forth. The evaluation should also give information regarding the level of expertise required. Even though a specific method may be related to the desired area (for example telecommunication), it may still not be an appropriate method to use, if the experience and competence needed far exceeds the demands of the decision problem. The same logic applies to the third criterion. Seen together, the criteria should (at least) answer the following questions: What type of method is this, and what does it demand in terms of experience, competence and required resources? 3.1 Conducting an evaluation of a risk analysis method The evaluation should provide a brief overview or summary of the most important elements of a method, as 1376

well as more in-depth comments related to different activities of the risk management process, and determine to what extent these activities are covered by the method. Figure 3 illustrates how a short summary of the method can be displayed in tabular form: This table, along with a short description, can give a brief overview of important properties of the method. Providing information concerning which phases of the risk management process are adequately covered, can assist in determining whether the demands of the decision problem coincide with the properties of the method. A summary table can provide useful information which the method selection can be based upon. However, since methods with different properties can be equally suitable for a complex decision problem, and since many methods could have the same properties, a more detailed picture is required. This must provide more specific information regarding the method s risk management process, while at the same time incorporate the aforementioned criteria of the evaluation process. Clearly, if this is to be structured, then there must be a link connecting the risk management framework, the selection process, and the evaluation approach. A questionnaire (see Appendix) is used for the evaluations. This questionnaire is based upon the three evaluation criteria; methodology, experience and competence needed, and required resources. The questionnaire comprises the three main activities in the BAS 5 risk management process: to establish context, risk and vulnerability assessment, and risk treatment. To gain a general overview of the method, the questionnaire contains questions regarding whether the method is quantitative or qualitative, extensiveness, prior experience with use of the method, who the target user is, required resources, expertise and amount of information, and other relevant questions. This can be Properties of the method Focus Attributes Level of detail Simplified Standard Detailed Phasescovered by themethod Establish context Accidents Security Description Analysis Coarse Detailed Type of ICT General Risk- and vulnerability assessment Intentional threats Holistic (security, environment, economy, reputation ) Specific type (specify) Risk treatment Comments Figure 3. Table summarizing the most important properties of a method. seen as an elaboration of the properties in the summary table. Further, the questionnaire is divided into risk management phase-specific questions. In establishing the context of a decision problem, it is essential that a method clearly defines the problem and gives guidance to information gathering and planning. Hence, this section focuses on issues such as whether basic principles and key terms are well-defined, which assumptions the method is based upon, whether purpose, problem definition and decision foundation is clarified, and whether planning and system description constitute part of establishing the context in the method. With respect to the risk and vulnerability assessment section of the questionnaire, the issues regard whether such elements as identification of unwanted events, hazard identification, cause and consequence analysis, evaluations of barriers, uncertainties and probabilities, sensitivity analysis and dependencies are covered by the method and to what degree. Finally, in the risk treatment section, the review concentrates on how the results of the previous assessment section are treated, and how and to what extent the results constitute a part of the decision foundation. That is, the questions relate to type of evaluation of the results, use of risk acceptance criteria, whether the method is conditioned on assumptions, whether a holistic view is used in the method, how implementation of mitigating measures are ensured, etc. On the basis of a conducted evaluation, the following points of departure can be stated: 1) The questionnaire, when filled out, constitutes an in-depth evaluation allowing for distinction between similar methods and for reviewing suitability of different methods with respect to complex decision problems. 2) The in-depth evaluation constitutes a basis for summarizing the basic properties for the method. 3) The in-depth evaluation constitutes a basis for classifying a method as a simplified, standard or model based risk assessment method. Assuming that, by conducting several evaluations, the above objectives are fulfilled; how do these evaluations contribute to the selection of the most suitable method? Given a well-defined decision problem, this can be classified into one out of three categories of different problem complexity, corresponding to the classification of methodologies. The problem of selecting the most suitable method is then reduced to selecting among methods within one group. Further, methods within a specific group can to a large extent be separated on the basis of the in-depth evaluations, so that the selection problem will further reduce itself to (potentially at least) selecting between a few methods of similar character. 1377

4 EXAMPLES OF EVALUATIONS AND APPLICATIONS THEREOF In this section we give an example of how an evaluation can be conducted, and show how an evaluation can be used to select a specific method. A computer software company is used as a starting point for the example. The company has recently become aware of a threat, in the form a malicious virus, which could potentially breach the company s security firewalls and cause severe damage. On this basis, the company would like to known in what way, and to what degree, they are vulnerable to this threat. A risk and vulnerability analysis will be used to gain knowledge on the threat. The company has defined a preliminary decision problem: Which mitigating means are best suited to protect the company against the identified threat? Through the planning stage of the risk management process, it is decided, in part based upon readily available resources and competencies, that the primary purpose of the analysis should be, giving further insight into the threat posed, and giving a coarse view of the risk picture. Depending upon the outcome of the preliminary analysis, further analysis of a more sophisticated nature will be conducted. Hence, the company has (presently) decided that the decision problem conforms to using a simplified risk and vulnerability assessment. However, there are many simplified risk and vulnerability assessment methods available, so a choice has to be made. Now, we could suppose that evaluations for different methods were readily available (as would be the case when applying the framework), but for the purpose of outlining the evaluation process itself, we shall first see how an evaluation can be conducted, before illustrating how the resulting evaluation can be used. It is important to note that the evaluations are not related to a specific decision problem. The following example uses an excerpt from the questionnaire, where two methods (named method 1 and method 2) are evaluated. For the sake of simplicity, only questions regarding the risk and vulnerability assessment activity are included. The evaluation questions are denoted Q, and corresponding evaluations are denoted E1 and E2 for method 1 and method 2, respectively: Q: Is there a structured hazard identification process based upon multiple and diverse sources? E1: The hazard identification process is primarily based upon checklists. E2: Identification is performed by selecting amongst predefined attack components, but there is no specification of how information should be gathered or from what sources. Q: Does the method include cause- and consequence analysis with assessments of uncertainties and probabilities? E1: Causes are indirectly identified through hazardand threat identification, but are not treated formally. Uncertainties/probabilities are assessed on a coarse scale related to vulnerabilities. E2: The method does not support identification of causes. A coarse assessment of the risk of attack is presented, which is not synonymous with assessments of consequences. Uncertainties/ probabilities are not assessed. Q: Are potential risk reducing measures identified and evaluated? E1: Measures are identified by means of a stringent procedure which follows the ISO standard closely. The focus is on measures which reduce the greatest and most probable risks. E2: Measures are identified by using brainstorming, and checklists categorized by barriers, detection and reaction. The measures are ranked by using a cost/benefit-analysis. Evaluation related to establishing context and risk treatment is conducted in a similar fashion. Now, assuming that the process of selecting a specific method in this case consists of choosing between method 1 and method 2, how can the above evaluation be of guidance? The starting point should be the decision problem at hand, and the specified purpose of the analysis. As we are interested in addressing which means are most properly suited for mitigating a viral threat, we must look into the evaluations to see how these conform to our decision problem and analysis purpose. Both method 1 and method 2 include identification of unwanted events. However, method 1 is more directed towards security issues. Also, both methods make use of predefined sources to identify hazards, but neither can be said to use diverse sources in doing so. Method 1 supports, to a certain extent at least, identification of causes, as opposed to method 2. Both methods support identification and evaluation of risk reducing measures, but have different approaches; method 1 follows a stringent procedure, while method 2 uses brainstorming and checklist. One could intuitively argue that method 1 would be the better choice, since it is more focused on security threats, and thus is more specifically related to the decision problem at hand. However, one could also argue that method 2 is better suited, as the identification and evaluation of risk reducing measures are of a less formal nature, which corresponds well to the company s available resources and competencies. This highlights an important point; the evaluations of specific methods do not support a mechanical, straight-forward selection process. They do however constitute part of a basis from which a well-informed selection can be made. It would neither be appropriate nor feasible to have normative guidelines imposed on the selection process, as this would 1378

completely ignore the diversity in decision problems and corresponding analysis purposes. Another point to note is that although the evaluations may highlight specific strengths or advantages of a method they may equally expose weaknesses or flaws of a method. While justifying the selection of a method (partly) on the basis of its strengths could be reasonable, ignoring its flaws would be equally unreasonable. Before a selection is made, consideration should therefore be taken concerning how flaws can be redeemed, and what is required to adapt the method to the specified purpose. 5 DISCUSSION AND CONCLUSIONS This paper presents and discusses an evaluation process that can be applied to risk and vulnerability analysis methods within the ICT sector. The idea is to carry out evaluations on a number of risk and vulnerability methods in order to describe their properties, and to categorize them as regards e.g. level of detail, point of focus etc. Then, when a specific risk and vulnerability analysis is being planned, or a threat is identified, the method evaluations can be used as background information for the analysis selection process. We conclude that the evaluation process can be useful since a high number of risk and vulnerability methods exist, and the suggested framework provides a systematic method characterization. The evaluation framework simplifies the selection process, since the user, by means of limited resources, can get an overview of which methods that exist, and their pros and cons. However, the framework does not support a mechanistic selection process, but rather constitutes a basis for making a well-informed selection: In most cases, it will still be difficult to make a simple selection of one method ahead of others, as different methods have different strengths and weaknesses. REFERENCES AS/NZS 4360, 2004. Australian/New Zealand Standard: Risk management. Aven, T. 2006. A unified framework for risk and vulnerability analysis covering both safety and security. Reliability Engineering & System Safety. To appear. ISO 2002. Risk management vocabulary. ISO/IEC Guide 73. Wiencke, H. S., Aven, T. & Hagen, J. 2006. A framework for selection of methodology for risk and vulnerability assessments of infrastructures depending on information and communication technology. Safety and Reliability for Managing Risk, 2297 2304. APPENDIX EVALUATION OF METHODS FOR RISK AND VULNERABILITY ANALYSIS Method: Method X Developed for: Enterprises with relations to information systems Released: 2005 (Latest software version), 2004 (Latest documentation version) Developer: Developer X Application: Assessment of risks related to information systems. Compatible with ISO 13335, ISO 15408 and ISO 17799. Reference projects: The user manual, best practices and software can be downloaded from www.homepageformethodx.com Properties of the method Level of detail Simplified Standard Detailed Phasescovered by themethod Establish con-text Risk- and vulnerability assessment Description Focus Accidents Intentional threats Attributes Security Holistic (security, environment, economy, reputa-tion, ) Analysis Coarse Detailed Type of ICT General Specific type (specify) Risk treatment Comments Focuson attack methods, threat agents, threats and vulnerabilities. Security related to confidentiality, integrity and availability. Detailed analysis in terms of amount and type of information required, but does not require use of special analysis techniques. Information system Brief general description of the method: Method X is a method developed by Developer X for assessing and treating risks related to information systems. It is used both within the public and private sector in numerous countries. The method is compatible with international standards such as ISO 13335, ISO 15408 and ISO 17799, and is to a certain extent based upon these standards. Method X is also compatible with other security tools for information systems, such as PSSI and TDBSSI. The method has a comprehensive approach, as it primarily is component-based, and attempts to identify and describe functions, information, attack methods, threat agents, threats, vulnerabilities, risks, security criteria, security objectives, generic measures, etc. within a bounded system area. A study of the organization constitutes part of the analysis. The extent of the analysis can be adjusted, from covering the entire organization to focusing on a certain branch within the organization. The method is conducted by using a software program, where required data from the analysis is entered. An important point to note is that although Method X is meant to cover all phases of the risk management process, it has a complex and rigid structure, and can therefore not be readily applied without first understanding its structure and procedure. Time and resources required should also be evaluated related to this. To use Method X, it is recommended that the following evaluation is used in conjunction with the method, so that limitations and shortcomings may be adjusted. 1379

Aven CH172.tex 17/5/2007 10: 56 Page 1380 1380

Aven CH172.tex 17/5/2007 10: 56 Page 1381 1381