Effective Techniques for Risk Measurement Steven Ross Executive Principal Risk Masters, Inc. Agenda The Failure of Current Techniques A Fresh Approach to Risk Measurement The Theory Behind the Techniques A Practical Example
The Failure of Current Techniques Measuring and Managing Risk It is axiomatic that If it can t be measured, it can t be managed And yet, standard techniques for continuity risk management do not address risk measurement Risk Management is an aspect of many disciplines Finance Insurance Military Enterprise Risk Management In Business Continuity Management we fall back on the simplistic classic formula
The Classic Formula Risk = Impact Probability, where Impact = Expected cost per incident Probability = Expected number of incidents/time (usually one year) The classic formula deals with exposure, not risk Annualized loss expectancy from predictable causes Risk is the measure of the uncertainty of loss Failure of the Classic Formula We do not know what the impact of rare events will be Therefore, we fall back on worst case But the worst case is only one of many outcomes of a given incident The rate of occurrence of catastrophic events is unknowable Thus differentiated probability is meaningless How many airplanes have ever flown into buildings? Tsunamis that killed hundreds of thousands? No matter the number of occurrences, it might happen today
Failure of the Classic Formula, continued Thus, Risk = Impact Probability is the product of the unknown and the unknowable! No wonder the classic formula fails As a result, we get useless risk assessments that tell us that Tornadoes are a risk in Kansas Ice storms are not a risk in Miami Unmitigated risks are the greatest of all Electromagnetic pulse Animal or insect infestation And yet all the major standards are based on the classic formula What the Standards Say BS 25999 Risk is an average effect by summing the combined effect of each possible consequence weighted by the associated likelihood of each consequence ISO 27005 Risk estimation [is the] process to assign values to the probability and consequences of a risk NFPA 1600 Risk assessment categorize[s] threats, hazards, or perils by both their relative frequency and severity
And Do Not Say None of them even mention risk measurement None address the underlying rationale for Multiplication of impact and probability Limiting risk to only impact and probability Towards a New Formula Risk = ʄ (impact, probability)
Towards a New Formula Risk = ʄ (impact, probability, credibility, resources, scale, duration, mean time to repair, mean time to recurrence ) And many other factors that Can be described but not quantified Attract differing viewpoints as to values and weighting A Fresh Approach to Risk Measurement
Overview of the Approach Measure the effect on critical resources, not the threats to them Categorize the impacts Scale the categories Determine the credibility of each level of risk Consider frequency of occurrence In each step, there are variables to consider for each risk being measured Availability of people, premises, information, networks, raw materials Destruction, inaccessibility, unavailability, unusability, incapacity Total loss, significant damage, moderate damage, minimal damage Credible or not credible Examples of risks that would not fit into the classic formula Office facilities VOIP telephony Loss of all Not credible that intact but not if the Internet is personnel vs. all personnel are accessible down loss of a few lost but credible people to lose some Often, occasionally, rare Bad weather often, terrorism rarely Technique #1 Focus on Resources Measure the effect on critical resources, not the threats to them The set of causes is infinite and unknowable The set of resources is finite and known, e.g., Working premises Human resources Data Equipment Information systems Voice and data networks Raw materials Thus the measurement of risk is the consequential effect of disruption of these resources
Technique #2 - Categorization Destruction (the resource no longer exists) Consider the smoking hole Inaccessibility (the resource exists but we cannot get to it) Consider offices on the fiftieth floor when the elevator does not work Unavailability (the resource exists but is rendered inoperable) Consider hacks that stop Internet web sites Unusability (the resource exists but it is malfunctioning) Consider a VOIP telephone systems if Internet connectivity is lost Incapacity (the resource exists and functions as expected, but not at a sufficient level) This usually occurs at a gradual pace, but consider a computer virus that slows a network to a crawl Technique #2 Categorization, continued There are other categories that might apply in specific circumstances Not all categories apply to all resources Unusable people?
Technique #3 - Scale Each of the impact categories might occur at different levels, e.g., Total loss (i.e., worst case) Most of the resource affected Some of the resource affected Unit damage Inconsequential effect Each of these presents its own distinct risk profile Technique #4 - Credibility Some risks exist but need not be taken seriously, in context If a risk is credible, then some response is required If only risk acceptance The test of credibility is entirely subjective, based on the perspective of the observer Multiple observers might provide a better measurement Fuzzy but correct sets of data points are better than precisely wrong ones
Technique #5 - Frequency Related to, but not the same as, probability Enables the distinction between high frequency-low impact and low frequency-high impact events Fuzzy terminology is helpful in distinguishing levels of risk, e.g. Routine Frequent Sometimes Rare Never The Theory Behind the Techniques
Risk is Not an Absolute Risk measurement depends on Who is doing the measuring What is at risk To what degree of accuracy Within which bounds Let s do an experiment! Accuracy and Precision The goal of risk measurement should be accuracy, forsaking precision Fuzzy mathematics enables this A methodology for systematically handling concepts that embody imprecision and vagueness
Fuzzy Sets and Systems The mathematics of fuzzy set theory was originated by L. A. Zadeh in 1965 Fuzziness describes objects or processes that are not amenable to precise definition or precise measurement Fuzzy systems Processes that are too complex to be modeled by using conventional mathematical methods Vaguely defined and have some uncertainty in their description The uncertainty and fuzziness arising from interrelated humanistic types of phenomena such as Subjectivity Thinking Reasoning Cognition Perception Fuzzy Sets and Systems Fuzziness in thinking and reasoning processes is an asset since it makes it possible to convey a large amount of information with a very few words Uncertainty characterized by structures that lack sharp (well-defined) boundaries A modeling link between the human reasoning process, which is vague, and computers, which accept only precise data
An Example of Fuzziness Conventionally, we might say that temperature is an absolute (e.g., 20 o, 30 o 100 o ) But we do not perceive temperature that way Rather as very cold, cold, moderate, hot, etc. The determination of the temperature is subjective, with varying degrees of certainty An Example of Fuzziness, continued Temp. in Farenheit Very cold Cold Cool Moderate Warm Hot Brain baking 20 0.850 0.150 0 0 0 0 0 30 0.275 0.450 0.275 0 0 0 0 40 0.100 0.400 0.400 0.100 0 0 0 50 0 0.300 0.500 0.300 0 0 0 60 0 0 0 0.300 0.500 0.200 0.000 70 0 0 0 0.100 0.800 0.100 0.000 80 0 0 0 0 0.400 0.500 0.100 90 0 0 0 0 0.100 0.700 0.200 100 0 0 0 0 0 0.400 0.600 110 0 0 0 0 0 0.100 0.900 Note that all the rows add up to 1 (or 100%)
An Example of Fuzziness, continued Similarly, we do not experience risk in a discrete manner In real life (e.g., buying property insurance) we consider Extent of loss (total loss, partial damage) Types of incidents (fire, flood, earthquake) Resources protected (jewelry, furs) We rarely consider probability An Example of Fuzziness, continued Risk Category Total Loss Most Some Individual Units Inconsequential Destruction 0.850 0.150 0 0 0 Inaccessibility 0.275 0.450 0.275 0 0 Unavailability 0.100 0.400 0.400 0.100 0.000 Unusability 0 0.300 0.500 0.300 0.000 Incapacity 0 0 0.200 0.300 0.500 We have introduced confidence in place of probability The high value points to the consensus measure of risk Values may be attributed by a single analyst or by a panel Important that the panel members have a roughly equivalent degree of expertise And that there is consistency in the definition of terms
A New Formula It is important that all values in a set add up to 1 Thereby assuring the completeness of the fuzzy set Risk = ʄ 0,1 destruction, inaccessibility, unavailability, unusability, incapacity Put another way, adding confidence to one fuzzy description of risk takes away from the others A Practical Example
The Context Company has a data center in an office park or campus The company Backs up data daily and stores if off-site Has a recovery hot site Has a diverse network What is the risk of an incident affecting its data center? Resources and Categories Resources Equipment Data Network People Risk Categories Destruction Fire Fire Examples of potential causes Fire Earthquake Inaccessibility Campus incident Not applicable Campus incident Campus incident Unavailability Hacking, virus Software failure CO failure Pandemic Unusability Maintenance error Maintenance error Internet failure Not applicable Incapacity Maintenance error Maintenance error Internet overload Strike Outside the data center Note that the examples are indications of credibility, not systematic statements of threats or hazards
Scaling and Credibility Risk Factors Risk Scales Resource Category Total Most Some Individual Units Inconsequential Equipment Destruction Credible Credible Credible Not credible Not credible Inaccessibility Credible Credible Credible Not credible Not credible Unavailability Credible Credible Credible Credible Not credible Unusability Credible Credible Credible Credible Not credible Incapacity Credible Credible Credible Credible Not credible Data Destruction Not credible Not credible Credible Credible Credible Inaccessibility Credible Not credible Credible Credible Not credible Unavailability Credible Credible Credible Credible Credible Network Destruction Not credible Not credible Not credible Credible Credible Inaccessibility Credible Credible Credible Credible Credible Unavailability Credible Credible Credible Credible Credible Unusability Credible Credible Credible Credible Not credible Incapacity Credible Credible Credible Credible Credible People Destruction Not credible Not credible Credible Credible Does not apply Inaccessibility Not credible Credible Credible Credible Does not apply Unavailability Credible Credible Credible Credible Credible Scaling and Credibility Risk Factors Risk Scales Resource Category Total Most Some Individual Units Inconsequential Not credible because backed Equipment Destruction Credible Credible Credible Not credible Not credible Inaccessibility Credible Credible Credible up data would Not credible not be totally Not credible lost Unavailability Credible Credible Credible Credible Not credible Unusability Credible Credible Credible Credible Not credible Incapacity Credible Credible Credible Credible Not credible Data Destruction Not credible Not credible Credible Credible Credible Inaccessibility Credible Not credible Credible Credible Not credible Unavailability Credible Credible Credible Credible Credible Network Destruction Not credible Not credible Not credible Credible Credible Inaccessibility Credible Credible Credible Credible Credible Unavailability Credible Credible Credible Credible Credible Unusability Credible Credible Credible Credible Not credible Incapacity Credible Credible Credible Credible Credible People Destruction Not credible Not credible Credible Credible Does not apply Inaccessibility Not credible Credible Credible Credible Does not apply Unavailability Credible Credible Credible Credible Credible
Scaling and Credibility Risk Factors Risk Scales Resource Category Total Most Some Individual Units Inconsequential Equipment Destruction Credible Credible Credible Not credible Not credible Inaccessibility Credible Credible Credible Not credible Not credible Unavailability Credible Credible Credible Credible Not credible Unusability Credible Credible Credible Credible Not credible Incapacity Credible Credible Credible Credible Not credible Data Destruction Not credible Not credible Credible Credible Credible Inaccessibility Credible Not credible Credible Credible Not credible Unavailability Credible Credible Credible Credible Credible Network Destruction Not credible Not credible Not credible Credible Credible Inaccessibility Credible Credible Credible Credible Credible Unavailability Credible Credible Credible Credible Unusability Credible Unusability Credibleand Incapacity Credible do Credible Not credible Incapacity Credible not Credible apply to people Credible Credible Credible People Destruction Not credible Not credible Credible Credible Does not apply Inaccessibility Not credible Credible Credible Credible Does not apply Unavailability Credible Credible Credible Credible Credible Scaling and Credibility Risk Factors Risk Scales Resource Category Total Most Some Individual Units Inconsequential Hard to see an inconsequential Equipment Destruction Credible Credible Credible Not credible Not credible Inaccessibility Credible impact Credible on equipment Credible Not credible Not credible Unavailability Credible Credible Credible Credible Not credible Unusability Credible Credible Credible Credible Not credible Incapacity Credible Credible Credible Credible Not credible Data Destruction Not credible Not credible Credible Credible Credible Inaccessibility Credible Not credible Credible Credible Not credible Unavailability Credible Credible Credible Credible Credible Network Destruction Not credible Not credible Not credible Credible Credible Inaccessibility Credible Credible Credible Credible Credible Unavailability Credible Credible Credible Credible Credible Unusability Credible Credible Credible Credible Not credible Incapacity Credible Credible Credible Credible Credible People Destruction Not credible Not credible Credible Credible Does not apply Inaccessibility Not credible Credible Credible Credible Does not apply Unavailability Credible Credible Credible Credible Credible
Frequency Risk Factors Risk Scales Resource Category Total Most Some Individual Units Inconsequential Equipment Destruction Rare Rare Infrequent Not credible Not credible Inaccessibility Rare Rare Rare Not credible Not credible Unavailability Rare Rare Infrequent Infrequent Not credible Unusability Rare Rare Sometimes Sometimes Not credible Incapacity Rare Rare Infrequent Sometimes Not credible Data Destruction Not credible Not credible Infrequent Frequent Frequent Inaccessibility Rare Not credible Sometimes Sometimes Not credible Unavailability Rare Rare Rare Frequent Frequent Network Destruction Not credible Not credible Not credible Rare Rare Inaccessibility Rare Rare Sometimes Frequent Frequent Unavailability Rare Rare Sometimes Sometimes Frequent Unusability Infrequent Infrequent Infrequent Sometimes Not credible Incapacity Rare Rare Infrequent Infrequent Frequent People Destruction Not credible Not credible Rare Infrequent Does not apply Inaccessibility Not credible Rare Infrequent Infrequent Does not apply Unavailability Rare Rare Sometimes Frequent Does not apply Frequency Risk Factors Risk Scales Resource Category Total Most Some Individual Units Inconsequential Equipment Destruction Rare Rare Infrequent Not credible Not credible Inaccessibility Rare Rare Rare Not credible Not credible Unavailability Rare Rare Infrequent Infrequent Not credible Unusability Rare Rare Sometimes Sometimes Not credible Incapacity Rare Rare Infrequent Sometimes Not credible Data Destruction Not credible Not credible Infrequent Frequent Frequent Inaccessibility Rare Not credible Sometimes Sometimes Not credible Unavailability Rare Rare Rare Frequent Frequent Network Destruction Not credible Risk Factors Not credible Not credible Risk RareScales Rare Inaccessibility Rare Sometimes Frequent Frequent Unavailability RareResource Category Rare Total Sometimes Most Sometimes Some Individual UnitsInconsequential Frequent Unusability Infrequent Infrequent Infrequent Sometimes Not credible Equipment Inaccessibility 0.600 0.300 0.100 0.000 0.000 Incapacity Rare Rare Infrequent Infrequent Frequent People Destruction Not credible Not credible Rare Infrequent Does not apply Inaccessibility Not credible Rare Infrequent Infrequent Does not apply Unavailability Rare Rare Sometimes Frequent Does not apply Not surprisingly, those risk factors with the greatest impact are those that are often the rarest
Frequency Risk Factors Risk Scales Resource Category Total Most Some Individual Units Inconsequential Equipment Destruction Rare Rare Infrequent Not credible Not credible Inaccessibility Rare Rare Rare Not credible Not credible Unavailability Rare Rare Infrequent Infrequent Not credible Unusability Rare Rare Sometimes Sometimes Not credible Incapacity Rare Rare Infrequent Sometimes Not credible Data Destruction Not credible Not credible Infrequent Frequent Frequent Inaccessibility Rare Not credible Sometimes Sometimes Not credible Unavailability Rare Rare Rare Frequent Frequent Network Destruction Not credible Not credible Not credible Rare Rare Inaccessibility Rare Rare Sometimes Frequent Frequent Unavailability Rare Rare Sometimes Sometimes Frequent Unusability Infrequent Infrequent Infrequent Sometimes Not credible Incapacity Rare Rare Infrequent Infrequent Frequent People Destruction Not credible Not credible Rare Infrequent Does not apply Inaccessibility Not credible Rare Infrequent Infrequent Does not apply Unavailability Rare Rare Sometimes Frequent Does not apply Not surprisingly, those risk factors with the greatest impact are those that are often the rarest Frequency Risk Factors Risk Scales Resource Category Total Most Some Individual Units Inconsequential Equipment Destruction Rare Rare Infrequent Not credible Not credible Inaccessibility Rare Rare Rare Not credible Not credible Unavailability Rare Rare Infrequent Infrequent Not credible Unusability Rare Rare Sometimes Sometimes Not credible Incapacity Rare Rare Infrequent Sometimes Not credible Data Destruction Not credible Risk Not Factors credible Infrequent Frequent Risk Scales Frequent Inaccessibility Rare Not credible Sometimes Sometimes Not credible Unavailability Rare Rare Rare Frequent Frequent Incapacity Does not People apply Does Unavailability not apply Does not 0.050 apply 0.150 Does not 0.500 apply Does 0.300 not apply Network Destruction Not credible Not credible Not credible Rare Rare Inaccessibility Rare Rare Sometimes Frequent Frequent Unavailability Rare Rare Sometimes Sometimes Frequent Unusability Infrequent Infrequent Infrequent Sometimes Not credible Incapacity Rare Rare Infrequent Infrequent Frequent People Destruction Not credible Not credible Rare Infrequent Does not apply Inaccessibility Not credible Rare Infrequent Infrequent Does not apply Unavailability Rare Rare Sometimes Frequent Does not apply Resource Category Total Most Some Individual Units Inconsequential But some less impactful events may pose the greatest overall risk
Frequency Risk Factors Risk Scales Resource Category Total Most Some Individual Units Inconsequential Equipment Destruction Rare Rare Infrequent Not credible Not credible Inaccessibility Rare Rare Rare Not credible Not credible Unavailability Rare Rare Infrequent Infrequent Not credible Unusability Rare Rare Sometimes Sometimes Not credible Incapacity Rare Rare Infrequent Sometimes Not credible Data Destruction Not credible Not credible Infrequent Frequent Frequent Inaccessibility Rare Not credible Sometimes Sometimes Not credible Unavailability Rare Rare Rare Frequent Frequent Network Destruction Not credible Not credible Not credible Rare Rare Inaccessibility Rare Rare Sometimes Frequent Frequent Unavailability Rare Rare Sometimes Sometimes Frequent Unusability Infrequent Infrequent Infrequent Sometimes Not credible Incapacity Rare Rare Infrequent Infrequent Frequent People Destruction Not credible Not credible Rare Infrequent Does not apply Inaccessibility Not credible Rare Infrequent Infrequent Does not apply Unavailability Rare Rare Sometimes Frequent Does not apply But some less impactful events may pose the greatest overall risk Impact and Ranking Risk Factors Impact Resource Category Catastrophic Significant Some Minor Inconsequential Equipment Destruction Inaccessibility Unavailability Unusability Incapacity Data Destruction Inaccessibility Unavailability Network Destruction Inaccessibility Unavailability Unusability Incapacity People Destruction Inaccessibility Unavailability This is an example of a risk assessment derived from fuzzy risk measurement What does this tell us? Insufficient recoverability established for Company s data center and equipment After that, their worst cases are inaccessibility and unavailability, not destruction of other resources
Thank You What you don t know is far more important than what you do know Nassim Nicholas Taleb, The Black Swan