Sensitivity of an Environmental Risk Ranking System SUMMARY Robert B. Hutchison and Howard H. Witt ANSTO Safety and Reliability CERES is a simple PC tool to rank environmental risks and to assess the cost-benefit of proposals. It is based on a methodology which effectively combines environmental science, engineering and site operations expertise. The operations at each site are examined to generate a list of postulated incidents and associated estimates of the incident likelihood and the quantity of material released from the site. For each material-site combination, transfer effectiveness estimates are assigned based on the pathway to the receptors. Incident risks are ranked using these estimates along with ratings for the impact of each hazardous material; and ratings for the ecological sensitivity of each site. Initial estimates are based on subjective assessments which are refined by additional data when available. Estimation effort needs to be directed to refining those estimates which have the greatest impact on the results. The ranking method is described and approaches to determine the sensitivity of the results to individual estimates are examined. INTRODUCTION Many organisations have a number of sites with various surrounding land types and use a wide variety of environmentally hazardous materials. With an intention to reduce the environmental risk of possible accidental releases from their operations, the organisation needs to be able to assess which site and which particular improvement options should be addressed first. Management has a duty to spend the environmental improvement budget in an efficient and cost effective fashion. Naturally, sufficient funds need to be provided to comply with legal requirements but many organisations desire to further reduce environmental risks. Once the scenarios for environmental damage have been ranked, it is appropriate to assess how precise or vague are the rankings. Even within the bounds determined by the choice of value judgements, the particular scores assigned to each scenario will have a degree of uncertainty. Knowledge of the size of this uncertainty will enable the decision makers to understand the difference between scores of different scenarios. In addition, knowledge of the source of the uncertainty will enable effort to be spent in the area which will reduce the overall uncertainty of the results. The model described herein has been effectively utilised in a number of Australian organisations to rank their exposure to environmental risk from operations at a number of sites (Hutchison, et al. 1996). It is the intent of this study to investigate uncertainties in the model so that the best possible results are obtained with the minimum amount of effort spent on determining factors that do not significantly affect the result. MODEL DESCRIPTION The overall method is shown in Figure 1 which sets out the development of the various estimates. The differing expertise of personnel within the organisation is utilised in estimation of different factors incorporated into the technique. The expertise of environmental specialists is used to produce hazard ratings for the different chemicals used on the sites and site sensitivity ratings, while site based personnel s knowledge is used in the estimation of accident scenario size and frequency. Incident records are linked to the material and site data files. Incident consequence scores are computed using the material and site ratings from these files and incident specific data. The incident consequence score and an estimate of the incident frequency are combined to give the incident risk score.
Codes are defined later in the paper Figure 1 Model for estimating environmental risk The model presented here is congruent with AS/NZS 4360 (Hutchison, Perera & Witt 1996) and uses a technical panel to produce the scores used for ranking the accident scenarios. The reasoning for assessments and the data used in the assessment is documented to allow for revision and checking. There are three consequence scores that are developed, one for human health effects (CSh), one for the local environment (CSe) and one for the global environment (global warming and ozone depletion - CSg). We have chosen to explicitly assign the weightings for each of these consequence scores and have combined these scores by addition. The explicit assignment of weightings enables the degree of anthropogenic orientation to be documented. The addition of consequence scores results in a single total consequence score (CS) which is used for consequence ranking of the scenarios. Equations Used The quantity released is an estimate of the amount of material released from the site per incident, taking into account the effectiveness of the mitigation systems. Quantity released (QR) = Quantity spilt x fraction not contained by mitigation systems The algorithms for the consequence scores (CS) are: Human: CSh Ecological: CSe Global: CSg = QR x HRh x SSh x TRh = QR x HRe x SSe x TRe = QR x HRg where HR = material hazard rating; SS = site sensitivity rating; TR = transfer route score. The total incident consequence score is the sum of the consequence scores for each of the three types of effects. This total score, multiplied by the estimated incident frequency (F) yields the incident risk score (RS) which is used to rank the environmental risk of the incidents. CS RS = CSh + CSe + CSg = CS x F A fictitious example could be a site using aromatic solvents stored in a bunded tank located next to a stream leading to wetlands. Details of the calculations are given in Table 1. Table 1: Sample calculations for fictitious environmental risk
Quantity spilled from tank Effectiveness of bund in containing spill Quantity Released Likelihood of spill 10 000L 90% 1 000L 0.01 p.a. human local environment global environment 0.40 0.70 0.10 Aromatic Solvents hazard ratings Site environs sensitivity 0.30 0.60 N.A. Transfer routes efficiency 0.20 0.90 N.A. Consequence Scores 0.024 0.378 0.1 Total Consequence Score 0.502 Risk Score 0.005 Technique Application The technique developed builds on lists of hazardous substances and operating practices at each site in the organisation. This could be generated during environmental audits, other site visit reports or by site staff. Using an experienced analyst, the site knowledge of local staff and various identification techniques such as a Rapid Ranking Study or brainstorming, a list of potential incidents is generated. An incident record is created for each postulated release at each location in which a hazardous material is stored or used. The likely quantity to be released is calculated from estimates of the quantity spilt and estimates of the effectiveness of mitigation systems. If historical data are not available, the frequency (or likelihood) of the event is based on industry experience or generic failure data. This work is best done by a team, or drafted by one individual and reviewed by a multidiciplinary team. The approach can be similar to the Rapid Ranking approach advocated by Tweeddale, Cameron & Sylvester (1992), with the team comprising a vertical slice of the organisation, including persons with shop floor knowledge of operating practice. In large operations the team membership would change as different activities are covered. Expertise of environmental scientists and engineers is utilised to derive hazard and site sensitivity ratings. The hazard rating for effects on humans (HRh) is derived by summing weighted scores for the toxicity, fire risk, aesthetic impact and public perception. The environmental hazard rating (HRe) for the chemicals is derived from similar consideration of the toxicity to flora and fauna, biodegradability and the propensity to affect habitats through fire, visual impact, degradation of the soil, etc. As ecotoxicity develops the methods for estimating the environmental hazard score may become less dependent on judgements by environmental experts. The global hazard rating (HRg) is similarly derived from the materials ozone depletion capacity, greenhouse contribution and bioaccumulation. Bioaccumulation is included in the global hazard rating as it will affect flora and fauna for a long time and is very likely to extend for a considerable distance around the site. SENSITIVITY ANALYSIS Elliott and Horowitz (1994) advise that ranges of risk, rather than single-value conservative risk estimates should be communicated to legitimise risk analysis whereas Garetz (1993) advocates the use of single point estimates in order to simplify the analysis and reduce the disputation on what the results mean. In this system we have chosen to use best estimate single point scores to simplify the results. Additional analysis of the results is suggested to determine the factors that have the most uncertainty. This would be the starting point for any additional refinement of the scores. In a risk score calculated by the product of many factors, it is intuitively obvious, that the only parameters that contribute significantly to the uncertainty of the final risk are those with the largest errors (Seiler and Alvarez 1996). Seiler and Alvarez also point out that the most uncertain parameters are the ones which have the most uncertain probability distribution and standard errors. This leads to the necessity of spending effort attempting to refine the estimates of those parameters that are intrinsically difficult to quantify, whereas refining estimates of more easily assessed parameters does not improve the overall result. This system calculates the risk score from multiplication of the primary factors, QS, (1-ME), F and CS. Thus for the best accuracy with the least effort, each of the values assigned to these factors should have the same level of confidence. In other words, the standard errors of the distributions of real scores
about the estimated values should be similar for each of the factors. This will enable the final risk score, RS, to have the lowest spread of values about the calculated value based on the estimated factors. Through analysis of information used in implementations of this system, estimates of the precision of the parameters can be made. Three different techniques are utilised. The first technique uses the discrimination between parameter values for different model scenarios. Where only three different parameter values are entered for 200 scenarios, it can be concluded that the degree of uncertainty for this parameter is large. The second technique uses the precision of the parameter estimates. This would ascribe a higher level of precision to a parameter given a score of 2.34 than a parameter given a score of 6. The third technique requires judgement of experienced risk analysts to estimate the likely error in the different parameters. This third technique permits assessment of uncertainties due to both lack of knowledge and lack of precision in the parameter value estimates. In all analyses of this nature, the possibility of correlations between input data must be incorporated in the results. In this model a significant degree of correlation is likely to be present between the Hazard Ratings for humans, the local environment and the global environment. ie if the hazard to people is underestimated it is likely that the hazard to the environment would also be underestimated. It is important to note that these correlations only exist in the calculations determining the consequence score. RESULTS One large implementation of this system was analysed as part of this study. The table of estimated accuracy of parameter values (Table 2) shows that the confidence which could be read into the parameter values entered by either discrimination or numerical precision was much higher than the confidence in the values given by risk analysts. Table 2: Estimated precision of parameter values used in an implementation of the environmental risk ranking system. The values are the factors by which the estimated values are multiplied or divided to be confident that the real value will lie within that range. Parameter Value Discrimination Value Precision Estimated by Risk Analysts Quantity Spilt (QS) 1.01 1.01 3 1 - Mitigating Effectiveness (1-ME) 1.4 1.1 3 Likelihood (F) 1.1 1.01 10 Hazard Rating Human 1.1 1.001 2 Hazard Rating Environment 1.1 1.001 5 Hazard Rating Global 1.1 1.01 3 Transfer Route Human 1.5 1.2 2 Transfer Route Environment 1.5 1.2 3 Site Sensitivity Human 1.03 1.03 2 Site Sensitivity Environment 1.3 1.1 3 Consequence Score 5 The likelihood of scenarios is seen to have the highest uncertainty with a factor of 10 being the estimated range of values about the estimated value. This suggests that significant effort be applied to improve the values used in this area. However, the final scenario scores are likely to be biased in the same direction, with all consequences underestimated or overestimated, etc. This systematic error would result in little error to the ranking of the risks. As the ranking of the risks is the most important aspect of the system, the accuracy of the scores is of less importance than the ranking of scenarios. However, this does not lessen the importance of being diligent in assigning best estimates to all the scores as rankings are also possible based on the consequence score, the frequency or the quantity of material lost to the environment. It can be argued that even though the real results have the uncertainty ranges shown, the ranking of the results does not change due to similar assumptions and processes being involved in the determination of both values. Alternatively, if different people have been involved in the determination of the values, completely different assumptions may have been used and the uncertainty ranges may reflect the uncertainty in the relative values as well as the real values. It is important to have at least
one person who is involved in all the assessments so that views of different personnel can be moderated to reduce the variability. Figure 2: Discrimination of parameter values The choices made by the team in assigning scores to the various factors in one implementation shows the confidence that they have in the values or the confidence they have in the relative rankings between scenarios. The transfer route human values have all been chosen to be either 1%, 10%, 20%, 40% or 60% (see Figure 2). This suggests that the team either had very little information to base their decisions on or could only perceive five distinct classes of values due to the inherent variability of this factor. This could be due to difficulty in reconciling the different factors which comprise the transfer routes to local human populations. Care must be taken to ensure that the users are aware of the limitations of the system. Even though the use of a single team to produce the estimates will result in rankings that have validity, the uncertainty in the numerical results remain. Application of the results of the ranking system must be made in awareness of the basis for ranking and the explicit value judgements which are incorporated. In the method given here, only the consequence score is broken down into other non-multiplicative factors. The consequence score is the sum of three factors of consequence scores for each of human, environmental and global effects. The basis for this breakdown in factors is to enable the value judgements and the assumptions to be documented and explicitly considered. This is particularly relevant as the consequence score is the one factor in this system which is quite different to the factors currently used in risk assessments and ranking methods for risks to people and property. The 200 scenarios were represented by four scenarios equally spread by risk score. Figure 3 shows the possible variation in risk scores using the estimated precision factors of the risk analysts given in Table 2. This shows that even with the very low precision estimates of the scores and assuming complete independence of the scores for the scenarios, the scenarios can be separated into four categories with high levels of confidence. As the estimates of the scenario values are not likely to be independent but will vary in line with each other, the separation into separate classes of risk score is likely to be even greater.
Figure 3: Sensitivity of scenario risk scores assuming independence of factor scoring CONCLUSIONS The use of a consequence score broken down into separate factors has enabled value judgements, assumptions and lack of knowledge to be faced explicitly. Despite the difficulties which the value judgements and lack of knowledge pose, a numerical score has been able to be assigned and the documentation of reasoning allows the score to be defended. Even incorporating high levels of uncertainty and assuming independence of scores, the risk scores can be separated into separate groups. Although the scores may not be significantly different from one score to the next one, they are different from one group to the next. This work shows that it is possible to analyse the results from a preliminary environmental risk ranking study to enable efforts to refine the results to be directed to those areas of the analysis which are less precise. The approach used for this environmental risk ranking system has been demonstrated through use to be efficient and effective. Decisions on the allocation of available funds for environmental risk reduction can be made on a systematic scientific/engineering basis. It also enables operators to improve their understanding of the environmental issues associated with the operations. REFERENCES AS/NZS. Australian and New Zealand Standard 4360:1995. Risk Management. Elliott E D and Horowitz A B. 1994. Risk-based environmental priorities: What priority? Water Resources Regulations. p24-31. Garetz W V. 1993. Current concerns regarding the implementation of risk-based management: How real are they? Comparative Environmental Risk Assessment. Ed. Cothern C R. Hutchison R B, Perera J, Witt H H, Brodie M H, Ross K J and Elliot G I. 1996. Application of an environmental risk ranking system. paper in preparation. Hutchison R B, Perera J, Witt H H. 1996. Preliminary Environmental Risk Ranking. First Annual Risk Engineering Seminar, 11 April 1996. The Munro Centre for Civil and Environmental Engineering, The University of New South Wales. Seiler F A and Alvarez J L. 1996. On the selection of distributions for stochastic variables. Risk Analysis. Vol. 16 no. 1 pp5-18. Tweeddale H M, Cameron R F & Sylvester S. 1992. Some Experiences in Hazard Identification and Risk Shortlisting. J. of Loss Prev. Process Ind. vol. 5(5) p279-88