What do you think? Best Practice Control Effectiveness

What do you think? Best Practice Control Effectiveness 1

Content Why focus on controls? How do we consider the effectiveness of controls - Bowtie Analysis (BTA) + Control Effectiveness Analysis* Where to from here?

A B C D E 1 1 2 4 7 11 2 3 5 8 12 16 3 6 9 13 17 20 4 10 14 18 21 23 5 15 19 22 24 25 3 3

What do you think? The Risk Matrix does it help in Task Planning? Consequence Likelihood 5 Almost Certain 4 Likely 3 Possible 2 Unlikely 1 Rare 1 Minor Medium (19) Medium (20) Low (23) Low (24) Low (25) 2 Low Significant (10) Medium (16) Medium (18) Low (21) Event Risk Rating 3 Medium Significant (6) Significant (9) Significant (13) Medium (15) 4 High High (3) High (5) Not suited to job at hand Task Planning RA Low (22) Medium (17) Significant (8) Significant (12) Medium (14) 5 Major High (1) High (2) High (4) Significant (7) Does not add value to the discussion Significant (11) OK for finding higher level priorities 4 4

Swiss Cheese Model Risk Management Controls 5

The Hierarchy of Controls: Most effective ELIMINATION SUBSTITUTION ENGINEERING More people dependent ADMINISTRATIVE PPE Least effective 6

What makes a control effective? Design Hierarchy of Control AND? PRESENT - at the right location AVAILABLE - applies as planned (automatically / manually*) WORKS - works as planned / required (maintained / calibrated / checked) SURVIVES - not compromised (independent / survivable) (* = People dependent) 7

What do you think? Low Risk CONTROLS High Risk Risk is determined by the effectiveness of controls 8

Communication and consultation Establish the context Control decisions Risk Assessment Risk Identification Risk analysis Risk evaluation Monitoring and review Control Checking Control placement Risk treatment 9

ACARP C23007: SELECTION AND OPTIMISATION OF RISK CONTROLS ACARP C23007 Slide 11

Motivation: The top factors for incidents are people not properly identifying risks, controls not being in place, or the controls not being effectively implemented or maintained. (ICMM 2013) ACARP C23007 Slide 12

Communication and consultation ISO31000 Establish the context Identify relevant unwanted events prospectively & retrospectively Risk Assessment Risk Identification Risk analysis Risk evaluation Risk treatment We need enhanced focus of best practices for risk treatment Monitoring and review Select best risk treatment option for each unwanted events If best risk treatment involves installing risk controls on site Identify optimum controls to achieve required risk reduction using bowtie analysis Select method(s) for measuring operational effectiveness of controls Implement then measure control Slide 13 effectiveness and monitor control assurance management systems ACARP C23007

Objective: If best risk treatment involves installing risk controls on site Identify optimum controls to achieve required risk reduction using quality bowtie analysis Getting the controls right for an unwanted event a. Describe unwanted event for the bowtie knot b. Determine scope of analysis c. Identify threats that could cause event d. Identify possible consequences of event e. Select optimum set of controls to manage causes and consequences of the event f. Identify failure modes for important controls g. Determine items for control assurance mgt

Key steps: Identify optimum controls to achieve required risk reduction using quality bowtie analysis a. Describe unwanted event for the bowtie knot b. Determine scope of analysis c. Identify threats that could cause event d. Identify possible consequences of event e. Select optimum set of controls to manage causes and consequences of the event f. Identify failure modes for important controls g. Determine items for control assurance mgt ARP C23007

Identify optimum controls to achieve required risk reduction using quality bowtie analysis a. Describe unwanted event for the bowtie knot b. Determine scope of analysis What do you think? ACARP C23007 Slide 16

Output: Basic bowtie diagram Hazard c. Identify threats that could cause event d. Identify possible consequences of event e. Select optimum set of controls to manage causes and consequences of the event f. Identify failure modes for important controls Slide 17

Legend: Good Caution Action required Control not considered 18

e. Select optimum set of controls to manage causes and consequences of the event Controls are: Acts a description of what a person should do Objects a device that works without an act(s), Systems combination of act(s) and object(s)

Specifiable Measurable Auditable

What do you think? A control for mitigating child pedestrian fatalities outside schools. The control is drivers driving vehicles at a speed that is specified as 40km/hr or less which is measureable via speed cameras and auditable with speed versus fatality data. ACARP C23007 Slide 21

Start Is it, of itself a physical object, technological system, and/or human action? No Not a control Yes Does it, of itself, arrest or mitigate an unwanted event sequence? No Yes Is the required performance specifiable, measurable, and auditable? No Yes A CONTROL Slide 22

e. Select optimum set of controls to manage causes and consequences of the event Controls that minimise exposure Controls that detect and deflect threats Last chance intervention controls Protection controls Isolation/ containment controls Recovery/. Restoration controls Hazard Control Control Control Control Control Control Control Control Control Decreasing intervention time for ARRESTING CONTROLS Increasing itervention time for MITIGATION CONTROLS ACARP C23007 Slide 23

Output: Advanced bow tie diagram Hazard Control Failure Mode identification (Defeating Factors)

e. Select optimum set of controls to manage causes and consequences of the event Control Effectiveness Legend: Good Caution Action required Control not considered 25

Control Type e. Select optimum set of controls to manage causes and consequences of the event BASIC Control Effectiveness Rating 6 Eliminate A = or > 90% Control Quality B 60 90% C 30 60% D < 30% It is available and effective? Consider Hierarchy of Control 5 Substitute / Minimise 4 Engineering 3 Separate Legend: Good effectiveness Satisfactory but improvable Inadequate - action required 2 Administrative / Procedural 1 PPE 26

Determine quality of control with matrix on the right. Then use control quality and your assessment of control impact to determine adequacy of control with matrix below

OBJECT ACT SYSTEM

Determine quality of control with matrix on the right. Then use control quality and your assessment of control impact to determine adequacy of control with matrix below What do you think?

Key steps: Identify optimum controls to achieve required risk reduction using quality bowtie analysis a. Describe unwanted event for the bowtie knot b. Determine scope of analysis c. Identify threats that could cause event d. Identify possible consequences of event e. Select optimum set of controls to manage causes and consequences of the event f. Identify failure modes for important controls g. Determine items for control assurance mgt ARP C23007

Output: Advanced bow tie diagram Hazard Monitoring, maintaining and improving controls CONTROL ASSURANCE MANAGEMENT SYSTEM (CAMS) Operations activities Maintenance activities Engineering activities Management activities CAMS: Activities that ensure people and equipment are ready and able to perform the control activities as required when required (i.e. activities that ensure controls sustain effectiveness over time)

Focus areas for developing quality bowties Think about facilitation (someone who understands RM and bowtie fundamentals and someone who understands the context) Think about who to involve in process Design output to suit end user. Use a standardise description of unwanted event Have a clear definition of control Include assessments of control design quality, control assurance management system requirements and overall adequacy of control regime Think about how you would measure effectiveness ACARP C23007 Slide 33

How many controls are enough? 35

More on Measuring Control Effectiveness: Measuring control effectiveness should be done for important controls to allow for the tracking and actioning of control performance over time. ACARP C23007 - Summary Slide 36

More on Measuring Control Effectiveness: QUANTATIVE ANALYSIS: Uses actual data to determine the 3 components of control effectiveness: The ability of the control to function as required. The availability and use of the control when required. The extent to which the control a) minimises exposure, detects/deflects or intervenes to prevent threats becoming unwanted events or b) mitigates the severity of the consequence. For example the effectiveness of alcohol testing = ACARP C23007 Slide 37

Measuring Control Effectiveness: TREE (SEMI-QUANTATIVE) ANALYSIS:

Measuring Control Effectiveness: Proximity detection systems (humans respond to alarms) ILLUSTRATION DATA ONLY Numbers have not be derived from real data Verification data Q1. Proximity detection system reliably 0.25 N Y 0.75 picks up objects as required when required? Q2. Operator competent to respond to 0.01 N Y 0.99 proximity detection system feedback? Q3. Operator aware and does respond 0.40 N Y 0.60 correctly to proximity detection feedback? Q4. When object in proximity detected 0.00 N Y 1.00 and correct actions taken were they effective in avoiding accident? % fleet fitted x routine reliability test information Driver training/competency testing, driver interviews Vehicle data reports, incident reports, Analysis of incidents reports and vehicle data records ESTIMATED EFFECTIVENESS OF CONTROL 45% ACARP C23007 Slide 39

Measuring Control Effectiveness: CATEGORISATION (QUALITATIVE) ANALYSIS: Based on peoples judgements Examples of categories used to assess control effectiveness Three category control effectiveness scale Good effectiveness Satisfactory but could be improved Four category control effectiveness scale High Effectiveness Satisfactory Effectiveness Five category control effectiveness scale Excessive too many controls [that adversely effectiveness] Acceptable no action Inadequate action required Unsatisfactory Acceptable but current controls could be enhanced Replace Not acceptable refinement to current controls needed Inadequate more/better controls needed Slide 40

Control Type BASIC Control Effectiveness Rating 6 Eliminate A = or > 90% Control Quality B 60 90% C 30 60% D < 30% It is available and effective? Consider Hierarchy of Control 5 Substitute / Minimise 4 Engineering 3 Separate Legend: Good effectiveness Satisfactory but improvable Inadequate - action required 2 Administrative / Procedural 1 PPE 41

Conclusions: What do you think? Improvement of risk controls and their effectiveness will positively impact the safety, operational efficiency and cost performance Improvements will only be achieved if systems, tools and people drive a shift in mindset to the importance of managing controls. Potential impacts and improvements include the following: Development of quality of bowtie analysis and quality control assurance management systems. Further work on effectiveness and dependency analysis for control sets Development of knowledge management systems to help quantify control effectiveness. Utilise and build on the knowledge in RISKGATE. Involve regulators, EMESRT etc. ACARP C23007 Slide 42