Safety Risk Impact Analysis of an ATC Runway Incursion Alert System. Sybert Stroeve, Henk Blom, Bert Bakker

Transcription

1 Safety Risk Impact Analysis of an ATC Runway Incursion Alert System Sybert Stroeve, Henk Blom, Bert Bakker EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October 2006

2 Contents Motivation Example application Systemic approach Risk results Conclusions EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

3 Runway incursion: Recognised as important air traffic safety issue EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

4 Complexity of aerodrome operations Complexity of accident risk assessment Complexity of operations Many agents (humans/systems) Many interactions Highly dynamic Performance deviations Complexity of risk assessment Multiple agents Dependencies between agents Dynamics of agents Nominal/non-nominal conditions EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

5 Three types of accident models (Hollnagel, 2004) 1. Sequential accident models Accident = sequence of events e.g. fault trees, event trees, domino theory Causes Pivotal HAZARD Event FTA S F ETA S F S Effect A Effect B Effect C F Effect D Consequences 2. Epidemiological accident models Accident = like spreading of disease (latent/environmental conditions, barriers) e.g. Reason s Swiss cheese model, Bayesian belief networks EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

6 Three types of accident models (Hollnagel, 2004) 3. Systemic accident models Accident = emergent from variability of a complex system e.g. control theory, chaos theory, stochastic resonance Compared to sequential / epidemiological accident models: No fixed cause-effect relations Dynamic / non-linear behaviour Performance beyond event probability Complex multi-agent interactions EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

7 Contents Motivation Example application Systemic approach Risk results Conclusions EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

8 Active runway crossing operation: Effectiveness of ATC runway incursion alerting? Human operators Pilots take-off aircraft Pilots taxiing aircraft Runway controller Ground controllers Technical systems VHF R/T communication Active stopbar Runway incursion alert Ground radar Procedures Crossing clearance by runway controller Stopbar switching Read-back EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

9 Contents Motivation Example application Systemic approach Risk results Conclusions EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

10 Safety risk assessment cycle 1 Determine operation 2 Identify hazards 3 Construct scenarios 0 Operational development Identify objective Decision making 7 Identify safety bottlenecks Iterate (option) Assess risk 6 tolerability 4 Assess 5 frequency Assess severity EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

11 Risk assessment by combination of two models: Monte Carlo Simulation + Bias & Uncertainty EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

12 Monte Carlo simulation model of multi-agent runway incursion scenario Key aspects of agents, e.g. SA / task performance of operator Flight phase / aircraft performance Modes within key aspects, e.g. Task: monitoring / alert reaction Flight phase: taxi / take-off Dynamics within modes, e.g. Task performance time Take-off acceleration profile Interactions Between modes Between key aspects of an agent Between agents EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

13 Parameter values in MC simulation model Types Technical systems, e.g. accuracy, availability, update rate, aircraft thrust Human performance, e.g. task duration, decision parameter, likelihood of misunderstanding Context, e.g. taxiway layout, visibility Sources Technical system specifications Human factors literature Incident databases Interviews with operational experts Measurement data of real operations Measurement data of real-time simulations Simulation results from other relevant models EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

14 Performing Monte Carlo simulation Model implementation in dedicated Delphi / Java software MC simulation speed-up by risk decomposition MC simulation of conditional collision risks given an event, e.g. R/T system not functioning Alert system not functioning Pilots taxiing aircraft are lost Visibility condition Assess event probability Combine conditional risks and event probabilities MC simulation: about 10 5 to 10 7 simulations per condition Results Conditional collision risks at various aggregation levels Overall collision risk EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

15 Bias and uncertainty assessment Types of differences between simulation model & reality Numerical approximations Parameter values Formal model structure Non-covered hazards Operational concept differences Assessment steps Identify differences between simulation model and reality Assess size of each difference Assess risk sensitivity for parameter values Assess effect of each difference on the risk Combine the joint effect of the differences on the risk EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

16 Contents Motivation Example application Systemic approach Risk results Conclusions EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

17 Monte Carlo simulation results Conditional collision risk (per take-off) SA PF taxiing aircraft Proceed taxiway Without RIAS With RIAS Cross runway Proceed taxiway Cross runway Visibility Unrestricted m EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

18 Bias and uncertainty assessment: Effects of model-reality differences (examples) Significant effects (>30%) Type of manoeuvre of taking-off aircraft to avoid collision Conflict decision process by pilots of taking-off aircraft Speed of taxiing aircraft Monitoring frequency by pilots of taxiing aircraft Deceleration of taking-off and taxiing aircraft Time before braking is initiated by pilots of taking-off aircraft Small effects (<13%) Acceleration profile during the take-off run Performance of R/T communication systems Performance of surveillance systems Performance of runway incursion alert system Task scheduling of runway controller EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

19 Monte Carlo simulation + bias & uncertainty results 10-2 Without RIAS With RIAS Conditional collision risk (per take-off) SA PF taxiing aircraft 10-6 Proceed taxiway Cross runway Proceed taxiway Cross runway Visibility Unrestricted m EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

20 Contents Motivation Example application Systemic approach Risk results Conclusions EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

21 Conclusions A wide scope safety assessment (including performance of relevant human operators) is needed to evaluate the effectiveness of a runway incursion alert system Systemic accident models can effectively analyse the dependent dynamics of multiple agents in aerodrome operations (which is difficult by other model types) The MC simulations indicate that the effectiveness of ATC runway incursion alerting is small in good visibility, but significant in reduced visibility conditions Bias and uncertainty assessment supports informed decision making by addressing specific aspects of aerodrome operations at a particular airport EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

22 Discussion EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

23 Step 0: Identify objective Close co-operation with decision makers Aim: safety risk assessment for decision support of implementation redevelopment certification Safety context What are the safety criteria, target levels of safety? Scope Boundaries of the operation? Absolute or relative information? What types of risks? EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

24 Step 1: Determine operation Goal Understanding of operational concept by safety assessors Freeze operational concept during assessment cycle Check for holes and inconsistencies (should be repaired by concept developers) Input Description of the operation from concept developers Output Concise, structured, consistent operational concept human operators technical systems procedures environment EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

25 Step 2: Hazard identification brainstorm Shifting the boundary between imaginable and unimaginable hazards Open-minded and experienced operational experts Pure brainstorming No analysis / solutions / mitigation open atmosphere: promotes creativity of participants seemingly unimportant hazards trigger more relevant ones analysis of one hazard may take too much time hazards outside scope are removed during later analysis EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

26 Step 3: Construct scenarios Event n Condition Hazardous situation Event m Cluster B Conflict Cluster J, ATCo resolution Cluster K, Pilot resolution Hazards' combined effects EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

27 Step 4: Identify severities How severe can the consequences of a scenario be? consequences and their severities often depend on conditions, geometry and resolution usually a spectrum of severities applies Example severity classes Minor, Major, Hazardous and Catastrophic Severity assessment usually performed by safety experts consultation of and review by operational experts EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

28 Step 5: Assess frequency Assess frequency of each possible severity per scenario First assessment cycle Interviews with operational experts Incident/accident databases Optional subsequent cycle Monte Carlo simulation EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

29 Step 6: Assess risk tolerability For each conflict scenario indicate identified severity/ frequency combinations determine associated risk tolerability classification Example Severity Frequency Catastrophic Hazardous Major Minor Probable Unacceptable Unacceptable Unacceptable Tolerable Remote Unacceptable Unacceptable Tolerable Extremely remote Unacceptable Tolerable Extremely improbable Tolerable EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

30 Step 7: Identify safety bottlenecks In case of (possibly) unacceptable risk identify which hazards/conditions contribute significantly to the large risk Bottlenecks give operational developers a clue where they might improve EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

31 Contextual Control Mode Model (Hollnagel, 1993) degree of control strategic tactical scrambled opportunistic subjectively available time EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

32 Uncertainty assessment matrix Risk uncertainty Major Parameter value uncertainty Considerable Significant Minor Small Major Major Major Major Considerable Significant Minor Risk sensitivity Considerable Significant Minor Small Major Major Considerable Significant Minor Major Considerable Significant Minor Small Considerable Significant Minor Small Significant Minor Small Minor Small Small EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October

33 Bias assessment matrix Risk bias Probability assumption does not apply Typical Regular Frequent Less frequent Infrequent Unlikely Bias due to non-applicability Major Considerable Significant Minor Small Major Considerable Significant Minor Small Considerable Significant Minor Small Significant Minor Small Minor Small Small EUROCONTROL Safety R&D Seminar, Barcelona, Spain, October