VisionMonitor PSPMS an airline operational data model for a predictive safety performance management system P. Ulfvengren, J. Rignér Industrial Management, KTH Royal Institute of Technology, Stockholm, Sweden M.C Leva, N. McDonald Aerospace Psychology Research Group School of Psychology Trinity College Dublin, Ireland Cpt. M. Ydalus Head of Safety Office STOOB Airline operations, SAS, Stockholm, Sweden Introduction It takes all the running you can do, to keep in the same place. If you want to go somewhere else, you must run at least twice as fast as that. -Lewis Carroll s Through the Looking glass Today s business environment is constantly changing and boundaries in the design envelope (Rasmussen, 1997) keep pushing the operational point towards the boarder of acceptable safety limit. New demands increase pressure on capacity, safety, quality, security, cost, environment, regulations and competition. This makes the pressure in the operational process constantly changing. To keep operating safe under these constantly new conditions requires a flexibility and control over safety related performance along with other important performance measures reflecting the success of the new demands on the system. This means that operations will have to constantly transform. In turn this increases the organizational processes delivering improvement to operations get feedback of what influences the new demands on a strategic level have on the operational process. In earlier research (McDonald et al., HILAS, 2009) a theoretical framework with a human factors approach has been developed. It is believed that this complementary view is necessary to identify areas of improvement with the most relevance and leverage on safety critical systems change. The purpose with Management control is to have intentional impact on operations, executives and personnel towards certain objectives. Support for these tasks are: formal controls such as budgets and 1
performance measures; Organizational structure such as allocation of responsibilities and accountabilities and incentives. Less formalized control such as organizational culture, learning and authorization (Kulven et al., 2010). Performance management (PM) is an activity aimed at delivering improvement of performance. Traditionally performance management in business used financial measures. Using key performance indicators at various departments allowed aggregation and collection of measures to gather an overall summary of results of the business as well as to identify specific areas performance and need for improvement. The overall goal is to improve operational process outcome. Here the performance management has a focus on performance related to a successful change and safe operations. Performance management is one angle of management which needs to be integrated with other organizational focuses such as change and risk management. Management approaches such as Lean help a very pressed industry to survive and stay in a cost cutting mode. Perhaps we are moving towards a point where the system is so stretched and where this is manifested in fatigue, and stress and see that production pressure impact safety? This link is not known and possible impact on safety is manifested in a parallel management system, in reports, and SPI s. This is not always known in the system measuring Lean KPIs. Safety deficiencies are still often explained by simple human factors such as human error and compliance to routines and human performance. 30 years of HF history has not proved a concept that directs actions to real system change and improvement. Identifying HF factors even at a systemic level (individual technology, joint cognitive systems and latent conditions) may help develop indicators but not identifying system antecedents that influence human performance in the system operations. Industrial leaders admit they have difficulty with change but it is not the actual change that is difficult. As soon as we know what needs to change and have an idea how it is more or less just to put it on the right track. However it is identifying what needs to change, to know it has changed and actually had the desired effect that is difficult (MASCA, 2011). Safety is a field that does not lend itself easily to traditional performance measures. In safety research there are many models explaining what happens when safety fails such as weaknesses in technology, organizations and culture. What is needed is to implement a new HF framework that may govern system performance that will improve safety. When logic is laid, tools and systems are required to support system performance management with a system integrating risk, change and learning. (1+2) Literature review / Situation today The Performance Review Commission (PRC) produced a report (Eurocontrol, 2009) ATM Airport Performance (ATMAP) Framework. IT describes a framework for performance measures for airport airside and nearby airspace performance. It is made up by the following components; Scope definition within which measurements are conducted; list of factors affecting performance; 2
Relevant Key Performance Areas (KPAs) to describe performance objectives of an airport community; Key Performance Indicators (KPIs) to measure KPAs; Data requirements to populate KPIs and various approaches on how to use the KPIs in order to assess and evaluate performance. (3+4) The ARMS concept, Airline Risk Management Solutions (Nisula, 2008), has been called the next generation methodology for operational risk assessment. Operational risk assessment is a core part of the safety management system. It is a result of a consensus among practitioners that all existing methods had serious shortcomings. The new methodology for global risk assessment builds on four terms rather than the traditional two in a risk matrix, namely probability and severity. Instead there are in ARMS two matrixes, which include two terms between probability and severity. The first matrix includes frequency and avoidability and will result in a number value. The second matrix includes recoverability and gravity and will result in a letter value. The combined assessments give values of likelihood and severity as before in an unacceptable zone and in an acceptable zone. ARMS also present a new framework addressing risk controls in order to prevent and avoid threats, avoid and recover from undesired events and recover and mitigate accidents. ARMS ensure urgent issues are addressed timely and is a practical and acceptable method that support the risk assessment phase for industry. Another recent initiative is the APF (Aerospace Performance Factor) (Eurocontrol, 2009). The working group behind the APF argued that it changes how aviation views on safety information and expects improvements in competitive efficiencies, improvements in insurance costs and a step towards operational risk forecasting as a result. The idea is to integrate safety information into organizational and operational decision making by breaking down silos and then merge data sources. This allows safety information to come to full field of view which is argued to be proactive and enhance risk forecasting. The main process is around organizing and prioritizing safety information. Safety data and information sources are mapped with regard to origin in operations and system level. Then a pair wise comparisons at all levels is done by use of elements of Analytical Hierarchy Processing (AHP). This methodology is used in multi objective decision making analysis. The last step in the process is to display information for decision makers: Various graphs are produced that may visualize various levels leading up to the APF level. The APF is the sum of weighted occurrences over appropriate denominator. The denominator is chosen appropriate to the domain, i.e. Eurocontrol use the number of flight hours flown in the time interval of interest, but for an airline it might be the number of flight sectors flown. (5) ICAO has produced two main draft documents of a Safety Management Manual (SMM) in 2006 and 2008 (ICAO, 2006, 2008). The manual is intended to provide States with guidance to develop a regulatory framework and guidance material for implementation of a Safety Management System (SMS) by service providers as well as for the implementation of a State Safety Program (SSP). Only SMS will be discussed here. Guidance is given for SMS planning and operations following ICAO SMS framework: Safety policy and objective, Safety risk management, Safety assurance and Safety promotion. 3
States shall require an SMS from service providers that are exposed to safety risks. In the Annexes to the ICAO SMS document it is also established that the SMS shall be accepted by the state. The SMS shall, as a minimum: identify safety hazards; ensure the implementation of remedial action necessary to maintain agreed safety performance; provide for continuous monitoring and regular assessment of the safety performance; and aim at a continuous improvement of the overall performance of the safety management system. The notion of Safety performance is expressed as an essential ingredient of the effective operation of an SMS and progress towards a performance based regulatory environment. Safety performance for service providers are proposed to be expressed in terms of compliance to agreed on and set safety requirements, safety performance targets and safety performance indicators. The safety performance of an SMS is not directly related to the quantification of highconsequence outcomes (safety measurement) but rather to the quantification of the lowconsequence process (safety performance measurement). Safety performance indicators are described as short term, tactical, measurable objectives reflecting the safety performance of an SMS. They are expressed in numerical terms; they should be obvious, measurable and linked to the safety concerns of an SMS. Safety performance indicators reflect safety performance measurement exclusively. The safety performance indicators of an SMS should not reflect safety measurement. State and individual organizations will separately agree on safety performance targets for each aviation provider, therefore safety performance indicators may differ between individual service providers. Safety performance targets are described, by ICAO, as long term, strategic measurable objectives reflecting the safety performance of an SMS. Safety performance targets are expressed in numerical terms, they should be obvious, measurable, acceptable to stakeholders, and be linked to the safety performance indicator (short term objective) of an SMS. Safety requirements are described as the tools and means needed to achieve the safety performance indicators and safety performance targets of an SMS. This may further be interpreted as what is required to do or to implement to achieve the level of safety performance of the process, relating to the safety performance indicator. Indicators and targets may be the same or may be different. ICAO uses a few examples explaining that safety performance indicators are set in agreement between the individual service provider and the state. Resources, costs and urgency for a safety performance indicator to improve towards a more demanding target may vary and sometimes it may be considered enough to maintain the safety performance. In this case the target and indicator are the same. In one example a low consequence process is described as number of foreign objects debris (FOD) over a period of time or for a number of flights. A range of different safety parameter indicators and targets are believed to provide a better insight. The SPI must be agreed upon with the authority to be reasonable and relevant for strategic management safety training, believed to contribute to increased safety by improving what is identified as prioritized processes and assumed to reduce latent conditions. 4
Safety performance for service providers are proposed to be expressed in terms of compliance to agreed on and set safety requirements, safety performance targets and safety performance indicators. As per ICAO(2008, p. 204) safety performance monitoring and measurement shall include the following: hazard reporting; (mandatory, voluntary and confidential reporting) safety studies; safety reviews; (conducted during introduction and deployment of new technologies, change or implementation of procedures etc.) audits; safety surveys; and internal safety investigations. Reactive, proactive and predictive safety management perspectives are discussed in the second edition of the guidelines and framework from ICAO (2008). The traditional approach has been to react to undesirable events by investigations finding cause and then through regulations and rules avoid it happens again. Today, ICAO (2008) makes a distinction where reactive method responds to the events that already happened, such as incidents and accidents. The proactive method looks actively for the identification of safety risks through the analysis of the organisations activities. The predictive method captures system performance as it happens in real time normal operations to identify potential future problems. The ICAO Safety Management Manual (SMM) draft documents have been on the table and discussed by most service providers and authorities for some time and they are supposed to influence regulations worldwide. The ICAO manual will be the basis for EASA when formulating common requirements in Europe for SMSs and for national civil aviation authorities in their function as national regulators. Two aspects of the proposed ICAO safety management system framework distinguish it from previous organisational frameworks for managing safety. These concern, first, the integration of the management of safety into the overall management system of the organisaton; and second that this should be a performance based regulation, capable of demonstrating its implementation and effectiveness in terms of measurable operational outcomes which are related to safety. The previous generation of safety regulation was based on the requirement for an independent quality or safety system with separate accountability to the accountable manager for the operational organisation. While this sought to ensure that commercial interests do not override those of safety, the critical weaknesses of this form of regulation concern the implementation of safety recommendations to achieve verifiable change, and the difficulty of ensuring that a safety management system, which might look fine on paper, is actually working in practice. The new generation of SMS The new generation of SMS seeks to implement a performance based regulation in which it is recognised that modern organisations require greater integration of their management processes (e.g. supply, operations, standardisation and control functions) to deliver better outcomes overall, simultaneously satisfying commercial, quality and safety requirements. Within this framework the safety functions have to deliver accountability for safety outcomes, as well as demonstrating the integrity of safety management processes. This creates a paradox for regulation: it has to prescribe what has to be 5
done to assure that safety is properly addressed; but actually assuring this requires a broader focus than just on safety related functions, because safety performance depends on the integrated performance of the whole system; however regulators cannot prescribe what is beyond their remit which relates to those aspects which are (specifically) related to the safety of the system. Therefore, in order to understand how to implement an effective safety management system we need a model that goes a long way beyond the borders of what has traditionally been included under the rubric of safety management. The HILAS project, while one of its drivers has always been the improvement of safety (most particularly with respect to the human factor), has taken just such a broad remit. It has developed a platform for the integration of human requirements across the industry lifecycle which includes the organization, management and improvement of flight operations and aircraft maintenance (not just the safety aspects). The HILAS management system has taken an integrated approach to the management of performance, risk and organizational learning and change, thus it addresses precisely the basis for the performance outcomes required by the ICAO framework, but does so in a way that is not specific to safety, but relates to the effective functioning of the organizational processes in delivering all their operational goals. HILAS has also provided the opportunity to examine in depth other fundamental challenges of regulation in complex systems, including: How to assess risk in ultra safe systems How to develop integrated risk management across multi organizational system of systems like aviation How to bridge the gap between knowing what needs to be done to improve the system and actually doing it. How to address the cultural aspects of implementing organizational and management systems across diverse regions of the world. A core challenge to developing a performance based regulatory framework in aviation is the fact that aviation is what is often termed an ultra safe system in that it is impossible to differentiate between organisations by means of safety failures they are, fortunately, too few. How do we then differentiate between relatively safe and unsafe performance? What can be monitored is the normal variation of operational performance (reports, audits, flight data, etc), on the basis of which we draw inferences about the probability of system failure. This assumes a linear relationship between minor failure and major catastrophe. Without an understanding of the underlying system mechanisms, it is doubtful whether this is sufficient to anticipate or prevent complex system failures. What is needed is a more complex in depth analysis of system performance in order to understand both what has happened in the past and how to anticipate future potential failure often in the context of changed circumstances. This analysis needs to draw in the different organisations who make up the operational system (flight operations, maintenance, air traffic, airports, etc.). If we do not achieve this, the danger is that we will be monitoring Safety Performance Indicators which do not adequately reflect the state of the system. 6
Because transparent and accountable implementation and change have always been problematic issues in safety management and its regulation, HILAS has placed a great deal of emphasis on the necessary sequence leading to system change, as follows: Providing the tools to allow all stakeholders to contribute to adequate solutions to complex system problems Assurance that appropriate change has actually happened following recommendations Establishing that the outcome has improved (i.e. that risk has been measurably reduced) Enabling other organisations to learn effectively so that the system as a whole improves. These stages define the type of performance outcomes which need to be regulated, if regulation is to be effective. (6+7+8+9) The research gap Aviation is known to have enormous amount of data and to develop indicators are not the challenge. The ATMAP has a well founded performance measurement framework but aims mainly to assess and evaluate performance. In both ARMS and APF focus lies on identifying risks, acquiring data, and assessing risk and at the most gets the management attention and decision support to act timely. The real challenge is to manage the system outcome by letting the indicators drive change. What to do after risks are identified, classified, prioritized and performance is evaluated and not sufficient? What to do when action is needed? Clearly it is ICAOs intention to move away from assessing safety merely on numbers on operational outcome. This proposal is arguing for the research gap on the essential and undisputable need for a proven change capability in order to discuss safety performance measures. A measure on safety performance could be time from identifying risk to measurable change in operational data. The result of an effective improvement will be identified and measured and visual in the ARMS, APF and ATMAP but none of the concepts support any industry to achieve improvements. Since SMS assess safety performance it would be logical to develop a safety performance management system. However safety and safety performance should not be managed in isolation. Performance management for continuous innovation in aviation is a concept better describing the integration of change and in the aviation domain that by nature is a high risk industry with safety risks. Concern for the future safety of the aviation system requires us to address many issues in which tried and tested processes; methods and tools are not well known and generally available. These issues include, for example: How to monitor the performance of complex systems in a valid manner How to ensure effective learning and change both within and between organisations How to ensure systems of systems like aviation achieve goals in an integrated way, and how to design future integrated aviation systems to deliver operational safety requirements. 7
How to regulate for a global system which involves many different types of organisations, in different regions, with different cultures, interacting with each other. Performance based regulations Developing performance based regulation for safety management inevitably will touch on these issues if such regulation is to play an effective role in ensuring and improving aviation safety. The HILAS framework provides ways of addressing these issues that not only fulfills the Safety Management System requirements of ICAO, but also demonstrates the integration of SMS in an overall operational management concept and shows how this can foster learning between organizations and innovation for safer systems in the future. Another identified gap is that existing management systems and financial governance and approaches such as lean are all made for managing profitability, competitiveness and survival, but this is not enough for safety of or managing safety performance. The PSPMS (predictive safety performance management system) needs to fill a gap of other theoretical frameworks that may be used to build a management structure, which address safety risks and operational performance in an integrated manner. A Key Performance Area for the Aviation Transport System for the future is innovation in human systems and change in high risk industries. 8
Performance Management cycle for PSPMS: 1. A performance measure and monitoring framework. 2. Capability to aggregate data to appropriate level for decision makers. 3. A structural and procedural framework that uses this data for strategic decision making and priority. 4. A structural and procedural framework to de aggregate the data and triggering appropriate change process and team in relation to the triggering performance measure(s). 5. De aggregation should triggering appropriate change process and team in relation to the triggering performance measure(s). 6. Complex system analysis including process modeling and analysis, evaluating and developing SPI s to identify antecedents and suggest mitigating action. 7. Implement change, using CMS with Knowledge Transformation Process, appropriate learning and training and integration with other ongoing change initiatives. 8. Evaluation framework. Develop new SPIs to monitor progress and measure Safety performance. 1. A structural and procedural framework that uses this data for strategic decision making and priority. 3. Capability to aggregate data to appropriate level for decision makers. 4. A performance measure and monitoring framework. 2. A structural and procedural framework to deaggregate the data 5. De aggregation should triggering appropriate change process and team in relation to the triggering performance measure(s). 8. Evaluation framework. Develop new SPIs to monitor progress and measure Safety performance. 6. Complex system analysis including process modeling and analysis, evaluating and developing SPI s to identify antecedents and suggest mitigating action. 7. Implement change, using CMS with Knowledge Transformation Process, appropriate learning and training and integration with other ongoing change initiatives. 9
References: 1. European Aviation Safety Plan 2011 2014 (EASA, 2011). Annual Safety review 2010, (EASA, 2010). 2. Flight path 2050. 3. Human Factors performance indicators for the energy and related process industries (Energy Institute, 2010). 4. Developing process safety indicators (HSE, 2009). 5. Resilience engineering (Eurocontrol, 2009). 6. Understanding and Evaluating the Federal Aviation Administration Safety Oversight system (Hansen et al., 2006), 7. History of Aviation Safety Oversight in the US (Hansen et al., 2005). 8. Human Performance in Air Traffic Management Safety (Eurocontrol, 2010). 9. New HF (McDonald et al. 2009) 10