Why assess operational behaviour? Seeing is Believing. Drift into Failure. The Problem with Learning. What to assess?

Transcription

1 Why assess operational behaviour? Seeing is Believing Assessing CRM Competencies Dr. Nicklas Dahlstrom Human Factors Manager The Problem with Learning Drift into Failure Normal people doing normal work may lead to unsafe actions Safe actions according to SOPs Unsafe actions Safe actions Why assess operational behaviour? What to assess? Information about drift of operational behaviour towards unsafe actions Complements other reporting and evaluation systems (more technical) Can be used for proactive analysis of operations Increased safety! activities Training Reporting Evaluation Assessment Assessment of: Reactions - Participants feelings Learning - Attitude and knowledge change Behaviour - Behavioural change Organisation - Impact on organisation (O Connor et al., 2003) 1

2 Behavioural Markers Specific and observable behaviours Behaviours that contribute to successful or less successful operational performance (Cause effect relation between behaviour and performance) Based on analysis of data (Accidents, incidents or other data from reporting systems...) Simple, clear and adapted to the actual operational environment CRM-skills? Technical Skills Non-Technical Skills Assessment of Are CRM-skills they identifiable? should always be linked to Are a technical they separable? consequence Background NOTECHS Early work Draft of new European regulations, including regulations regarding assessment of CRM-skills NASA/University of Texas, Behavioural marker system KLM, FAS - Feedback Appraisal System Recent work NOTECHS, JAR-TEL and other research projects Today Development, introduction and use of different systems for assessment of CRM-skills NOTECHS There is an obvious need for a valid and reliable generic behavioural marker system that can be made available to those airlines that do not have the resources or expertise to develop their own system. O Connor et al., 2003 (NOTECHS research report) 2

3 2. IMPORTANT CAUTION FOR USE NOTECHS is not intended to fail additional crewmembers during mandatory checks or indeed on any other occasion, as compared to the situation. Nontechnical skills cannot provoke a FAILED condition out of the context of a related objective technical consequence leading to a compromised flight safety in the short or long term. 3.2 THE FIVE NOTECHS PRINCIPLES 4 ACCEPTABLE OR UNACCPEABLE RATING REQUIRED The JAR-OPS requires the airlines to indicate whether the observed NTS are acceptable or unacceptable. (Amalberti & Valot, JAR-TEL research report) At the pass/fail level the results were more mixed A global pass/fail decision is not inherent to the NOTECHS system and therefore of less relevance to the method and NTS evaluation. Also, for the purpose of training the pass/fail rating is of less importance than the ratings at the category level. (O Connor et al., (NOTECHS research report) Behaviours and Wording Behaviours and Wording Example from NOTECHS: Elements and behaviours of the category Cooperation Supporting Helps other crew members in demanding situation Conflict solving Keeps calm in conflicts Takes command if situation requires Intervenes if task completion deviates from standards Example from NOTECHS: Behaviours of the category Leadership Encourages crew participation Distributes tasks among the crew Behaviours and wording A central question is what types of performance items to include. Unfortunately we know of no empirical data to guide carriers in writing these items. After rewording we noted an increase of 63% evaluator agreement in a subsequent session. we have found that the distribution of grades assigned by evaluators is affected by the choice of labels. Although it is likely that the number of scale points and scale labels do influence an evaluators ability to rate performance, these features are often determined by corporate history and may be resistant to change even in the face of empirical data. At the least, carriers should continually monitor their grade distributions to determine how well their grading is actually reflecting the desired levels of discrimination in performance. CRM scales showed lower reliability across judges and evidence of halo. (Brannick, Prince, Salas, 2002) 3

4 (NOTECHS Rating Form) (NOTECHS Rating Form) The very poor label sounds to have a strong psychological impact on the trainee. The 2-point scale for the final judgement Fail or Pass that is viewed negatively by instructors in that the context for retraining should be replaced to Further training is required / Acceptable. (JAR TEL WP4 Report, 2001) (CAMS Rating Form) Instructor or Rater? Assessment includes: - Observation of specific technical and CRM behaviours - Evaluation of behaviours according to effectiveness - Weighing results to arrive at different scores Problems: - Personal constructs used for evaluation - Opportunity to observe relevant behaviours - Experience and training regarding evaluation - Organisational constraints (Baker and Dismukes, 2002) Instructor or Rater? Specific or global? Calibration training - Transfer of calibration? - Decay of calibration? Standards - Identifiable? - Conserves current practises? Conflict of roles - Observer or evaluator? - Supporter or judge? When performance improvement rather than evaluation is the main goal of the simulation (as in LOFT), a few themes are necessary for feedback. For example, a crew cannot assimilate feedback about 50 behaviours; they need to see a pattern or theme that is illustrated by behavioural examples. Therefore, global CRM dimensions may prove useful in performance improvement even though such scales are problematic from a reliability standpoint. (Brannick, Prince, Salas, 2002) 4

5 Competence Markers and Training PAMs in Practice Knowledge Procedures Handling Automation Reliability Easily identifiable, can have clear links and short-term term change may be possible Workload Situational Awareness Decision Making Communication Leadership & Teamwork Resilience Not easily identifiable, linked in complex ways and often require long-term change Regulatory requirement Checking tool Performance measure Feedback for improvement PAMs & Safety Summary Safety Boundary Outline of benefits and problems of assessment of CRM Skills in an operational context Specific points in regards to problems to resolve when designing and implementing a system for assessment of CRM Skills Safety Margin Assessment of CRM Skills In Practice! Goal of CRM Training Thank you! Awareness Airmanship Suspicion Resilience nicklas.dahlstrom@emirates.com 5