EUROPEAN CIVIL AVIATION CONFERENCE CONFÉRENCE EUROPÉENNE DE L AVIATION CIVILE

Transcription

1 EUROPEAN CIVIL AVIATION CONFERENCE CONFÉRENCE EUROPÉENNE DE L AVIATION CIVILE PRINCIPLES FOR MEASURING THE QUALITY OF TRAINING ENDORSED AT DGCA/146 (18 MAY 2016) EXECUTIVE SUMMARY This document presents the key principles of quality control of training. Training remains an issue of critical importance for the aviation community, as most key processes and decisions still rely on the human factor. Therefore, it is necessary to set quality control mechanisms so as to ensure the expected outcomes are achieved. Quality control of training should consider organisational issues, design and delivery of training, and training evaluation. Systematic approaches to training design should follow Instructional Systems Design approaches, focusing on learning objectives, controlled learning experiences, predefined performance criteria to measure whether objectives have been achieved, and the collection of evaluative information. Best practices involve a typical five-steps sequence by which training is analyzed, designed, developed, implemented and evaluated (ADDIE). Evaluation is a key factor in quality control of training. Its results should be used to constantly improve all aspects of the training process. In order to evaluate a training programme, a well-established method by Kirkpatrick & Kirkpatrick (2006) suggests the following four levels of evaluation: Reaction (i.e. trainees reactions to the training programme) Learning (i.e. what the trainees actually learned) Behaviour (i.e. whether the newly acquired knowledge is being transferred into the trainee s daily work) Results (i.e. whether the training programme was beneficial at the organisational level). Quality control should focus on the management of the training programme within the organisation, the design of instruction and training, and the training evaluation results. To assess the main objectives and the impact of a training system in an organisation the following questions should be taken into consideration: 1) What is the purpose of training? 2) Is this possible to solve it through a training strategy or is there another organisational solution available? 3) If a training a strategy is to be put in place, what is the final goal to be obtained? What is the performance expected from the staff compared to their initial starting point (before receiving training)? 4) Considering the performance expected, what are correspondent learning outcomes (knowledge, skills and competencies)?

2 2 5) Considering all the above, what kind of training path is to be developed (considering the training cycle)? 6) How can the outcome of the training be evaluated in terms of the trainee s development, the training management and the impact on the organisation (short and medium term for all three aspects)? PRINCIPLES FOR MEASURING THE QUALITY OF TRAINING Introduction 1. Training remains an issue of critical importance for the aviation community in all its components, in regard to the number of key processes and decisions still relying on the human factor. Moreover, training aviation staff represents significant efforts for all stakeholders (e.g. State authorities, operators, Air Navigation Service Providers) in terms of human resources and budgets. Therefore, it is necessary to set quality control mechanisms so as to ensure the expected outcomes are achieved and that training offers a return on investment. 2. High quality training programmes are managed on an organisational level and are coherent with other organisational processes. Furthermore, they follow a systematic instructional design approach and attach great importance to training evaluation. Consequently, quality control of training should consider i) organisational issues, ii) design and delivery of training, and iii) training evaluation. This quality control approach should apply to both regulated and non-regulated training programmes. Organisational issues 3. On an organisational level, several aspects are important to assure that high quality can be achieved with a training programme. These aspects should be considered and assessed when quality control is implemented. For instance, the supervision of trainees (on the job post) after training during the adaptation phase. 4. The content of different training courses should be co-ordinated to guarantee a coherent learning experience. Initial training, recurrent training, computer-based training (CBT) and/or on-the-job training should all target relevant competencies and should ideally build up on each other. 5. It is a common misconception in organisational training that everyone who has ever learned anything as a participant is able to become a trainer as well [6]. Instructors should complete train-the-trainer courses to become familiarized with educational principles, instructional techniques, and group facilitation. Recurrent train-the-trainer courses should be offered on a regular basis. 6. A process to assure regular updates of training content is a key factor of high training quality. It is necessary to adapt the training content to changing requirements of the workplace or changes in the business environment. Instructional Design 7. High quality workplace training should be planned and implemented using a systematic approach to foster learning and development to improve individual, team or organisational effectiveness [2].

3 3 8. A systematic approach to training design is Instructional Systems Design (ISD; [2]). ISD emphasizes the importance of learning objectives, controlled learning experiences, predefined performance criteria to measure whether objectives have been achieved, and the collection of evaluative information. 9. Different ISD models exist, and most of them have similar components. A basic, well-known model is ADDIE, containing five phases: Analyze, Design, Develop, Implement and Evaluate ([1]; see figure 1). Evaluation Figure 1: ADDIE model of instructional design 10. In the analysis phase, a training needs analysis is conducted. The tasks required on the job and the competencies, i.e. the knowledge, skills, abilities and other characteristics (KSAOs) that are necessary to complete the tasks are determined. Based on the KSAOs, learning objectives are formulated. The objectives should include the required performance in the form of observable behaviours at the completion of training, the conditions under which the behaviour will occur, and criteria of acceptable performance ([4]). 11. Furthermore, learning objectives should be defined according to SMART criteria, to include: Specific for example, the learning objectives to be reached by the target audience should be clearly identified. Measurable the learning objectives should be measurable (either with a quantitative or qualitative approach). Different measurement tools could be used to evaluate the achievement of these objectives. Attainable the learning objectives should be attainable by the target audience, and different training methods implemented to achieve this. Relevant the training content is relevant to the target audience/participants, and to their expectations. The learning objectives are also realistic, i.e., there is adequacy between the training content and the pre-requisites defined for participating in the training. Time-bound the duration necessary to reach the learning objectives should be set in advance, and the duration of the training course or programme tailored to this. 12. In the design phase, a blueprint for the training course is established. Based on the learning objectives, course topics, instructional units (lessons), and specifications for learning assessments are defined. In this phase, it has to be considered whether forms of computer-based training (CBT) or blended learning (combination of face-to-face training

4 4 and CBT) are suitable to achieve the defined learning objectives. Training content, learning activities and materials are produced and tested. 13. In the development phase, the blueprint is put into practice. Instructors are trained on how to deliver the course. Training methods and media are chosen in a way that maximizes learning. Ideally, different methods are combined, e.g. classroom training, real equipment demonstration, group discussion, individual or group exercises, to name a few. Furthermore, training should deal with realistic workplace problems and issues, activate prior knowledge of trainees, include demonstrations and opportunities for active practice, and foster integration of new learned material into previous experiences of trainees ([5]). 14. The implementation phase is characterized by the realisation of the course. Adequate training facilities are of high importance. 15. In the evaluation phase, trainees performance, the training programme, and its impact on the organisation are assessed. Evaluation is a key factor in quality control of training. Its results should be used to constantly improve all aspects of the training process. Gathering evaluative information over a prolonged period of time, it is possible to establish a benchmarking process at an organisational level that shows the training programme s, and the company s, improvements. Implications of quality control of training - Evaluation 16. No matter what kind of evaluation is being run, the evaluation criteria always have to be set in advance. Ideally they are defined based on carefully elaborated learning objectives (see previous section). 17. In order to evaluate a training programme, a well-established method by Kirkpatrick & Kirkpatrick ([5]) in 2006 suggests four levels of evaluation: I. Reaction II. Learning III. Behaviour IV. Results 18. The Reaction stage is concerned with trainees reactions to the training programme, i.e. whether they liked it or not, whether they thought it was relevant for their jobs or not, whether they think it allows them to perform better or not, etc. The trainees reactions can be measured by means of questionnaires. The construction of a reliable, valid and standardized questionnaire is a task that should not be taken too lightly. For this topic alone a wide and encompassing range of scientific literature exists. 19. The Learning stage answers the question, if and what the trainees actually learned. Traditionally, the amount of accumulated knowledge and skill is being measured by some form of final exam. This can either be a written or oral exam in a classroom, or a practical exam where the trainee s behaviour is rated and observed by an assessor. 20. The Behaviour stage relates to whether the newly-acquired knowledge is being transferred into the trainee s daily work or not. For this purpose various methods of observational and interviewing techniques can be used to assess a trainee s performance on the job. This allows deriving the impact of the training programme when comparing the current performance with the previous performance data from the time before the training programme. 21. The Results stage examines whether the training programme was beneficial at the organisational level. After all, there is no sense in introducing a new training programme on a wide scale if the costs are too high in relation to the additional benefits

5 5 it provides. The challenge here is to find the key performance indicators (KPI) that validly represent the trainees performance. Evaluating people s performance by using bad KPIs is ethically questionable, unfair, and decreases the trainees motivation, as they might feel that no matter how hard they try, they have no influence on the result. 22. At an organisational level, results accumulated by Kirkpatrick s four levels method need to be carefully interpreted and be viewed in the broader organisational context. Kirkpatrick s four levels method is a tool to assess training efficiency; however it fails to adequately display innate problems of the organisation as a whole that could possibly have an effect on training results. Training of very high quality, for example, may still deliver poor results if the trainees salaries are being cut at the same time and/or work motivation is low. 23. In order to get a systematic and complete overview of training efficiency, it is recommended where possible to perform all four levels of analysis, analyze every training programme, and repeat the analysis with every reoccurrence of the training. Kirkpatrick s Evaluation levels Reaction Learning Behaviour Results Examples Trainee questionnaires/surveys, interviews and reports National certification tests, quantitative performance evidence, Computer-based test (CBT) performance Threat Image Projection (TIP) in aviation security, covert tests Reduced absenteeism, reduced number of incidents, better customer services 24. Quality control of training should focus on the management of the training programme within the organisation, the design of instruction and delivery of training, and the training evaluation results. 25. Questions that should be asked include, but are not limited to, the following: a) Is the training programme centrally managed in a way that allows coordination of the content of different training courses? b) Do instructors complete train-the-trainer courses and recurrent refreshers? c) Are the course contents compliant with relevant regulations? d) Are the course contents regularly updated to assure that changing requirements are accounted for? e) Is the training programme designed based on a sound training needs analysis? f) Have clear and transparent learning objectives been defined, specifying required behaviour after training? g) Are multiple suitable methods used in training? h) Are training materials clear and adequate and do they support learning? i) Does the training follow basic instructional principles for adult education, such as problem-orientation, activation of prior knowledge, demonstration, active practice, and opportunities to integrate learned content ([5])?

6 6 j) Does the organisation evaluate its training programme, preferably on multiple levels ([3]) and in accordance with the learning objectives? k) Do the results of the evaluation show that the training is effective? l) Are the results of the evaluation used to establish a benchmarking process and improve the training programme? m) Are the results of the evaluation used to modify the content of the training course(s)/training programme? Appendix: References [1] Gagné, R. M., Wagner, W. W., Golas, K. C., & Keller, J. M. (2005). Principles of instructional design. Belmont: Wadsworth. [2] Goldstein, I., & Ford, J. K. (2002). Training in organizations. Belmont: Wadsworth. [3] Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels. San Francisco: Berret-Koehler Publishers. [4] Mager, R. F. (1975). Preparing instructional objectives. Belmont: Fearon Publishers. [5] Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50, [6] Salas, E., Wilson, K. A., Burke, C. S., & Bowers, C. A. (2002). Myths about crew resource management training. Ergonomics in Design, 10, [7] ICAO Manual on the Approval of training organisations (Doc 9841 AN/456), second edition, 2012 END

7 7