How To Evaluate And Review Flight Training Manawatu

Transcription

1 Report of External Evaluation and Review Hawk Enterprises Limited trading as Flight Training Manawatu Highly Confident in educational performance Confident in capability in self-assessment Date of report: 31 July 2014

2 Contents Purpose of this Report... 3 Introduction TEO in context Scope of external evaluation and review Conduct of external evaluation and review... 5 Summary of Results... 7 Findings... 9 Recommendations Appendix MoE Number: 7540 NZQA Reference: C14687 Dates of EER visit: 28 and 29 April

3 Purpose of this Report The purpose of this external evaluation and review report is to provide a public statement about the Tertiary Education Organisation s (TEO) educational performance and capability in self-assessment. It forms part of the accountability process required by Government to inform investors, the public, students, prospective students, communities, employers, and other interested parties. It is also intended to be used by the TEO itself for quality improvement purposes. Introduction 1. TEO in context Name of TEO: Type: Location: Delivery sites: Hawk Enterprises Limited trading as Flight Training Manawatu (FTM) Private training establishment (PTE) Feilding Aerodrome, RD 5, Feilding As above First registered: 12 December 2002 Courses currently delivered: New Zealand Diploma in Aviation (Aeroplane and Helicopter) (General Aviation) (Level 5) New Zealand Diploma in Aviation (Aeroplane and Helicopter) (Flight Instruction) (Level 6) New Zealand Diploma in Aviation (Airline Preparation) (Level 6) Training schemes: Private Pilot Licence (Level 3) Commercial Pilot Licence Multi Engine Rating Course (Level 3) Multi Engine Training Scheme (Level 5) Flight Instruction Training Scheme (Level 6) Code of Practice signatory: Yes; approved for students aged years and 18 years upward 3

4 Number of students: Domestic: 14 (no Māori or Pasifika students are currently enrolled) International: 27 (international students are predominantly from India) Number of staff: Scope of active accreditation: Distinctive characteristics: Recent significant changes: Previous quality assurance history: Other 13 full-time equivalents Domains and unit standards in Aviation, up to and including level 6. FTM operates from Feilding Aerodrome in Manawatu. The current owner has run the school for the past 18 years. FTM was, for several years, offering the NMIT Diploma in Aviation Science (Level 6) under a partnership arrangement with Nelson Marlborough Institute of Technology (NMIT). This partnership ceased at the end of 2012, at which time FTM became directly funded by the Tertiary Education Commission (TEC). The change from one qualification and funding source to another has led to a number of anomalies in TEC qualification completion data as students have transitioned between qualifications and funding providers. To varying degrees, this has affected all flying schools who were party to NMIT partnership arrangements. This is FTM s second external evaluation and review (EER). The previous EER, held in March 2013, resulted in summative judgements of Confident in educational performance and Not Yet Confident in capability in self-assessment. The certification and operation of organisations conducting aviation training and assessments in New Zealand is prescribed under Part 141 of Civil Aviation Rules governed by the Civil Aviation Authority of New Zealand. All theory and practical (flight test) assessments are conducted independently by Aviation Services Ltd, which operates under licence to the Civil Aviation Authority. All pilot licences are issued by the authority. FTM was last audited by the Civil Aviation Authority in September 2013, at which time FTM was found to be compliant with all of the 668 Civil 4

5 Aviation Rules that were tested during the audit. 2. Scope of external evaluation and review The scope for the EER consisted in the mandatory focus areas: Governance, management and strategy International student support 1 In addition, the following focus area was selected: New Zealand Diploma in Aviation including all levels and strands delivered by FTM The above focus area encapsulates all of the teaching and programme delivery of FTM. The diploma provides flight training up to a Commercial Pilot Licence with an Instrument Rating. In the second year of training, FTM students are split into three streams: General aviation stream including GPS (navigation), dangerous goods, air operations and aerobatic flight Airline preparation stream including the Airline Transport Pilot Licence (ATPL) theory examinations with a component in the Multi Crew environment Instructor stream including the qualification required to become a C- Category Flight Instructor As students from the three streams are taught through individual learning plans within a single cohort, it was considered appropriate to include all activities in the scope. 1 Traditionally, international students have enrolled for programmes leading to Civil Aviation Authority licences, such as the Commercial Pilot Licence or Multi Engine Rating, rather than a full diploma. 5

6 3. Conduct of external evaluation and review All external evaluation and reviews are conducted in accordance with NZQA s published policies and procedures. The methodology used is described fully in the web document Policy and Guidelines for the Conduct of External Evaluation and Review available at: The TEO has an opportunity to comment on the accuracy of this report, and any submissions received are fully considered by NZQA before finalising the report. The EER was conducted in late April Prior to the EER visit, the lead evaluator discussed the visit by telephone and with the chief executive of FTM. A one and a half day visit was made to the FTM site at Feilding Aerodrome, where the evaluation team, consisting of two evaluators, reviewed a range of documentation and met with the chief executive/chief flying instructor, training manager/deputy chief flying instructor, course director, student support coordinator, two groups of instructors, two groups of students and four independent advisory board members. 6

7 Summary of Results Statement of confidence on educational performance NZQA is Highly Confident in the educational performance of Flight Training Manawatu. Key reasons for this include: Students at FTM are achieving excellent results (refer Findings 1.1 for detailed figures). Course completion in 2013 was 93 per cent for international students and 100 per cent for domestic students. Eighty-five per cent of domestic graduates from the last five years of FTM programmes are now working in the aviation industry (refer Findings 1.2). FTM is in regular contact with many of its graduates, and the results indicate that the organisation is preparing students well for employment in the aviation industry. FTM and its staff engage well with the industry at a local level and are valued for the positive contribution they make. A more systematic engagement approach, especially with employer organisations, would lead to more useful feedback and stronger links with potential employers of graduates. The EER team found consistent evidence of excellent teaching throughout the organisation, brought about by a systematic and deliberate strategy (refer Findings 1.4). Feedback from students indicates that FTM programmes have an appropriate mix of theory and practice, and that they are delivered and assessed in a manner that enables the students to understand and apply the material being presented. FTM has excellent student support structures that are appropriate to the needs of both international and domestic students and are helping them to achieve (refer Findings 1.5). The organisation is well managed and has a clear philosophy and purpose. The training is well resourced and FTM uses its resources effectively. 7

8 Statement of confidence on capability in self-assessment NZQA is Confident in the capability in self-assessment of Flight Training Manawatu. Key reasons for this include: Records and discussions indicate that self-assessment has improved since the first EER, exemplified in the way it is being practised and beginning to be embedded across the organisation. FTM has systems to gather and collate feedback from learners. The quality and validity of the information collected is sound and the information is then used to inform improvements to teaching practice and course delivery. Although FTM has very positive engagement with the aviation industry on a local level, a more systematic approach to gathering feedback from industry and graduates on a wider basis is needed and will provide useful information that can be used to bring about further improvements. Staff at FTM demonstrated a good understanding of the factors that lead to student achievement. Despite achieving excellent results, FTM staff still regularly formally and informally discuss and assess ideas for further improving achievement and making the courses more useful and enjoyable for students. Staff members are well supported by management and have plenty of opportunities for reflection on their role, which benefits students through improved teaching techniques, student support and graduate outcomes. FTM has followed through on opportunities for improvement identified in the last EER, and has engaged well with the key evaluation questions. The evaluation team believes that as self-assessment processes continue to be implemented at FTM, significant and meaningful ongoing improvements will result. FTM is aware that this good start to improving self-assessment processes needs to be fully embedded as a planned and coordinated approach across the organisation. 8

9 Findings How well do learners achieve? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. The students who have studied at FTM have achieved excellent results in the completion of courses and examinations. The 2013 course completion rate was 93 per cent for international students and 100 per cent for domestic students. FTM has benchmarked its course completions against other similar providers and, although scoring better than other providers in the sector, still aspires to maintain and continuously improve student achievement. For instance, in 2013 FTM identified that the 7 per cent of international students who did not pass their courses were all doing a significant part of their theory through self-study. As a result, in 2014 all international students are required to attend classes daily. To date, this strategy is proving to be successful, as borne out in the improved engagement of international students in their study. Early indications are that the strategy will result in increased course completion by international students. The organisation has a strong focus on, and commitment to, student achievement. During 2013, FTM developed and implemented systems to provide management, advisory committee members and staff with accurate data of pass rates for both theory and flight tests, to enable staff to fine-tune strategies for sustaining and improving achievement. A dedicated staff member has been appointed to collate this data, and early indications are that the increased level of awareness of progress among students, instructors and management will assist FTM to maintain the current level of domestic student achievement and improve international results. Students are debriefed by their instructor after every flight to identify what the student did well, what he/she did less well, and where improvements could be made, either in the student s performance or FTM s instruction. The training manager and chief executive regularly conduct interviews with students to review progress and gain feedback on the course. These interviews are also a forum for the chief executive to discuss career aspirations/goals with the students and offer advice on further training and study. Every student has a record book in addition to their logbook, which includes achievements and running records. The record book gives students detailed 2 The findings in this report are derived using a standard process and are based on a targeted sample of the organisation s activities. 9

10 feedback about how they are progressing and provides handover notes when they have a change in flight instructor. Staff at FTM demonstrated a good understanding of the factors that lead to student achievement, and formally and informally analyse and discuss ideas for improving achievement. The instructors take a very collegial approach and all share the same office space, so they can communicate and share information informally on a daily basis. FTM also has a compulsory staff meeting every two weeks where progress and the achievement of students are the main topics of conversation. FTM currently has no Māori or Pasifika (domestic 3 ) students. In line with commitments to the TEC, the organisation is in the early stages of developing strategies that will improve participation by Māori and Pasifika. 1.2 What is the value of the outcomes for key stakeholders, including learners? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. Thirty-four of the 40 FTM domestic student graduates from 2009 to 2012 are now in aviation-related employment (see Table 1). Table 1. Graduate employment outcomes (domestic students), Total Number of graduates Number in aviationrelated employment Percentage 92% 92% 71% 75% 85% This good level of graduate employment is due to the focus that FTM has put on preparing graduates for the aviation workforce. FTM considers that the merit of the programme lies not only in the long-term professional outcomes for students but also in attributes such as the professionalism, confidence and positive attitudes they acquire on the courses. The immediate outputs from this training are licences and ratings as awarded by the Civil Aviation Authority, and these also meet the requirements of the International Civil Aviation Organisation. FTM also prepares (Indian) students to sit the Indian Directorate General of Civil Aviation examinations. Students achievement of these licences and ratings equip them to apply for a range of roles within the aviation industry in their respective home countries. 3 FTM currently has one international student from a Pacific country. 10

11 International students are gaining formal Civil Aviation Authority licence qualifications, and those interviewed during this evaluation clearly stated that it was their intention to return to their home countries and gain employment as pilots following graduation. They were confident that they were well prepared to do so. However, accurate employment outcomes for international graduates have been difficult to ascertain. FTM interviews all international graduates on completion, but finds it difficult to systematically follow up with graduates once they return to their home countries. The organisation endeavours to keep in touch with international graduates through a variety of social media, informal networks and communications, and has some good anecdotal evidence of the employment successes of individual international students in the aviation industry. There are also a number of students who, after returning to their home country to complete licence conversions, return to FTM to undertake further training. The information that FTM gains from the engagement with domestic and international students is gathered, analysed and discussed. Analysis is then appropriately used to make improvements to programmes. 1.3 How well do programmes and activities match the needs of learners and other stakeholders? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. FTM courses are structured to offer three different diplomas to domestic students, and various Civil Aviation Authority licences to international students. The New Zealand diplomas are also available to international students, some of whom have expressed a strong interest in completing these qualifications. The overall delivery structure of the Diploma in Aviation programme at FTM is designed to enable students to complete the theory component of the qualification simultaneously with their practical flying instruction. FTM reports that it has adopted this structure because it engages students well and enables them to immediately apply the theory learned to their practice. FTM s involvement with outside organisations such as Aviation Services Ltd, the Civil Aviation Authority and the Aviation Industry Association informs FTM practice to maintain a high standard of training. These organisations provide valuable feedback. FTM has agencies in India which handle the recruitment of students and regularly provide advice about Indian aviation requirements. The FTM chief executive visits India at least annually to keep in touch with the Indian student market and airline requirements. FTM recently purchased a Diamond DA20 training aircraft as a first step to incorporating the latest technology into the training fleet. However, the DA20 does not comply with Indian Directorate General of Civil Aviation requirements. 11

12 Therefore, students wishing to return to India cannot have DA20 hours recognised towards their flying time. FTM has an established advisory committee which comprises both staff members and several prominent industry members, which assists the organisation to ensure it is meeting industry and learners needs. Meeting minutes and discussions with individual members indicates that the advisory committee is meeting regularly, functioning well, receiving good information from the organisation, engaging with external stakeholders outside of the group, and engaging with the organisation to provide operational and strategic advice. FTM deliberately looks for feedback from employers and graduates. However, FTM should now adopt a more formal process for analysing, discussing and summarising this valuable feedback so that it can be systematically used to make improvements. The EER team noted, for instance, that there was little engagement with third-level airline operations that could become a significant source of employment for FTM graduates. FTM has a number of instructors available and generally seeks to ensure that instructors are matched to the learning styles of individual students. The systematic approach by the teaching staff to meet the individual, personal and academic goals of every student is a strong feature of the organisation. Student evaluations are conducted to a regular schedule and gather students feedback on their experiences of different aspects of the programme, including content and delivery, facilities and staff effectiveness. The information is then used to bring about useful improvements to teaching and programme design. 1.4 How effective is the teaching? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. There was clear evidence of excellent teaching practice taking place at FTM. All the staff interviewed are enthusiastic and passionate about the curriculum and their teaching and are well supported by management. Every staff member at FTM has a role in student learning, and staff take a collegial, whole-of-organisation approach to ensuring that all students achieve. Instructors and students relate well to each other. The strong rapport between students and staff was evident from discussions and evaluation survey outcomes. Students like the open, friendly style and find their learning activities interesting and challenging. Students have good access to staff outside of formal classes and can make suggestions about improvements. Students noted that the staff were responsive to any concerns or issues raised. 12

13 The staff are well qualified, with B Category and C Category instructors on permanent staff. In addition to extensive industry and professional experience, and an obvious passion and ability to engage students, 83 per cent of FTM instructors have completed the National Certificate in Adult Education and Training, a strategy FTM has adopted to provide students with effective teaching. This is an excellent example of good practice and management commitment to educational performance. Staff appraisals are completed twice a year and include goal-setting for staff to encourage and support them to further their qualifications. All flight instructors have at least three reviews per year: an annual instructor renewal with an independent qualified and examiner; an interview with the chief flying instructor or a person appointed by the instructor; and a flight check with the chief flying instructor. FTM has implemented this regime to ensure that staff members are monitored continually during the year to improve the instruction for students. This review process does not preclude regular and ongoing feedback on a formative basis. Given that all summative assessment is through external examination, FTM does not require a moderation system. The chief executive does, however, closely monitor examination results and matches these results with the various instructors to identify trends, positive or negative, that may indicate good practice or areas for improvement. 1.5 How well are learners guided and supported? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. FTM has robust and client-friendly systems for the pastoral care of its students. The organisation is providing a safe and supportive learning environment for its students and is in close contact outside of the normal hours of tuition. The small numbers of students means that they are given individual support. There is a designated support staff member with 24-hour phone contact should the students require assistance when away from the site. Accommodation for international students is managed in-house by the international student coordinator. The coordinator visits all accommodation sites on a regular cycle to inspect and evaluate suitability. The international student coordinator has overall responsibility for ensuring that Code of Practice requirements are met, and staff regularly attend professional development offered by the Code Office. The organisation periodically (at least once a year) thoroughly self-reviews its compliance with the code. Pre-enrolment information is comprehensive and designed to ensure that students make an informed decision before committing to a significant personal and financial 13

14 investment. International students receive a comprehensive orientation programme in their first week of study. FTM has detailed and robust procedures for resolving complaints. Feedback from students was that any complaint or feedback is addressed promptly and appropriately. 1.6 How effective are governance and management in supporting educational achievement? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. FTM is a privately owned and operated aviation training provider with a clear set of goals and objectives. The organisation has developed and embedded effective systems for monitoring student achievement and supporting staff to improve educational achievement. FTM s management recognises that the key to success of the business lies in the success of the students and the value they gain from the training. FTM actively supports strategies to increase student achievement and to support students to meet their employment goals. FTM has employed very highly qualified and experienced staff, who are managed effectively and provided with professional development opportunities. The organisation is well equipped with physical and learning resources for the number of students. There are systems in place to monitor resourcing to ensure there are always sufficient resources, mainly aircraft and instructors, to meet the needs of the students. Monitoring of performance within the organisation is regular, transparent and robust, and the organisation encourages opportunities for reflection on its role and how to continue to make ongoing and continuous improvements to meeting the needs of students and other stakeholders. Self-assessment is being well led by management but is yet to be universally understood and adopted throughout the organisation. The next challenge for FTM is to sustain and increase the current levels of educational outcomes by embedding systematic, whole-of-organisation analysis and ensuing action from self-assessment activities. 14

15 Focus Areas This section reports significant findings in each focus area, not already covered in Part Focus area: Governance, management and strategy The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Good. 2.2 Focus area: International student support The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Excellent. 2.3 Focus area: New Zealand Diploma in Aviation including all levels and strands delivered by FTM The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Good. 15

16 Recommendations NZQA recommends that Flight Training Manawatu: Continue to develop and embed systems across the organisation for the purpose of data collection and analysis that will inform self-assessment practice across the whole organisation. Engage and develop external stakeholder networks (such as third-level airlines) for the purpose of understanding employer requirements/needs and graduate pathways. Continue to work to improve Māori and Pasifika participation and achievement through engagement with Māori networks. Explore and take up opportunities for professional networking with other training providers to help FTM staff gain a more in-depth understanding of self-assessment in practice. 16

17 Appendix Regulatory basis for external evaluation and review External evaluation and review is conducted according to the External Evaluation and Review (EER) Rules 2013, which are made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. Self-assessment and participation in external evaluation and review are requirements for maintaining accreditation to provide an approved programme for all TEOs other than universities. The requirements are set through the NZQF Programme Approval and Accreditation Rules 2013, which are also made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. In addition, the Private Training Establishment Registration Rules 2013 require registered private training establishments to undertake self-assessment and participate in external evaluation and review, in accordance with the External Evaluation and Review Rules (EER) 2013, as a condition of maintaining registration. The Private Training Establishment Registration Rules 2013 are also made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. NZQA is responsible for ensuring non-university TEOs continue to comply with the rules after the initial granting of approval and accreditation of programmes and/or registration. The New Zealand Vice-Chancellors Committee (NZVCC) has statutory responsibility for compliance by universities. This report reflects the findings and conclusions of the external evaluation and review process, conducted according to the External Evaluation and Review (EER) Rules The report identifies strengths and areas for improvement in terms of the organisation s educational performance and capability in self-assessment. External evaluation and review reports are one contributing piece of information in determining future funding decisions where the organisation is a funded TEO subject to an investment plan agreed with the Tertiary Education Commission. External evaluation and review reports are public information and are available from the NZQA website ( The External Evaluation and Review (EER) Rules 2013 are available at while information about the conduct and methodology for external evaluation and review can be found at 17

18 NZQA Ph E [email protected] 18