Report of External Evaluation and Review

Transcription

1 Report of External Evaluation and Review Glasgow Training Services Confident in educational performance Confident in capability in self-assessment Date of report: 16 January 2014

2 Contents Purpose of this Report... 3 Introduction TEO in context Scope of external evaluation and review Conduct of external evaluation and review... 5 Summary of Results... 6 Findings... 8 Recommendations Appendix MoE Number: 8708 NZQA Reference: C12413 Date of EER visit: 15 and 16 October

3 Purpose of this Report The purpose of this external evaluation and review report is to provide a public statement about the Tertiary Education Organisation s (TEO) educational performance and capability in self-assessment. It forms part of the accountability process required by Government to inform investors, the public, students, prospective students, communities, employers, and other interested parties. It is also intended to be used by the TEO itself for quality improvement purposes. Introduction 1. TEO in context Name of TEO: Type: Location: Delivery sites: Glasgow Training Services Ltd Private training establishment (PTE) 39 Portview Crescent, New Plymouth Training is delivered at client company premises and at Glasgow Training s Portview Crescent site. First registered: 1 August 1992 Programmes currently delivered: All short courses (one to three days) include one or more unit standards from the New Zealand Qualifications Framework and are individually designed to meet specific client company training requirements, and include aspects of: Drilling Rigging Working at heights Working in a confined space Occupational health and safety practice Scaffolding Code of Practice signatory: Number of students: No Approximately 300 domestic students attended courses over the past 12 months, equating to approximately six equivalent full-time students (EFTS). 3

4 International: nil Number of staff: Scope of active accreditation: One full-time manager, one full-time trainer and one part-time trainer Drilling Hydrocarbon (to level 6) Occupational Health and Safety Practice (to level 4) Scaffolding (to level 2) Glasgow Training is also accredited to deliver a range of individual unit standards within Core Generic, and Fire Rescue and Emergency Response. Distinctive characteristics: Glasgow Training is a small, specialist PTE which has been providing industrial safety training to the engineering, construction and drilling industry sectors in the Taranaki, Wanganui and Taupo regions for over 20 years. Courses are all between one and three days, and are designed to address specific client company employees training needs. Recent significant changes: Previous quality assurance history: None Previous NZQA external evaluation and reviews (EER) (December 2009 and November 2011) both resulted in the judgement that NZQA was Confident in Glasgow Training s educational performance, and Not Yet Confident in the organisation s capability in self-assessment. External moderation of assessment material and assessors decisions has been carried out by the standard-setting bodies NZQA, NZITO (New Zealand Industry Training Organisation), EmQual (Emergency Management Qualifications), The Skills Organisation and MITO (Motor Industry Training Organisation). The standard-setting bodies have noted that all assessor decisions are at the national standard for the materials moderated, but in some cases modifications are required or suggested to assessment materials. 4

5 2. Scope of external evaluation and review Two focus areas were selected: Governance, management and strategy Industry training The governance, management and strategy focus area is mandatory for all EERs. The second focus area in effect includes all training activities offered by Glasgow Training, because this is the only training offered. In effect, this was an evaluation of the whole organisation. 3. Conduct of external evaluation and review All external evaluation and reviews are conducted in accordance with NZQA s published policies and procedures. The methodology used is described fully in the web document Policy and Guidelines for the Conduct of External Evaluation and Review available at: The TEO has an opportunity to comment on the accuracy of this report, and any submissions received are fully considered by NZQA before finalising the report. Two evaluators conducted this evaluation over one and a half-days at the provider s New Plymouth site. The evaluators met with all personnel: the owner and the two trainers. A range of key industry stakeholders, including employers who contract training from Glasgow Training, were also interviewed by phone, to gain their perspective. During the evaluation, the evaluators reviewed a range of key organisational documents, student survey results, assessment records, electronic records and spreadsheets of training activity and achievement. 5

6 Summary of Results Statement of confidence on educational performance NZQA is Confident in the educational performance of Glasgow Training Services Ltd. Glasgow Training is providing effective training that meets the safety and technical requirements of its client companies, and matches the learning needs and learning challenges of its trainees. The vast majority (approximately 96 per cent) of trainees achieve the course outcomes and are awarded between one and two unit standards per course. These are often the first educational achievements for these trainees. This is a significant factor, with attendance at the courses resulting in trainees gaining both improved technical skills and soft or transferable skills such as confidence, self-esteem, self-awareness, safety awareness and observation skills in their workplace, and in improved confidence towards future learning. Glasgow Training s courses are meeting a range of industry needs, such as improved health and safety on worksites, and contribute to client companies improved ability to meet contractual requirements and occupational safety and health compliance requirements, for example with their multinational parent oil exploration companies. Training has resulted in significant changes in attitudes and behaviours in regard to safety on worksites. In one specific case, a company has changed work practices, improving its employees safety as a result of Glasgow Training courses. Courses are structured and delivered in ways that meet trainees learning requirements. This was confirmed through trainee survey results and comments, and feedback from client companies. Many examples were provided where trainees have gained new insights into the theory underpinning a range of safety calculations or practices, as a result of the tutors breaking down complex information or providing practical experience. Trainees are provided with a high level of support to achieve success; tutors persist with them until they achieve competence, often spending extra time with them and providing opportunities for further practical demonstration, until trainees have shown sufficient evidence of competency. The educational performance of Glasgow Training was confirmed and validated at this evaluation through the organisation s data, and is at a similar but slightly higher level to the previous EER in The factors influencing this evaluation reaching Confident rather than Highly Confident in educational performance are linked to some gaps in documentation that would otherwise provide a higher level of surety and robustness. For example, there is a need for more complete documentation of the internal moderation of assessment materials and decisions. Similarly, the organisation s self-assessment processes, while improved significantly from 2011, require some further formalisation to provide objectivity and a basis for further indepth analysis of educational performance. 6

7 Statement of confidence on capability in self-assessment NZQA is Confident in the capability in self-assessment of Glasgow Training Services Ltd. Glasgow Training has a well-developed and consistent culture of reflection and analysis, occurring on a daily basis among the three personnel. It was clear to the evaluators that staff are continually questioning their practices, and exploring options for improvement, for example to course structure, teaching strategies, resources, how best to address trainees barriers to learning, building new resources or improving assessment practice. Glasgow Training has made significant improvements to its self-assessment since the previous EER in Improvements noted at this evaluation were: More complete documentation and analysis, such as programme reviews More comprehensive trainee record-keeping, including paper files and the computer-based spreadsheet Improved reporting of course results and issuing of trainee completion certificates Trialling a new client company survey (still a work in progress at this evaluation) to improve the level of input from client companies towards achieving ongoing improvements to courses. More regular face-to-face visits with client companies supplement this practice, and is already resulting in a good level of stakeholder feedback. The combination of these improvements is providing Glasgow Training with better and more complete information that is more accessible and is being used more to analyse performance and bring about ongoing improvements, such as the building of a confined space area to improve trainees practical exercises. The organisation has strong and effective processes to monitor and review how well its training addresses client company training needs and trainees educational needs and barriers. The effectiveness of this needs assessment was evident through trainee achievement rates, trainee satisfaction rates, client company satisfaction and repeat (and expanding) corporate business. Glasgow Training has a 20-year history in industry training, and is held in high respect in its community and among oil and gas exploration, engineering and construction industries as a provider of quality training. This was evidenced through staff anecdotes, ongoing repeat business and growth in new clients, as well as through stakeholder interviews at this evaluation. This reputation has been built primarily through personal relationships and informal face-to-face meetings with industry. The organisation s self-assessment practices have developed from these informal processes and, as noted, the organisation has developed a more structured approach to self-assessment, including improved record-keeping, documentation and analysis, resulting in improvements. 7

8 Findings How well do learners achieve? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. The context of this training is such that Glasgow Training specifically designs each course (one to three days) to meet client companies requirements, and each course addresses the requirements for one to two unit standards. Tutors noted that they persist with trainees until they achieve competence, and this results in approximately 96 per cent of all trainees achieving success, up slightly from the previous EER. Only eight students in the last 12 months have not achieved competence, and each involved specific situations that the tutors were able to discuss in detail. Unsuccessful students receive a Glasgow Training certificate noting they have attended the course but have not yet achieved all aspects of the unit standard in question. The companies involved view this practice as helping them to provide the on-site support and supervision required to assist employees to develop further and be safe at work. Glasgow Training staff related many anecdotes where trainees and company representatives had noted additional benefits resulting from the training: Aha! moments, and understanding reasons behind safety processes, including improved awareness of safety matters and greater awareness of the safety of their team mates. While client companies contacted by the evaluation team noted that they had not carried out specific studies within their organisations, their impressions were that the training had resulted in wider safety awareness and practical benefits across the organisation, beyond the achievement of the unit standard. Since the most recent EER in 2011, Glasgow Training has put in place improved self-assessment practices, including record-keeping of trainee achievements. This has included an electronic record (student achievement database) as well as improved filing of student assessment material. These (new) practices resulted in at least one unexpected benefit to a client company: Glasgow Training was able to provide documentation of proof of safety training that permitted a key group of workers entry to a hazardous worksite, who would have been prevented entry without these documents. There is still room for Glasgow Training to improve its analysis of trainee achievement data. However, with the small number of trainees involved, the informal analysis combined with the improved documentation noted, is providing 1 The findings in this report are derived using a standard process and are based on a targeted sample of the organisation s activities. 8

9 the organisation with an in-depth level of knowledge and understanding of how well its learners are achieving. 1.2 What is the value of the outcomes for key stakeholders, including learners? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. The principal value added through Glasgow Training is improved worker safety in the oil and gas drilling, engineering and construction industries. The added value goes beyond the course participants, with many anecdotes related at this evaluation of situations where trainees awareness of hazards and risks in the workplace, including the safety practices of their workmates, has been heightened and applied through improved practices. This was noted by Glasgow Training s tutors, trainee survey responses and client company comments as part of Glasgow Training s own self-assessment, providing a strong confirmation of this wider value to businesses and team safety. One particular example of this added value was noted where, following a Glasgow Training course, a client company changed a significant aspect of its safety procedure, thus improving workers safety when working at heights. Client companies contacted at this evaluation noted that the value of the training included meeting contractual requirements (with parent oil exploration multinationals) to provide trainees with training in safe working practices (ensuring employee safety) and meeting compliance with occupational safety and health legislation. In addition, according to a client, the training to the unit standards added another level of external validation of the skills gained. Several client company personnel were contacted at this evaluation and all said they would recommend Glasgow Training to other companies without hesitation. Glasgow Training is proactive in working with its client companies to identify additional training that will benefit them. This was discussed by Glasgow Training staff and confirmed by client companies when contacted by the evaluators, and is perceived as adding value beyond delivering the minimum required. Glasgow Training says it is held in high respect by its clients, and this was confirmed through the interviews with client companies, with one noting, Glasgow Training provides an interactive, engaging learning environment; they are keen and observant, second to none. The evaluators noted a strong culture of reflection, with daily discussions among staff. Staff were able to discuss (with the evaluators) many situations where they had made both small and large changes to add value to the training day by day, for example taking trainees outside the training room to experience correct and 9

10 incorrect harness attachments so that they physically feel the benefit of the correct methods. Glasgow Training surveys all students at the end of each course, and the information gained is reviewed and used well to adjust programmes where this is required. For example, changes have been made to course presentations, handout materials and practical exercises such as in confined spaces, as noted. In response to the 2011 EER, Glasgow Training developed a client survey to gain better documented input. To date, this has not proved a successful strategy, with face-to-face regular discussions being more effective in gaining feedback from companies about the value of training. The client survey is currently being reviewed and is to be combined with the face-to-face meetings to capture the best of the oral feedback and documenting this to a greater degree for future reflection. 1.3 How well do programmes and activities match the needs of learners and other stakeholders? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. As noted, trainees are employees of client companies. Glasgow Training maintains very close contact with its client companies, and this results in short courses designed specifically to meet those clients needs. The training needs assessment is carried out by the client companies themselves, with the programme delivery designed by Glasgow Training. All evidence reviewed at this evaluation identified that the programmes very closely match both the client companies training requirements and trainees needs, including individual trainees specific requirements and barriers and challenges to learning. This is further explored in section 1.4 of this report. All training is well designed and addresses the unit standard outcome statements, as well as addressing any company-specific training, for example using company equipment such as breathing apparatus, overhead cranes and harnesses. The effect is that current industry good practice is covered through addressing the unit standards (written and designed by standard-setting bodies to stay current with industry good practice), within the context of client companies specific technical training needs. Trainee course surveys indicate a very high level of satisfaction with the training matching their needs, with very few suggestions for changes or improvement. However, Glasgow Training has made additions and changes to its programmes and facilities in response to trainee feedback, for example constructing an enclosed area for developing the practical skills for working in confined spaces. 10

11 Glasgow Training receives pre-course information from companies about trainees specific learning needs, and this is supplemented through tutor observations during the course. All matters reviewed at this evaluation indicated that trainees personal or learning barriers are well addressed through close one-to-one attention and additional time as required. The owner of Glasgow Training regularly visits client companies to monitor their satisfaction with the training and to identify where emerging training needs are not yet being addressed. This process is effective and results in adjustments to training to match clients changing needs. To a large extent, Glasgow Training s self-assessment processes are oral and relationship-based rather than highly documented. However, as noted, trainee satisfaction surveys are well documented and reviewed, and show that trainees needs are well catered for and met. The informal nature of Glasgow Training s selfassessment is in the main fit for purpose for an organisation with two and a half fulltime staff and only six equivalent full-time students. However, the organisation has developed further recording and documentation to strengthen its self-assessment practices, although, as noted, client company surveys were trialled and found not to be effective. This feedback process is being adapted to provide better documentation, which is likely to support the organisation to reflect more objectively on its strengths and areas for improvement. 1.4 How effective is the teaching? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. Glasgow Training has recruited two highly motivated and passionate tutors with significant and relevant industry experience and ongoing links to the industry sectors in which they are teaching, which provides opportunities for them to stay current with industry changes. The tutors have attended courses related to adult teaching, and during this evaluation were able to discuss a wide range of teaching strategies and motivational techniques reflecting good adult teaching practice. Tutors break up courses into manageable chunks of learning, intermixing theory and practical elements, ensuring trainees have sufficient time to demonstrate their understanding of the background theory as well as their competence with specific equipment. Many examples were provided where trainees who had previously not achieved well at school or in other training had gained a clearer understanding of the background reasons for, or explanations of, the underpinning theory, which can involve relatively complex mathematics and physics, such as dealing with the use of cranes, loads, stresses and strop angles. Tutors related a number of incidents where they had met past trainees in the street and other social settings and were personally thanked for the quality of the training 11

12 and the difference they had made in some cases to trainees lives. Client company feedback to Glasgow Training has also noted that they prefer to wait for a space to put their employees in training with Glasgow Training rather than send them to alternative courses, because of the quality of the teaching and learning. A gap in both educational performance and self-assessment relates primarily to the level of documentation. The evaluators received extensive information from the tutors, client companies and standard-setting bodies that provided reasonable assurance that the trainers are addressing the requirements and outcome statements of the unit standards. There was no evidence to suggest that trainees were at risk of entering the workplace unprepared or unsafe. However, Glasgow Training does not yet have complete documentation of internal moderation of its assessment tools and assessors decisions. The risk here is in terms of staff ability to objectively review the assessment material and assessors decisions, to identify areas to improve their practices and avoid repeating past mistakes. The mitigating factor is that there are only two tutors and six equivalent full-time students, and there are regular discussions and reflection between the two tutors, as well as contact with companies, and external moderation undertaken by the standardsetting bodies. The size of this training organisation is such that the manager and trainers observe each other and provide a strong level of peer review and support, identifying areas for improvement and mitigating any risks. 1.5 How well are learners guided and supported? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. Learners are provided with appropriate information through their own company and by Glasgow Training prior to attending their short courses, and are comprehensively supported while on the course. This was borne out in discussions with tutors, and noted in student surveys and client companies comments. Tutors discussed a range of processes they use to socialise the group at the beginning of a session and to avoid causing any individual to feel foolish in the group, maintaining their mana and building their confidence. As noted, tutors will work with trainees until they are able to show competence over time and in a variety of situations. There is a clear passion among the staff for working with their trainees, who have usually not experienced educational success in the past and may enter courses with some trepidation. The high success rate of the courses and the high esteem in which Glasgow Training is held among its client companies and the local community attests to the support and guidance provided to the trainees, often extending beyond the completion date of the course. Tutors spoke of actively supporting the trainees to build a culture of responsibility and care, and of trainees observing their workmates to identify potential risks in the 12

13 workplace to pre-empt rather than merely respond to accidents. This was commented on by stakeholders, who see an improved awareness and effective use of safety practices and personal protection equipment. The evaluators consider the organisation to be strong in its knowledge and understanding of how well it is meeting trainees support and guidance needs and continually improving its performance. However, as noted, some improvement in documentation and record-keeping would support a more robust process. 1.6 How effective are governance and management in supporting educational achievement? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. Glasgow Training management, in effect the single owner and his two staff, have a clear and effective focus on supporting trainee achievement. This was evidenced at this evaluation by the 96 per cent trainee achievement rate (a slight improvement on the previous EER), significant ongoing client company repeat business, and newly developing business from word-of-mouth referrals. The organisation has limited but fit-for-purpose documentation stating its intent and purpose. As with many small-to-medium enterprises in New Zealand, the organisation s philosophy and mode of operation are largely lived day to day, with emerging issues being discussed among the three personnel, and changes made to improve processes and practices on a daily basis. The organisation has been delivering training for approximately 20 years, and during this time has gained respect within the oil and gas exploration, engineering and construction industries within its region. This was well evidenced at this evaluation through conversations with staff, student evaluations, the organisation s documentation and interviews with client company personnel. This information was also essential within the organisation s own self-assessment practices. This indicates that the organisation s current processes are fit for purpose for the size and complexity of the organisation, and no significant areas of concern were noted by the evaluators. However, as noted, considerable benefit may be gained through more extensive or complete documentation, to enable the organisation to reflect more objectively on its strengths and areas for improvement. The owner has hired well-experienced and motivated tutors who have gained the respect of client companies and trainees. The organisation is well resourced in terms of training space and equipment, appropriate to the nature of the industries and technical requirements of companies. Much of the training delivered is on client company worksites using their equipment, and as such effectively addresses the technical training needs at each site. 13

14 Organisational self-assessment is primarily built on the strength of personal relationships with client companies and the community, and as such is very strong. This was confirmed through all aspects of this evaluation. The evaluators observed a well-developed culture of reflection, and analysis of and robust debate about all aspects of the organisation s operations. The gap in self-assessment practices is, as noted, in the reliance on organisational discussions and face-to-face relationships with client companies, which has the potential for losing organisational knowledge and missing opportunities to improve objective reflection where matters are less than fully documented. Glasgow Training has made a range of improvements to its self-assessment since the previous EER in Improvements noted at this evaluation were: More complete documentation and analysis, including programme reviews Improved trainee record-keeping, including paper files and computer-based records Improved reporting of course results and issuing of trainee completion certificates Trialling a new client company survey (still a work in progress at this evaluation) to improve the level of input from client companies towards ongoing improvements to courses. However, this aspect is also supplemented by more regular face-to-face visits with client companies, which is resulting in an improved level of stakeholder feedback. The combination of these improvements is providing Glasgow Training with better and more complete information that is more accessible, and is being used more often to analyse its performance and bring about ongoing improvements. The evaluation team saw sufficient evidence that Glasgow Training has effectively taken steps to improve its self-assessment processes and practices, and is using the results to bring about meaningful improvements. Educational performance continues at a slightly improved level than was evident at the previous evaluation. Glasgow Training s educational performance for this key evaluation question is rated as Good rather than Excellent. The primary reason is the level of documentation and formal analysis, as noted. Arguably, the most important area requiring improvement is in the documentation of internal moderation of assessment materials, to confirm they are fit for purpose, and the moderation of assessors judgements, to confirm they are consistent, reliable and at the national standard. 14

15 Focus Areas This section reports significant findings in each focus area, not already covered in Part Focus area: Governance, management and strategy The rating in this focus area for educational performance is Good. The rating for capability in self-assessment for this focus area is Good. 2.2 Focus area: Industry training The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Good. 15

16 Recommendations NZQA recommends that Glasgow Training: Continue to develop the documentation of trainee achievement demographics (by gender, ethnicity, etc), to provide more complete data for analysis and improvement Continue to develop the documenting of internal moderation of assessment materials and of assessors decisions for reflection and analysis Continue to explore options to gather documented feedback from client companies, to improve the objectivity of reviewing how well courses match their needs Continue to develop organisational self-assessment practices, and ways to document information currently held anecdotally by individuals, to improve the identification of patterns or trends over time. 16

17 Appendix Regulatory basis for external evaluation and review External evaluation and review is conducted according to the External Evaluation and Review (EER) Rules 2013, which are made by NZQA under section 203 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. Self-assessment and participation in external evaluation and review are requirements for maintaining accreditation to provide an approved programme for all TEOs other than universities. The requirements are set through the NZQF Programme Approval and Accreditation Rules 2013, which are also made by NZQA under section 203 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. In addition, the Private Training Establishment Registration Rules 2013 require registered private training establishments to undertake self-assessment and participate in external evaluation and review, in accordance with the External Evaluation and Review Rules (EER) 2013, as a condition of maintaining registration. The Private Training Establishment Registration Rules 2013 are also made by NZQA under section 203 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. NZQA is responsible for ensuring non-university TEOs continue to comply with the rules after the initial granting of approval and accreditation of programmes and/or registration. The New Zealand Vice-Chancellors Committee (NZVCC) has statutory responsibility for compliance by universities. This report reflects the findings and conclusions of the external evaluation and review process, conducted according to the External Evaluation and Review (EER) Rules The report identifies strengths and areas for improvement in terms of the organisation s educational performance and capability in self-assessment. External evaluation and review reports are one contributing piece of information in determining future funding decisions where the organisation is a funded TEO subject to an investment plan agreed with the Tertiary Education Commission. External evaluation and review reports are public information and are available from the NZQA website ( The External Evaluation and Review (EER) Rules 2013 are available at while information about the conduct and methodology for external evaluation and review can be found at 17

18 NZQA Ph E qaadmin@nzqa.govt.nz 18