Report of External Evaluation and Review Quality Environmental Consulting Limited Confident in educational performance Confident in capability in self-assessment Date of report: 3 September 2014
Contents Purpose of this Report... 3 Introduction... 3 1. TEO in context... 3 2. Scope of external evaluation and review... 5 3. Conduct of external evaluation and review... 5 Summary of Results... 6 Findings... 8 Recommendations... 16 Appendix... 17 MoE Number: 7880 NZQA Reference: C15037 Date of EER visit: 12 and 13 June 2014 2
Purpose of this Report The purpose of this external evaluation and review report is to provide a public statement about the Tertiary Education Organisation s (TEO) educational performance and capability in self-assessment. It forms part of the accountability process required by Government to inform investors, the public, students, prospective students, communities, employers, and other interested parties. It is also intended to be used by the TEO itself for quality improvement purposes. Introduction 1. TEO in context Name of TEO: Type: Location: Delivery sites: Quality Environmental Consulting Limited (QEC) Private training establishment (PTE) 62D Victoria Street, Cambridge Training takes place at the above site, online or in the workplace nationwide. First registered: 5 January 2009 Courses currently delivered: Code of Practice signatory?: Number of students: Number of staff: Scope of active accreditation: Courses in handling and managing hazardous substances, and health and safety in the workplace No Domestic: 71 students Six full-time equivalents Programme accreditation Training Scheme: Approved Handler Hazardous Substances (level 3) Domains: Occupational Health and Safety Practice (to level 4) Workplace Health and Safety Management (to level 4) 3
Safety and Legislation for Energy and Chemical Plant (to level 4) Hazardous and Special Waste Management (to level 4) Solid Waste Disposal (to level 4) Solid Waste Handling (to level 4) Unit standard 8086 Demonstrate knowledge required for quality auditing (level 4) Distinctive characteristics: Recent significant changes: Previous quality assurance history: QEC provides specialised regulatory and technical consultancy, working across the fields of environment, safety, hazardous substances, food safety and quality management. Short-course training is provided in three delivery modes on site (QEC premises), online and in the workplace. Other than the directors, all staff have been appointed in the past year. There have been some short-term appointments to the key position of training manager, and recruitment for a tenured position is under way as part of the PTE s growth strategy. The previous NZQA external evaluation and review (EER) of QEC was in June 2010. At the time, NZQA was Confident in the educational performance of the PTE and Confident in its capability in self-assessment. National external moderation is exempt for 2013 and 2014, following good results in 2012. Health and safety is accredited to tertiary level for the ACC Workplace Safety Management Practices standard. QEC is an accredited agent for the Warranty Standards Certification Programme. Other: QEC is an approved trainer for the Plumbers, Gasfitters and Drainlayers Board. 4
2. Scope of external evaluation and review The scope for the EER consisted of the mandatory focus area of governance, management and strategy and one training area. Focus area selected: Hazardous Substances The current course delivery is mostly within the hazardous substances area and comprises the two-day Approved Handler: Hazardous Substances course, as well as half-day refresher/relicensing courses in special chemical areas. 3. Conduct of external evaluation and review All external evaluation and reviews are conducted in accordance with NZQA s published policies and procedures. The methodology used is described fully in the web document Policy and Guidelines for the Conduct of External Evaluation and Review available at: http://www.nzqa.govt.nz/providers-partners/registration-andaccreditation/external-evaluation-and-review/policy-and-guidelines-eer/introduction. The TEO has an opportunity to comment on the accuracy of this report, and any submissions received are fully considered by NZQA before finalising the report. QEC submitted a brief self-assessment summary prior to the EER, and a prescoping telephone conversation was held with the lead evaluator and the managing director of the organisation. The evaluation team of two visited the Cambridge premises over two days. Interviews were conducted with the two owners who are also the directors, the managing director, the health and safety consultant/trainer, the office coordinator, and students. A range of documentation was reviewed, including course evaluations, meeting minutes, moderation records, handbooks, office procedures manual, and internal audit reports. The evaluators also observed the use of online task and time management tracking software. 5
Summary of Results Statement of confidence on educational performance NZQA is Confident in the educational performance of Quality Environmental Consulting Limited. QEC offers a wide range of specialised short courses for industry. The organisation is strongly focused on successful outcomes for learners and has a clear understanding of the workplace requirements and risks associated with regulatory and legislative compliance training. All learners successfully complete courses and acquire relevant and up-to-date workplace safety skills. At the time of the EER, QEC had not yet begun the process of systematically collecting and analysing data on its Māori and Pasifika learners. Discussion with stakeholders confirmed their high regard for the quality and integrity of QEC training. This organisation has developed a reputation and very strong business relationships and networks for consistently delivering rigorous and relevant training that is responsive to the specialised needs of the compliance, legislative and regulatory safe workplace industry. Staff are passionate about their work and are well connected to key national organisations and local employer groups. Clients and employers recognise the integrity and quality of QEC training and value the focus on the needs of the end-user and their individual workplace requirements. Staff are recognised as highly skilled and experienced in workplace safety. Business growth and opportunities are the result of repeat business and referrals from clients from other businesses. QEC provides a coherent set of services in consultancy, quality auditing and the ongoing training for these areas. Overall, the evaluators found that QEC is focused on the needs of the learners, quality training and positive outcomes. While there are opportunities for improvement to enhance the effectiveness of processes contributing to learning and learner support, no major weaknesses were identified. 6
Statement of confidence on capability in self-assessment NZQA is Confident in the capability in self-assessment of Quality Environmental Consulting Limited. There are a range of mechanisms used to understand the needs of learners and stakeholders. Systematic learner and employer feedback, observations in the workplace, and significant engagement with client companies and businesses are examples of these mechanisms. The resulting information is used to inform improvements. QEC has an effective and systematic organisational online process for capturing feedback and areas for improvement. There is a strong focus on managing feedback, priorities and improvements through the use of the online task management and monitoring tool. This enables a systematic and purposeful capturing of feedback and ideas for improvement that are then allocated time, resources and priority, followed by monitoring of completion within required timeframes. The reports from this tool are regularly used by staff to review progress. Course review and development is occurring on an ongoing basis, with staff continuously reflecting on their practice, quality of course materials and processes. There is an opportunity for systematic analysis and self-assessment of completion data at organisational level to understand the achievement of learners as well as trends and benchmarks. There are also opportunities to evaluate the effectiveness of changes and improvements made at an organisational level. 7
Findings 1 1.1 How well do learners achieve? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Adequate. Learners acquire relevant workplace safety skills and knowledge from their training at QEC. Learners complete this training because they need to hold relevant Approved Handler and other licences in their workplace. Staff provided evidence that all students pass we keep working with them until they pass. QEC uses benchmarks for competency which align with regulatory and licensing requirements to ensure that the completed training has integrity and any risks are well managed. Discussion with stakeholders confirmed their high regard for the quality and integrity of learner achievement from this organisation. QEC monitors closely the learner achievement results for each course and uses this data to improve the way learners are taught and supported at the next delivery of that course. The evaluation team noted that there is an opportunity for systematic analysis and self-assessment of completions data at organisational level to understand the achievement of learners. There would be benefit in evaluating differences in completion times, the achievement of priority groups including Māori and Pasifika learners and other cultures, re-assessments, and patterns of achievement of online learners, on-site learners and those in workplace-based training. 1.2 What is the value of the outcomes for key stakeholders, including learners? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. The evaluation team is confident that there is value for learners who complete training courses at QEC. 1 The findings in this report are derived using a standard process and are based on a targeted sample of the organisation s activities. 8
Employers value the outcomes of QEC training in hazardous substances and have confidence in the visible improvement in safe working practices, and the way that trained students use their new knowledge and skills to train others in their workplace. It was evident that QEC actively engages with employers to provide consultancy and training to meet employer needs and industry compliance and regulatory standards. Stakeholders indicated that QEC regularly provides updates on regulatory and compliance changes, and improved workplace safety practices and risks through audits, seminars, workshops, site visits, phone calls and targeted newsletters. Learners are fully employed and motivated to complete training alongside their employment to ensure currency of their licence and workplace safety knowledge and understanding. The training aligns with employee and supervisor responsibilities in the workplace and the risks associated with workplace safety. Management has active affiliations with national workplace safety organisations: the New Zealand Institute of Hazardous Substances Management at executive committee level, and with the New Zealand Institute of Safety Management. These affiliations enable the contribution of knowledge and experience at national and regional levels to ensure QEC is at the forefront of industry alignment with current and future compliance, and regulatory and legislative requirements and developments. External stakeholders including client companies, business networks, past students, and health and safety groups are kept well informed of QEC developments, updates to legislation, current industry issues, and consultancy and training opportunities through visits, phone calls, the website and targeted newsletters for specific groups. Employers highly value that QEC responds to their specific needs: [QEC] understand what our workplace issues are and are willing to deliver the training in our workplace to directly meet our specific needs. The evaluators heard from client companies and employers that they highly value the way QEC delivers training and assessment that has integrity and currency and is engaging for the learners. Almost all QEC training is repeat business by employers for whom QEC is the preferred service provider, and client referrals from word-of-mouth recommendations. QEC is also meeting the needs of client businesses when training gaps are identified during consultancy and audit activities. Formal feedback opportunities from employers were not being used consistently and have been replaced by direct one-to-one contact by phone or visits. This has resulted in an improved level of feedback which is recorded in the online task management and monitoring system. Improvements are identified and prioritised in a comprehensive job log that provides actions, priorities, allocations and timeframes which can be viewed by all staff and effectively monitored to completion. 9
Workplace performance and competence of past students is observed frequently by QEC staff during engagement in consultancy and auditing activities in workplaces. The evaluators heard that past students actively engage with QEC staff during workplace visits to share how they have used the learning from the training to improve the storage of chemicals, or the processes of safe handing, and how they have informed other staff of improved workplace safety practices. 1.3 How well do programmes and activities match the needs of learners and other stakeholders? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. The needs of the key stakeholders of QEC are being well met by the courses on offer. The provision of training online, on site or in the workplace meets the needs of a diverse range of learners, most of whom are in employment. Where training is delivered in individual workplaces, course design is customised to the needs of the particular employer. Initial consultation with the employer is used to verify training needs and development of course materials tailored to the context and the standard operating procedures of the workplace. The employer gives final approval of the training materials prior to delivery. Course materials and delivery strategies are adjusted to meet the needs of learners. In a pre-delivery scoping meeting, information on the learner profile is shared between trainers to ensure that the course materials are current and relevant to the learner group. Improvements identified from trainer and student feedback following previous delivery of similar courses are used to improve course materials, resources and delivery. For example, the trainers photograph workplaces during consultancy and audit visits, and these are used in scenarios in the materials for relevant courses. Scoping meetings are embedded in the preparation phase of course delivery for both the on-site courses (open to employees from a range of businesses) and customised workplace-based training. Changes in legislation are monitored regularly, notified to staff, added to the online task management system, and included in newsletters. Course materials are systematically checked to ensure they are updated with legislative and regulatory changes. QEC anticipates and responds to change through the identification of new training opportunities. Management undertakes research of current reports and findings relating to workplace safety and regulatory issues. For example, the findings of the Pike River (mining disaster) report have informed a renewed focus on training for 10
supervisors and team leaders a new briefing course on health and safety legislation has been developed for management and supervisors. Hazardous substance training courses are delivered on site, using organisations standard operating procedures and real-life scenarios that are relevant to the industry and the chemicals used. The evaluators heard that a chemical spill was observed at a workplace and the incident was developed as a case study to be used at the next approved handler course for that chemical. Industry feedback confirms that QEC is highly responsive to the needs of its clients. The learner and client feedback from every training course is recorded in the online management system. This feedback is provided to management and the trainer for discussion on how well the training met the needs of the learner group. The outcomes of this discussion are recorded in a systematic way to ensure they are included in the pre-scoping meeting for the next delivery of that course. For instance, improvements were made to include binding and colour coding in the course workbook following learner feedback on difficulties understanding the course documentation. Both management and trainers were able to show multiple examples of how course content and delivery methodology had been changed as a result of feedback from learners and clients. 1.4 How effective is the teaching? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. Improving teaching and learning is a key focus for QEC. Courses have changed from teaching theory and practice separately to interactive and integrated learning activities using scenarios, problem-solving and group work. Learners are positive and enjoy the engaging delivery methods. The training at QEC is effective and characterised by highly qualified staff who are passionate about the subject. Course feedback is recorded and analysed to identify areas for improvement which are then logged into the online task management software for action. The evaluation team heard that trainers meet with management to discuss the feedback and identify improvements, which are systematically recorded in the course materials file so that updating is completed for the next delivery of the course. This reflective practice is highly valued by the staff and management. New staff go through staged internal training to build subject knowledge and teaching skills by first developing course materials, followed by observations of experienced trainers (co-teaching and supervised teaching), to taking training sessions on their own. The process has been developed in recognition of the risks 11
associated with hazardous substances training. The PTE is highly aware of the workplace risks in handling hazardous substances and will not pass a learner unless the employer formally verifies that the learning from the course is evident in the student s competence in the workplace. Meetings are held to share feedback from learners and employers, to update staff on legislative changes, and to discuss opportunities for improvement. Sharing complaints and compliments is embedded in the meeting agendas. Student feedback is embedded as part of the online, on-site and workplace-based courses and is systematically collected, analysed and used to make worthwhile improvements to both course and delivery strategies. Peer observation is completed regularly for all trainers and is an integral part of the ongoing improvement of delivery. The online task management and monitoring tool provides effective graphic summaries indicating strengths and areas for improvement for each trainer. Trainers reflect on the outcomes and discuss the feedback and improvements which then lead to improvement tasks being identified and recorded. The organisation could explore further professional development from learning and teaching specialists to extend learning and teaching strategies to support the change towards a more interactive learning environment. Identifying delivery strategies to meet the cultural needs of learners appears to be a gap. Online delivery is increasing with online enrolments for short, half-day refresher/relicensing courses reaching 50 per cent of the delivery. Students indicated a high level of satisfaction with the online course delivery option which enabled them to complete the refresher/relicensing courses out of work hours and at their own pace. The evaluators heard that assessment processes are robust, and that reassessment practice was used to ensure authentic learning and assessment to meet required standards of competency. Employers confirmed that the students who had completed courses from QEC were able to use the learning in their work to create safer workplace practices. External moderation evidence indicated that assessor judgements are upheld and meet the national standard. 1.5 How well are learners guided and supported? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. The level of support provided is appropriate to the learner profile, which is comprehensively established and planned for prior to delivery. This support is consistently available to learners whether in employment, on half-day courses, not 12
being assessed for unit standards, or in employment undertaking compliance training that is necessary for workplace safety. The learners are motivated they need the certification and licensing for their workplace roles. A participant information booklet and application form provide clear information for students about expectations for on-campus students and requirements of the course. Additional information is required on the hazardous substances enrolment form to provide details of relevant practical work experience to ensure the student has the necessary background to complete the training. Online learning support, both IT-related and academic, is available. Students confirmed that this support is accessible and useful. Intentionally small class sizes of between four and 10 learners per course ensures appropriate individual attention is given. Peer support is used as a strategy to assist learners in the classroom, for example linking an experienced student with a student who is perceived to be struggling. Trainers provide learners with support through additional class time with the trainer, one-to-one tutorial time, and phone call reminders to monitor and support progress in post-course assessment. The evaluation team heard that the trainers expect that all students will pass the course, so additional learning support is available until the students reach the required competency levels. Newsletters tailored to specific roles in workplaces are sent to employers and past students to provide information on changes to legislation, new courses, and related local news. This purposeful, ongoing communication is one of the strategies used to attract repeat business and give support to relevant businesses and students. Employees are sent reminders of due dates for relicensing and refresher training to ensure that knowledge and skills are current. There exists an opportunity for QEC to increase understanding and awareness of how the cultural needs of the students could be more appropriately supported. 1.6 How effective are governance and management in supporting educational achievement? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. QEC is managed effectively by two highly experienced directors, one with highlevel qualifications and experience in the chemical industry, environmental and workplace safety management and auditing, and one with a strong business management background. External consultants are engaged in areas where additional expertise is required to assist with business growth strategies. 13
Management has a clear vision: to provide high-quality, relevant workplace safety regulatory and compliance training which is tailored to meet the needs of individuals and businesses. There was good evidence that the organisation is effective in building and retaining strong business relationships and networks. The ongoing success of QEC is built on referrals, repeat business and being able to provide training where gaps are identified in the consultancy and audit services. The evaluators also heard evidence that QEC was the provider of choice for training in the specialised areas it offers. New business opportunities are being explored through collaboration with other TEOs in the region. The evaluation team saw evidence of comprehensive and well-organised systems and processes including the management of updates to relevant legislation. Staff are clear about their roles, delegations and lines of accountability. All staff actively use the online task and time management tracking software to monitor improvements, priorities and staff allocations. This tool is central to the management of self-assessment and improvement in the daily operations of the organisation. Further testament to the effectiveness of the systems and processes is that the new staff understand and are fully engaged with them. The directors are well respected by staff. The strengths-based formal performance review process and the open consultation when new initiatives are being considered are examples of effective staff management processes. There is an opportunity to evaluate the effectiveness of improvements at an organisational level and to undertake systematic analysis and self-assessment of completions data to understand the achievement of learners. 14
Focus Areas This section reports significant findings in each focus area, not already covered in Part 1. 2.1 Focus area: Governance, management and strategy The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Good. 2.2 Focus area: Hazardous Substances The rating in this focus area for educational performance is Good. The rating for capability in self-assessment for this focus area is Good. 15
Recommendations NZQA recommends that Quality Environmental Consulting Ltd: Strengthen data analysis and self-assessment of performance information at an organisational level. Analyse more systematically the achievement of Māori and Pasifika students. Explore further professional development to extend learning and teaching strategies for trainers to support the direction towards a more interactive learning environment. Strengthen understanding of the cultural needs of learners to inform teaching, guidance and support. 16
Appendix Regulatory basis for external evaluation and review External evaluation and review is conducted according to the External Evaluation and Review (EER) Rules 2013, which are made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. Self-assessment and participation in external evaluation and review are requirements for maintaining accreditation to provide an approved programme for all TEOs other than universities. The requirements are set through the NZQF Programme Approval and Accreditation Rules 2013, which are also made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. In addition, the Private Training Establishment Registration Rules 2013 require registered private training establishments to undertake self-assessment and participate in external evaluation and review, in accordance with the External Evaluation and Review Rules (EER) 2013, as a condition of maintaining registration. The Private Training Establishment Registration Rules 2013 are also made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. NZQA is responsible for ensuring non-university TEOs continue to comply with the rules after the initial granting of approval and accreditation of programmes and/or registration. The New Zealand Vice-Chancellors Committee (NZVCC) has statutory responsibility for compliance by universities. This report reflects the findings and conclusions of the external evaluation and review process, conducted according to the External Evaluation and Review (EER) Rules 2013. The report identifies strengths and areas for improvement in terms of the organisation s educational performance and capability in self-assessment. External evaluation and review reports are one contributing piece of information in determining future funding decisions where the organisation is a funded TEO subject to an investment plan agreed with the Tertiary Education Commission. External evaluation and review reports are public information and are available from the NZQA website (www.nzqa.govt.nz). The External Evaluation and Review (EER) Rules 2013 are available at http://www.nzqa.govt.nz/assets/about-us/our-role/rules/eer-rules.pdf, while information about the conduct and methodology for external evaluation and review can be found at http://www.nzqa.govt.nz/providers-partners/external-evaluation-andreview/policy-and-guidelines-eer/introduction/. 17
NZQA Ph 0800 697 296 E qaadmin@nzqa.govt.nz www.nzqa.govt.nz 18