Report of External Evaluation and Review

Transcription

1 Report of External Evaluation and Review TOSHA Limited Highly Confident in educational performance Highly Confident in capability in self-assessment Date of report: 28 January 2011

2 Contents Purpose of this Report...3 Introduction TEO in context Scope of external evaluation and review Conduct of external evaluation and review...4 Summary of Results...5 Findings...7 Recommendations...12 Further Actions...12 Appendix...13 MoE Number: 9682 NZQA Reference: C02361 Date of EER visit: 28 and 29 October

3 Purpose of this Report The purpose of this external evaluation and review report is to provide a public statement about the Tertiary Education Organisation s (TEO) educational performance and capability in self-assessment. It forms part of the accountability process required by Government to inform investors, the public, students, prospective students, communities, employers, and other interested parties. It is also intended to be used by the TEO itself for quality improvement purposes. Introduction 1. TEO in context Location: Type: 10 Devon Street, New Plymouth 2 Cawley Street, Auckland Private training establishment (PTE) First registered: 1994 Number of students: Number of staff: Scope of active accreditation: Sites: Distinctive characteristics: Recent significant changes: Previous quality assurance history: Other: concurrently on short courses throughout year Eight (four full-time, four part-time) Range of unit standards assessing health and safety competency in many contexts. Most courses delivered on client sites throughout the country, some generic courses in Taranaki. In its educational function, TOSHA Limited specialises in safety, health, and environment in a wide range of industrial sites and applications, in particular focusing on work at heights, hazardous substances management, fire safety, first aid, and driver education. Recently, there was a reduction in the engagement of contracted staff, but skilled teachers who are acknowledged experts in their fields were retained. The previous quality assurance visit by NZQA was an audit in 2008, and all relevant requirements were met. Education and training comprises 60 per cent of the work of this PTE. Another major focus is audit, a prominent feature in its own management. 3

4 2. Scope of external evaluation and review The agreed scope of the external evaluation and review of TOSHA Limited (TOSHA) included the following mandatory focus area: Governance, management, and strategy. The programme focus areas were: Elevated Platforms Hazardous Goods. 3. Conduct of external evaluation and review All external evaluation and reviews are conducted in accordance with NZQA s published policies and procedures. The methodology used is described fully in the document Policy and Guidelines for the Conduct of External Evaluation and Review available at: The EER was conducted over 12 hours by one lead evaluator and one external evaluator. They interviewed the managing director, training coordinators, quality systems controller, a teacher, and five external stakeholders. TOSHA has had an opportunity to comment on the accuracy of this report, and submissions received have been fully considered by NZQA before finalising the report 4

5 Summary of Results Statement of confidence on educational performance NZQA is Highly Confident in the educational performance of TOSHA. TOSHA distinguishes itself not only for the excellent achievement of its learners in a series of short courses, but also in its ability to create a working culture that mixes uncompromising attention to detail with an unusual degree of democratic involvement in course management. The outcome is high quality learning, as evidenced in achievement records, the satisfaction of participants, the respect of client companies, and their willingness to re-engage TOSHA for further training. TOSHA has an ongoing, proactive involvement with industry and with its standard-setting bodies, consulting, advising, constructing and adapting courses and standards to apply to a wide range of contexts and codes of practice. Statement of confidence on capability in self-assessment NZQA is Highly Confident in the capability in self-assessment of TOSHA. Auditing is a significant part of the TOSHA portfolio, and self-assessment is conspicuous in having its own quality assurance programme. The organisation s self-assessment is a detailed and unremitting process, based on clearly identified outcomes and a comprehensive approach to every part of the process. This involves ongoing critical analysis of every activity, removal of redundancies and immediate and monitored remedial response to failure, to whatever degree. This process is performed in a democratic management culture which urges enquiry, debate, and a drive for improvement in all outcomes. Little is left to chance, but there is potential for further improvement. Aggregated achievement records across all courses, notwithstanding the variety of contexts and contents, could provide internal benchmarks on overall performance and give valuable information about trends. This information is already analysed course by course. Aggregation would be a simple, and potentially useful, further step in self-assessment. Another development might involve a more aggressive approach to external professional development for teachers. Funded training is available for relevant and proven professional development courses, so this would be a minor change to a self-assessment programme of rare thoroughness and steely focus on the utility of everything written and done. Finally, while there is ample anecdotal information on improvement to the personal welfare of participants, the major focus in a series of very short courses has been on occupational safety and health, with a positive, but informal, interest in the personal welfare of learners. 5

6 Findings1 1.1 How well do learners achieve? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. Assessment records seen by the EER team indicate 90 per cent or greater first-time competency outcomes, and around 90 per cent passes in theory. These are excellent outcomes in a complex teaching operation involving a wide variety of clients with unique situations and needs, in courses with a maximum duration of three days. A fund of anecdotal evidence exists on the added value of the learning to candidates lives and careers. This feature was discussed at the EER, and the managing director has since made amendments to participant feedback to collect more formal data on learners personal achievements. Courses are reviewed in an exhaustive and informative self-assessment process involving formal feedback from course participants, clients, parent companies, and the trainers. Participant feedback is collected via an 18-point questionnaire, inviting detailed responses and summary comment. The teacher s report is essentially a self-assessment document, supplemented by a signed and dated note on remedial action taken where required. Immediate remedial action is the rule on any matters of concern. Training coordinators monitor courses daily and discuss outcomes and issues at weekly management meetings. All evaluations are charted and returned to the client company. Self-assessment, exemplary in its range and application, might, as already mentioned, be enhanced with an integrated report of achievement in all courses, to provide benchmarks year by year and to give warning of developing trends in learning outcomes. This information is readily available, since outcomes are charted course by course, but an integral report might informatively present the global picture. Given the prodigious variety of individual short courses, uniquely structured for clients in diverse industries, the managing director is uncertain of the value of an integral record, but is planning overall exception reporting to give early warning of possible negative trends in learning. Participant, client, and company feedback is uniformly positive on the quality and relevance of learning. Clients interviewed by the EER team corroborate this, some of them commenting on dramatic improvements in individual and company performance. 1 The findings in this report are derived using a standard process and are based on a targeted sample of the organisation s activities. 6

7 1.2 What is the value of the outcomes for key stakeholders, including learners? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. TOSHA is directly and continually involved with client companies and industry training organisations (ITOs) in advising, selecting, adapting, or in one case amalgamating standards to match the needs of clients, ITO standards and codes of practice, and the requirements of the New Zealand Qualifications Framework (NZQF). Training is only undertaken when the client is able clearly and concisely to answer the question: What exactly do you want? A proposal is then sent to the client, for confirmation or otherwise of its suitability. TOSHA has an extensive catalogue of unit standards, and courses are designed and delivered only when confirmation is received that the client has ITO approval for the standards involved. Every course is set up according to an unvarying protocol, detailing content, delivery, timing, assessment, and subsequent follow-up, and any special conditions and needs. A confirmation letter to the client specifies all details of the transaction. It is monitored step by step by the training coordinator, in day-by-day consultation with the trainer and the client. With increasing focus on the value of the learning to the personal lives of participants, personal outcomes will become a more prominent feature of self-assessment. Client consultation continues after completion, both informally and formally, to ensure that no part of the process is without comment from the recipient. The feedback received from stakeholders by the EER team emphatically confirmed the relevance and meticulous focus of TOSHA s programmes to their own contexts. 1.3 How well do programmes and activities match the needs of learners and other stakeholders? The rating for performance in relation to this key evaluation question is Excellent The rating for capability in self-assessment for this key evaluation question is Excellent. Public courses with generic content are periodically offered, but the preferred teaching model is to deliver at the working site of the learners, where materials, working conditions, situations, and site materials are resources for teaching, ensuring maximum relevance to the employment situation. Sites are examined prior to a proposal being made and are prepared as required for the learning event. Needs assessment for learners is discussed prior to the course with client managers, who nominate participants and are welcome to attend courses, both for their critical review and to enhance their own learning. Needs assessment in such short courses is moment by moment, requiring personal attention to each of a small group of participants. Teachers use a variety of delivery methods to accommodate a range of learning styles and approaches and to overcome resistance in 7

8 participants with earlier negative experiences of learning. Ad hoc adaptations are made as needed during delivery. Excellent first-time achievement figures indicate the relevance and accessibility of course content. Formal feedback from clients and participants is strongly positive, and some have reported enthusiastically on life changes. For example, one participant declared that the course lit my fire has sorted my life out, and he is now a branch manager with his organisation. As mentioned, more formal accumulation of data on added value has been discussed and planned. The managing director engages in an ongoing, energetic dialogue with teachers, clients, and parent companies, travelling countrywide, consulting and advising on learning requirements and opportunities, and on a variety of other matters relating to health and safety standards and issues. 1.4 How effective is the teaching? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. There is clear evidence of achievement in learning outcomes. Contributing factors are the quality and style of teaching, supported by a meticulous programme of monitoring and assessment. A range of materials, teaching approaches, and styles is adapted to the learning site and the learning needs of the participants. A training manual is written for every course and also condensed into a pocket-sized volume for easy and regular access. There is intensive preparation for assessments, one-to-one attention where required, and unlimited telephone and access to teachers. Assessments may be completed one-to-one, and those unable to complete first time are prepared for reassessment with one-to-one guidance and unlimited telephone and access to the teacher, although this is rarely necessary. There is ample scope and help for reassessment. Twice-yearly external moderation of assessments is conducted both by an external provider and by the relevant ITO for every standard taught. Teachers are in daily contact with their training coordinator, who reports on progress or otherwise to the weekly management meeting. A summary report from the teacher assesses the success of each course and of the teaching, with a complete and dated response to matters requiring attention. Teachers are experts in their specialist areas and have a teaching qualification. They are funded to attend external professional development events and participate in annual assessor meetings with four ITOs to discuss content, assessment, and delivery. Refresher training is an ongoing, internal process of mutual peer assessment, with two-way observation and mutual exchange of views and advice. Teachers are required to telephone each other weekly. In a culture of shared information, with open discussion and debate on any matter, there is company-wide involvement in course design. 8

9 1.5 How well are learners guided and supported? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. Courses last no more than three days, often less, and guidance for students during this time and beyond is generous, with unlimited and telephone availability of teachers to clients and participants. Teaching and assessment are often conducted on a one-to-one basis, with extra time and help given for reassessments. The pocket-sized training manuals are helpful not merely in preparation for assessment, but for subsequent reference in practice. Special adaptation is made for learning needs, for example literacy challenges and disability, or more frequently, fear and resistance to learning. Achievements are celebrated and published, custom-designed certificates and ID cards are awarded, and the class elects and celebrates an exceptional learner in each course. Teachers form personal relationships with learners and foster a cooperative learning environment among them, creating a base for application of learning in the working environment and for further workplace learning. The erosion of resistance to learning and the creation of a cooperative and committed workplace were features warmly commented on by clients interviewed by the EER team. 1.6 How effective are governance and management in supporting educational achievement? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. All written material is subject to discussion, debate and revision involving all participants TOSHA is jointly owned by two directors, one of whom is the managing director. The responsibility for the culture, vision, and policies of TOSHA resides with the governing body. The management of all operations is conducted within a flat management structure, with a guiding principle of no stone unturned. Values required in every staff member are drop-dead honesty, commitment to both clients and learners, and awareness of the social importance of the kaupapa of health and safety. These values are supported by an ongoing, proactive involvement with industry, by comprehensive documentation of all procedures, by a climate of cooperation, encouraging questioning and debate, and by the recognition of staff achievement. An excellence cup is awarded weekly to one staff member. An annual team meeting sets goals and targets for the year. All staff prepare a personal annual plan, which is monitored monthly. The procedures manual is a living document, under review from all quarters. Staff are encouraged to question the purpose or utility of any part of it and to table proposals at the weekly staff meeting for deletion, revision, or replacement. Every procedure and written document is subject to the same continuous scrutiny. The average roll-over time of operational policies is three to four months. 9

10 This is a performance and self-assessment regimen requiring concentration and commitment from every staff member, for which ample evidence was offered at the EER. A reiterated comment among staff was that, we are all in this together. This comment represents a management approach that is at once demanding and liberal, creating a strongly cooperative culture which nonetheless encourages independent thought and ownership of the kaupapa. 10

11 Focus Areas Educational performance and assessment capability in all focus areas are addressed in Part Focus area: Governance, management, and strategy The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Excellent. 2.2 Focus area: Elevated Platforms The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Excellent. 2.3 Focus area: Hazardous Goods The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Excellent. 11

12 Recommendations There are no recommendations arising from the external evaluation and review, other than those implied or expressed within the report. Further actions The next external evaluation and review will take place in accordance with NZQA s policy and is likely to occur within four years of the date of this report. 12

13 Appendix Regulatory basis for external evaluation and review Self-assessment and external evaluation and review are requirements of course approval and accreditation (under sections 258 and 259 of the Education Act 1989) for all TEOs that are entitled to apply. The requirements are set through the course approval and accreditation criteria and policies established by NZQA under section 253(1)(d) and (e) of the Act. In addition, for registered private training establishments, the criteria and policies for their registration require self-assessment and external evaluation and review at an organisational level in addition to the individual courses they own or provide. These criteria and policies are set by NZQA under section 253(1)(ca) of the Act. NZQA is responsible for ensuring TEOs continue to comply with the policies and criteria after the initial granting of approval and accreditation of courses and/or registration. The Institutes of Technology and Polytechnics Quality (ITP Quality) is responsible, under delegated authority from NZQA, for compliance by the polytechnic sector, and the New Zealand Vice-Chancellors Committee (NZVCC) has statutory responsibility for compliance by universities. This report reflects the findings and conclusions of the external evaluation and review process, conducted according to the policies and criteria approved by the NZQA Board. The report identifies strengths and areas for improvement in terms of the organisation s educational performance and capability in self-assessment. External evaluation and review reports are one contributing piece of information in determining future funding decisions where the organisation is a funded TEO subject to an investment plan agreed with the Tertiary Education Commission. External evaluation and review reports are public information and are available from the NZQA website ( Information relevant to the external evaluation and review process, including the publication Policy and Guidelines for the Conduct of External Evaluation and Review, is available at: 13

14 NZQA Ph E eeradmin@nzqa.govt.nz 14