Report of External Evaluation and Review

Transcription

1 Report of External Evaluation and Review New Zealand Institute of Sport Highly Confident in educational performance Highly Confident in capability in self-assessment Date of report: 18 July 2012

2 Contents Purpose of this Report... 3 Introduction TEO in context Scope of external evaluation and review Conduct of external evaluation and review...5 Summary of Results... 6 Findings... 8 Recommendations Appendix MoE Number: 8490 NZQA Reference: C06793 Date of EER visit: 8, 9 and 10 May

3 Purpose of this Report The purpose of this external evaluation and review report is to provide a public statement about the Tertiary Education Organisation s (TEO) educational performance and capability in self-assessment. It forms part of the accountability process required by Government to inform investors, the public, students, prospective students, communities, employers, and other interested parties. It is also intended to be used by the TEO itself for quality improvement purposes. Introduction 1. TEO in context Name of TEO: Type: Location: Delivery sites: New Zealand Institute of Sport (NZIS) Private training establishment (PTE) Westpac Stadium, 105 Waterloo Quay, Pipitea Wellington 382 Manukau Road, Greenlane, Auckland First registered: 20 March B Wharenui Road, Riccarton, Christchurch Courses currently delivered NZIS offers two foundation programmes: NZIS Certificate in Sport and Recreation (SportsZone) (Level 2) NZIS Certificate in Sports Studies (Level 3) Other certificate programmes include: NZIS Certificate in Personal Training (Level 3) NZIS Pre Police Proficiency Certificate (Level 3) Diploma programmes currently offered by NZIS include: NZIS Diploma in Sport Management and Exercise Prescription (Level 5) NZIS Diploma in Sport (Advanced Sport Management) (Level 6) 3

4 NZIS Diploma in Sport (Advanced Exercise Prescription) (Level 6) Code of Practice signatory? Number of students: Yes. Approved for year olds and 18 years up. At the time of the external evaluation and review (EER), only four international students were studying with NZIS. Domestic: in 2011 approximately 550 students studied with NZIS. Approximately 30 per cent were Māori and approximately 35 per cent were Pasifika students. In 2011, over 90 per cent of NZIS students were aged 20 years or under. International: four Number of staff: Distinctive characteristics: Recent significant changes: Previous quality assurance history: 25 full-time equivalents NZIS delivers qualifications for the sport, fitness, and recreation industry. It incorporates into its mission a commitment to an effective and culturally relevant learning environment with a commitment to Pacific and Māori communities. NZIS has recently completed the purchase of the New Zealand College of Massage, which is colocated with the NZIS campus in Auckland. At the previous NZQA quality assurance visit, an audit in 2008, NZIS met all the requirements of the quality assurance standard then in place. In 2011 the organisation did not meet moderation requirements for one of the two unit standards moderated, although the report noted an improvement on Scope of external evaluation and review The scope for the external evaluation and review consisted of the mandatory focus area: Governance, management, and strategy In addition, the following focus areas were selected: NZIS Diploma in Sport Management and Exercise Prescription (Level 5) Foundation programmes including: NZIS Certificate in Sport and Recreation (SportsZone) (Level 2) 4

5 NZIS Certificate in Sports Studies (Level 3) Christchurch earthquake The focus areas above were selected to provide a reasonable representation of NZIS s activities across the three campuses. The level 5 diploma is NZIS s largest programme and the foundation programmes are distinct in that they have large numbers of under 20-year-olds funded predominantly through Youth Guarantee. The Christchurch earthquake was a focus area chosen in conjunction with the provider, having been a major event affecting the organisation in the previous year. NZIS s former Christchurch campus is located in the red zone. 3. Conduct of external evaluation and review All external evaluation and reviews are conducted in accordance with NZQA s published policies and procedures. The methodology used is described fully in the web document Policy and Guidelines for the Conduct of External Evaluation and Review available at: The TEO has an opportunity to comment on the accuracy of this report, and any submissions received are fully considered by NZQA before finalising the report. Prior to the EER visit, the evaluation team was supplied with a self-assessment document which summarised the organisation s achievement data and quality improvements, processes, and practices. A three-day EER visit included a visit to the new Christchurch campus and the Wellington campus. A videoconference interview was conducted with the staff from the Auckland campus. The evaluation team, consisting of the lead evaluator and two other evaluators accompanied by an NZQA observer, met with the chief executive officer, the deputy chief executive officer, campus supervisory tutors, course leaders, the academic advisor, tutors, students, some graduates, the board of directors, and the advisory board. A large number of documents was also reviewed as part of the enquiry, including the investment plan, the business plan, organisational review documents, course and tutor evaluations, staff and student files, moderation records, external stakeholder liaison and feedback, and minutes of academic meetings. 5

6 Summary of Results Statement of confidence on educational performance NZQA is Highly Confident in the educational performance of New Zealand Institute of Sport Limited. The table below outlines the improving course and qualification achievement rates for NZIS. Course and qualification achievement at NZIS Course completion % Qualification completion % It is highly likely that this increase in achievement can be attributed to quality improvements including the appointment of dedicated pastoral care staff, introducing weekly guidance and support meetings, and implementing individual education plans and homework centres. While these achievement rates are on a par with other PTEs, they are high when compared with other organisations with similar demographic profiles. There is good evidence of positive outcomes for graduates from NZIS programmes. Data collected by the organisation demonstrates that approximately 90 per cent move on to either further education or full or part-time relevant employment. This is an excellent result for any tertiary provider. The organisation emphasises the acknowledgement and celebration of Pasifika cultures and this is reflected in the strong sense of family, or aiga, among staff and students, and a philosophy of respect for all. Taking care of the whole student and fostering a culture of achievement and professionalism are evident in the organisation s policies and the practices of NZIS staff. This commitment to support was thrown into stark relief during the tumultuous days following the February 2011 earthquake in Christchurch. There is strong leadership guiding the organisation and a clear vision that is shared across the three campuses. This vision, along with a highly strategic view of the tertiary education environment, has enabled the organisation to respond effectively to shifting policy decisions and growth. 6

7 Statement of confidence on capability in self-assessment NZQA is Highly Confident in the capability in self-assessment of New Zealand Institute of Sport Limited. Data informs decision-making at all levels of NZIS. For example, achievement data is collected and analysed from the level of each individual student, through different courses and cohorts, campuses and programmes. While not every course or programme has achieved high completion rates, those few that did not are identified and plans are in place for improvement. Course and tutor evaluations have a good response rate and tutors and supervisors use the information to inform performance reviews and plan professional development. This culture of enquiry is evident at the governance level and board of directors, which use regular independent, organisation-wide reviews to check assumptions and inform planning. Throughout the EER the evaluation team heard from staff and students about how feedback was sought and given and how quality improvement processes were routine and ongoing. Quality improvements include improved pastoral care mechanisms, individual education plans, revised qualifications with greater relevance to industry, a new organisational structure, and the appointment of a resource teacher to support the development of tutors. The organisation had a continuity plan in place when the February 2011 earthquake struck. The careful approach to decision-making, informed by good information from key people on the ground in Christchurch, resulted in the organisation keeping nearly all the students engaged in learning and achieving in 2011 at similar rates to The use of a Facebook page to connect with staff and students separated by the earthquake, and Survey Monkey to gauge student and whānau responses to different options, were examples of ensuring the response was well informed. 7

8 Findings How well do learners achieve? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. As noted above, for the period NZIS has shown a steady increase in both course completion and qualification completion rates. These rates are in line with averages for the sector. However, this is strong achievement given the high numbers of young learners, the strong representation of Māori and Pasifika students, and the significant numbers who are completing qualifications at levels 4-6 on the New Zealand Qualifications Framework. Of note is that Christchurch students achieved at a rate equal to the other campuses and other years, despite 2011 being an extremely difficult year with weeks of study lost. There is good evidence that students are gaining knowledge and skills that support further study and career development. Students and staff reported gains in literacy and numeracy for foundation-level students, and across all programmes students gain work-ready skills that enable them to engage with potential employers and future clients. The feedback from work experience supervisors is particularly valuable for demonstrating these skills. The organisation systematically collects and analyses achievement data at the individual, course, campus, and programme level and therefore has comprehensive information to inform quality improvements. Staff at all levels are familiar with and use this data to guide evidence-based decision-making. While not all courses are achieving at a satisfactory rate, the lower-performing courses have been identified and targeted for support. The organisation s understanding of literacy and numeracy gains and the acquisition of work-ready skills is less well developed. 1.2 What is the value of the outcomes for key stakeholders, including learners? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. Data from graduates has been systematically collected to identify their destinations after leaving NZIS. This data shows that over 90 per cent move into full-time or part-time relevant employment or further training. At all three campuses there are strong links with the sport and recreation industry which confirms this flow of 1 The findings in this report are derived using a standard process and are based on a targeted sample of the organisation s activities. 8

9 graduates into relevant work. The evaluation team heard multiple examples of industry employers contacting NZIS to employ graduates, and this indicates that the programmes are relevant and prepare students well for employment. NZIS looks for ways to add value to the training it delivers. For example, the organisation has taken students on overseas sporting and cultural tours. A recent example included a sporting and cultural exchange to China. The tours are organised in advance by NZIS and costs are kept to a minimum to enable all eligible students to travel. NZIS also provides multiple opportunities for the students to engage in sporting activities and events. The organisation has a number of sporting academies which support and encourage high-performing athletes to improve in their chosen field. In a regular sports exchange event, students from across the three campuses are brought together to compete across a number of sporting codes. The organisation is also involved in a large number of community sport and recreation events. These give students the opportunity to develop their sport management and customer service skills while they work with the community. Examples of events include the National Secondary School Volleyball Championships, Rugby World Cup Fanzones and Sportacular in Christchurch, and involvement in school sports days across the three cities. Working with Pasifika communities has been another focus and NZIS staff and students have been actively involved with Pasifika community events including the annual Pacifically Wellington festival and running fitness and aerobic sessions with church and community groups. Two indicators of value include the large number of word-of-mouth referrals and the huge practical support given to the institute from students and their families in Christchurch after the earthquake, particularly to help fit out interim buildings on a new campus. The funding given to NZIS for an Ako Aotearoa research project looking into supporting educational success for Pasifika learners will add value to the wider education community. Of value to the students who study with NZIS are the multiple pathways and exit points. A significant number of foundation students continue to study with NZIS at diploma level. For students who have completed the advanced diploma, NZIS has recently signed an articulation agreement with Victoria University in Melbourne, Australia. This enables students to complete a degree with two years crosscredited from their NZIS level 6 diploma and one additional year s study in Melbourne. NZIS is systematically collecting feedback from learners, quantitative data about graduate destinations, and qualitative data from the many stakeholder groups. Graduates views about the value of their training, and the records of success of those students who move to higher levels of study, could be more systematically collected and better understood. 9

10 1.3 How well do programmes and activities match the needs of learners and other stakeholders? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. NZIS has a comprehensive understanding of the needs of students. Students are interviewed on entry and processes are in place to ensure that students are enrolled in the programme that best matches their needs. For the foundation students, a literacy and numeracy test ensures that those with extra needs are identified for support early. NZIS is committed to providing a culturally safe environment for students and has a number of strategies in place to ensure this, including cultural advisors, cultural ambassadors, and cultural training for staff. NZIS is well connected with industry, with both formal and informal networks, including an advisory board, work-experience placements, and graduate and staff networks. An example of matching industry needs is the new NZIS Diploma in Sports Management and Exercise Prescription, which was developed in response to sector and student feedback about the outdated nature of the older programme. The sports labs give students the opportunity to conduct real tests that mirror industry analysis. These labs and the community events and work experience give students the opportunity to apply their learning in realistic contexts. There are multiple mechanisms in place to ensure that useful feedback from key stakeholders is gathered. There is good evidence that this feedback is regularly analysed and used to inform planning and processes. The development of the sixmonth certificate programme for personal trainers is one example of the organisation responding to industry needs. After the Christchurch earthquake, staff and students and their families were surveyed online to see what they wanted in terms of their study with NZIS. A high response rate showed an overwhelming wish for NZIS to continue to operate in Christchurch. In response, a potential new campus was found. Students and their families actively supported NZIS staff to turn an abandoned bowling club into a teaching facility. 1.4 How effective is the teaching? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. Across the campuses and programmes, the evaluation team heard how tutors use a range of strategies to engage the students. Tutors are supported to develop with multiple opportunities for professional development, including in-house workshops and external professional development opportunities. The recent appointment of a learning resource specialist is a positive initiative appreciated by tutors across the campuses. By interviewing and observing tutors at work across the different 10

11 campuses, an organisation-wide strategy for quality improvements in teaching is being developed. One common theme is the need for a range of classroom management strategies for those tutors working with younger learners; another was the need to run cultural training refreshers. Some tutors are being given individual support. This process is occurring alongside but separately to the routine performance observation and appraisal system. The organisation provides regular opportunities for teaching staff to meet and reflect on courses and programmes. This includes all staff gathering at the end of the year for reflection and at the beginning of the year for planning. In addition, an academic committee oversees assessment and moderation and ensures their effectiveness. These scheduled meetings and regular contact between the tutors teaching the different courses ensures there is consistency between the campuses. A recently installed videoconferencing facility has improved the ability to communicate across the organisation, and the implementation of a shared student management system and shared drive will further support this. Staff have a good mix of industry and teaching qualifications and maintain currency through attendance at industry conferences and events. Students are given regular opportunities to provide feedback on their tutors through a student committee and online surveys. These surveys and the evaluation team interviews with current students indicate a high level of satisfaction with the tutors and courses. 1.5 How well are learners guided and supported? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. A key feature of NZIS is the wrap-around support provided to students to enable them to achieve. Tutors are tasked with ensuring students remain engaged with the programme, and attendance and punctuality issues are addressed quickly. Students causing concern are given individual education plans which enable their progress to be closely monitored. Dedicated pastoral care staff liaise with students and their families and attempt to remove barriers to learning. The establishment of homework centres, where students can get one-to-one academic support, emerged from an organisational review in 2010 and are proving popular with students. The sports academies are another mechanism to offer holistic student-centred support and guidance. Students are welcomed to the organisation with an orientation which introduces them to their tutors and staff. All new students then participate in a camp which focuses on outdoor education activities and team-building. Evaluations show that this experience is valued and popular with students. Across the organisation, from staff and students the evaluation team heard about an open-door policy and students and staff being shown mutual respect. 11

12 The response of the organisation to the February 2011 earthquake in Christchurch demonstrated its commitment to guidance and support. The sole tutor in charge on campus in the central business district on the day had only recently started with NZIS but had been through emergency response training. She responded accordingly, ensuring all students were safely evacuated and accounted for. The majority of students were just beginning their week-long orientation camp and were immediately anxious to return to their families. Staff spent a long night ensuring that all students were returned safely to their families. A Facebook page was rapidly set up and enabled communication between staff and students to continue despite cell phones being down. Staff were told to prioritise their families and ensure their safety, outside of the city if necessary. No time limit was put on their leave. The management team used a survey to ask Christchurch students and their families what they wanted to do, and a new site was sought based on this information. With support from students and their families, the abandoned building on the new site was repaired and classes resumed within just three weeks. Ongoing support was provided, including counselling being offered to students and staff. Senior management from the national office were on site for four days a week for approximately three months. Only eight of the original 124 students decided not to continue with their studies. The support and guidance needs of learners and the effectiveness of the processes to meet their needs are reviewed regularly. The pastoral care staff and tutors meet weekly on each campus to identify and respond to the emerging issues of the learners. The learners are regularly surveyed to identify their concerns and aspirations. They can also give feedback via their student representatives. Organisational reviews have been the impetus for new support initiatives such as the dedicated pastoral workers and the homework centres. 1.6 How effective are governance and management in supporting educational achievement? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. This organisation has been shaped by strong leadership with a clear vision. It is clear that the vision is shared across the different campuses and helps frame planning and strategy. The senior management team has a strong focus on education and positive education outcomes, particularly for Māori and Pasifika students, and invests in ensuring quality is delivered to those ends. NZIS is an organisation that is driven by data, both qualitative and quantitative. Ensuring that there is sufficient information for good decisions to be made was a feature noted across a number of scenarios, including new programme development, the refit of new campuses, and the response to the Christchurch earthquake. Another characteristic of this provider is regular and systematic 12

13 reviews of performance and sound planning at all levels of the organisation based on often systematically collected data. The board of directors takes a strategic view and ensures that senior management is challenged in its thinking. The board has directed a number of internal reviews, which over the years have highlighted issues that have shaped the direction of the organisation. The most recent review noted that growth in the organisation had resulted in some systems and processes not functioning satisfactorily. Resulting from this review, new developments included the refitting of a new campus in Auckland, improved communication mechanisms, the restructure of academic leadership roles, and the appointment of an information technology manager. For a number of reasons, 2011 was a challenging year for NZIS: the introduction of the new diploma programmes, the increasingly unsatisfactory campus in Botany, Auckland, and the Christchurch earthquake stretched available resources. While some planned activities needed to go on hold, students continued to learn and achieve at higher rates than in previous years. There is a clear commitment from staff to ensuring the quality delivery of foundation and sport and recreation programmes. Across the campuses the longevity of a significant number of staff indicates that they feel valued and connected to the organisation and vision. 13

14 Focus Areas This section reports significant findings in each focus area, not already covered in Part Focus area: Governance, management, and strategy The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Excellent. 2.2 Focus area: NZIS Diploma in Sport Management and Exercise Prescription (Level 5) The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Good. 2.3 Focus area: Foundation programmes The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Good. 2.4 Focus area: Christchurch earthquake The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Excellent. 14

15 Recommendations There are no recommendations arising from the external evaluation and review, other than those expressed or implied within the report. 15

16 Appendix Regulatory basis for external evaluation and review Self-assessment and external evaluation and review are requirements of programme approval and accreditation (under sections 249 and 250 of the Education Act 1989) for all TEOs that are entitled to apply. The requirements are set through the Criteria for Approval and Accreditation of Programmes established by NZQA under section 253(1)(d) and (e) of the Act and published in the Gazette of 28 July 2011 at page These policies and criteria are deemed, by section 44 of the Education Amendment Act 2011, to be rules made under the new section 253. In addition, for registered private training establishments, the criteria and policies for their registration require self-assessment and external evaluation and review at an organisational level in addition to the individual programmes they own or provide. These criteria and policies are also deemed, by section 44 of the Education Amendment Act 2011, to be rules made under section 253. Section 233B(1) of the Act requires registered PTEs to comply with these rules. NZQA is responsible for ensuring non-university TEOs continue to comply with the rules after the initial granting of approval and accreditation of programmes and/or registration. The New Zealand Vice-Chancellors Committee (NZVCC) has statutory responsibility for compliance by universities. This report reflects the findings and conclusions of the external evaluation and review process, conducted according to the EER process approved by the NZQA Board. The report identifies strengths and areas for improvement in terms of the organisation s educational performance and capability in self-assessment. External evaluation and review reports are one contributing piece of information in determining future funding decisions where the organisation is a funded TEO subject to an investment plan agreed with the Tertiary Education Commission. External evaluation and review reports are public information and are available from the NZQA website ( Information relevant to the external evaluation and review process, including the publication Policy and Guidelines for the Conduct of External Evaluation and Review, is available at: NZQA Ph E qaadmin@nzqa.govt.nz 16