Report of External Evaluation and Review

Transcription

1 Report of External Evaluation and Review Change Training Consultants Limited trading as New Zealand School of Business & Government Not Yet Confident in educational performance Not Yet Confident in capability in self-assessment Date of report: 12 September 2012

2 Contents Purpose of this Report 3 Introduction 3 1 TEO in context3 2 Scope of external evaluation and review6 3 Conduct of external evaluation and review6 Summary of Results 8 Findings 11 Recommendations 21 Appendix 22 MoE Number: 7346 NZQA Reference: C07517 Date of EER visit: and 28 May

3 Purpose of this Report The purpose of this external evaluation and review report is to provide a public statement about the Tertiary Education Organisation s (TEO) educational performance and capability in self-assessment It forms part of the accountability process required by Government to inform investors, the public, students, prospective students, communities, employers, and other interested parties It is also intended to be used by the TEO itself for quality improvement purposes Introduction 1 TEO in context Name of TEO: Type: Location: Change Training Consultants Limited trading as New Zealand School of Business & Government (Change Training) Private training provider The head office and site of the Wellington Campus is located at level 8 Grand Plimmer Tower, 26 Gilmer Terrace, Wellington Change Training also operates from three other campuses: Auckland: Level 8, 115 Queen Street Katikati: 109 Main Road Christchurch: Westfield Mall, Riccarton Delivery sites: Katikati, Auckland, Christchurch, Wellington First registered: 24 July 2000 Courses currently delivered Code of Practice signatory Number of students: See scope of accreditation Yes At the time of the external evaluation and review (EER): 350 international students 85 per cent Indian, 70 per cent male 130 domestic students 74 per cent European, 75 per cent female 3

4 Number of staff: At time of EER: 18 full-time staff Seven part-time Scope of active accreditation: Change Training is currently accredited to deliver the following certificates and diplomas: New Zealand Institute of Management (NZIM) Certificate in Management (Level 4) NZIM Certificate in Language, Culture and Management (Level 4) NZIM Certificate in Practice Management (Health) (Level 4) NZIM Diploma in Practice Management (Health) (Level 5) NZIM Diploma in Management (Level 5) NZIM Diploma in Management (Advanced) (Level 6) NZ Diploma in Business (NZDipBus) (Level 6) Change Training also administers short courses in public policy development, foundation skills, financial management, economic analysis, and performance management training for the public sector Distinctive characteristics: Recent significant changes: Change Training offers online learning to domestic students and blended learning to international students It also offers programmes developed in conjunction with NZIM Several key programmes offered at Change Training incorporate NZDipBus papers The organisation has four campuses across New Zealand The Christchurch campus is co-located with another organisation that employs staff on behalf of Change Training There have been a number of staffing changes since the last EER, with four appointments yet to be filled Significant changes have occurred at the Christchurch and Katikati campuses, with changes in premises at Christchurch and considerable staff changes at the Katikati campus 4

5 Since the EER the ownership of Change Training has changed with new directors and senior management being appointed Previous quality assurance history: Results from the previous EER carried out in March 2011 were Not Yet Confident in educational performance and Not Yet Confident in capability in self-assessment In that EER report, NZQA recommended that Change Training: Ensure entry requirements for international students enrolling in the NZIM Diploma in Management align with NZIM s mandatory and discretionary guidance Continue to develop and implement identified responses to improve student outcomes Continue development in the collection of data for self-assessment and build capability in analysing this data Develop and maintain a comprehensive self-assessment regime As a result of its 2009 and 2010 national external moderation results, and the findings of the March 2011 EER, Change Training was required to submit assessment materials and learner samples for an increased number of prescriptions NZQA advised Change Training that if moderation results for resubmitted and new moderation were not satisfactory, NZQA would consider actions around imposing conditions around registration and accreditation requirements for delivery of NZDipBus prescriptions The latest moderation results from NZQA, dated 7 May 2012, state that Change Training is regarded as not meeting national external moderation requirements for the 2011 semester 2 moderation round At the time of the EER, Change Training was working through the moderation results and was preparing documentation to appeal four of the moderation decisions 5

6 2 Scope of external evaluation and review The scope of the EER of Change Training included the following focus areas: NZIM Diploma in Management (Level 5) The NZIM Diploma in Management programme was selected as it is the main programme in which international students enrol Online domestic students The online delivery of the NZDipBus is a programme of key interest to NZQA The online domestic students study a variety of NZDipBus prescription papers wholly online The following mandatory focus areas were also included: Governance, management, and strategy International student support 3 Conduct of external evaluation and review All external evaluation and reviews are conducted in accordance with NZQA s published policies and procedures The methodology used is described fully in the web document Policy and Guidelines for the Conduct of External Evaluation and Review available at: The TEO has had an opportunity to comment on the accuracy of this report, and any submissions received were fully considered by NZQA before finalising the report The EER was conducted over four days by two evaluators, and a fifth day was spent phoning advisory board members, domestic online students, and online delivery tutors The evaluation team spent one day at each of the four campuses at Auckland, Katikati, Christchurch, and Wellington The evaluation involved engagement with: The directors of Change Training Programme management and tutors at all four campuses A learning support coordinator International student support staff International students at each campus Graduates 6

7 Advisory board members Online domestic students Online tutors The evaluation also involved a review of the website and relevant documentation such as the investment plan, the post-eer improvement plan, quality improvement plan, AKA student tracking data, Campus Visit Report data, a copy of the Action Learning Programme trial delivery plan, the student handbook, achievement data, student evaluation forms, meeting minutes, self-assessment reports, annual reviews, executive reports, advisory minutes, individual learning plans, stakeholder feedback, moderation reports, and associated correspondence 7

8 Summary of Results Statement of confidence on educational performance NZQA is Not Yet Confident in the educational performance of Change Training Consultants Limited trading as New Zealand School of Business & Government Change Training has undergone further growth in student numbers since its last EER in March 2011, with over 50 per cent more international and domestic students However, this has not been complemented by a comparable increase in full-time teaching staff, with a decrease from 16 to 15 full-time staff and an increase in part-time staff from four to seven The evaluators were concerned at the high student-tutor ratio seen at the Christchurch campus, with 66 international on-site students and one part-time tutor working 20 hours per week In 2009, 97 per cent of the organisation s international students successfully completed the NZIM Diploma in Management; however, in 2010 this dropped to 80 per cent Domestic students who study wholly online do not achieve as well as international students, due in some part to the nature of the students and the parttime nature of the study Only 10 per cent of the domestic students successfully completed NZDipBus papers in 2009, although this rose in 2010 to 33 per cent As noted, at the 2011 EER there were gaps and areas of weakness affecting learner outcomes that have still not been addressed NZQA has identified critical issues with external moderation and the design and marking of assessments which detract from the validity of learner achievement Change Training has made, and is still in the process of making, a number of major changes and early signs indicate these will improve the outcomes achieved by the students, and will also improve engagement across the four campuses The most comprehensive of these changes are a new student tracking system (AKA) implemented in October 2011, and the Campus Visit Report and Engagement and Achievement Records, both new in 2012 The AKA student attendance and progress tracking system initiative has led to some immediate improvements, with the attendance rates and pass rates of students lifting significantly at three of the campuses At the Christchurch campus, which Change Training shares with another provider, the AKA tool has uncovered anomalies in attendance registers and student attendance and this, along with a tutor-student ratio of 1:66, is yet to be effectively managed The senior management team receives the AKA reports on achievement rates and attendance course by course and semester by semester, and the team is now tracking the differences across campuses, but no benchmarking of the performance 8

9 of Change Training against other, similar providers or against published NZDipBus prescription data 1 has been carried out The organisation could not comprehensively demonstrate the long-term benefits of its programmes Staff capture information on the employment or further study destinations of most of the students, but this information is not summarised or analysed and used to make improvements to the design, delivery, or assessment of the programmes The organisation continues to offer good pastoral care to international students Students at all campuses value the combined online and face-to-face delivery and the level of tutor interaction and support, which meets their expectations Change Training is beginning to respond to the gaps in performance noted in the 2011 EER These include the AKA tracking system, monthly Campus Visit Reports, further staff development in assessment and moderation, Action Learning Programmes, and individual learning plans As these are still in the early stages of implementation, it is too early for the organisation to give a full analysis on the effectiveness of their impact on educational performance Statement of confidence on capability in self-assessment NZQA is Not Yet Confident in the capability in self-assessment of Change Training Consultants Limited trading as New Zealand School of Business & Government Since the previous 2011 EER, Change Training has been aware of key areas where it needs to improve its self-assessment capability and has identified a number of initiatives to meet these including student evaluations, the AKA student tracking system, (which gives comprehensive data on at-risk students), a Campus Visit Report, and some analysis of achievement data However an absence of indepth analysis of the data collected to fully understand and develop strategies around achievement issues has limited the organisation s ability to fully extend the process of improvement of self-assessment Two of the four sites have individual staff driving their own effective improvements, which are not disseminated to all sites, and some staff are not involved in any organisational processes of self-reflection Ongoing staff training in assessment and moderation is not improving assessment and marking as evidenced in the latest NZQA external moderation outcomes Change Training recognises the gaps in some areas of its data collection and is moving to fill them There are no meaningful comparisons of the achievement data across time or comparisons with the pass rates of other, similar providers The ability to identify, understand, and action the outcomes from good self-assessment 1 9

10 is still at the development and trial stages It is therefore difficult for Change Training to gauge how well it is doing The governance function is not being assessed against any criteria to ensure that weaknesses are being identified and improvements are being made to enhance the management structure, increase teaching staff, and improve the achievement of students through staff training, data collection, and analysis and benchmarking The organisation has some excellent methods or tools to support improvement Examples are the Engagement and Achievement Records which have detailed information on learner engagement and achievement and the assistance of a capable advisory group representing key industry members; new graduate and business survey tools; and the trial of a new model of face-to-face delivery, the Action Learning Programme However, the initiatives identified are in the development and initial trial phases and it is too early to assess their impact or the extent to which they will provide adequate and effective data to improve educational performance 10

11 Findings 2 11 How well do learners achieve? The rating for performance in relation to this key evaluation question is Adequate The rating for capability in self-assessment for this key evaluation question is Poor Change Training has high retention and achievement rates for international students studying in New Zealand (93-97 per cent retention between 2008 and 2011) In 2008, 93 per cent of students successfully completed the NZIM Diploma in Management, after which there was a drop in the number of qualification completions from 98 per cent in 2009 to 80 per cent in 2010 There are no final figures available for 2011 given that students can enrol at any time throughout the year; a number of students who enrolled in 2011 are yet to complete their studies Year Course completion Qualification completion Retention % 93% 93% % 97% % 8075% 888% Achievement rates for domestic students have risen, with 33 per cent successfully completing NZDipBus papers in 2010 in contrast to 10 per cent in 2009 Achievement statistics for the international students studying the NZIM Diploma in Management and the domestic students studying NZDipBus papers are good, but an adverse moderation report from NZQA stating that Change Training did not meet national external moderation requirements for the 2011 semester 2 moderation throws doubt on the validity and reliability of the achievement data presented The moderation results showed seven of 15 prescriptions met NZQA moderation requirements, with eight resubmissions being required At the time of the EER, Change Training was working through the moderation results and was preparing documentation to appeal four of the moderation decisions Change Training has a database which captures achievement information for each student across all campuses, but there has been little analysis of it After requests from the evaluation team, the pass rates and grades in the different papers were supplied and there was evidence of some analysis of these This analysis is yet to 2 The findings in this report are derived using a standard process and are based on a targeted sample of the organisation s activities 11

12 be fully developed using the AKA process to enable an in-depth, organisation-wide understanding of how differing groups of students achieve across different campuses, programmes, and learners The newly implemented AKA report provides regular reporting and monitoring of students where each student s progress and attendance are identified AKA allows for systematic evaluation of achievement rates across campuses and this has been working well, although it is still in its infancy and is not yet used to its full potential This new process identified a problem with the Christchurch campus where students were achieving less and had a lower attendance rate than the other three sites The attendance at this site is now being monitored more carefully; however, the evaluators saw two consecutive campus visit reports by the chief executive relating to low attendance with no actions outlined or a lift in attendance numbers being noted from one visit to the next Students at the Christchurch campus attend three days per week for four hours each and are able to attend on the other two days at any time, although their tutor only works for a total of 20 hours per week Students stated their concern that 12 face-to-face, tutor-directed contact hours did not meet Immigration New Zealand requirements of 20 hours per week The evaluators could see little evidence that Change Training is meeting learning needs with a tutor-student ratio of 1:66, and no other mitigating processes to compensate for this The AKA, Campus Visit Report, and Engagement and Achievement Records are used to collect and monitor attendance and risk data and to ensure consistency across campuses However, these processes have only recently been implemented and the data collected is in its infancy; it is too soon to note with any accuracy whether the information gained will be of value There has been no meaningful comparison or benchmarking of the data across time or comparisons with the pass rates of other, similar providers and little evidence of achievement data being used as a basis for staff reflection for decision-making across sites other than benchmarking the AKA results and Campus Visit reports 12 What is the value of the outcomes for key stakeholders, including learners? The rating for performance in relation to this key evaluation question is Adequate The rating for capability in self-assessment for this key evaluation question is Poor The programmes offered at Change Training prepare students for success in gaining employment or to pathway to further study The value of the outcomes for the students is further confirmed by staff records showing the employment of graduates often leading to permanent residency status, which is one of the preferred outcomes for students The evaluators heard from the organisation that the online domestic students are able to applying their learning, although this has not been verified with formal feedback or surveys, The evaluators heard from 12

13 those students interviewed that many are already working and apply their learning on the job The students also commented that they find value in the flexibility of the delivery The close relationship that the directors, managers, and teaching staff often have with international students means that the organisation informally keeps in touch with international graduates after they complete their studies The evaluators heard several examples of international graduates having received promotions, obtained employment, and moved on to further study after leaving Change Training, although in many cases there is not a direct link between the job and the course studied A small number of students interviewed intend to enrol in a level 6 NZDipBus with Change Training following completion of the NZIM Diploma in Management Change Training does not systematically follow up with graduates to obtain feedback about the value of having studied at Change Training The evaluators heard anecdotal feedback, sighted one set of six-month graduate surveys from the Katikati campus, and sighted lists of businesses where graduates were working but not the positions they held The value of the learning in respect of employment has not been fully evaluated by Change Training, therefore there is no means of knowing the value-add of tuition provided and whether it has assisted students to progress into sustainable employment with higher remuneration In addition, although the directors have informal relationships established with key stakeholders, the organisation does not systematically engage with employers of its graduates to understand how well graduates are prepared for future pathways Change Training has not tracked whether the level of motivation or overall skills level of learners has improved, although current students report that they are acquiring new skills and developing their existing skills The key priority and plan to develop a systematic process to survey and understand graduates and stakeholders and to employ an additional staff member to engage with businesses has not occurred, and therefore the value of Change Training s contribution to key stakeholders has not been ascertained More formal mechanisms are now being developed and there is recognition that some of the data, such as for graduate employment destinations, needs to be more specific Once again, there also needs to be more analysis of the data so that conclusions can be reached and decisions made that will bring about systematic improvements in educational performance 13

14 13 How well do programmes and activities match the needs of learners and other stakeholders? The rating for performance in relation to this key evaluation question is Adequate The rating for capability in self-assessment for this key evaluation question is Adequate The main methods being used to check whether the courses are matching student needs and whether learner support is effective are course evaluations and contact between students and staff Learner feedback has only recently been collected and is still being analysed fully to provide the information required for relevant changes to be made to the teaching, the delivery, or the resourcing of the programmes Change Training now has a standard assessment process to determine whether students have sufficient English competency from the beginning of their study and to ensure they receive the necessary support to progress to an equivalent IELTS level 60, although the evaluators queried the practice of taking students with an overall 55 IELTS when 81 per cent have been identified in the at-risk category in the AKA report Further information received from NZIM following the EER visit outlined a serious issue with regard to Change Training s English language policy and procedure, poor quality of assessments, and plagiarism This further highlights inconsistencies in Change Training s ability to assess correctly the English competency of enrolling students and its lack of careful self-assessment and staff training with regard to assessment and moderation, as noted again in section 14 The blended-learning delivery option offers flexibility for students to undertake study alongside employment and other interests The level of support received from staff on site exceeds expectations at most campuses International students interviewed were also satisfied with the online resources, which they find interactive and engaging, as well as the on-site library and other learning resources With the implementation of the site tutor report, there is now clearer identification of how individual students are progressing to enable corrective measures to be taken when required The online learning option available to domestic students offers students flexibility to accommodate study around other commitments Change Training had identified that a number of students do not complete their studies because they struggle with online learning, and it has now implemented an introductory online module in response to this issue, along with the online forum to increase the ongoing support to learners The online forums are evaluated with an online feedback form to ascertain their success but due to the newness of the online survey the analysis of the data is still in its infancy Change Training has also identified that a number of international students do not manage to complete their study in the time allotted This does not reflect the policy statement in the quality management system which states that course and programme outcomes are achievable within the timeframe of the course/programme 14

15 Further analysis around the issues of attendance, delivery, support, and assessment relative to the inability of students to complete in the obligatory time is required to ensure the programme matches the needs of the learners and for the organisation to meet its educational performance outcomes and its responsibilities under the Code of Practice One of the key statements in the previous EER was the need to implement a systematic process to obtain and analyse learner and stakeholder feedback Progress in these areas is slow, and although there has been some feedback sought on graduate destinations, the numbers are not clear with regard to the relevancy of the learning to employment Given the gravity of the first EER finding, this is a weak response Change Training has estimated that 80 per cent of the graduates reach employment goals, but the TEO was unable to provide a clear process or rationale as to how this figure was reached Until the organisation systematically engages with business stakeholders it will be unable to fully understand the extent to which programmes and activities are matching stakeholders needs 14 How effective is the teaching? The rating for performance in relation to this key evaluation question is Adequate The rating for capability in self-assessment for this key evaluation question is Poor International students interviewed thought highly of the support of the teaching staff as many come from similar cultural backgrounds, which allows them to empathise with the students Students were very complimentary of teaching staff at all campuses and noted that the level of tutor knowledge and support exceeded the expectations of the students, and which motivated them to attend The online domestic students also found their tutors motivating, interesting, knowledgeable, and helpful, with good engagement, and were keen to point out that the tutors stay in close contact and go the extra mile with their support One aspect that was mentioned frequently during the EER was the effective use of real-life examples by most tutors to aid learning and understanding The evaluators heard inconsistent commentary from both international and domestic students regarding feedback on assignment work before and after marking The majority of students interviewed found final assessment results to be clearly explained, with good feedback being helpful in identifying weaknesses and strengths for future assignments Most considered the ability to submit assignments for feedback prior to assessment highly informative for learning However, as in 2011, the evaluators again heard variable understanding by students and staff on the number of resubmissions permitted, thereby potentially limiting the fairness and consistency of assessment processes if not uniformly applied There is some question around the new resubmission policy not being enforced and students still referring to the old resubmission policy Domestic 15

16 student referred to submitting assignments for checking before actually submitting them for marking Staff are well qualified in their subject areas Staff appraisals are works-in-progress and there is not sufficient evidence of managerial oversight in this process, although some performance processes are ongoing, such as monitoring and discussing student survey results with those individual staff where results are poor Professional development is mainly in-house training supplemented by attendance at workshops delivered by NZQA, the Ministry of Education, and the New Zealand Association of Private Education Providers A greater level of management support is required to support the further knowledge and expertise of staff, especially in the moderation and assessment area and the organisation has plans to make staff development in 2012 a top priority Change Training did not meet all NZQA external moderation requirements for NZDipBus prescriptions in 2011 NZQA reports that the organisation did not meet the national standard in relation to three NZDipBus prescriptions in 2009, two in 2010, and eight in 2011 An action plan was created following the 2010 moderation outcomes for staff to be trained in assessment and moderation practices, but clearly this has not been achieved and other measures have not been effective or convincing in addressing the issues Most teaching staff, including online tutors, are not aware of a problem with moderation, although they are heavily involved with some of the papers that do not meet the national standard This highlights a significant level of risk and is a reflection of poor management and programme oversight Further analysis and review around moderation processes and practice is needed to ensure the validity of assessments and marking improves significantly and does not bring into question the validity of the students results 15 How well are learners guided and supported? The rating for performance in relation to this key evaluation question is Adequate The rating for capability in self-assessment for this key evaluation question is Adequate International students receive comprehensive information about living in New Zealand and studying at Change Training from the organisation s student handbook This contains comprehensive coverage of relevant aspects of the Code of Practice, the prospectus, website, and a comprehensive orientation programme Students appreciate being collected at the airport and the Change Training welcome packs contain all relevant Code of Practice information The packs have recently been changed as a result of staff team input on a staff development day An annual Code of Practice review is completed, although Change Training has not undertaken a self-assessment of its implementation of the code International students at the Auckland, Katikati, and Wellington sites could clearly identify the 24-hour support person for emergency contact However, both 16

17 management and students at the Christchurch campus were unclear as to who was the 24-hour contact for students With the diverse cultures of students at each campus, the evaluators were concerned that the organisation did not have a first language support person available or within the community for those students from countries other than where the majority were from, ie Japan and Germany, as this does not reflect effective pastoral care for all its students Student feedback evaluations have only been administered for the first time this year Therefore, the organisation is not yet in a position to assess the efficacy of the guidance and support it offers to students Students at the Auckland and Katikati campuses commented to the evaluators on the lack of sufficient texts in the library Extra tutorials, special lectures, and classes are offered to the AKA-identified at-risk and poorer performing students at the Auckland, Wellington, and Katikati campuses, where staff are timetabled to offer additional support to students on a one-to-one basis to assist them in their understanding and learning A similar regime is offered at Christchurch Although the support is monitored through the AKA reporting process the organisation has not assessed the full extent to which this extra support is working and making a difference to student learning and achievement Those online tutors interviewed by the evaluation team engage regularly with learners and work with students to ensure any potential issues are addressed and to offer effective support for their learning The majority of domestic students spoken to considered that they were well supported by their online tutors However, feedback to the evaluators from one domestic online student noted that some prescription courses were more easily managed, delivered, and supported than others Management outlined that they were aware of the differences in tutor facilitation and support and had recently developed and implemented a survey to obtain learner feedback from all domestic online students around guidance and support However, the results have yet to be collated, analysed, and actioned to enable informed improvement for teacher facilitation and support mechanisms The level of guidance and support provided to international and domestic online students has improved, and the recently introduced online student forum and introductory workshops for new online learners introduced in 2011 are providing some support to the students 17

18 16 How effective are governance and management in supporting educational achievement? The rating for performance in relation to this key evaluation question is Adequate The rating for capability in self-assessment for this key evaluation question is Adequate Management closely monitors the programme attendance figures along with the feedback from staff and usually acts quickly when changes are required to policy, staffing, or funding for resources The organisation has responded to the demands of growth in most areas and allocated resources to support learning and teaching, although the 1:66 tutor-student ratio for the one part-time tutor at the Christchurch campus should be of concern, along with the extra workload for the one tutor/campus manager at the Katikati campus, which is one tutor down The board and the management team ensure there are minimal barriers to learning through the appointment of well-qualified staff and the use of modern online and face-to-face teaching facilities and classrooms, although the Christchurch campus students had some concerns around the use of a tutor laptop to teach and demonstrate rather than a data show, and various students at two campuses were keen to have more texts in the library There is very little separation between governance and management and this was also noted by those spoken to on the advisory committee There is an accepted focus on management and operational issues at the expense of the helicopter view of governance This issue is compounded by the fact that three of the four members on the governance board are management/operational staff of Change Training The governance function is not being assessed against any criteria and no evidence was seen by the evaluation team of any structure or strategy to assess the effectiveness of the management team in supporting educational achievement to ensure that weaknesses are being identified and improvements are being made to enhance the achievements of the students Student progress is discussed at a monthly management meeting, although no formal actions are prompted or actively monitored The principal s liaison with management and staff ensures he is actively involved in the day-to-day operations of the institute, including the Campus Visit Report process However, no actions have been taken with the attendance problem following two recent campus visit reports completed by the principal at the Christchurch campus Managerial and support staff have an open-door policy whereby it was noted by both students and teaching staff that they could comfortably approach either with queries and comments around teaching and learning improvements This is good practice, but there is no evidence that the information generated is systematically used to make changes While Change Training values its staff and appears to encourage staff input, the evaluation team heard a couple of examples where staff had been left wondering whether their input had been taken into account Evidence of some collegial and 18

19 cross-cultural activity was heard at the Katikati campus where staff and students visited the Auckland campus and performed a customary Indian dance and joined in a shared lunch This initiative was organised between the two campuses and not driven from management As noted in the previous EER report, management should work to facilitate greater collegial opportunities across campuses and consider the extent to which staff are involved in self-reflection activities This will require the management of the physical distance between staff and campuses, and also crosscultural differences The advisory group has strong subject matter expertise relevant to the programmes taught at Change Training Advisory group members confirm that the directors seek input into course and organisational development, but noted that there was a need to better determine the type of information needed to help Change Training look at its data more effectively and for more in-depth conversations around guidance about the opportunities that the organisation should progress to be more effective in its engagement with the community The group wanted more regular meetings and clearer indications of meeting dates and times throughout the year instead of the rushed approach used at present It is of some concern that not all key staff are aware of the specific issues identified by NZQA at the previous EER or through national external moderation of assessment, as management should be involving staff in solving the moderation issue Directors and the management team should monitor the actions arising from the plans adopted as a result of the organisation s analysis of external moderation outcomes A number of key improvement processes noted in the previous EER have been or are becoming established across the organisation Examples of such processes include: a newly prescribed process and statements regarding assessing learner achievement or English language support (including external NZQA approval); a new professional development policy; the trial Action Learning Programme, Engagement and Achievement Records, and Campus Visit Report; and the development of the AKA analysis tool to better benchmark and understand learner achievement across the organisation It is clear that Change Training is aware of further performance issues and gaps in its self-assessment capability These include a systematic process to engage employers, graduates, and students; uniformity regarding student evaluations; effective staff professional development plans; effective assessment and moderation activities; active and timely stakeholder involvement; strategies around the risk areas of the high tutor-student ratio; external moderation outcomes; and ongoing attendance problems at the Christchurch campus The improvement plan as implemented since the previous EER has not resulted in significant improvement to lift NZQA s confidence in this organisation 19

20 Focus Areas This section reports significant findings in each focus area, not already covered in Part 1 21 Focus area: Governance, management, and strategy The rating in this focus area for educational performance is Adequate The rating for capability in self-assessment for this focus area is Adequate 22 Focus area: International student support The rating in this focus area for educational performance is Good The rating for capability in self-assessment for this focus area is Adequate 23 Focus area: NZIM Diploma in Management The rating in this focus area for educational performance is Adequate The rating for capability in self-assessment for this focus area is Poor 24 Focus area: Online and distance delivery The rating in this focus area for educational performance is Adequate The rating for capability in self-assessment for this focus area is Poor 20

21 Recommendations There are no recommendations arising from the external evaluation and review, other than those expressed or implied within the report 21

22 Appendix Regulatory basis for external evaluation and review Self-assessment and external evaluation and review are requirements of programme approval and accreditation (under sections 249 and 250 of the Education Act 1989) for all TEOs that are entitled to apply The requirements are set through the Criteria for Approval and Accreditation of Programmes established by NZQA under section 253(1)(d) and (e) of the Act and published in the Gazette of 28 July 2011 at page 3207 These policies and criteria are deemed, by section 44 of the Education Amendment Act 2011, to be rules made under the new section 253 In addition, for registered private training establishments, the criteria and policies for their registration require self-assessment and external evaluation and review at an organisational level in addition to the individual programmes they own or provide These criteria and policies are also deemed, by section 44 of the Education Amendment Act 2011, to be rules made under section 253 Section 233B(1) of the Act requires registered PTEs to comply with these rules NZQA is responsible for ensuring non-university TEOs continue to comply with the rules after the initial granting of approval and accreditation of programmes and/or registration The New Zealand Vice-Chancellors Committee (NZVCC) has statutory responsibility for compliance by universities This report reflects the findings and conclusions of the external evaluation and review process, conducted according to the EER process approved by the NZQA Board The report identifies strengths and areas for improvement in terms of the organisation s educational performance and capability in self-assessment External evaluation and review reports are one contributing piece of information in determining future funding decisions where the organisation is a funded TEO subject to an investment plan agreed with the Tertiary Education Commission External evaluation and review reports are public information and are available from the NZQA website (wwwnzqagovtnz) Information relevant to the external evaluation and review process, including the publication Policy and Guidelines for the Conduct of External Evaluation and Review, is available at: NZQA Ph E qaadmin@nzqagovtnz wwwnzqagovtnz 22