Report of External Evaluation and Review

Transcription

1 Report of External Evaluation and Review Design and Arts College of New Zealand Limited Confident in educational performance Confident in capability in self-assessment Date of report: 1 April 2014

2 Contents Purpose of this Report... 3 Introduction TEO in context Scope of external evaluation and review Conduct of external evaluation and review... 6 Summary of Results... 7 Findings... 9 Recommendations Appendix MoE Number: 8656 NZQA Reference: C12795 Date of EER visit: November

3 Purpose of this Report The purpose of this external evaluation and review report is to provide a public statement about the Tertiary Education Organisation s (TEO) educational performance and capability in self-assessment. It forms part of the accountability process required by Government to inform investors, the public, students, prospective students, communities, employers, and other interested parties. It is also intended to be used by the TEO itself for quality improvement purposes. Introduction 1. TEO in context Name of TEO: Type: Location: Delivery sites: Design and Arts College of New Zealand Limited (D&A) Private training establishment Head Office: 61 Cambridge Terrace, Christchurch Christchurch campuses at: 54 Oxford Terrace and 66B Wharenui Road First registered: 16 December 1991 Auckland campus: 3 Wakefield St, Auckland City Courses currently delivered: Certificate in Design and Arts (Foundation) (Level 4) Certificate of Art (Advanced) (Level 5) Diploma of Art (Level 6) Diploma of Art (Advanced) (Level 7) Certificate in Contemporary Photography (Level 5) Diploma of Contemporary Photography (Level 6) Diploma of Architectural Design and Technology (Level 7) Diploma of Interior Design and Decoration (Level 5) Certificate of Communication Arts and Design (Level 5) Diploma of Communication Arts and Design 3

4 (Level 6) Diploma of Fashion Design (Level 5) Diploma of Fashion Design (Advanced) (Level 6) Certificate of Makeup Design and Production (Level 4) Code of Practice signatory Number of students: D&A is a signatory to the Code of Practice for the Pastoral Care of International Students for students aged 18 years and over. Domestic: 182 equivalent full-time students (11 per cent Māori, 4 per cent Pasifika) International: five full-time students Number of staff: Scope of active accreditation: Distinctive characteristics: Recent significant changes: 25 full-time equivalents In addition to the programme accreditations listed on the previous page, the provider holds accreditation for a variety of domains and unit standards in the design and arts fields, up to level 6. D&A provides education and training to domestic and international students and delivers 13 levels 4-7 certificate and diploma courses across the design and art fields: Fine Arts, Photography, Architecture, Interior Design, Graphic Design, Fashion Design, and Makeup Design. There are intakes in January and July of each year, the academic year is 45 weeks, and courses vary in length from six months to three years. D&A also jointly delivers the fourth year of a level 7 Bachelor of Fine Arts degree in partnership with Whitecliffe College of Arts and Design. In 2010 D&A was purchased by Arowana and Co, an Australian-based fund management company. Arowana has purchased five New Zealand private training establishments, including D&A, which it operates under the collective brand of Intueri Education Group. D&A recently established a campus in the Auckland CBD. In 2013 the organisation has delivered only the Certificate in Design and Arts (Foundation) in Auckland, but intends to establish a wider portfolio of programmes in Auckland from Previous quality This is D&A s second scheduled external evaluation and review (EER). The first EER was conducted in late 2009, 4

5 assurance history: at which time NZQA was Confident in D&A s educational performance and Highly Confident in its capability in selfassessment. Other: Prior to the Christchurch earthquakes in September 2010 and February 2011, D&A was located in premises in the Christchurch CBD. D&A was severely affected by the earthquakes, resulting in the total loss of premises in Worcester St and the subsequent need to teach from temporary premises for a substantial period. Fine Arts courses are still in temporary premises and will finally be moving to more permanent premises in January Student numbers, particularly international students, have significantly reduced since the earthquake, with consequent downstream reductions in funding and staffing. 2. Scope of external evaluation and review The scope of the EER consisted of the mandatory focus areas: Governance, management and strategy International students In addition, the following focus areas were selected: The Fine Arts portfolio of qualifications (levels 4-7) including the Certificate in Design and Arts (Foundation) (Level 4) through to the Advanced Diploma in Fine Arts (Level 7), for a total of 420 credits. The portfolio enables students to staircase from level 4 to higher qualification levels. Also, some students progress from level 7 to the fourth year of the Whitecliffe degree (on site in Christchurch). The Foundation programme is also offered in Auckland. Diploma of Interior Design and Decoration (Level 5). This is D&A s largest single programme, with increased demand arising from the Christchurch rebuild. 5

6 3. Conduct of external evaluation and review All external evaluation and reviews are conducted in accordance with NZQA s published policies and procedures. The methodology used is described fully in the web document Policy and Guidelines for the Conduct of External Evaluation and Review available at: The TEO has an opportunity to comment on the accuracy of this report, and any submissions received are fully considered by NZQA before finalising the report. The EER was conducted over two and a half days. Prior to the EER visit, the lead evaluator communicated by telephone and with the provider to discuss and agree to the scope of and process for the EER. The evaluation team consisted of two evaluators. A half-day EER visit by the team was made to the D&A site in Auckland, including a visit to Whitecliffe College of Arts and Design, followed by two days spent at the Christchurch site. At both sites, the evaluation team reviewed a range of documentation and met with the owners, management, teaching and student support and administration staff, and students. External stakeholders were contacted by telephone. 6

7 Summary of Results Statement of confidence on educational performance NZQA is Confident in the educational performance of Design and Arts College of New Zealand Limited. Students at D&A are achieving very good results. Although the EER team applauds these results, because of the gaps identified below and in the body of this report, NZQA does not have a high level of confidence in D&A s educational performance. Tertiary Education Commission (TEC) educational performance data indicates that qualification completion at D&A was on average 91 per cent in 2011 and 78 per cent in (During the same period, course completions increased by 5 per cent.) D&A attributes the 2012 drop in performance to the fall-out from the Christchurch earthquakes. This is consistent with anecdotal evidence from other Christchurch providers. Qualification completion for semester one 2013 has increased to over 80 per cent, and subsequent semester results are expected to reflect similar improvement. D&A has identified that although achievement levels for Māori and Pasifika students are improving, these students still fall below the rest of their cohorts, as shown in Table 1 in Findings 1.1. D&A is still developing strategies to address Māori and Pasifika achievement. As student numbers increase at the Auckland campus, where there is already a considerably higher proportion of Pasifika enrolments, this gap will need to be addressed sooner rather than later. The organisation is strongly employment-focused, and driven, at local level, by the belief that students must receive good value from their experience and their qualification. This belief is borne out by, for instance, the fact that 100 per cent of interior design graduates in 2012 are now working in the industry. Progression to higher-level study from the Foundation programme in 2012 was double the national average for similar programmes. Feedback to the EER team from students indicates that D&A programmes have an appropriate mix of theory and practice, and that they are delivered and assessed in a manner that enables the students to understand and apply the material being presented. Teachers are enthusiastic and relate well with their students. However, the high proportion of contract teaching staff and their perception of the nature of their tenure, limits the opportunity for those staff to participate in, and contribute to, the academic and professional development activities of the organisation. D&A students are well supported, particularly through the goodwill and enthusiasm of the staff. The students interviewed at this evaluation were mostly positive about their experience at D&A. Staff are very student-focused but feel that they are not well supported by the governance of the organisation. 7

8 The organisation is well managed at the local level, although the aftermath of the Christchurch earthquakes and recent changes brought about by the change of ownership have resulted in confusion about roles, structures and systems which, if not addressed, may put at risk the excellent outcomes that D&A has achieved in recent years. Statement of confidence on capability in self-assessment NZQA is Confident in the capability in self-assessment of Design and Arts College of New Zealand Limited. D&A has been in business for 30 years and is long-term in its thinking. The organisation has established a very reflective culture that encourages students, staff and stakeholders to discuss their performance and put forward ideas for improvement. The good results to date have come about by the organisation knowing and conducting its business soundly and drawing on the depth of experience its staff have to offer. However, the new management structure of the Intueri Education Group has reduced staff ability to positively influence change. While D&A has summarised a wealth of educational performance information, the organisation acknowledges that more could be done to analyse it and use that analysis to bring about useful improvements. Staff are closely attuned to the needs of the students and the art and design industry, but the part-time nature of the employment of most of the staff means that they have limited opportunity to participate in ongoing and meaningful self-assessment. For instance, staff teaching one of the focus programmes reported that they had not met as a team for several months, and therefore the sharing of information about student progress and ideas for supporting improved achievement, while still occurring, tended to be informal and ad hoc. D&A has systems for gathering learner, graduate and industry feedback, but there was patchy evidence that this information was being systematically used to make improvements to the programmes. The challenge for D&A is to ensure that the good practice evident at programme level is expanded to the wider organisation s systems for quality improvement. D&A management and governance are not sufficiently focusing their selfassessment efforts on developing a systematic approach to gain a greater level of understanding of educational performance. Therefore, they are unable to use any such understanding to bring about improvements linked to valid and reliable achievement data and valued outcomes for learners. 8

9 Findings How well do learners achieve? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. Students at D&A are achieving very good results, as demonstrated by the course and qualification completion rates listed below. D&A attributes the 2012 drop in qualification completion to the consequences of the Christchurch earthquakes. Table 1: Course and qualification completion rates, Course completion Qualification completion Course completion (Māori learners) Course completion (Pasifika learners) (provisional) 74% 79% 81% 91% 78% 86% 60% 63% 70% 56% 65% 75% Although the Certificate in Fine Arts (Foundation) is open-entry, D&A has an extensive interview and information process to enable students to self-select from an informed position. It is therefore expected that every student who enrols on the programme will succeed, and staff work tirelessly to keep all students focused on completion. D&A makes good use of benchmarking information from multiple sources and has consistently exceeded achievement commitments to the TEC as detailed in the investment plan. For instance, course and qualification completions in 2012 were, respectively, 9 per cent and 8 per cent above the TEC investment plan commitment. D&A monitors student attendance in recognition of the correlation between attendance and achievement. However, follow-up of students when nonattendance occurs is not always immediate due to lack of resources or communication between staff. 1 The findings in this report are derived using a standard process and are based on a targeted sample of the organisation s activities. 9

10 Staff at D&A demonstrated a good understanding of the factors that lead to student achievement and regularly analyse and discuss ideas for improving achievement. However, they do not have the opportunity to formally meet regularly to communicate ideas for improving course delivery and learner achievement. Fortunately, the current staff members have a very collegial approach and communicate informally by , telephone and text to share information. Students reported that they received timely, accurate and supportive feedback on their progress from the D&A staff. Students meet with a staff member regularly for one-to-one discussion around progression, achievement and goal-setting. Staff and employers reported that students are learning useful technical skills, and staff see positive changes in students confidence and attitudes as they progress. 1.2 What is the value of the outcomes for key stakeholders, including learners? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. Students gain worthwhile skills in the programmes that D&A delivers, which make them employable in the art and design industries. Recent graduates and employers report that the skills and knowledge gained were immediately applicable to the workplace and are valued by both the employee and employer. Apart from graduates of the Foundation programme, 100 per cent of D&A graduates from 2012 are now in relevant full-time employment or relevant further study. It is notable that D&A considers that the relevance of subsequent employment to the programme studied is a critical indicator of value. Progression to higher-level study from the Foundation programme was 44 per cent in 2012, which is higher than the 22 per cent national average for similar programmes. D&A considers that the merit of the programmes lies not only in the long-term professional outcomes for students but also in attributes such as the confidence, self-esteem and positive attitudes they acquire on the courses. The introduction of internships into many of the programmes has helped students to gain meaningful interaction with potential employers, and feedback from internship employers has enabled D&A to gain a better appreciation of the degree to which students have acquired the attributes mentioned above. D&A staff conduct exit meetings with every graduating student and then follow up with a conversation with all graduates three months after graduation to ascertain their destination, and also to gain feedback from graduates and their employers about how well the programme prepared them for work or further education. The organisation also keeps in touch with graduates through Facebook, by involving them in training and through a variety of informal communications. Information 10

11 gained from this engagement is then gathered, analysed and discussed. Analysis is in turn used appropriately to make improvements to programmes. 1.3 How well do programmes and activities match the needs of learners and other stakeholders? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. D&A has in the past year put in place processes to ensure that regular and ongoing interaction with its stakeholders at all levels of the organisation is a core feature of the operation. Mechanisms developed include ongoing contact with employers and industry professional bodies, client surveys, graduate surveys and participation in professional bodies and industry forums. The organisation has also developed a useful discussion record form for staff to record details of engagement with industry or professional bodies so that the information can then be used in subsequent self-assessment. Student evaluations are conducted to a regular schedule and gather students feedback on their experiences of different aspects of their programmes and college life, including programme content and delivery, facilities and staff effectiveness. However, several staff reported that they had not personally seen the results of student evaluations for several years. A more systematic approach to recording, analysing and discussing this information with staff may help identify trends and lead to further enhancements to programme delivery. Efforts by the teaching staff to meet the individual, personal and academic goals of every student are a strong feature of the organisation. This activity tends to rely on the relationships that tutors have with their students rather than a more systematic approach across the organisation. The design of the various suites of programmes enables students to progress to the limit of their capability or need, then exit with a valid qualification. D&A has engaged well with the current NZQA Targeted Review of Qualifications process. D&A staff are represented on working and advisory committees for the Design and Arts review. This in itself is testimony to the high regard in which the sector holds D&A, its staff and its programmes. D&A staff are variously involved, both professionally and informally, in the design and arts sector and are well respected for the part they play, often in leadership roles. These ongoing connections benefit students in that D&A staff are able to keep abreast of developments in the industry and tailor their teaching to address emerging changes in industry. 11

12 1.4 How effective is the teaching? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Adequate. Teachers and students relate very well to each other. Students spoke highly of D&A s teaching staff and the evaluators saw evidence of good teaching taking place by passionate teachers. Students liked the open, friendly style and found their learning activities interesting and challenging. Students report that assessments are returned to them promptly and that they contain comprehensive feedback, indicating that staff acknowledge that timely feedback to students is important so that students have the opportunity to reflect and build on their learning. Students interviewed noted that staff were responsive to concerns or issues raised. Students have limited access to teaching staff outside of formal class hours. The current high proportion of part-time teachers 2 presents a challenge to D&A to ensure that students have reasonable access to staff outside of class hours and that part-time staff contribute to the academic capital of the organisation. For instance, although the contracts for part-time staff technically allow for input other than time directly related to teaching, staff who were teaching one of the focus programmes reported that they had not met as a team for several months, and therefore the sharing of information about student progress and ideas for supporting improved achievement, while still occurring, tended to be informal and ad hoc. Part-time staff are not subject to the same degree of induction, performance review and management as full-time staff, and reported that they have less access to institute-sponsored professional development. This view was subsequently challenged, in that four part-time staff were participating in professional development that was financially supported by the organisation. D&A is providing support to several of its staff to undertake adult teaching qualifications with a view to having all teaching staff either (adult teaching) qualified or in training by the end of Professional development planning for teaching staff is ad hoc. For instance, D&A has recognised that the lower achievement rates for Māori and Pasifika students need to be addressed, but this is not evident in professional development planning for teaching staff. A more systematic approach to professional development planning linked to self-assessment findings may lead to improvements in student outcomes. D&A has plans to put in place a peer observation and review process for teaching staff, but this only occurs informally at present. Feedback from students, both formal and informal, provides teaching and management staff with valuable information which is often used to identify new or 2 Approximately two-thirds of teaching staff are part-time. 12

13 alternative learning strategies and to generally improve educational outcomes for students. Despite evidence that student feedback is analysed and reported on, some teaching staff reported that the information from student feedback does not always reach them. As the organisation s self-assessment processes expand, a more systematic approach, which should include reporting back to students on the actions taken following the gathering of feedback, would be preferable and may improve educational outcomes for students. 1.5 How well are learners guided and supported? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. D&A staff are passionate about the care and support they provide for the students. The family culture at programme level is clearly evident, with open doors and an easy informality between staff and students. Although D&A staff provide good support, in the intuitive belief that there is likely to be a positive correlation between student support and achievement, to date the organisation has not formally analysed the link between the two. A more analytical and systematic approach to student support may lead to improved outcomes for students. Pre-enrolment information and guidance is appropriate to the programmes. When students enrol, they know what to expect in the programme and what outcomes are likely. The students receive a comprehensive handbook and are well informed and guided about what to expect in the programmes. There is a designated international student support staff member in Christchurch with 24-hour phone contact should the students require assistance when away from the site. At present, there are no international students in Auckland, but it is intended that the Auckland campus manager will fill this role. Accommodation for international students is managed in-house by the international support coordinator, although there is only a small number of students in homestay accommodation. The international student support coordinator has overall responsibility for ensuring that Code of Practice requirements are met, and in the past year he has attended professional development offered by the Code Office. D&A has completed a selfassessment of compliance with the code using the template provided by the Code Office, and has furnished the required annual attestation to the Code Office, confirming compliance. 13

14 1.6 How effective are governance and management in supporting educational achievement? The rating for performance in relation to this key evaluation question is Adequate. The rating for capability in self-assessment for this key evaluation question is Adequate. D&A has a sound understanding of the financial aspects its business. Its purpose and direction are clear and articulated, although there is evidence of incongruence between the corporate and operational areas in the way in which strategy and purpose are interpreted and embraced. There appears to be a lack of corporate understanding of educational processes. The new corporate structure means that some reporting is vertical within D&A while other reporting is horizontal within the Intueri Education Group. This in itself is not a problem, in fact it should be a strength, but at this point many staff are unclear about their lines of reporting and responsibilities. Staff performance management systems are ad hoc and staff receive little feedback on their performance. This in turn translates into ad hoc professional development planning. The campuses in Oxford Terrace, Christchurch and in Auckland are ideally located, spacious and well-appointed with physical and learning resources. However, the campus at Wharenui Rd, Christchurch, where Fine Arts and Photography students are taught, is inadequate and unsuitable. While it is acknowledged that the premises have been temporary since the earthquakes and that D&A has recently acquired new premises for 2014, the Wharenui campus has been used by D&A for two and a half years and the organisation could have done more, at minimal expense, to make the premises more appealing, workable and safe for students and staff. For instance, a coat of paint to the interior, the provision of a temporary office for staff, and some modifications to the plumbing so that the pipes do not freeze in winter, making the toilets inoperable, would have been worthwhile improvements to student and staff comfort. There was evidence that resourcing for programmes is not planned consistently or effectively. For instance, in Christchurch, Foundation students had little access to cameras for photography. In order to meet the needs of the curriculum, the tutor often brought her own cameras to class for student use. The tutor said she had regularly requested new and additional cameras, but had been told there were insufficient funds. Meanwhile, at the Auckland campus, the evaluators noted a set of at least a dozen high-quality new cameras which the staff said were rarely used. The current roll-out of a new student management system should result in good quality data becoming increasingly available to management to enable them to monitor and improve student achievement. This information also needs to be 14

15 distributed across the organisation so that it is readily accessible to teaching staff and that they are able to use it effectively to inform improvements. This report should serve as a clear signal to D&A and its governing body that it is below minimum expectations in terms of the effectiveness of its governance and management in supporting educational achievement. The challenge for D&A is to direct its efforts towards developing a greater level of understanding of educational performance and bringing about improvements by more systematic analysis and consequent action from self-assessment activities. 15

16 Focus Areas This section reports significant findings in each focus area, not already covered in Part Focus area: Governance, management and strategy The rating in this focus area for educational performance is Adequate. The rating for capability in self-assessment for this focus area is Adequate. 2.2 Focus area: International students The rating in this focus area for educational performance is Good. The rating for capability in self-assessment for this focus area is Good. 2.3 Focus area: Fine Arts portfolio of qualifications (levels 4-7) The rating in this focus area for educational performance is Good. The rating for capability in self-assessment for this focus area is Good. 2.4 Focus area: Diploma of Interior Design and Decoration (Level 5) The rating in this focus area for educational performance is Good. The rating for capability in self-assessment for this focus area is Good. 16

17 Recommendations NZQA recommends that Design and Arts College: Continue development in the collection of quality data for self-assessment Build capability in analysing this data, discussing it as a whole organisation, including all staff, and using it to make improvements Ensure staff undertake a performance appraisal at least once each year Ensure that the information gained from students is appropriately analysed and used to improve teaching and programme design Improve staff understanding of lines of reporting and responsibility Ensure that the new premises for Fine Arts and Photography are appropriately equipped to enable the courses to be adequately taught and assessed. 17

18 Appendix Regulatory basis for external evaluation and review External evaluation and review is conducted according to the External Evaluation and Review (EER) Rules 2013, which are made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. Self-assessment and participation in external evaluation and review are requirements for maintaining accreditation to provide an approved programme for all TEOs other than universities. The requirements are set through the NZQF Programme Approval and Accreditation Rules 2013, which are also made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. In addition, the Private Training Establishment Registration Rules 2013 require registered private training establishments to undertake self-assessment and participate in external evaluation and review, in accordance with the External Evaluation and Review Rules (EER) 2013, as a condition of maintaining registration. The Private Training Establishment Registration Rules 2013 are also made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. NZQA is responsible for ensuring non-university TEOs continue to comply with the rules after the initial granting of approval and accreditation of programmes and/or registration. The New Zealand Vice-Chancellors Committee (NZVCC) has statutory responsibility for compliance by universities. This report reflects the findings and conclusions of the external evaluation and review process, conducted according to the External Evaluation and Review (EER) Rules The report identifies strengths and areas for improvement in terms of the organisation s educational performance and capability in self-assessment. External evaluation and review reports are one contributing piece of information in determining future funding decisions where the organisation is a funded TEO subject to an investment plan agreed with the Tertiary Education Commission. External evaluation and review reports are public information and are available from the NZQA website ( The External Evaluation and Review (EER) Rules 2013 are available at while information about the conduct and methodology for external evaluation and review can be found at NZQA Ph E qaadmin@nzqa.govt.nz 18