Report of External Evaluation and Review

Transcription

1 Report of External Evaluation and Review Sheffield Language School Co Limited trading as Sheffield College of Studies Confident in educational performance Confident in capability in self-assessment Date of report: 26 July 2011

2 Contents Purpose of this Report... 3 Introduction Sheffield in context Scope of external evaluation and review Conduct of external evaluation and review...4 Summary of Results... 5 Findings... 7 Recommendations Further Actions Appendix MoE Number: 7513 NZQA Reference: C04794 Date of EER visit: 25 May

3 Purpose of this Report The purpose of this external evaluation and review report is to provide a public statement about the Tertiary Education Organisation s (TEO) educational performance and capability in self-assessment. It forms part of the accountability process required by Government to inform investors, the public, students, prospective students, communities, employers, and other interested parties. It is also intended to be used by the TEO itself for quality improvement purposes. Introduction 1. Sheffield in context Location: Type: Level 7, 246 Queen Street, Auckland Private Training Establishment First registered: 2002 Number of students: Number of staff: International: 98 equivalent full-time students 12 full-time equivalents Scope of active accreditation: Certificate in English for Health Professionals (Level 5) Premier Intensive English Language Course Premier Standard English Language Course Premier Intensive IELTS (International English Language Testing System) Course Premier Intensive Academic English Course Sheffield TESOL (Teachers of English to Speakers of Other Languages) Course (Introductory) (Level 4) Sites: Distinctive characteristics: Recent significant changes: Previous quality assurance One site Sheffield College of Studies (Sheffield) offers a range of General English courses to international students. The courses are from two to 48 weeks in duration. Started Certificate in English for Health Professionals course in February 2011, but this has been discontinued for lack of students. Sheffield substantially met the requirements of the last 3

4 history: quality assurance visit by NZQA, an audit in Scope of external evaluation and review The focus areas selected for this EER were: Governance, management, and strategy International student support General English. Governance, management, and strategy and international student support are mandatory focus areas. General English was chosen as a focus area as it effectively covers most of Sheffield s current provision. 3. Conduct of external evaluation and review All external evaluation and reviews are conducted in accordance with NZQA s published policies and procedures. The methodology used is described fully in the web document Policy and Guidelines for the Conduct of External Evaluation and Review available at: The EER team comprised two lead evaluators. The team visited the Queen Street site over one and a half days and met with the director, the director of studies, the administration assistants who support the students and ensure compliance with the Code of Practice for the Pastoral Care of International Students, tutors from all programmes, and students from the IELTS programme. Sheffield Language School Company Limited (trading as Sheffield College of Studies) has had an opportunity to comment on the accuracy of this report, and submissions received have been fully considered by NZQA before finalising the report. 4

5 Summary of Results Statement of confidence on educational performance NZQA is Confident in the educational performance of Sheffield Language School Co Limited trading as Sheffield College of Studies. Sheffield has collected good quality data including completion and achievement results and information on graduate destinations. The achievement data has been well analysed and the achievements are good. The destination data continues to be gathered and Sheffield is researching better, more efficient processes of getting good quality information. Learners complete courses and acquire good English language skills. Course completion and achievement rates by class range from 79 to 100 per cent. Students spoken to by the evaluation team affirmed that they were acquiring good language skills and the courses were meeting their needs. The graduate destination information for 2010 shows that about 50 per cent go on to further study, 26 per cent to employment in New Zealand, and 24 per cent return to employment or further study in their home country. Some graduates added comments indicating that their study at Sheffield had been very valuable to them. Evidence for good achievement by learners and the value of the outcomes for them and other stakeholders include: High course completion rates Favourable student evaluations Good evidence of meeting needs from student comments and evaluations and graduate destination data Good evidence of adequate quality in processes and outcomes Evidence of graduates being accepted for higher-level courses at other tertiary institutes Some internal pathways from general English to the IELTS and TESOL courses. While there is generally good evidence that Sheffield is meeting the needs of learners and other stakeholders, and of quality in the processes that contribute to learning, much of this evidence is generated at present from manual examination of files. A newly installed student management system should enable reports to be produced more easily. The director of studies is pivotal in the current system. There is some evidence of wider, more holistic outcomes such as contributions to the community and fostering cultural experiences. 5

6 Statement of confidence on capability in self-assessment NZQA is Confident in the capability in self-assessment of Sheffield Language School Co Limited trading as Sheffield College of Studies. Sheffield staff have embraced the need for self-assessment and are collecting data and information purposefully to measure progress and look for worthwhile improvements. The quality of the data is good and staff have already used some analysis of data to make improvements and inform practice. For example, Sheffield has changed the previous system of half-term or six-week intervals between assessments to a system where students are assessed at least on some elements of English every week. There are indications that this new system is improving outcomes for the learners, although it is too early for hard evidence of this. While NZQA is confident in Sheffield s capability in self-assessment, more evidence of wide consultation and reflection by staff and good information on the effects of the improvements that have recently been put in place would be required for a higher confidence-level judgement. TEO response Sheffield Language School Co Limited trading as Sheffield College of Studies acknowledges the factual accuracy of this report. 6

7 Findings How well do learners achieve? The rating for performance in relation to this key evaluation question is Good. Learners complete courses and acquire good English language skills. For students to successfully complete their course, they must meet the attendance requirement and achieve the class standard, i.e. Intermediate level. Sheffield thus measures completion and achievement rates and these range from 79 to 100 per cent across the levels. Also, there is good evidence that at least 75 per cent of students who study at Sheffield for at least 12 weeks became capable of studying at a more advanced level. For the pre-intermediate programme in 2010, 96 per cent of the students who studied for at least 12 weeks progressed to a higher level. These rates represent good achievement. Learners at Sheffield complete useful qualifications and many engage with further study. The evaluation team s analysis of Sheffield s records of those who studied either general English or for IELTS testing during 2010, showed that about 43 per cent went on to further study in New Zealand, 28 per cent to work in New Zealand, and 22 per cent returned home to either work or further study. Sheffield has achieved very good rates of positive outcomes. While these results are good, better data on improvement of IELTS scores, wider information on trends, benchmarking with similar institutes, and analysis of this data would be needed to prove excellence in this area. The rating for capability in self-assessment for this key evaluation question is Good. Sheffield has started to improve the collection and analysis of data. Its own self-assessment, for example, shows that the Elementary area has the lowest rate of improvement in English skills, and staff are reflecting on this to see what improvements could be made. The trend data collected shows that the 2010 results for all levels show improvements over Sheffield has started collecting information on graduate destinations but has yet to analyse and reflect on this to gain the full benefits. Self-assessment has led to the change of testing regime from once every six weeks to weekly testing. This weekly testing picked up a need for better teaching of grammar which has been addressed, and the staff believe that the extra assessment is resulting in faster learner progress. Sheffield has also identified that it needs to ascribe accurate IELTS scores on entry to the IELTS class so that improvements can be measured accurately. 1 The findings in this report are derived using a standard process and are based on a targeted sample of the organisation s activities. 7

8 1.2 What is the value of the outcomes for key stakeholders, including learners? The rating for performance in relation to this key evaluation question is Good. Graduates of Sheffield gain employment and contribute to their communities. Staff have collected a comprehensive file of letters and s and are now supplementing these sources with information from Facebook. There was good evidence of graduates obtaining work in New Zealand and overseas because of the knowledge and skills they gained through Sheffield. These longer-term outcomes are of value to the learners, their communities, and the wider society. Learners gain value from the pathways provided, both internally within Sheffield and to further education at other tertiary education organisations (TEOs). About half of the students undertaking the IELTS course over the last 18 months and about 7 per cent of the TESOL classes have come directly from Sheffield training. Over 50 per cent of Sheffield graduates progress to further study in New Zealand or overseas. These too are valuable outcomes for the learners and their communities. Sheffield makes a contribution to community life. Most notably, Sheffield supported the production of Our Stories, a Cultural Stage and Auckland Playback Theatre production, part-funded by Auckland City. Sheffield provided free space for rehearsals, and students from other TEOs participated along with a large number of Sheffield students. Sheffield staff and students affirmed that those studying at Sheffield enjoy a rich multicultural experience. Over 30 nations are represented in the student body and, although Koreans are the largest group, comprising about a third of the students, they speak to each other and their classmates in English which contributes greatly to the inclusive culture of the school. Sheffield also actively promotes weekend excursions which enable students to gain a wider New Zealand experience. These wider cultural experiences add an important value to studying at Sheffield. The rating for capability in self-assessment for this key evaluation question is Adequate. Self-assessment in this area is just beginning at Sheffield and it was clear that better quality data is being gathered to measure the value of outcomes. Some changes have been made as a result of value considerations. One example is that three levels of certificates have been introduced so that learners have a document of value. Those who only participate for short times are awarded attendance certificates. Those who attend for longer receive attainment certificates for participating at a given level without completing the course, or achievement certificates if they complete. Tracking Sheffield graduates has started but is still in the data collection stage, with some limited analysis so far. As Sheffield staff do more analysis and reflect on the results, they should get more value from more coherent and comprehensive self-assessment. 8

9 1.3 How well do programmes and activities match the needs of learners and other stakeholders? The rating for performance in relation to this key evaluation question is Good. Sheffield has updated and improved its placement tests and procedures in the light of its self-assessment. The written section of the test now takes one hour and can be applied and scored by administration staff. Listening and speaking skills are validated by a classroom teacher later the same morning. This establishes that the learner is in the most appropriate ability group. Students arrive every week so it is important to have accurate, efficient, and robust procedures to ensure that students are studying at the most appropriate level. Sheffield identifies its stakeholders as the students, their families, and their agents. The director has surveyed agents and found them to be happy with the ESOL instruction provided at Sheffield. Sheffield operates in a very competitive environment and the good opinion of agents is important to economic survival. Sheffield is just starting to develop relationships with TEOs that offer more advanced training and education. Sheffield s business plan identifies a number of providers that it has agreements with, but few of these feature as destinations for the graduates Sheffield has surveyed. Auckland University of Technology (AUT) is not on the list, but about 20 per cent of recent graduates have enrolled there. Further self-assessment may show which organisations Sheffield should seek to form partnerships with to meet the needs of its students. The rating for capability in self-assessment for this key evaluation question is Good. Self-assessment of information gathered by Sheffield has led to the better matching of student needs by: Introducing weekly testing and feedback Distributing detailed weekly timetables so that students can plan their work Increasing the frequency of student evaluations. Students now evaluate the Sheffield experience twice per course. The evaluation forms are analysed by the director of studies and feedback is given to tutors and administration staff. Sheffield is currently meeting the needs of those students who want a multicultural experience, but it will have to continually assess the balance between enrolling large numbers of students from one country and maintaining the present culture. Sheffield plans to improve its self-assessment information by conducting initial IELTS testing of those joining the IELTS course. Also, while there is anecdotal evidence that those proceeding overseas are meeting the needs of their communities, more valid and robust information may well lead to better connection between Sheffield and valued overseas pathways for graduates. This also applies to seeking feedback from institutes, such as AUT, that enrol graduates into higher-level courses to affirm the students are being well prepared. 9

10 1.4 How effective is the teaching? The rating for performance in relation to this key evaluation question is Good. Teachers and learners at Sheffield relate effectively to one another. There was good evidence from student evaluations and from the teachers and learners interviewed by the evaluation team of interactions that encourages learning. Classes are small and are split up if they grow above 15 learners. There was also evidence that learners collaborated well with one another. For example, there is a high level of compliance with the English-only policy. These positive relationships between learners and teachers and between learners are likely to encourage engagement with learning. Assessment at Sheffield provides learners and teachers with useful feedback on progress. Learners are getting formative assessment every week and staff believe that the extra assessment effort that the weekly system requires from them is resulting in faster learner progress. Moderation, both internal and external, is thorough. Sheffield has an arrangement with a nearby PTE to moderate each other s assessments. This is providing a productive flow of ideas, for example with respect to innovative assessment processes, and is a form of benchmarking. Learning activities and resources at Sheffield are effective in engaging learners. There was evidence that: The teachers are well qualified The director of studies appraises staff systematically and professional development is provided Students apply their learning by paired activities in class. The results that learners achieve at Sheffield affirm the effectiveness of the learning activities. The rating for capability in self-assessment for this key evaluation question is Good. Self-assessment of effective teaching has led to the following improvements: Professional development for staff, targeting grammar teaching Changes to the staff appraisal form Building professional development into staff activities using collective strengths, for example using the TESOL tutor to facilitate staff discussion of ideas and methods for grammar teaching Improving assessment, for example by compiling judgement criteria for speeches and presentations. This is good use of self-assessment to improve educational performance. 10

11 1.5 How well are learners guided and supported? The rating for performance in relation to this key evaluation question is Good. Sheffield provides its learners with comprehensive and timely study advice and information. The evidence for this includes: An informative and up-to-date website and prospectus A thorough student orientation package and handbook A standard induction presentation which has evolved through self-assessment of student evaluations Evidence from students interviewed by the evaluation team Good signage throughout the premises identifying who to go to for information, assistance, and complaints Good information on the Code of Practice for the Pastoral Care of International Students (COP) Good evidence of self-audit of performance and processes to ensure compliance with the COP. Sheffield staff can counsel the majority of students in the student s first language if necessary. Administration has good systems for checking attendance and is proactive in contacting students by telephone on the same day if they are absent without excuse. This close attention and support contributes to a learning culture where learners feel valued, supported, and understood. The rating for capability in self-assessment for this key evaluation question is Good. Sheffield s self-assessment of the results of its guidance and support work with students has resulted in some worthwhile improvements. Examples are Improvements to the induction process and better documentation to ensure consistency Improved documentation and procedures for ensuring good attendance Minuted monthly meetings of student representatives and responding to suggestions Improving the evaluation form that students complete. The review team viewed three iterations of this Undertaking exit surveys and improving the exit survey form. More analysis and reflection is needed in some areas, and systems for assessing the effects of improvements still need to be developed. 11

12 1.6 How effective are governance and management in supporting educational achievement? The rating for performance in relation to this key evaluation question is Good. Governance and management support educational achievement by: Setting clear goals for student outcomes Recruiting and retaining qualified staff Having a sound quality management system, regularly reviewed and being improved with the addition of processes Maintaining adequate resources. Sheffield has clear purpose and direction. Its mission is to offer educational programmes that prepare students to be competent and able in a rapidly changing international environment. The governance and management team comprises the director, the administrative manager, and the director of studies with advice from independent professionals as required. In addition to daily communication, formal minuted monthly meetings are held, and administration now participates in these also. Strategically, the organisation is seeking to offer courses within Sheffield that students can progress to. Sheffield is exploring the delivery of a diploma in language, culture, and management. This diploma and other options are clearly in line with Sheffield s purpose. This clear purpose and direction focuses decision-making. Leadership is effective at Sheffield. The director of studies has a key role, holding formal monthly meetings with the tutors and administration as well as undertaking daily informal communication. Communication and shared leadership at Sheffield is generally good, although some tutors felt that the director could have more frequent involvement with the students. Unpredictable change has been managed effectively. Sheffield applied for and was approved to teach English for Health Professionals (OET) and started a course. Unfortunately, other TEOs also ran courses for health professionals without approval, so this, and policy changes by the Nursing Council of New Zealand, have rendered the course uneconomic. The last student on the Sheffield OET course is now in the IELTS class and receiving additional support with medical terminology. This proactive response to change enhances Sheffield s credibility and integrity. The rating for capability in self-assessment for this key evaluation question is Good. Sheffield s self-assessment of its governance and management has led to improvements. Examples are: Using a strategic advisory committee including an executive from another TEO Including administration staff in management meetings and staff meetings Radically reviewing the quality management system to make it relevant and useful. 12

13 Self-assessment could be improved by involving all staff even more in reflecting on analysed data. Though there were many signs of self-assessment activity and consequent changes, it is too early to tell whether some of these changes are worthwhile improvements. In some cases, Sheffield was still working out how the effects of these changes might be measured. 13

14 Focus Areas This section reports significant findings in each focus area, not already covered in Part Focus area: Governance, management, and strategy The rating in this focus area for educational performance is Good. The rating for capability in self-assessment for this focus area is Good. 2.2 Focus area: International student support The rating in this focus area for educational performance is Good. The rating for capability in self-assessment for this focus area is Good. 2.3 Focus area: General English The rating in this focus area for educational performance is Good. The rating for capability in self-assessment for this focus area is Good. 14

15 Recommendations NZQA recommends in addition to those recommendations implied or expressed within the report, that Sheffield: continues to review the data it collects for self-assessment builds capability in analysing this data, and establishes a comprehensive self-assessment regime. Further actions The next external evaluation and review will take place in accordance with NZQA s policy and is likely to occur within four years of the date of this report. 15

16 Appendix Regulatory basis for external evaluation and review Self-assessment and external evaluation and review are requirements of course approval and accreditation (under sections 258 and 259 of the Education Act 1989) for all TEOs that are entitled to apply. The requirements are set through the course approval and accreditation criteria and policies established by NZQA under section 253(1)(d) and (e) of the Act. In addition, for registered private training establishments, the criteria and policies for their registration require self-assessment and external evaluation and review at an organisational level in addition to the individual courses they own or provide. These criteria and policies are set by NZQA under section 253(1)(ca) of the Act. NZQA is responsible for ensuring non-university TEOs continue to comply with the policies and criteria after the initial granting of approval and accreditation of courses and/or registration. The New Zealand Vice-Chancellors Committee (NZVCC) has statutory responsibility for compliance by universities. This report reflects the findings and conclusions of the external evaluation and review process, conducted according to the policies and criteria approved by the NZQA Board. The report identifies strengths and areas for improvement in terms of the organisation s educational performance and capability in self-assessment. External evaluation and review reports are one contributing piece of information in determining future funding decisions where the organisation is a funded TEO subject to an investment plan agreed with the Tertiary Education Commission. External evaluation and review reports are public information and are available from the NZQA website ( Information relevant to the external evaluation and review process, including the publication Policy and Guidelines for the Conduct of External Evaluation and Review, is available at: NZQA Ph E eeradmin@nzqa.govt.nz 16