Report of External Evaluation and Review

Transcription

1 Report of External Evaluation and Review Torchbearer Trust of New Zealand trading as Capernwray Bible School Confident in educational performance Confident in capability in self-assessment Date of report: 7 May 2015

2 Contents Purpose of this Report... 3 Introduction TEO in context Scope of external evaluation and review Conduct of external evaluation and review... 6 Summary of Results... 7 Findings Recommendations Appendix MoE Number: 8564 NZQA Reference: C18171 Date of EER visit: 25 and 26 February

3 Purpose of this Report The purpose of this external evaluation and review report is to provide a public statement about the Tertiary Education Organisation s (TEO) educational performance and capability in self-assessment. It forms part of the accountability process required by Government to inform investors, the public, students, prospective students, communities, employers, and other interested parties. It is also intended to be used by the TEO itself for quality improvement purposes. Introduction 1. TEO in context Name of TEO: Type: Torchbearer Trust of New Zealand trading as Capernwray Bible School Private training establishment (PTE) First registered: 16 April 1992 Location: Delivery sites: Courses currently delivered: Monavale Homestead, 3553 Cambridge and Te Awamutu Road, Cambridge 2/128 Horahora Road, Tirau Certificate in Christian Development and Leadership using Adventure-based Learning (Level 3) Certificate in Bible Training and Preparation for Christian Service (Level 3) Business Ministry Leadership programme (local course) Code of Practice signatory: Number of students: Signatory to the Code of Practice approved for student both under and over 18 years. Domestic: nil International: currently 65 students are enrolled in courses. Number of staff: Distinctive characteristics: 12 full-time and four-part time-staff. A number of contract and guest tutors lecture on both the certificate courses. Capernwray Bible School provides residential 3

4 Christian education. Capernwray is part of an international network of 26 Torchbearers Bible Schools. Most students are international and aged between 18 and 22 years, 80 per cent of whom are from North America. All students live on site for the duration of their study. Recent significant changes: These include: The appointment of a new national director/principal in January 2014 Changes to board membership since the previous external evaluation and review (EER) in 2011 Cessation of operations at a third site in Geraldine in May 2014 Previous quality assurance history: NZQA conducted a routine validation visit in October This resulted in three required actions and one recommendation. The required actions were carried out within the timeframe stipulated by NZQA, and were: Introduce processes and procedures to ensure internal pre- and post-assessment moderation is undertaken Update the student handbook to include correct NZQA contact information Complete conflict of interest form for a trustee The previous EER report of Capernwray was published in July NZQA was Highly Confident in Capernwray s educational performance and Highly Confident in Capernwray s capability in self-assessment. 4

5 2. Scope of external evaluation and review Following a scoping meeting between the lead evaluator and Capernwray s national director and the business manager, the following focus areas were agreed for inclusion in the EER. Governance, management and strategy This is a mandatory focus area. Other focus areas selected were: International student support All current Capernwray students are international. Certificate in Christian Development and Leadership using Adventure-based Learning (Level 3) (referred to as ABS in this report) This is a six-week course which may precede the Certificate in Bible Training and Preparation for Christian Service. This certificate is based in Tirau. Certificate in Bible Training and Preparation for Christian Service (Level 3) (referred to as Bible Training in this report) This is a 20-week course based in Cambridge. The six-week Adventure-based Learning Certificate can be cross-credited directly to this course. Capernwray students enrol on either or both of the above courses. In addition to these focus areas, Capernwray also offer the following programmes: Business Ministry Leadership programme This is a local course run on an as-needs basis. This course has been offered for several years and runs for 10 weeks, usually with eight to 12 students. This course was not included as a focus area as there are no courses on offer currently. Online learning Online learning consists of two modules made up of a series of units from the Certificate in Bible Training and Preparation for Christian Service. These are interest-only modules, and although initially designed to be assessed, there have been no students who completed both modules. There is no awarding of a certificate or recognition provided on completion. The number of enrolments in the past has been low and Capernwray has a clear intention to discontinue offering this mode of delivery. Therefore, online learning was not selected as a focus area. 5

6 3. Conduct of external evaluation and review All external evaluation and reviews are conducted in accordance with NZQA s published policies and procedures. The methodology used is described fully in the web document Policy and Guidelines for the Conduct of External Evaluation and Review available at: The TEO has an opportunity to comment on the accuracy of this report, and any submissions received are fully considered by NZQA before finalising the report. The EER of Capernwray Bible School took place with a team of two evaluators over two days, at both sites in Cambridge and Tirau. The EER included interviews with the national director, a board member, the academic dean, business manager, coordinator of ministries, pastoral dean, adventure ministry director and tutors. Stakeholder interviews included half of the total student body, locally based graduates and their employers. A range of documentation was reviewed, including: achievement data, various meeting minutes, academic processes, including moderation and programme review, and a variety of self-assessment processes, surveys, evaluations and reports. 6

7 Summary of Results Statement of confidence on educational performance NZQA is Confident in the educational performance of Torchbearer Trust of New Zealand trading as Capernwray Bible School. Key reasons for this are: Capernwray students, graduates and stakeholders strongly identify that the most important outcomes for students enrolling at Capernwray are engagement in a course that holistically enhances their well-being, develops and challenges them personally, and results in an experience of personal and spiritual growth. Capernwray has comprehensively and consistently met these outcomes for both courses over several years. Student achievement of the certificates is strong. The retention of students and certificate completion rates are consistently very high across all courses. On average, two or three students will either withdraw or not successfully complete the course out of the total number of approximately 145 students per year. These results are consistent with the previous EER. Capernwray, its staff and students actively engage with their local church communities and are valued for the positive contribution they make. As part of their practical studies, students and graduates have organised or participated in worthwhile events and services in the community over several years. All courses and associated activities are highly relevant and meet the theory, ministry and community components of the courses. Applying the theory learning to physical activities in the ABS course and in ministry placement for the Bible Training course are highly valued, optimise learning opportunities and effectively engage students. The effective support of students is a key priority. Capernwray is a residential education provider. Student welfare, support, attendance and progress, and academic support in ministry and community service are monitored closely by staff, some of whom are also residential and accessible 24/7. The exemplary pastoral care of students is a major contributor to supporting students to meet their personal needs and the highly valued key outcomes of personal and spiritual growth. Capernwray is well managed, with a clear mission and direction which anticipates change, monitors outcomes and is grounded in the philosophical and Christian practice of the Torchbearers International Schools. The PTE 7

8 has an established culture of support for staff and students and is well resourced to support learning and teaching. Teachers have the experience and relevant industry and academic qualifications to teach the subject areas. Internal and external moderation is occurring reliably. Not all assessors hold assessing and moderation unit standards, and mixed assessment grading methods creates the potential for inconsistency in assessor judgements. Academic rigour needs to be further improved by reviewing the assessment methodology and moderation of assessment guidelines. This has some impact but is not significant enough to undermine the substantial evidence that the most important and highly valued outcomes of students are being convincingly met. Statement of confidence on capability in self-assessment NZQA is Confident in the capability in self-assessment of Torchbearer Trust of New Zealand trading as Capernwray Bible School. Key reasons for this are: The national director leads highly effective self-assessment which is ongoing and comprehensive. Self-assessment at Capernwray includes the systematic review of all aspects of the school academic, pastoral and operational which is practised organisation-wide and involves all staff. Capernwray has a significant number of authentic processes which inform management and staff about key areas that support the educational performance of the students, including monitoring and analysis of achievement, course and tutor reviews, and Capernwray s priority area of pastoral care of students. Achievement and pastoral care areas have a numerical target for comparison and are benchmarked cohort to cohort and year to year. Capernwray s self-assessment processes, practices and activities are purposeful and embedded, providing meaningful information for review, and largely showing evidence of ongoing improvement and, importantly, areas of possible improvement. There is systematic collection and analysis of useful data. Appropriate staff are engaged in analysis and either discuss or are informed of the results. There is evidence of worthwhile improvements, and changes are monitored and reviewed to determine the impact and effectiveness of the change. Self-assessment at Capernwray is clearly effective and authentic, resulting in good quality, valid information. Self-assessment in the areas of the achievement and assessment methodology and graduate outcomes could be further strengthened by: 8

9 Reviewing and developing a clearer basis on which assessors base their decisions to validate the course and qualification achievement reported. Evidencing the benefits of the courses in greater detail for graduates to substantiate the anecdotal information collected. 9

10 Findings How well do learners achieve? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. Capernwray students are motivated and enthusiastic, and are gaining a variety of skills, including leadership, teamwork and cross-cultural experiences. A high proportion of students achieve the certificates. Successful completion of the relevant certificate demonstrates achievement in three areas in the Bible Training course: academic, ministry and community, as well as two areas in the ABS course: academic and community. Students improve their well-being and enhance their abilities and attributes, mostly through the ministry and community components as they participate in and contribute to local community churches. These components are monitored and graded during the course by Capernwray and self-reviewed by the students. Progress, both qualitative and quantitative, is closely monitored and reviewed in all areas. The national director reports to the board on achievement quarterly. Most importantly, all faculty and pastoral staff are involved in discussions about the progress and assessment of each student. Students are supported to complete assessment requirements. The certificate achievement results are summarised in Table 1 and show that most students successfully complete the course they are enrolled in and are awarded the corresponding certificate. Capernwray management and staff believe this is an excellent outcome. 1 The findings in this report are derived using a standard process and are based on a targeted sample of the organisation s activities. 10

11 Table 1: Qualification completion and student retention for selected Capernwray programmes (Certificate in Christian Development and Leadership using Adventure-based Learning (Level 3) (ABS) and Certificate in Bible Training and Preparation for Christian Service (Level 3) (Bible Training)) Year ABS 2 Bible Training 3 All courses at Capernwray Total Completions Withdrawal and did not complete Completions Withdrawal and did not complete Completions Withdrawal and did not complete Assessment of academic, ministry and community areas uses mixed-assessment grading methods, but in the absence of a well-defined guide for each course component, the likelihood of subjectively is increased, in particular when determining pass, merit and distinction passes. The validity of the certificate achievement is fundamentally assured in that students exceed the course components for each area; however, a clearer basis is needed on which assessors can base their decisions to validate the consistently excellent course achievement reported. Capernwray s completion and retention rates across courses year to year are consistent and therefore not deemed the best data on which to benchmark achievement. Instead, student feedback, specifically around pastoral care, is considered most relevant and important to both Capernwray and the students. This data is collected and analysed, and changes have occurred within Capernwray in terms of processes, roles and activities to improve the experience and therefore the educational performance of the students. 2 ABS Certificate in Christian Development and Leadership using Adventure-based Learning (Level 3) 3 Bible Training Certificate in Bible Training and Preparation for Christian Service (Level 3) 11

12 1.2 What is the value of the outcomes for key stakeholders, including learners? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. Students at Capernwray are largely young adults looking to have a gap year to find or confirm their future direction. For most of the students, this is their first experience of living outside of their own country. Capernwray certificate courses provide for the students an opportunity to be challenged and reflect on their future. Achieving and being awarded the national certificates is of little value to students as the New Zealand qualification has limited recognition when graduates leave these shores. Students identify the most important outcomes as the personal development and spiritual growth they experience, which the students believe will prepare them to embark on the next phase of their lives. Capernwray is assured that these anticipated outcomes are being met early on within both courses. This is evidenced in the mid-course evaluations and feedback from small group, one-toone interactions and the assessed and monitored course criteria within the ministry and community components. Most students have enrolled at Capernwray on the recommendation of Torchbearers communities and Capernwray graduates. After attending the six-week or 20-week certificate courses, most graduates leave New Zealand and return home. A small number of graduates take up internships, returning to New Zealand on a visa to work with churches and with Capernwray using the team and leadership skills developed on the courses. Capernwray has regular contact with some graduates and stakeholders, who are mostly pastors who employ some graduates in local churches. A number of graduates choose to extend their study and repeat, by choice, the 20-week Bible Training course to gain further personal and spiritual benefits. Capernwray communicates by , Facebook and through the network of Torchbearers International Schools. Capernwray receives approximately six requests every week from graduates wishing to work voluntarily at the school. Capernwray has a large amount of anecdotal information about graduates and the way in which the programme has contributed to their lives, but it does not have a definitive list of graduate outcomes or a formal analysis of graduate feedback which could then be used to make improvements and substantiate the value of the outcomes in the longer term. 12

13 1.3 How well do programmes and activities match the needs of learners and other stakeholders? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. Programmes are highly relevant and well matched to the needs of Capernwray international students. The ABS and Bible Training courses specifically provide the learning, activities, experiences and challenges that students are looking for to achieve their individual goals of spiritual and personal growth. Students enrolling in the six-week ABS course choose this as they have less time and want more physical challenges which the course provides. Students who complete the six-week ABS course have the option to cross-credit to the Bible Training course, of which approximately half do. In the Bible Training course a small number of students choose to extend their study and repeat the course to gain further benefits from the ministry and community components of the course and classroom-based learning. Capernwray has significant learning and teaching resources, including international guest tutors and speakers on a rotation, therefore the content is not repeated in consecutive courses and continues to meet the needs of these repeating students. Applied learning is a key feature of Capernwray certificates and is effective in engaging students in learning. ABS uses both outdoor and group activities to emphasise and challenge each student s personal development and learning. In the Bible Training course, ministry placements are purposefully designed and organised to link academic learning to a practical application within local churches. Review of courses occurs reliably after every cohort. This is exemplary practice in that both the ABS and Bible Training courses are delivered several times each year and have been for several years. Nonetheless, comprehensive reviews occur after each cohort to highlight areas of strength and areas to consider for improvement. This maintains the high relevance of the courses and the many benefits to the students and graduates. Capernwray provides accurate and useful pre-course information prior to enrolling and at enrolment, which includes confirmation of the ESOL (English for Speakers of Other Languages) capability of the students. Pre-course information is reviewed and changed in response to the various student evaluations and student council suggestions. Action plans track the implemented change to monitor the impact. A comprehensive orientation to both courses occurs and, most importantly, includes a thorough introduction to the on-site community living arrangements, ministry work experience expectations, behaviour, participation and the contribution each student is expected to make. 13

14 The goals of each student are identified on entry and are reflected on throughout the course. There are many ways of assessing the academic, personal and spiritual needs of the students. A holistic approach is given to needs assessment using the various group and one-to-one sessions as well as formal surveys and reviews. The residential facility provides a safe learning experience within a supportive family environment. 1.4 How effective is the teaching? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. There was clear evidence of effective teaching practice taking place at Capernwray. All staff are enthusiastic and passionate about the curriculum and teaching and they are well supported by management and the academic dean. A new position has been created this year to further focus Capernwray on the quality of the teaching and programme development. The teaching is highly organised, and assignment and class schedules are supplied well in advance to provide students with the best opportunity to plan and structure their time. Students track their progress through the prompt return of assessments, the quality and frequency of the feedback in the academic, ministry and community components, and in the ABS certificate at the debrief which occurs after every activity. Feedback from Capernwray staff to students is delivered with support and a plan for improvement in any areas where the student is challenged. Teachers and students relate well to each other, and constant access to the teaching staff is enhanced through the residential and community living. Capernwray understands how effective the teaching is through the analysis of comprehensive surveys completed by the students for each lecturer every teaching block, and at mid- and end-of-course evaluations. Observations of teaching practice are scheduled and are reliably occurring. These processes are providing authentic and useful data which Capernwray uses to inform decision-making which has resulted in a number of changes and are part of an effective teaching and learning strategy. Teaching staff all have experience and relevant industry and academic qualifications to teach subject areas. However, not all staff have education and assessment qualifications. There is some impact from this, in that the mixed assessment grading methods, and the numbering and grading and weighting criteria of these achievement results, are not clearly defined and transparent. As such, these processes do not adequately confirm the consistency of assessor judgements. The most significant impact is differentiating between pass, merit and distinction passes. This is somewhat mitigated by the following: internal moderation occurs systematically and reliably; all staff who are assessing 14

15 participate; and the most crucial academic staff the academic dean and primary tutor have the relevant education and assessment qualifications and experience, as does the routinely contracted external moderator. 1.5 How well are learners guided and supported? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. Learner guidance and support at Capernwray is highly effective in supporting the international students towards achieving spiritual and personal growth and educational success. Capernwray understands and effectively executes its responsibilities under the Code of Practice for the Pastoral Care of International Students, including those under 18 years. Under-18 student numbers are low and they invariably turn 18 while studying at Capernwray. All staff at Capernwray are responsible for the pastoral care and support of students and understand that this is a required function of their role. There is a wide range and number of avenues for students to access support and guidance academic, personal and spiritual. Some staff live on site and are present at meal times and in the evenings. This close contact and high ratio of staff to students 24/7 provides a variety of informal and naturally occurring opportunities for staff to engage with and monitor students well-being as well as to enjoy each other s company. Staff provide feedback and discuss at weekly meetings any concerns about individual student. Structured, activity-based groups meet routinely, led by Capernwray staff: family groups and Bible and prayer groups, and a new gender-based support group. The group numbers are small and have different leaders and students within each group, increasing the accessibility of a variety of staff with whom students can form a supportive relationship. One-to-one mentoring is an option for all students, and these mentoring times are scheduled weekly; at least one-third of students take up this opportunity. The student council, the national director and all staff provide academic and other support. Professional spiritual and personal counselling external to Capernwray is available and used as a matter of course. The full range of Capernwray self-assessment processes both informal and formal and including surveys (student, stakeholder, staff) and evaluations inform Capernwray about the pastoral care needs and satisfaction of students. Capernwray recognises this process of self-assessment as crucial if students are to benefit fully from the certificate course and grow spiritually and personally. Pastoral care is one of Capernwray s key areas targeted for ongoing and comprehensive selfreview. Following a full review in 2014, Capernwray found that its very high expectations for the pastoral care of students were not borne out as strongly as expected. In response, a new full-time pastoral care position has been introduced for 15

16 2015. The impact of this is being monitored, and there are indications of improved responses. Full analysis of the data is scheduled to occur within the existing review processes. 1.6 How effective are governance and management in supporting educational achievement? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. Capernwray is one of 26 Torchbearers International schools, and as such has a shared philosophy and beliefs which the national director and board members use to guide the school. The national director, board members and academic dean regularly travel to international conferences and have ongoing and frequent communication with other Torchbearers Schools and staff, some of whom are guest lecturers at Capernwray, sharing their expertise in specialist subject areas. Ongoing self-review by the Capernwray board and management occurs systematically within a holistic view of the organisation, beyond the standard fiscal and business responsibilities which nevertheless have clear reporting and decisionmaking processes. Capernwray has embedded effective systems for monitoring and reporting to governance about priority areas: achievement of the course certificates and the pastoral care of students. Capernwray recognises pastoral support as having the most significant impact not only on the educational performance but also the spiritual and personal growth of the students. Capernwray's relationship with the Torchbearers International Schools encourages governance and management to evaluate constructively, and there were multiple examples of Capernwray s governance and management being prepared to make difficult decisions to ensure the PTE continues to operate as per the Torchbearers white paper, Statement of Faith, and to meet the needs of the students that Capernwray attracts. Examples of improvements or change following review in 2013 and 2014 that were evident to the evaluation team included: changes to facilities; a supported restructure of staff positions and roles, including board membership and skills; and review of courses with proactive future planning. Capernwray s continuous self-review processes reveal the positive impact of these changes. Capernwray facilities support the learning activities and accommodation needs of international students. Both the Monavale and Tirau sites are well resourced and based in attractive settings, enhancing the experience for residential student living. Staff are highly valued, most of whom have been at Capernwray for several years. There is a supportive culture among staff and students; professional development is planned, ongoing and relevant for all staff, including the board and volunteer staff. Volunteer staff have the same professional benefits as permanent staff, with 16

17 clear job descriptions and performance reviews. How the board and national director formally review their own performance and sustain their effectiveness is still to be fully evidenced. This is an area they are aware of and they are considering suitable options. This is reasonable considering that the national director has been in the role for just one year and final changes to members of the board are yet to be finalised. 17

18 Focus Areas This section reports significant findings in each focus area, not already covered in Part Focus area: Governance, management and strategy The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Excellent. 2.2 Focus area: International student support The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Excellent. 2.3 Focus area: Certificate in Christian Development and Leadership using Adventure-based Learning (Level 3) The rating in this focus area for educational performance is Good. The rating for capability in self-assessment for this focus area is Good. 2.4 Focus area: Certificate in Bible Training and Preparation for Christian Service (Level 3) The rating in this focus area for educational performance is Good. The rating for capability in self-assessment for this focus area is Good. 18

19 Recommendations NZQA recommends that Capernwray Bible School: More fully evidence the benefits of the courses for graduates to substantiate the anecdotal information collected. Review and develop a clearer basis on which assessors base their decisions to validate the course achievement reported. Review the assessment methodology and moderation of assessment guidelines. The last two points are important as Capernwray s local qualifications will soon expire, and delivery of the new New Zealand qualifications will be required to meet national standards. 19

20 Appendix Regulatory basis for external evaluation and review External evaluation and review is conducted according to the External Evaluation and Review (EER) Rules 2013, which are made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. Self-assessment and participation in external evaluation and review are requirements for maintaining accreditation to provide an approved programme for all TEOs other than universities. The requirements are set through the NZQF Programme Approval and Accreditation Rules 2013, which are also made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. In addition, the Private Training Establishment Registration Rules 2013 require registered private training establishments to undertake self-assessment and participate in external evaluation and review, in accordance with the External Evaluation and Review Rules (EER) 2013, as a condition of maintaining registration. The Private Training Establishment Registration Rules 2013 are also made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. NZQA is responsible for ensuring non-university TEOs continue to comply with the rules after the initial granting of approval and accreditation of programmes and/or registration. The New Zealand Vice-Chancellors Committee (NZVCC) has statutory responsibility for compliance by universities. This report reflects the findings and conclusions of the external evaluation and review process, conducted according to the External Evaluation and Review (EER) Rules The report identifies strengths and areas for improvement in terms of the organisation s educational performance and capability in self-assessment. External evaluation and review reports are one contributing piece of information in determining future funding decisions where the organisation is a funded TEO subject to an investment plan agreed with the Tertiary Education Commission. External evaluation and review reports are public information and are available from the NZQA website ( The External Evaluation and Review (EER) Rules 2013 are available at while information about the conduct and methodology for external evaluation and review can be found at NZQA Ph E qaadmin@nzqa.govt.nz 20