Report of External Evaluation and Review Ecumenical Board of Theological Studies Confident in educational performance Confident in capability in self-assessment Date of report: 31 July 2013
Contents Purpose of this Report... 3 Introduction... 3 1. TEO in context... 3 2. Scope of external evaluation and review... 4 3. Conduct of external evaluation and review... 5 Summary of Results... 6 Findings... 8 Recommendations... 15 Appendix... 16 MoE Number: 7269 NZQA Reference: C11278 Date of EER visit: 21 and 22 May 2013 2
Purpose of this Report The purpose of this external evaluation and review report is to provide a public statement about the Tertiary Education Organisation s (TEO) educational performance and capability in self-assessment. It forms part of the accountability process required by Government to inform investors, the public, students, prospective students, communities, employers, and other interested parties. It is also intended to be used by the TEO itself for quality improvement purposes. Introduction 1. TEO in context Name of TEO: Type: Location: Delivery sites: Ecumenical Board of Theological Studies Private training establishment (PTE) 7 Somerfield Street, Christchurch None (distance learning provider) First registered: 12 August 1998 Courses currently delivered: Diploma Licentiate in Theology (Level 6) Scholar in Theology Code of Practice signatory?: Number of students: No Domestic: approximately 100 students per year (approximately 65 per cent of students are female; most students are Pakeha and aged between 50 and 80 years; 12 per cent identify as having disabilities) International: nil Number of staff: Scope of active accreditation: Distinctive characteristics: Two management and administration staff and a pool of approximately 49 tutors and moderators Diploma Licentiate in Theology (Level 6) The Ecumenical Board of Theological Studies has oversight and governance of the education arm, Ecumenical Institute of Distance Theological Studies (EIDTS), which provides Christian theological education by distance learning, using 3
paper-based resources and an (optional) residential school. The courses cover theology, biblical studies, church history and ministry and mission. Tutors and moderators, located throughout New Zealand, are contracted annually for one or more courses appropriate to their area of expertise. Recent significant changes: Previous quality assurance history: Other: As a result of the loss of a primary source of funding (from the Tertiary Education Commission (TEC)) EIDTS will cease operations in April 2015. EIDTS met all requirements at the previous NZQA quality assurance visit, an audit in 2007. EIDTS is a charitable organisation, governed by the Ecumenical Board of Theological Studies, which comprises representatives of the key stakeholder churches (Presbyterian, Methodist and Anglican churches and the Salvation Army). The 2013 investment plan from EIDTS was declined by the TEC because the education provision is not consistent with its funding policy, which prioritises the completion of qualifications. 2. Scope of external evaluation and review The focus areas selected were: Governance, management and strategy Diploma Licentiate in Theology The first focus area is mandatory. The second focus area is the only NZQAapproved qualification offered. 4
3. Conduct of external evaluation and review All external evaluation and reviews are conducted in accordance with NZQA s published policies and procedures. The methodology used is described fully in the web document Policy and Guidelines for the Conduct of External Evaluation and Review available at: http://www.nzqa.govt.nz/providers-partners/registration-andaccreditation/external-evaluation-and-review/policy-and-guidelines-eer/introduction. The TEO has an opportunity to comment on the accuracy of this report, and any submissions received are fully considered by NZQA before finalising the report. Two evaluators visited EIDTS over one and a half-days. Interviews were conducted in person with the director, academic registrar and deputy chair of the academic committee. In addition, phone interviews were held with the chair of the board and one board member, four tutors, four current students and two graduates. A wide range of documents and other resources was reviewed, including learner achievement and feedback data, enrolment, course and assessment materials, planning documents, meeting minutes and stakeholder feedback. 5
Summary of Results Statement of confidence on educational performance NZQA is Confident in the educational performance of Ecumenical Board of Theological Studies. EIDTS effectively meets the needs of its stakeholders, including learners, for high quality and balanced theological education in an accessible form. Distance learning is facilitated by well-structured and comprehensive paper-based course materials and access to highly qualified subject experts. Flexibility and responsiveness to individual circumstances and study goals are key characteristics of EIDTS engagement with all students. This reflects an excellent understanding of the learner cohort and the church communities to which they belong. Many learners are more than 40 years old (up to 80 years), and fully employed (either on a paid or voluntary basis) in their church and/or community and seek to complete only one or two courses. Student feedback to EIDTS and the evaluators confirms their high levels of satisfaction with their learning experience and the support and encouragement they receive. Students are acquiring theological knowledge and related practical skills (such as preaching and pastoral care), and developing greater self-confidence, which equip them well for their paid and unpaid work in their church and/or the wider community. The learning includes developing critical thinking skills and a greater understanding of Christian theology across a range of denominations. Stakeholder feedback held by EIDTS and gathered by the evaluators confirmed the high value of individual learning outcomes. EIDTS is also contributing to the development of social capital and strengthening the communities in which the learners are working or volunteering. EIDTS defines learner achievement in terms of the students goals, noting that most do not seek to complete a qualification, and this is reflected in the results data. Course completion rates for 2009 to 2011 were 75 per cent, 78 per cent and 72 per cent respectively (2012 completion data was not available at the time of the visit). EIDTS reports that, while up to 20 per cent of students require extensions beyond one year, approximately half of these go on to successfully complete their course/s. Qualification rates for the same period were 21 per cent, 31 per cent and 29 per cent, which are well below the national median. The evaluators acknowledged that this reflected the particular circumstances and goals of EIDTS s learners. At the same time, NZQA would also note that these results do not exactly match the expectation of the Crown s funding body, the Tertiary Education Commission (TEC). Overall, when all the indicators of successful learning are considered, there is a solid body of evidence indicating that EIDTS s governance, management and staff work effectively together to improve educational performance, and that the 6
organisation s structure and management and academic and communication processes reflect their commitment to responding to changing contexts and ensuring valued outcomes. Statement of confidence on capability in self-assessment NZQA is Confident in the capability in self-assessment of Ecumenical Board of Theological Studies. EIDTS is methodical and effective in its approach to self-assessment, and has wellestablished processes for understanding and improving educational performance. These include regular engagement with all key stakeholders (including learners) to gather feedback and share information in order to understand developments and priorities. Feedback is analysed for useful information for enhancing teaching and learning. Course reviews, both scheduled and ad hoc, ensure that changes are made that meet learner needs and deliver education that is relevant and useful for their roles within their churches and the wider community. Examples include the development of a new course in rural ministry and the complete rewriting of a course in response to a changing church context. Academic standards are overseen by the academic committee. A recent revision of assessment practices and the development of assessment guidelines has reinforced those standards, providing greater clarity for tutors, moderators and students. Evidence of these processes, together with a regular cycle of reflection and planning, gave the evaluators confidence in the capability in self-assessment among governance, management and staff at EIDTS. The evaluators found that valid and reliable data, both qualitative and quantitative, is being collected from a range of sources and analysed by management. This information is informing strategy and decision-making, including investment decisions. However, a greater emphasis on identifying and understanding trends and more consistency in following up on the impact of changes would further strengthen EIDTS self-assessment capability. 7
Findings 1 1.1 How well do learners achieve? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. EIDTS defines learner achievement according to the study goals of each individual student, noting that, for many students, their objective may not be a whole qualification but one or two courses. Some students enrol just for interest, while others seek to prepare for, or enhance, their ministry. In each case, the achievement for the individual is determined by the acquisition of meaningful skills and knowledge that enhance learners critical thinking skills, understanding of Christian theology and ability to contribute positively to their church and wider community. Achievements include developing practical skills (such as preaching) and improved self-confidence and well-being. The evaluators found good evidence of students enhancing their abilities and attributes in this way, in records of stakeholder feedback as well as interviews conducted during the evaluation visit. One such example was of an older student who reported gaining confidence to take religious services in rest homes. EIDTS works hard to assist students to achieve their goals, and has recently focussed on understanding the reasons for withdrawal and offering additional support, acknowledging the particular challenges of learning at a distance and the demographics of their learners. It was found that additional support had not resulted in fewer non-completions or withdrawals overall. However, workload issues for students who are funded by the denominational churches remains an important factor in non-completion. Up to 20 per cent of students require extensions beyond one year. Approximately half of these go on to successfully complete their course/s. Course completion rates have been consistently good for a number of years (75 per cent, 78 per cent and 72 per cent for 2009-2011). Qualification completion rates for the same period were 21 per cent, 31 per cent and 29 per cent, reflecting the study goals and circumstances of EIDTS learners. EIDTS management collates and reports achievement data regularly to the academic committee and board, although further analysis of the data might yield useful information for development and improvement. 1 The findings in this report are derived using a standard process and are based on a targeted sample of the organisation s activities. 8
1.2 What is the value of the outcomes for key stakeholders, including learners? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. EIDTS is highly valued for its unique educational offering, which provides a focus for ecumenical dialogue and opportunities for the upskilling of both clergy and laity, particularly, but not exclusively, belonging to the four stakeholder churches. For some students, study through EIDTS is equipping them for their voluntary work in the wider community. Benefits reported to the evaluators include the development of an open and enquiring theological standpoint and knowledge and skills that support priesthood, and an enhanced ability to contribute to society, either within Christian communities or public organisations (such as schools, hospitals or prisons). In this way, EIDTS is enriching New Zealand society through strengthening communities. The wide range of courses leading to the level 6 Diploma Licentiate in Theology are complementary to the training offered by providers within various Christian denominations, and provides on-the-job training for clergy in some circumstances. Some churches seek to augment their own degrees or offer additional training for people working with children or youth. In addition, students who complete the diploma and wish to continue studying are eligible for cross-credits, for example towards degrees in theology. Evidence of the high value of the outcomes is available from a range of sources. These include the results of a graduate survey (undertaken in 2010) and the individual stories of graduating students published in EIDTS quarterly newsletter. EIDTS maintains destination information for Diploma Licentiate in Theology graduates and reports some limited ongoing contact with them. There was some evidence that information about work-related outcomes (voluntary or paid) is being actively used to identify and develop improvements such as the development of a certificate for lay ministry. 2 2 As a result of the closure of EIDTS in 2014, this development will not proceed. 9
1.3 How well do programmes and activities match the needs of learners and other stakeholders? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. Flexibility and responsiveness are key features of EIDTS educational provision, which are reflected in the nature of the courses themselves and the organisation s operational mode. New courses are developed in response to stakeholder feedback, such as those for child, youth and rural ministry. The courses provide for a balanced theological education and reflect the New Zealand context, including aspects of Māori Christian traditions. In addition to the regular cycle of course reviews, courses and resources are reviewed annually to address issues raised by learners or to incorporate developments within the subject area identified by the tutors, such as from new research or journal articles. Examples include the redesign and editing of a course to make it more user-friendly, and structuring assignments that require the student to engage with all the course materials rather than selected aspects. The distance learning mode enables access to affordable self-paced theological education, regardless of location. EIDTS has a good understanding of its learner profile and uses this understanding to ensure appropriate course resources are available. As a result, traditional paper-based course materials have been retained (rather than digital) and additional resources, such as a study and writing guide, are available. All learners are provided with readings and recommended texts and referred to theological libraries. At the annual residential school, students beginning their studies meet their tutor/s, can discuss their specific learning needs and are introduced to the EIDTS system of distance learning. In addition, tutorials are offered on essay writing, referencing and so on. The school also provides an opportunity for students to build relationships with each other and offer support to each other during the course. EIDTS invites written feedback from students at the residential course and twice during the academic year. The evaluators sighted examples of careful analysis of learner feedback and noted the subsequent discussions by the academic committee and board. Examples of improvement initiatives arising from this process include reviewing assessments to better match the course and closer monitoring of marking turnarounds (three-week target). 10
1.4 How effective is the teaching? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. The academic committee provides a strong framework for effective teaching at EIDTS, which is largely monitored through learner feedback. Learners benefit from receiving tutorial support from a highly qualified and enthusiastic community of scholars from a wide range of backgrounds, including university academics and practising clergy. The academic committee oversees tutor recruitment to ensure they are subject experts and to assess their teaching capability. The evaluators found clear evidence in written and oral feedback that tutors are engaging effectively with their learners. A high level of communication and support is effective in maintaining motivation and supporting students to progress through the course materials. This includes providing additional assistance, as required, for students struggling with study skills, such as essay writing. EIDTS has established good academic practices for course design and review, which was evident in the well-structured and comprehensive course materials. Assessment practice supports a high level of confidence in the academic standards at EIDTS, including the review of all marked assignments by a moderator with expertise in the subject. This provides markers with constructive feedback and the opportunity to check the assigned grade. Comprehensive feedback is also provided to students on each assessment. Assessment practice has been an important area for review and improvement over recent years. Assessment guidelines and templates were produced by the academic committee. Tutors reported positive outcomes, such as greater consistency in standards across courses. Students have asked fewer questions and responded well to having clearer assessment criteria. An annual planning weekend provides a valuable opportunity for tutors and moderators to meet together and participate in training and review activities. These include a structured reflective process, completed as pairs within subject clusters, which covers both the course/s taught over the previous year and a selfassessment, including setting goals for their own development as tutors. The outcomes of this process are collated and reviewed by management and the academic committee. 1.5 How well are learners guided and supported? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. Regular communication is an important feature of the guidance and support framework for EIDTS students. EIDTS recognises the importance for learner 11
achievement of maintaining regular contact with distance learners and monitoring participation. Management and administration staff establish the initial connection during the enrolment stage to ensure that the study plan is appropriate as well as to ensure learners understand the time commitment required. Tutors subsequently assume primary responsibility for the relationship and the learners progress. Attendance at a residential school at the beginning of study provides a useful introduction and welcome to the EIDTS community, and sets the scene for ongoing contact and support. Individual circumstances and learner needs are addressed on a case-by-case basis. Examples include arranging a mentor for a Pasifika learner and providing additional support for students and their supporters to attend the residential course or graduation. A quarterly newsletter supplements regular phone calls and emails, to maintain connection and a sense of belonging for students. The inclusion of graduates stories in the newsletter also provides inspiration and encouragement for enrolled students. EIDTS staff are active in seeking and using feedback to improve the effectiveness of the guidance and support framework. The course handbook and generic study and writing guide are revised regularly, with a particular focus on supporting learners who are re-entering study after a break. Regular conversations with key stakeholders are a useful source of information on individual and generic learner needs. A review of the format of the residential courses led to the combining of several residential block courses into one larger event at the beginning of 2013. Initial feedback suggests a positive response and learner achievement will be monitored to identify any impact. Analysis of the reasons for students ceasing to participate, as well as formal withdrawal, has contributed to a more managed approach to learner participation. Reasons for withdrawal include health issues (both mental and physical) and work commitments, sometimes for those individuals who have been required to enrol by the employing church. EIDTS plans a more active approach to reporting learner withdrawals to stakeholder churches in these instances. The evaluators also noted that the flexibility available in terms of course completion is much appreciated by the learner cohort. 1.6 How effective are governance and management in supporting educational achievement? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. The governance and management of EIDTS work closely together to support educational achievement. The structures, processes and communication practices are highly effective in ensuring that good information is collected and used to inform decision-making. Examples include the academic committee which holds responsibility for the academic integrity of EIDTS courses and has a clear role in 12
determining changes to be made in response to learner and tutor feedback. The board receives regular reports from the director on her stakeholder engagement activities, as well as updates from the board members who are representatives of the stakeholder churches. Regular strategic and business planning reflects a responsiveness to this feedback. Meeting minutes and annual academic reports provide evidence of both groups receiving regular data on student demographics and achievement as well as collated student feedback, and show a strong focus on the achievement of valued outcomes. The evaluators found that there was a good understanding of self-assessment at all levels of the organisation. A focus on review and improvement is embedded in day-to-day activities and is supported by valid and reliable information. In some instances, there are opportunities to extend the analysis to more clearly identify meaningful trends. It was not evident that, in every case, changes had been monitored to determine whether they had resulted in improved outcomes and in some instances, it is too soon to assess the impact of changes. However, in general, self-assessment processes are appropriate and effective, reflecting the nature and size of the organisation. EIDTS is a very small but resilient organisation. Prior to 2011, EIDTS operated out of private homes. Despite the impact of the Christchurch earthquakes and several relocations, EIDTS has established an office base and reports improved efficiency and administrative, planning and reporting processes. However, at the time of the evaluation visit, EIDTS was facing an uncertain future due to the loss of funding, and has indicated an intention to cease delivery in April 2015. As a result, several initiatives developed in response to feedback will not now proceed. These include an online component for several of the larger courses, and the introduction of a new certificate qualification for laity. The evaluators noted that a group had been formed to support the transition process. EIDTS is striving to ensure the best possible outcomes for students. These include offering additional courses, plus additional advice and support to students, to enable them to complete their qualification if at all possible. In addition, discussions are underway with other providers regarding cross-credit arrangements. 13
Focus Areas This section reports significant findings in each focus area, not already covered in Part 1. 2.1 Focus area: Governance, management and strategy The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Good. 2.2 Focus area: Diploma Licentiate in Theology (Level 6) The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Good. 14
Recommendations There are no recommendations arising from the external evaluation and review. 15
Appendix Regulatory basis for external evaluation and review External evaluation and review is conducted according to the External Evaluation and Review (EER) Rules 2013, which are made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. Self-assessment and participation in external evaluation and review are requirements for maintaining accreditation to provide an approved programme for all TEOs other than universities. The requirements are set through the NZQF Programme Approval and Accreditation Rules 2013, which are also made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. In addition, the Private Training Establishment Registration Rules 2013 require registered private training establishments to undertake self-assessment and participate in external evaluation and review, in accordance with the External Evaluation and Review Rules (EER) 2013, as a condition of maintaining registration. The Private Training Establishment Registration Rules 2013 are also made by NZQA under section 253 of the Education Act 1989 and approved by the NZQA Board and the Minister for Tertiary Education, Skills and Employment. NZQA is responsible for ensuring non-university TEOs continue to comply with the rules after the initial granting of approval and accreditation of programmes and/or registration. The New Zealand Vice-Chancellors Committee (NZVCC) has statutory responsibility for compliance by universities. This report reflects the findings and conclusions of the external evaluation and review process, conducted according to the External Evaluation and Review (EER) Rules 2013. The report identifies strengths and areas for improvement in terms of the organisation s educational performance and capability in self-assessment. External evaluation and review reports are one contributing piece of information in determining future funding decisions where the organisation is a funded TEO subject to an investment plan agreed with the Tertiary Education Commission. External evaluation and review reports are public information and are available from the NZQA website (www.nzqa.govt.nz). The External Evaluation and Review (EER) Rules 2013 are available at http://www.nzqa.govt.nz/assets/about-us/our-role/rules/eer-rules.pdf, while information about the conduct and methodology for external evaluation and review can be found at http://www.nzqa.govt.nz/providers-partners/external-evaluation-andreview/policy-and-guidelines-eer/introduction/. NZQA Ph 0800 697 296 E qaadmin@nzqa.govt.nz www.nzqa.govt.nz 16