Report of External Evaluation and Review

Transcription

1 Report of External Evaluation and Review The Shepherd s Bible College Charitable Trust trading as The Shepherd s Bible College Confident in educational performance Confident in capability in self-assessment Date of report: 16 January 2013

2 Contents Purpose of this Report... 3 Introduction TEO in context Scope of external evaluation and review Conduct of external evaluation and review...4 Summary of Results... 5 Findings... 7 Recommendations Appendix MoE Number: 7721 NZQA Reference: C08838 Date of EER visit: 13 and 14 November

3 Purpose of this Report The purpose of this external evaluation and review report is to provide a public statement about the Tertiary Education Organisation s (TEO) educational performance and capability in self-assessment. It forms part of the accountability process required by Government to inform investors, the public, students, prospective students, communities, employers, and other interested parties. It is also intended to be used by the TEO itself for quality improvement purposes. Introduction 1. TEO in context Name of TEO: Type: Location: Delivery sites: The Shepherd s Bible College Charitable Trust trading as The Shepherd s Bible College Private training establishment 354 Te Aute Road, Havelock North, Hastings One as above First registered: 21 April 2006 Courses currently delivered Certificate in Christian Studies (Level 5) Diploma of Biblical Studies (Level 5) Code of Practice signatory? Number of students: Yes, for students 18 years and over Domestic: 14 equivalent full-time students 67 enrolled students in total, the majority enrolled for interest only. Number of Māori and Pasifika unknown International: one equivalent full-time student Number of staff: Scope of active accreditation: Distinctive characteristics: 4.5 equivalent full-time staff Certificate in Christian Studies (Level 5) Diploma of Biblical Studies (Level 5) The Shepherd s Bible College (TSBC) was established to serve the local Riverbend Bible Church, (Brethren denomination) with the key purpose of preparing its students in sound theology, academic excellence, character 3

4 development and ministry competence. Recent significant changes: Previous quality assurance history: N/A At the most recent quality assurance visit in 2009, TSBC met all but one requirement. The requirement not met related to the achievement of goals and objectives. It was not confirmed that the college had undertaken a complete or effective self-assessment and review. The concern related to a failure to adequately identify a number of areas for improvement following the college s selfreview. TSBC has no requirements or arrangements for external moderation of its assessments or assessment decisions. 2. Scope of external evaluation and review This evaluation included the Diploma of Bible Studies (Level 5) and the mandatory focus area: governance, management, and strategy. TSBC also offers a certificate programme, but the diploma was chosen because this programme attracts the majority of students (only one student is currently enrolled in the certificate programme). 3. Conduct of external evaluation and review All external evaluation and reviews are conducted in accordance with NZQA s published policies and procedures. The methodology used is described fully in the web document Policy and Guidelines for the Conduct of External Evaluation and Review available at: The TEO has an opportunity to comment on the accuracy of this report, and any submissions received are fully considered by NZQA before finalising the report. Two evaluators visited the only training site in Havelock North for one and one halfdays. The evaluators interviewed members of the trust board, academic staff and management, and a group of students. A sample of stakeholders was interviewed by phone, and a range of the organisation s documents and records was reviewed. 4

5 Summary of Results Statement of confidence on educational performance NZQA is Confident in the educational performance of The Shepherd s Bible College. Between 2006 and 2012, 20 of the 23 students enrolled in the Diploma of Biblical Studies graduated with the diploma. While the majority of students at TSBC study for interest only, a few have subsequently enrolled in and completed the diploma. There is substantial evidence showing that all students achieve significant personal and spiritual growth, resulting in contributions to their local churches or in overseas missionary work. While the student numbers are low, these are significant achievements in terms of the qualification achievement rate and in the achievement of soft skills across all students. A high percentage of those who complete the diploma gain employment as pastors within the associated churches, or move on to further study, for example enrolling in the Bachelor of Ministry at the Adelaide College of Divinity, or in a programme of study at the Master s Seminary in the USA. The needs of the students and external stakeholders are well met, as evidenced through the organisation s self-assessment, including surveys and ongoing weekly involvement through its church networks. TSBC has appointed appropriately qualified tutors and programme leaders, and maintains effective and regular performance appraisals leading to professional development. The evaluation team noted a high level of respect and engagement between students, staff, and management, facilitating an effective learning environment. The governance structure is strong and independent of the college and includes external input from the wider church community. In the main, management stays up to date and current with regulatory changes, although there are gaps in this area. The evaluation team found no significant gaps in the educational performance at TSBC and noted that the organisation has effective processes to assure itself, its students, and stakeholders that it is providing highly effective education and support to its students, is meeting identified needs, and is adding significant value to its church community. However, the evaluation team considers that with the organisation s planned growth, there is room for improving the robustness of some systems and processes such as ensuring that the organisation stays up to date with all the requirements of a registered private training establishment. 5

6 Statement of confidence on capability in self-assessment NZQA is Confident in the capability in self-assessment of The Shepherd s Bible College. TSBC has strong informal connections and communication within its New Zealandbased and overseas network of churches, which provides regular ongoing and effective identification of education and training needs and feedback on the value the college adds to its community. The organisation has recently implemented a range of online surveys, such as with students at the end of each course, graduates intentions, and stakeholder feedback. Enrolment, attendance, and achievement data has been gathered over many years and is maintained on a bespoke database. However, while it is reviewed, there is limited analysis of this data. This combination of informally collected qualitative data, formal surveys, and quantitative data is being used to some extent to review the organisation s educational performance. TSBC is very small in terms of student numbers and programmes, and the informal processes noted are, in the main, fit for purpose for the organisation s current size and complexity. TSBC is not yet using this bank of qualitative and quantitative data to analyse and review its performance across such measures as gender, ethnicity, between courses, or across tutors. However, the organisation is planning to increase its programme offerings and increase student numbers. This may necessitate a more structured or formal self-assessment system to ensure that TSBC is meeting the increasing diversity of needs of its students and other stakeholders. The evaluation team considers that TSBC s self-assessment processes are currently providing assurance that the organisation is meeting the needs of its church community and the students are gaining value from their studies, given the college s present size and complexity. 6

7 Findings How well do learners achieve? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. Students are achieving at a high level, both in terms of graduating with the diploma where this is the enrolment aim, and in achieving wider and deeper understanding of their church and personal spirituality. In the past six years, 20 of the 23 students who enrolled in the diploma have graduated. These figures include some students who initially enrolled for interest only and later enrolled to complete the diploma. While this is a one-year programme, many students study part-time and a number of the students who enrolled over the last few years are still progressing towards completing the diploma. On a course-by-course basis, very few students do not complete their assessments and pass the courses. The majority of TSBC enrolments are for interest only, with no goal to complete the diploma but rather to engage in personal growth and develop the knowledge and skills for a deeper experience within the church and community. Most of the interest-only students complete all requirements for each course undertaken and receive a grade, and this is recorded along with all students in the student management system. The achievement levels of these students are on a par with students studying towards the diploma, indicating that achievement levels are high across all students. The organisation s self-assessment, through its informal feedback and surveys, indicates that most if not all students studying for interest only achieve their goals, and this was confirmed through the evaluators phone interviews with graduates during this evaluation. The organisation s processes for tracking and monitoring of student achievement are primarily based on recording achievements and outcomes. However, there is a significant amount of informal review and analysis and, given the very small student numbers, this is an effective process. The organisation has plans for growth and may leave itself vulnerable if it does not develop a more structured and documented approach to analysing student achievements, such as comparing achievements by gender, ethnicity, or across courses or tutors. 1 The findings in this report are derived using a standard process and are based on a targeted sample of the organisation s activities. 7

8 1.2 What is the value of the outcomes for key stakeholders, including learners? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. The outcomes being achieved at TSBC are of high value to the students and to their church communities. Male graduates of this programme tend to move into pastor or leadership roles within their church communities, while female graduates move into other roles such as church counsellors, in accordance with the church s beliefs. The organisation s self-assessment indicates that there is high value in the contributions made by graduates and non-graduates within TSBC s church communities in New Zealand and overseas. This was confirmed through phone interviews with stakeholders during this evaluation. A number of past students have gone on to work in New Zealand or overseas as Christian missionaries. Some graduates have moved on to further education, for example to study at the Master s Seminary in the USA, and one student is completing a Bachelor of Ministry at Adelaide College of Divinity. An example of the feedback from these students is that they, feel better prepared [for their studies] than other fellow students. While the enrolments at TSBC are relatively low, they primarily come through word of mouth, through past students and their communities seeing the value resulting from this education and training. The organisation s self-assessment of the value of its programmes is largely through the ongoing informal or anecdotal feedback from its community. However, an online stakeholder survey was recently conducted which shows a high level of community support. Graduates are surveyed on graduation, but longer-term outcomes are not formally followed up. However, the current small student throughput and the nature of the tight-knit church community means that TSBC has a reasonable knowledge and understanding of these longer-term outcomes. 1.3 How well do programmes and activities match the needs of learners and other stakeholders? The rating for performance in relation to this key evaluation question is Good. The rating for capability in self-assessment for this key evaluation question is Good. The programmes currently offered by TSBC match the needs of its stakeholders and students well. The high diploma achievement rate is one indicator of students needs being well matched. 8

9 The organisation is a signatory to the Code of Practice for the Pastoral Care of International Students and has enrolled a total of four international students to date. An area for improvement has been recently identified by management at TSBC in regard to how best to match the learning needs and expectations of students coming from countries with an educational system significantly different to New Zealand s. Specifically, this is in regard to students studying in New Zealand being required to carry out some independent research and learning and to show their understanding. One recently enrolled international student did not cope well with this and was subsequently removed from the programme. While this was dealt with appropriately by management, improvements to processes prior to the point of enrolment may have pre-empted this issue. The evaluation team noted that one international student, who has now left TSBC, did not have travel or medical insurance during the period of his enrolment. This is in breach of the Code of Practice. The Diploma of Biblical Studies is meeting TSBC s church community needs. This was confirmed through the organisation s self-assessment processes, including stakeholder surveys and feedback via TSBC s close and ongoing involvement with these church communities. As a result of student course surveys, the organisation has revised the structure of its programmes and moved from a four-term year to a two-semester format. This change was to some extent a result of students identifying concerns about the disruption caused by the four terms and a preference for longer, more continuous periods of study. This is seen as a significant improvement by staff and students. The organisation offers a range of options to best meet its part-time students study and life needs, with daytime, evening, or weekend block-course options to complete each course. This is valued by the students and helps them to fit study around work and other life commitments. The organisation is exploring offering a level 6 diploma programme in response to stakeholder-identified needs for more leadership skills. TSBC is currently working with a qualification developer, the Christian Theological Ministries Education Society (CTMES), a body representing 22 church-based private training establishments. CTMES is currently developing a Christian ministry diploma at level 6 to meet NZQA s current programme of reviewing all New Zealand qualifications. The organisation s self-assessment processes in this area are primarily reliant on the informal and ongoing engagement with its church communities in New Zealand and overseas, and these connections are strong and long-standing. However, with the planned growth, more formalised or documented processes may serve the organisation better. 9

10 1.4 How effective is the teaching? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Excellent. Learning and teaching at TSBC are highly effective. This is evidenced through formal student achievements, such as diploma achievement rates, but also through students development in personal, spiritual, and theological areas. Students spend some time during their study in internships within their church communities, providing opportunities to apply and embed their learning in practical situations, and this has added value to these church communities and the students. The churches provide the students with a mentor during these internships. The mentor role is structured and documented and valued by the students. The organisation has employed teachers with appropriate qualifications in their subject areas, and some have a specific training in adult education theory and practice. There is an appropriate performance appraisal process which is well linked to professional development, and teaching staff have access to a range of professional development options such as conferences. There is a well-structured and planned process to check that all assessment material is fit for its intended purpose, and assessors decisions are moderated post-assessment according to the organisation s moderation plan. The evaluation team considers that these processes are well planned and carried out, resulting in a good level of assurance in the validity of the assessments and of student achievements. The organisation is currently not required to have its assessments moderated by an external body. However, there may be value in exploring with other, similar private training establishments a mechanism for this to happen in the future. The organisation has an established culture of reflection by staff, and in their encouragement to students to reflect on their own learning. The organisation s processes for monitoring and reviewing the effectiveness of learning and teaching are strong and include well-embedded and documented processes as well as informal discussions and reflections. 1.5 How well are learners guided and supported? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. Students receive a high level of guidance and support throughout their period of enrolment. This includes information prior to enrolment via the organisation s website, an enrolment form, and a student handbook. The organisation runs discipleship labs which provide opportunities for discussion and where staff 10

11 interact with students to shepherd them along. There is one financial trust that students are referred to for financial support when they are struggling to pay course fees. These are seen as valuable support processes by both staff and students, and they fit well within TSBC s ethos. Students are provided with an orientation programme where they meet lecturers and board members. Some time is also spent familiarising students with the academic expectations of the programme, such as essay writing and academic referencing. Students are expected to attend at least 80 per cent of their classes, and the tracked attendance is closer to per cent, indicating a high level of student motivation and staff support. The monitoring and review processes in this area are largely through the informal interactions between staff and students and feedback from the wider church community. However, formal student surveys at the end of each course and at graduation contribute to the formal review. This combination is effective. As already noted, the organisation is exploring how best to support international students coming from different learning environments, and this is an area for future improvement. 1.6 How effective are governance and management in supporting educational achievement? The rating for performance in relation to this key evaluation question is Excellent. The rating for capability in self-assessment for this key evaluation question is Good. Both the trust board and the management team have a clear and effective focus on learner achievement. This is evidenced through the diploma achievement rates, the staff profile, and in references and commendations from the church community in regard to the skills, knowledge, and aptitudes of the graduates. TSBC is a trust established and supported by Riverbend Bible Church in Hastings. The trust board includes staff members, two elders, and one representative appointed by the elders from the church, as well as two external advisory members, providing the board with an appropriate level of independence from TSBC. The evaluation team considers that the board is operating effectively in its oversight and guidance role. In accordance with the trust s constitution, the principal of TSBC is the current pastor of Riverbend Bible Church, providing spiritual leadership, with operational management provided by an appointed vice-principal. This arrangement appears to be an effective combination in this context. Strategic planning and operational budgets are in place to provide and maintain resources appropriate for the programmes offered. However, it was noted that TSBC is reliant to some extent on financial gifts and support from its church 11

12 community within New Zealand and the USA. There is a level of vulnerability in this arrangement. As already noted, the organisation has plans for extending its programmes to include a level 6 diploma and to increase student enrolments. The evaluators observed some areas where governance and management were not fully cognisant of recent changes in the regulatory environment, such as the current Tertiary Education Strategy and some changes in the Gazetted Policies and Criteria for Ongoing Registration for Private Training Establishments. While these two areas have a relatively minor impact on student learning, the evaluators consider they will need to be strengthened prior to the planned organisational growth. Self-assessment is a core management function and TSBC has a good level of understanding of how well students are achieving and of the value of the outcomes, through the organisation s mix of formal and informal processes. However, management may need to consider how to strengthen or improve the robustness of its current processes for monitoring and reviewing organisational performance in order to cater for the planned growth. 12

13 Focus Areas This section reports significant findings in each focus area, not already covered in Part Focus area: Governance, management, and strategy The rating in this focus area for educational performance is Good. The rating for capability in self-assessment for this focus area is Good. The rating for performance in this focus area is slightly lower than in 1.6, and is a reflection of those points noted under 1.6 in relation to the board and management not staying up to date with current changes in its regulatory environment, as well as the issue noted under 1.3 regarding the one international student who did not have travel and medical insurance during his period of enrolment. 2.2 Focus area: Diploma of Biblical Studies (Level 5) The rating in this focus area for educational performance is Excellent. The rating for capability in self-assessment for this focus area is Good. 13

14 Recommendations In addition to any recommendations expressed or implied within the report, NZQA recommends that Shepherd s Bible College: Explore with appropriate bodies or professionals options for the external moderation of assessments, to provide an objective review of internal processes and findings. Consider extending the assessment processes for international students prior to enrolment, to better identify potential barriers to success in their studies at TSBC. Further develop self-assessment plans and processes to cater for planned growth, and to improve the organisation s capability to analyse its qualitative and quantitative data. Establish processes to improve management s awareness of and responsibilities in relation to the regulatory environment to ensure TSBC stays up to date with statutory requirements. 14

15 Appendix Regulatory basis for external evaluation and review Self-assessment and external evaluation and review are requirements of programme approval and accreditation (under sections 249 and 250 of the Education Act 1989) for all TEOs that are entitled to apply. The requirements are set through the Criteria for Approval and Accreditation of Programmes established by NZQA under section 253(1)(d) and (e) of the Act and published in the Gazette of 28 July 2011 at page These policies and criteria are deemed, by section 44 of the Education Amendment Act 2011, to be rules made under the new section 253. In addition, for registered private training establishments, the criteria and policies for their registration require self-assessment and external evaluation and review at an organisational level in addition to the individual programmes they own or provide. These criteria and policies are also deemed, by section 44 of the Education Amendment Act 2011, to be rules made under section 253. Section 233B(1) of the Act requires registered PTEs to comply with these rules. NZQA is responsible for ensuring non-university TEOs continue to comply with the rules after the initial granting of approval and accreditation of programmes and/or registration. The New Zealand Vice-Chancellors Committee (NZVCC) has statutory responsibility for compliance by universities. This report reflects the findings and conclusions of the external evaluation and review process, conducted according to the EER process approved by the NZQA Board. The report identifies strengths and areas for improvement in terms of the organisation s educational performance and capability in self-assessment. External evaluation and review reports are one contributing piece of information in determining future funding decisions where the organisation is a funded TEO subject to an investment plan agreed with the Tertiary Education Commission. External evaluation and review reports are public information and are available from the NZQA website ( Information relevant to the external evaluation and review process, including the publication Policy and Guidelines for the Conduct of External Evaluation and Review, is available at: 15

16 NZQA Ph E qaadmin@nzqa.govt.nz 16