Evaluating the Impact of Foundations and Graduate Certificate Programmes: Methodologies and Measures. Marguerite Clarke 1 July 2006

Transcription

1 Evaluating the Impact of Foundations and Certificate Programmes: Methodologies and Measures Marguerite Clarke 1 July 2006 Overview This paper reviews some of the approaches being used to evaluate the impact of foundations and Certificate programmes on teaching and learning in higher education. 2 It draws upon a convenience sample of ongoing and completed research from the UK, Australia and New Zealand in order to answer five main questions about these evaluations. 3 It concludes with general suggestions for conducting an impact evaluation of a foundations or Certificate programme. Findings 1. Why evaluate the impact of a foundations or Certificate programme? The primary goal of foundations and Certificate programmes is to improve teaching and learning in higher education. The main reason for evaluating these programmes is to collect evidence on the extent to which this goal has been achieved. The nature of the evidence collected and how it is analysed depends upon the context for the evaluation and the specific research question to be addressed (see Tables 1-3). Most evaluations attempt to get at the long-term (i.e., several years out) impact of the programme on teaching and learning. 2. What are some of the more commonly-used approaches/data collection methods? Evaluations of foundations and Certificate programmes tend to employ a mix of qualitative and quantitative approaches. The most popular data-collection methods are interviews, questionnaires and teaching/course evaluation instruments (Table 4). Most of the data collected pertain to participants satisfaction with the programme, their intentions to alter their practice and their perceptions of whether those changes to practice (if implemented) made a difference to student learning. Far rarer are evaluations that seek to document actual changes in practice or that try to causally link those changes to student learning outcomes. While such evidence is difficult to collect, it is vital for demonstrating the effectiveness of these programmes. 1 Dr Marguerite Clarke wrote this report at the request of Margaret Hicks while she working in the Flexible Learning Centre at the South Australia. 2 These are programmes that introduce academic staff to the theory, practice and scholarship of teaching and learning in higher education. 3 This research was identified through personal communications between Margaret Hicks and academic development staff in Australia and New Zealand.

2 3. Who tends to conduct these evaluations? Evaluations tend to be conducted by those running the programme. While evaluations by independent/external researchers are less common, the findings are more likely to be viewed as credible by external stakeholders. 4. What have they found? The majority of participants report positively on their programme experience (e.g., relevance, quality, support, flexibility) and on the perceived impact on their subsequent practice. Some of the more commonly reported impacts include a more student-centred approach to teaching, greater flexibility in teaching and assessment strategies and more openness to seeking and using student feedback. Unfortunately, there is little hard evidence in these reports to demonstrate a link between such changes to practice and improved student learning outcomes. 5. How are the findings used? Findings appear to be used primarily for internal accountability and for programme improvement. Conclusion Based on the above discussion, it would be worth keeping in mind the following when evaluating the impact of a foundations or Certificate programme: Given that the overarching goal of these programmes is to improve teaching and learning, any evaluation of their impact should aim to get at actual (as opposed to reported or perceived) changes in teaching practice and student learning outcomes. This requires the collection of pre- and post programme data on participant outcomes in these areas. Non-participant outcomes in these areas also should be gathered. (These function as a control or comparison group.) Post-programme outcomes should be monitored over several years. Ideally, this would involve the collection of longitudinal data on each programme cohort. The evaluation design should allow for the collection of multiple types of data on programme outcomes so as to allow for triangulation and verification of findings. The evaluation design should allow for quantitative analyses of the links between the programme and improved student learning outcomes. Some consideration should be given to using an independent person/reviewer to conduct the evaluation. Selected Bibliography Gibbs, G., & Coffey, M. (2004). The impact of training of university teachers on their teaching skills, their approach to teaching and the approach to learning of their students. Active Learning in Higher Education 5(1), Ginns, P., Kitay, J. & Prosser, M. (2006). Transfer of academic staff learning in higher education. Draft manuscript. (Presented at the 2005 Australian Universities Quality Forum?) Ginns, P., Kitay, J. & Prosser, M. (2006). Does learning last?: The influence of the Educational Studies (Higher Education) on teachers

3 experiences. PowerPoint presentation. Radloff, A. (2004, July). Review of the New South Wales Certificate of University Learning and Teaching. Available at T_Review_07_04.pdf/$file?GradCert_ULT_Review_07_04.pdf Prosser, M., Rickinson, M., Bence, V., Hanbury, A., & Kulej, M. (2006, May). Formative evaluation of accredited programmes. The Higher Education Academy report. Available at FormativeEvaluation.pdf Guskey, T. R. (2000). Evaluating professional development. Thousand Oaks, CA: Corwin Press. Kirkpatrick, D. (1998). Evaluating training programs. New York: Berrett-Koehler. Schulman, L. S., & Schulman, J. H. (2004). How and what teachers learn: A shifting perspective. Journal of Curriculum Studies, 36(2),

4 Table 1. Approaches to evaluating the impact of foundations and Certificate programmes: Australia Institution James Cook University Name of Programme Induction foundations programme Evaluation Question Evaluation Design/Indicators Findings Do participants (intend to) alter their practice as a result of the programme? Participant evaluations of the programme Pedagogical resource use (e.g., videos) Conference and workshop presentations on teaching and learning Case studies of participants accounts of how their practices changed Comparison of student evaluation scores for programme participants and non-participants. Participant evaluations of all programmes have been very positive. Participants indicated intentions to alter their practices as a result of the programme and there is evidence that many followed through. Student evaluation scores of participants are consistently higher than non-participants. (Some of these findings were presented at HERDSA 2006). Monash University Adelaide Semester-long Teaching for Learning programme Foundations programme and Higher Education programme (i) Do participants anticipate improvements in their practice as a result of having attended the programme? (ii) What are the effects of the programme on participants student evaluation scores? (i) What is the long-term impact of the foundations programme? Survey of 150 programme participants (covering 5-year period) Pre- and post- programme student evaluation data for participants At the end of the programme, participants were provided with a planning framework in which they stated what they intended to change in their teaching. These documents are now being used as a memory peg and framework for interviews with past participants to determine outcomes of the programme 18 months on. Most participants predicted an improvement in their practice. (Findings to be presented at AARE 2006). The greatest positive impact was for those starting out with mid-range student evaluation scores. Effects on those starting out with low or high evaluation scores were minimal. (Unclear when findings will be published.) Some findings, based on data from the 2004 cohort, were presented at HERDSA (ii) What is the long-term impact of the Certificate programme? Near the beginning of the programme, staff provided participants with the programme planning framework (the Attributes Continuum) so that they could self-assess. This is participants memory peg for reflecting on how far they have come at the end of the programme, and especially in subsequent years. The first cohort has just finished the programme so meaningful data won t be generated until mid-2007.

5 Canberra New South Wales Southern Queensland Sydney Higher Education University Learning and Teaching Tertiary Teaching and Learning Educational Studies (Higher Education) How did graduates experience the course and what have been the outcomes of their participation? What is the value of the programme in supporting the achievement of the university s goals for teaching and learning? To what extent has the programme impacted upon participants' own teaching contexts? What is the extent of learning transfer from the programme? What are the factors supporting or inhibiting transfer of learning back to the workplace? Semi-structured interviews with a sample of 23 graduates [Apart from this, evaluation of individual units has been regularly undertaken and a course review was completed in 2001] Collected data on participant satisfaction, learning, application and impact from current students, graduates, peers and supervisors using surveys, interviews and focus groups. Some document analysis. Using mix of formal and informal approaches [Previously, for other programmes, they have collected data using face-to-face and online focus groups and interviews, questionnaires, reflective journals and paper-based evaluations.] Semi-structured interviews with 15 alumni with a focus on their pre- and post-programme understandings of the theory, practice and scholarship of teaching and learning and their experiences of transfer of learning. Phenomenographic coding used to analyse the data. Interviewees indicated high levels of satisfaction with the applied nature of the coursework. However, while one-third reported that the workload was appropriate, one-quarter found it difficult to manage given other commitments. s felt that their students benefited from their improved teaching practice (e.g., changes in assessment, ability to give clearer and constructive feedback). Improved student ratings were also mentioned. (2-page summary available.) Participants expressed high degrees of satisfaction with their experience, described positive changes to their teaching (e.g., more student-centred), and identified positive impacts including positive feedback from students and peers. (See 2004 report by Radloff for full discussion.) Will not release formal report until early Participants went from teacher- to studentcentred approach in their teaching; and from less to more complex conceptions of the scholarship of teaching. Influence of colleagues, supervisors, reward structure and time pressures generally confirm expectations. (See AUQF 2005 paper and PowerPoint presentation by Ginns, Kitay, & Prosser for further details.)

6 Table 2. Approaches to evaluating the impact of foundations and Certificate programmes: New Zealand Institution Unitec Otago Name of Programme Diploma in Higher Education Post Tertiary Teaching Evaluation Question Evaluation Design/Indicators Findings What are the reactions of teaching staff to this programme? Does the rhetoric surrounding our stated goals match the practice? How successful are we at achieving what we say we are achieving? Questionnaire administered to all programme participants Interviews with staff from three different faculties, half of whom have completed this qualification and half have not. Interpretivist approach Observations of fortnightly workshop sessions Collection of assessment tasks from participants Collection of artefacts generated during fortnightly workshop sessions Interviews with participants population initially and representative sample for follow-up Administration of course evaluation instrument Request that participants administer the course evaluation instrument to their own students 2 observations of selected participants teaching Online discussions during the course (Possibly) Feedback from student representatives for each course taught by participants Qualified staff generally outperformed non-qualified in several areas, including flexibility in assessments and teaching style; student centeredness; ability to modify teaching strategies to own subject area; openness to student feedback; and capacity and confidence to discuss and challenge existing practices. (Unclear when findings will be published.) Ongoing.

7 Table 3. Approaches to evaluating the impact of foundations and Certificate programmes: United Kingdom Institution Gibbs and Coffey The Higher Education Academy Name of Programme 22 universities in 8 countries with training programmes between 60 and 300 hours in duration All HEA accredited programmes in teaching, learning and academic practice in the UK Evaluation Question Evaluation Design/Indicators Findings What is the impact of training of university teachers on their teaching skills, their approach to teaching and the approach to learning of their students? What are graduates perceptions of the quality of the programmes and the impact on their teaching practices? A training group of teachers and their students were studied at the start of training and one year later. A control group of new teachers who received no training were studied in the same way. Evidence of change over time was collected using three measures/scales: (i) student ratings of their teachers; (ii) the extent to which teachers described themselves as studentfocused in their approach to teaching; (iii) the extent to which these teachers students took a deep approach to learning Online survey of graduates of accredited programmes in 32 UK higher education institutions Focus group discussions and interviews with Pro-Vice-Chancellors and Vice-Principals, programme leaders, Heads of Departments and Faculties and programme participants in a representative sample of six institutions. Quantitative (survey) and qualitative (survey, focus groups, interviews) analyses Published journal article reports evidence of a range of positive changes in teachers in the training group (e.g., more studentfocused), and in their students (e.g., less likely to adopt a surface approach to learning), and a contrasting lack of change, or negative changes, in untrained teachers from the control group. Published report presents evidence that these programmes can be successful in helping participants become more student-focused in the way they see their teaching. This represents a very positive outcome in terms of the research relating teaching to student learning. However, the time and effort participants are expected to put into programme participation was questioned by a significant proportion of participants and by a number of heads of departments.

8 Table 4. Prevalence of various data collection methods in evaluating the impact of foundations and Certificate programmes Interviews Questionnaires Participant evaluations of programme Student evaluations of participants Focus groups Programme document and artefact analysis Participant assessment tasks and journals Classroom observations Number of Studies