THE INSTITUTE FOR TEACHING AND LEARNING EVALUATION AND QUALITY ASSURANCE (EQA) WORKING GROUP

Transcription

1 ORIENTATION PACK THE INSTITUTE FOR TEACHING AND LEARNING EVALUATION AND QUALITY ASSURANCE (EQA) WORKING GROUP This document is intended to assist the ITL to orient new members of the EQA Working Group to the purpose of the group, its ways of working and their role as their Dean s nominee on the group. The ITL convened the Evaluation and Quality Assurance Working Group in December 2000 to support faculties in implementing aspects of the University policy on The Management and Evaluation of Teaching and in making the best use of institutional and faculty level data. The EQA Working Group also provides an additional forum for consultation between the PVC (Learning and Teaching) and the faculties on relevant issues The Evaluation and Quality Assurance (EQA) working group has representatives from the different faculties who are nominated by their Deans. The members of the working group cooperate to: Facilitate the establishment of appropriate processes to implement academic board policies relating to the enhancement and assurance of learning and teaching in the participants faculties Support faculties in interpreting quality assurance data (eg SCEQ, USE, students open response comments etc) and identifying potential contributing factors Support faculties in identifying relevant issues to investigate further and the formulation of collaborative research strategies to effect such investigations Support faculties in using evaluation and quality assurance data in benchmarking activities Support faculties in planning effective responses to the teaching and learning issues identified in quality assurance data Provide leadership in their faculties in the area of evaluation and quality assurance of teaching and learning Support the dissemination and communication of information regarding strategic university initiatives in evaluation and quality assurance of teaching and learning to colleagues in their faculties Support the learning of colleagues in their faculties in relation to evaluation and quality assurance of teaching and learning Facilitate consultation between faculties and the PVC L&T and the ITL on issues relating to the evaluation and quality assurance of learning and teaching Contribute to the development and formulation of policy in relation to recognising and rewarding excellence in teaching Identify and prioritise strategic issues for the working group to address each year It is anticipated that the scope and purpose of the working group will evolve in response to the changing needs of the university community. The group meets three or four times a year and group members work between meetings to help achieve the aims of the group. This work might involve gathering information to share with other members of the working group at meetings, faculty based academic development activities relating to evaluation and quality assurance of teaching and learning and other activities as appropriate in their respective faculties.

2 Representation on the Working Group Agriculture, Food and Natural Resources Architecture Arts Dentistry Economics and Business Education and Social Work Engineering Health Sciences Law Medicine Nursing and Midwifery Pharmacy Science Summer School Sydney College of the Arts Sydney Conservatorium of Music Veterinary Science PVC Learning & Teaching Executive Officer (Quality Assurance) Institute for Teaching and Learning: Director Institute for Teaching and Learning: Survey Officer Institute for Teaching and Learning: EQA Project Leader (Chair) Policy Framework There are some key university policies and quality assurance & enhancement processes that are relevant to the work of the EQA Working Group. These are: Academic Board Resolutions on The Management and Evaluation of Coursework Teaching Assessment and Examination of coursework teaching Guidelines for Academic Board Reviews of Faculties Performance Based Funding of Teaching model: Teaching Dividend, Teaching Improvement Fund and the Scholarship Index. Information on the University s quality assurance framework for learning and teaching including many of the policies listed above is available at:

3 Student-Focused Teaching Evaluation Instruments At the heart of much of the EQA Working Group s work are the teaching experience surveys which underpin the university s learning and teaching evaluation and quality assurance strategies. At the University of Sydney, there has been a concerted effort in recent years to draw on the strengths of theory and evidence in the design and use of student evaluation of teaching surveys (Prosser & Barrie 2003). This has involved major changes in the collection of institutional data related to teaching and learning, in order to achieve a productive alignment between requirements to prove quality (to external bodies such as government and quality assurance agencies) as well as improve quality through feedback to university teaching staff to support curriculum and academic development. To this end, the ITL has developed an articulated, student-focused suite of survey instruments to support teaching evaluation and improvement. The development of these surveys was based in the research literature on student learning, university teaching and survey design and took account of the national higher education context, institutional benchmarking requirements and the university community s student feedback needs. The process of development of the surveys involved collaboration with international experts, consultation with the academic community (staff and students), and psychometric testing of the survey instruments. These surveys underpin much of the University s research into teaching and learning at both an institutional level and in faculties and individual units of study. An underlying theoretical perspective: The Course Experience Questionnaire (CEQ) Research over the past 25 years has repeatedly identified that the ways that higher education students approach their studies and the quality of their learning outcomes are not just functions of individual differences such as prior ability. The ways in which students experience the teaching and learning environment has a major impact. For example, a student who perceives that a particular course has a heavy workload, and largely requires rote learning of facts, will be likely to take a surface approach to learning, entailing the reproduction of content. Alternatively, if the student perceives a lower required workload, that the teaching emphasises self-directed learning; and that goals are clear, she will be more likely to adopt a deep approach to learning, focusing on understanding the underlying relations within the content. Researchers investigating the effects of such perceptions in a wide range of educational domains have found students who adopt a surface approach tend to have poorer learning outcomes than those adopting a deep approach. (For reviews of this research, see Prosser & Trigwell, 1999; Watkins, 2001). The Course Experience Questionnaire (CEQ) is a national survey of coursework graduates experiences that has been conducted by the Graduate Careers Council of Australia since It was developed by Professor Paul Ramsden (Ramsden, 1991; Wilson, Lizzio, & Ramsden, 1997) as a reliable and valid performance indicator of teaching quality in higher education. The core scales of the CEQ are intended to measure aspects of the student experience which have been demonstrated to be associated with deep and surface approaches to learning (Ramsden & Entwistle, 1981, Richardson 2005). The psychometric validity of the scales and items of the CEQ survey have stood up to over 20 years of rigorous investigation by researchers around the world. The version of the CEQ currently used by the University asks students to rate their experience of Good Teaching, Generic Skills development, and Overall Satisfaction with Degree Quality at the course (degree) level. The University of Sydney Student Course Experience Questionnaire (SCEQ) and the Unit of Study Evaluation (USE) both derive from the CEQ, and the theoretical and psychometric strength of the University of Sydney surveys is based in part in the findings of 20 years of research into the original CEQ.

4 The Student Course Experience Questionnaire (SCEQ) One of the limitations of the CEQ is its long lag: the CEQ is conducted the year after students graduate, and there is a delay in returning results to the University. In some cases, several years may pass before students exposed to a teaching initiative intended to enhance student learning graduate and rate their perceptions of this aspect of their university experience on the CEQ. To compensate for this limitation, the University of Sydney modified the CEQ for use with currently enrolled students. The modified CEQ was called the Student Course Experience Questionnaire (SCEQ). As a survey of enrolled students it provides academic staff and academic managers with a timelier indicator of teaching and learning quality. This approach to teaching quality assurance is now being taken by a number of Australian universities. The SCEQ survey or its sub-scales and items have been adapted for use by several universities including Queensland, Monash, and Wollongong, and the University of Oxford. The SCEQ has been run annually at the University of Sydney between 1999 and 2003, and from 2003 biannually. Deriving from the CEQ scales, the core SCEQ scales ask students about their perceptions of Good Teaching, Clear Goals and Standards, Appropriate Assessment, Appropriate Workload, Generic Skills, and Overall Satisfaction with Degree Quality. Other items and scales, developed as indicators of University of Sydney s strategic priorities, measure intellectual motivation, learning community, learning resources, supportiveness of the degree course administration, and benefits of contact with active researchers. New scales are being developed in 2005 to gather benchmarking data in relation to research-led-teaching, generic graduate attributes and e-learning. Students also provide ratings on use and quality of the various student services (e.g. library facilities). Lastly, students provide open-ended feedback on both the quality of their degree experience, and the quality of student services. The SCEQ is administered by the ITL to a random stratified sample of both undergraduate and postgraduate coursework students, using , postal, and phone contacts to achieve a mandated response rate greater than 50% for the undergraduate sample. Results of the SCEQ are publicly available through the ITL website SCEQ results are used for many purposes, including: Academic Board reviews Performance-Based Funding of Teaching (see Curriculum review and development Benchmarking with other universities Like the CEQ, the major SCEQ scales are both reliable (internally consistent) and valid in that the scale scores correlate with overall ratings of degree quality, and distinguish between different degrees and faculties (Ginns, Prosser & Barrie, in submission). Moreover, changes in SCEQ scale results between 1999 and 2003 indicate that students perceptions of key aspects of the teaching and learning environment are improving. These improvements have been more marked for commencing first year students, a result attributed to the University s extensive work in this area (Barrie, Ginns, & Prosser, in press). Prevailing themes in students responses to openended questions on the SCEQ have also shown marked changes over time, a result likewise attributed to faculties teaching and learning evaluation and improvement efforts (Symons, 2004). Results of the SCEQ are publicly available on the ITL website including trend reports over time.

5 The Unit of Study Evaluation (USE) The SCEQ provides data in relation to the quality of students experiences of their overall degree course. However, this information does not provide the staff responsible for a degree program with information about the quality of the various units of study that make up the degree. There is a need to link efforts to improve degrees through improvements in teaching and curricula of units of study, with the measures used to gather data on the quality of students overall experiences of the degree. That is, measures of students experiences of units of study need to be able to be related to measures of students experiences of the overall degree. The ITL s Unit of Study Evaluation (USE) system was developed with this need in mind (Barrie 2001; Prosser & Barrie 2003). The USE system is also designed to support relevant aspects of the University's policy on The Management and Evaluation of Teaching which was approved by Academic Board on 16 May, The key purpose of the USE system in this regard is to provide information for unit of study coordinators seeking to assess the effectiveness of units of study in order to plan and implement teaching and learning improvements. The focus of the system is on the unit of study and not the individual or the faculty. The first five USE survey items correspond to the factor scales of the university's primary teaching quality assurance survey - the Student Course Experience Questionnaire (SCEQ), which is used to monitor teaching quality and inform strategic teaching initiatives at a faculty level. Each SCEQ scale has a core item this item was used as the basis for the development of the corresponding USE item. The relationship of USE scores to SCEQ scores has been shown in several different faculties (Barrie & Ginns, in submission). Because of this correspondence between the USE survey items and the SCEQ factor scales, USE data can provide staff of the department, school or faculty with an indication of the relative contributions of different units of study, to performance on the SCEQ factor scales. The relationship of USE scores to SCEQ scores has been shown across different faculties. For instance, the correlation between facultyaverage SCEQ 2001 Good Teaching scores and faculty-average USE 2001/2002 scores for the USE Good Teaching item ( The teaching in this unit of study helped me learn effectively ) was (Barrie & Ginns, in submission). As well as the SCEQ related items, there are two items related to requirements of the university policy for integration between units of study, and responsiveness to student feedback. In addition to the 8 core items, faculties may add up to 4 additional items to gather information about faculty-specific teaching initiatives or priorities. The relationship of the USE items to the SCEQ factor scales and relevant aspects of Academic Board policy are summarised on the USE website at (Barrie 2001). Ginns and Barrie (2004) found, like the SCEQ, the USE was both reliable (internally consistent) and valid (scale scores correlate highly with overall ratings of unit of study quality, and the overall rating distinguishes between the quality of different units of study). The development of the USE items involved collaboration with international experts, consultation with the academic community, and piloting of the items with students, in addition to the investigation in the psychometric relationship between the USE items and the SCEQ scales. The interactive web ordering and reporting system was developed in consultation with university staff and the ordering and reporting process was approved by the Chair of Academic Board to ensure appropriate confidentiality of results is maintained. USE surveys can be ordered in two ways. Deans, Heads of Department/School or their delegates can order surveys for several units of study using the Faculty order form. Heads of Department/School should inform unit of study coordinator that the survey will occur, and indicate the process to be used for return of the USE survey results. Individual Unit of Study Coordinators can also order surveys using the individual order form. The placing of faculty orders and the determination of the additional four faculty items are usually the responsibility of the EQA Working Group representative in each faculty. See for details of the ITL Procedures approved by the Chair of Academic Board. Research Higher Degree Students Experiences of their Courses:

6 The teaching evaluation instruments discussed so far focus on the experiences of coursework students, making them less suitable for research higher degree (RHD) students. Recognising this problem, in 1996 the Department of Education, Training and Youth Affairs commissioned the development of the Postgraduate Research Experience Questionnaire (PREQ). From 2001, the PREQ has been part of the Graduate Destination Survey sent to RHD graduates. The PREQ is the research higher degree equivalent of the CEQ. It asks RHD students of all Australian universities about their perceptions of six aspects of their experience: Supervision, Intellectual Climate, Skills Development, Infrastructure, the Thesis Examination process, and the Clarity of Goals and Expectations. Because of the relatively small number of graduates per year, and the same time-lag problem which occurs for the CEQ, since 2002 the ITL has run an additional annual survey of enrolled RHD students based on the Australian Research Centre (ARC) funded research into the experience of such students (Ramsden, Conrad, Ginns & Prosser, 2003). This modified and expanded form of the PREQ, the Student Research Experience Questionnaire (SREQ), mirrors the major PREQ scales of supervision, intellectual/social climate and infrastructure; expands the original generic skills scale to include additional items related to independent learning ; and adds additional items on the frequency of various forms of supervision (e.g. one-on-one, group-based, phone, ), and usage and quality of support services. As with the PREQ, respondents to the SREQ may also give open-ended comments on their degree experience. For more details, see Like the PREQ, the SREQ scales have good levels of reliability and validity (i.e. correlate well with overall ratings of the quality of the RHD experience) (Ginns, 2004). Results from the PREQ and SREQ are being used at a variety of levels within the University: Schools, departments and faculties can use these results for planning initiatives and tracking the results The Academic Board uses the results as part of its Reviews Results can be uses to identify infrastructure priorities as part of submissions for funds from the Research Postgraduate Infrastructure Scheme (RPIS)

7 References Barrie S.C., (2001). Background to the Unit of Study Evaluation. accessed on 10 May Barrie, S., Ginns, P., & Prosser, M. (in press). Early impact and outcomes of an institutionally aligned, student focused learning perspective on teaching quality assurance. Assessment and Evaluation in Higher Education. Barrie & Ginns, (in submission). The linking of institutional change to improvements in teaching and learning in classrooms. Ginns, P. (2004). Quality Assurance and Improvement of the Research Higher Degree Experience at the University of Sydney. Paper presented at the Australian Universities Quality Forum, Adelaide, July 7-9. Ginns, P., & Barrie, S.C. (2004). Reliability of single-item ratings of quality in higher education: A replication. Psychological Reports, 95, Ginns, P., Prosser, M., & Barrie, S. (in submission). Students perceptions of teaching quality in higher education: The perspective of currently enrolled students. Prosser, M., & Barrie, S. (2003). Using a student-focussed learning perspective to strategically align academic development with institutional quality assurance. In R. Blackwell & P. Blackmore (Eds.), Towards strategic staff development in higher education. Buckingham: Open University Press. Prosser, M., & Trigwell, K. (1999). Understanding learning and teaching: The experience in higher education. Buckingham: Open University Press. Ramsden, P. (1991). A performance indicator of teaching quality in higher education: The Course Experience Questionnaire. Studies in Higher Education, 16, Ramsden, P., & Entwistle, N.J. (1981). Effects of academic departments on students approaches to studying. British Journal of Educational Psychology, 51, Symons, R. (2004). Getting the most out of the SCEQ data: meeting the needs of multiple stakeholders. Paper presented at the 2004 Evaluation Forum, Melbourne, Australia, November Watkins, D. (2001). Correlates of approaches to learning: A cross-cultural meta-analysis. In R. Sternberg and L. Zhang (Eds.), Perspectives on thinking, learning, and cognitive styles (pp ). New Jersey: LEA. Wilson, K.L., Lizzio, A., & Ramsden, P. (1997). The development, validation and application of the Course Experience Questionnaire. Studies in Higher Education, 22, Ramsden, P., Conrad, L., Ginns, P., & Prosser, M. (2003). Students and supervisors experiences of the context for postgraduate study. Paper presented at the Bi-annual European Association for Research on Learning and Instruction Conference, Padua, August 26 30, 2003.