Measuring the response of students to assessment: the Assessment Experience Questionnaire

Transcription

1 11 th Improving Student Learning Symposium, 2003 Measuring the response of students to assessment: the Assessment Experience Questionnaire Graham Gibbs and Claire Simpson, Open University Abstract A review of literature on student assessment (Gibbs and Simpson, in press) has identified eleven conditions under which assessment supports student learning. A questionnaire has been developed to measure the extent to which assessment is experienced by students as meeting these conditions. Questionnaire items were generated, based on pilot interviews with students on a range of science courses, and on published accounts of student experience of assessment, to form a prototype Assessment Experience Questionnaire (AEQ). This prototype AEQ contains six scales, each containing six items, that address the eleven conditions: 1 Time demands and student effort 2 Assignments and learning 3 Quantity and timing of feedback 4 Quality of feedback 5 Use of feedback 6 The examination and learning This prototype has been administered to a total of 1,050 students on seven science courses at Institution A and also to 529 students on seven science courses at Institution B. The three scales concerning feedback (Scales 3, 4 and 5) have also been administered to 700 students on a wide range of other courses at Institution A. The AEQ is being used to diagnose where assessment could be modified to improve student learning. In a three year study, modifications will be made to assessment patterns on the courses being surveyed and the AEQ administered on the revised courses to monitor the impact of changes. The paper describes the theoretical and empirical basis of the prototype AEQ, describe its characteristics, and outlines plans for its further development. Contact Graham Gibbs Student Support Research Group Open University Walton Hall Milton Keynes MK7 6AA [email protected] Acknowledgement This research reported in this paper is part of a project entitled Improving the effectiveness of formative assessment in Science in receipt of 250,000 from the Higher Education Funding Council for England. Their support is gratefully acknowledged

2 Introduction The research reported in this paper had two starting points. First, assessment is seen to exert a profound influence on student learning: on what students focus their attention on, on how much they study, on their quality of engagement with learning tasks, and, through feedback, on their understanding and future learning. As students become ever more strategic, the way assessment influences learning becomes ever more central in determining student learning outcomes and performance. Second, there is currently no appropriate instrument available for reviewing the way assessment influences learning on a specific course, that teachers could use to evaluate the design of their assessment systems. The item on the Course Experience Questionnaire (CEQ) (Ramsden, 1991) with the largest correlation with student performance, concerns tutor feedback. However the Assessment scale on the CEQ is concerned only with students perceptions of the extent to which assessment orients them to take a surface or deep approach. While this is important it is only one of many influences of assessment on student learning. It is likely to be useful to be able to identify some of these other influences with a similar type of instrument as the CEQ. The Assessment Experience Questionnaire (AEQ) is being developed as a tool for teachers to diagnose how well the assessment on their course is supporting their students learning, in order to be able to make principled changes to that assessment. The AEQ would then be re-administered in order to measure the extent of change, if any, in students perceptions and responses. The scales and items of the CEQ are based on empirical evidence, from qualitative and quantitative studies, concerning what features of course design relate to the extent to which students take a deep or surface approach to their learning. The items and scales of the AEQ are based on a much wider range of studies. The first stage of the development of the AEQ consisted of a literature review of ways in which assessment influences student learning (Gibbs and Simpson, in press). The review encompassed theoretical accounts of the way feedback works and of formative assessment in general, and also case study accounts of innovations in assessment that demonstrated improvements in student learning. The review identified a range of conditions under which assessment seems likely to support student learning, or obstruct or mis-orient learning. These eleven conditions are clustered under five headings in Table 1. Table 1 Conditions under which assessment supports student learning Quantity and distribution of student effort Condition 1 Condition 2 Assessed tasks capture sufficient study time and effort These tasks distribute student effort evenly across topics and weeks Quality and level of student effort Condition 3 Condition 4 These tasks engage students in productive learning activity Assessment communicates clear and high expectations to students - 2 -

3 Quantity and timing of feedback Condition 5 Condition 6 Sufficient feedback is provided, both often enough and in enough detail The feedback is provided quickly enough to be useful to students Quality of feedback Condition 7 Condition 8 Condition 9 Feedback focuses on learning rather than on marks or students themselves Feedback is linked to the purpose of the assignment and to criteria Feedback is understandable to students, given their sophistication Student response to feedback Condition 10 Feedback is received by students and attended to Condition 11 Feedback is acted upon by students to improve their work or their learning Condition 3 relates to the Appropriate Assessment scale on the CEQ but refers directly to the kind of learning activity students engage in when undertaking assignments or preparing for exams, rather than only students perceptions of the demands of assessment. Conditions 1, 4 and 6 are similar to three of the Seven Principles of Good Practice in Undergraduate Education (Chickering and Gamson, 1991). Condition 7 is derived from research on formative assessment in Schools (Black and Wiliam, 1998). Overall the Conditions are derived from a range of types of research, using a range of methodologies, in a range on contexts. They are not only based on insights from phenomenographic research. Development of the AEQ The first stage in the development of the AEQ involved open-ended interviews with Open University students about their experience of assessment, undertaken by lecturers during face to face summer schools at several locations nationally. The students came from a range of Physics and Chemistry courses. The interviewers attended a training session beforehand but there was no quality control over the interview process, and evidence from the interviews was not recorded or reported in a consistent way. The purpose was for the lecturers themselves to hear the kind of things that students said when describing their experience of assessment and to get a feel for the language students used. There were several eye-opening surprises, for example: the extent to which some students were strategic for example reading assignment questions first and then working back through course materials just for those sections that helped with the assignments; rather than reading all of the material in the way the author intended; some students not reading tutors' feedback on assignments at all, because it did not help directly with the next assignment; some students reading feedback on assignments in order to help during revision for the exam, but not when it was originally returned; some students admitting to faking good and tricking their tutor into believing that they had been studying hard and that they understood the material

4 It had originally been intended to develop a large set of potential questionnaire items relating to the 11 conditions and to trial them in order to build coherent factors and scales. However there was a pressing need to diagnose specific assessment problems on each course and to provide a before measure that could be used as a baseline against which to measure the impact of any changes introduced into assessment. As a result a prototype version of the AEQ was quickly constructed. To reduce the total number of questionnaire items, related pairs of Conditions were grouped into single Sections of the questionnaire, each containing six questions. Because of differences in assessment patterns between courses (for example some having exams and some not) separate Sections on exams and on assignments were used, instead of having a single Section for Conditions 3 and 4. The resulting prototype can be seen in Appendix 1 and its associated scoring sheet in Appendix 2. The combining of pairs of Conditions into single Sections turned out to be a mistake for some Conditions, but not for others (see the Factor Analysis below). Sample Three student groups have been sampled using the prototype version of the AEQ students on each of eight science courses at Institution A, a distance learning institution: a total of 1,050 students who were all part time and mature. The students were chosen randomly from large total cohorts. The AEQ was posted to students in a machine-readable form and posted back by students. There were 498 replies, a response rate of 47%. 2. Between 15 and 100 science students (depending on enrolment) randomly selected from each of seven science courses of varying enrolments at Institution B, a conventional face to face institution. The wording of some items on the AEQ needed to be modified in minor ways to make the questions relevant and comprehensible to conventional students. The total of 529 students were mainly full-time and years old. There were 278 replies, a response rate of 53%. The courses in samples 1 and 2 included Physics, Astronomy, Chemistry and Bioscience modules of 15, 30 or 60 credits (involving hours of student effort) and with a variety of patterns of assessment and types of coursework and end-of course assessment or exam. The patterns of assessment in Institutions A and B, while they varied between courses, showed broad institutional characteristics: assignments and in-course tests in Institution B were smaller in size, occurred closer together, and each assessed a smaller chunk (in terms of student learning hours or course credits) than did assignments or in-course tests in Institution A; In Institution A there is a convention of eight assignment and an exam for 60- credit courses and four assignments and an exam for 30-credit courses, with little variation from this convention. In contrast assessment patterns and assignment types were more varied in Institution B, with some courses not using exams; Exams in Institution B were larger (longer in duration) in relation to course size (study hours and credit weighting) than in Institution A; Feedback to students in Institution A is very largely through extensive written feedback on assignments, while in Institution B written feedback is less extensive, and oral and informal feedback, in and out of class, is much more frequent

5 distance learning students at University A. Only sections 3, 4 and 5 of the AEQ, concerning feedback, were used, as there was particular interest in this crucial component of distance education. The students were drawn randomly from 155 courses spread across all disciplines and academic levels. The questionnaire was delivered electronically via , and so only went to those student who had access to the internet. 434 questionnaires were returned, a response rate of 62%. Analysis Data from samples 1 and 2, from institutions A and B, were analysed separately as both the contexts and the students were markedly different. Mean scores both on individual items, and on scales, were different between the two samples, particularly for items and scales concerning the quantity and quality of feedback. Inter-institution differences were larger than inter-course differences within institutions, as shown in Table 2. Table 2 Analysis of institutional and course contributions to variance in student responses df F p Institute (1,762) < Course (12,762) 9.00 < It appeared that there were institutional patterns in the way assessment regimes operated and that the students responded to these patterns in a broadly similar way regardless of the particular course context they were in. Specific institutional differences in student response, and their interpretation, have been reported elsewhere (Gibbs et al 2003). However, separate factor analyses of samples 1 and 2 showed very similar patterns and so the data was combined for the purposes of factor analysis. Principal Component Analysis with Varimax Rotation with Kaiser Normalisation was conducted on the AEQ data from the combined sample of 731 students. Six factors emerged which together explained 50% of the variance. These factors do not correspond in a straightforward way with the Sections of the AEQ. This relative lack of correspondence was confirmed by low Cronbach Alpha reliability coefficients for some of the scales. These are shown below in Table 4. Table 3 shows the items organised under the six Sections of the AEQ. Only factor loadings of greater than 0.4 have been included. The six factors have now been labelled: 1 Quality of feedback 2 Use of feedback 3 Focus on assignments 4 Learning from the exam 5 Distribution of effort 6 Approach to the exam This analysis is interpreted to indicate that: students do not make fine distinctions between quantity, timing and quality of feedback (Factor 1, items in Sections 4 and 5 on the AEQ) but respond broadly positively (or negatively) to all features of feedback in a similar way; - 5 -

6 Table 3 Factor analysis of combined data from Institution 1 and Institution 2 Factor Item Section 2 Time demands and distribution of effort I do the same amount of studying each week regardless of whether -.57 an assignment is due* I can be quite selective about what I study and still do well * +.56 I only study things that are going to be covered in the assignments * +.57 I have to study regularly if I want to do well on the course -.60 On this course it is possible to do quite well without studying much* +.55 In weeks when assignments are due I put in many more hours +.71 Section 3 Assignments and learning Tackling the assignments really makes me think +.45 I learn more from doing the assignments than from studying the +.49 course material In completing the assignments you can get away with not understanding and still get high marks* The assignments give very clear instructions about what you are expected to do When I tackle an assignment it is not at all clear what would count as a successful answer* The assignments are not very challenging* -.44 Section 4 Quantity and timing of feedback On this course I get plenty of feedback about how I am doing -.72 The feedback comes very quickly -.69 There is hardly any feedback on my assignments when I get them +.76 back* When I get things wrong or misunderstand them I don t receive +.77 much guidance in what to do about it* I would learn more if I received more feedback* +.64 Whatever feedback I get comes too late to be useful* +.73 Section 5 Quality of feedback The feedback mainly tells me how well I am doing in relation to others* The feedback helps me to understand things better -.58 The feedback shows me how to do better next time -.53 Once I have read the feedback I understand why I got the marks I -.68 did I don t understand some of the feedback* +.63 I can seldom see from the feedback what I need to do to improve* +.71 Section 6 What you do with the feedback I read the feedback carefully and try and understand what the +.58 feedback is saying I use the feedback to go back over what I have done in the +.70 assignment The feedback does not help me with subsequent assignments* -.47 The feedback prompts me to go back over material covered earlier +.69 in the course I do not use the feedback for revising* -.57 I tend to only read the marks* -.47 Section 7 The examination and learning Preparing for the exam was mainly a matter of memorising* +.65 Doing the exam brought things together for me +.83 I learnt new things while preparing for the exam +.74 I understand things better as a result of the exam +.86 I ll probably forget most of it after the exam* +.42 In the exam you can get away with not understanding and still get +.46 good marks* % variance 17% 9% 8% 6% 5% 5% Items marked * are negatively scored students do distinguish between the quality of the feedback and the extent to which they do anything with this feedback (Factor 2, including items from Section 6 of the AEQ); students distinguish between two different kinds of distribution of effort: being selective about what to study in relation to assessment demands (Factor 3) and - 6 -

7 studying different amounts in different weeks in relation to the timing of assessment demands (Factor 5) students distinguish between two different aspects of examinations (items from Section 7 of the AEQ): the extent to which preparation for the exam and tackling the exam resulted in useful new learning (Factor 4) and the extent to which examinations were perceived only to involve memorisation (Factor 6). The lack of coherence of some of the Sections of the AEQ was confirmed by calculation of reliability coefficients for the six items in each Section. As can be seen in Table 4, Sections 1, 2 and 6 do not have satisfactory levels of internal reliability. Table 4 Internal reliability of AEQ scales and sample items Scale Cronbach Alpha 1 Time demands and distribution of student effort Assignments and learning Quantity and timing of feedback Quality of feedback Use of feedback The examination and learning 0.57 An identical form of factor analysis to that carried reported in Table 3 was conducted on the same kind of data from Sample 3 (where only Sections 3,4 and 5 of the AEQ were used) and resulted in four factors emerging which together explained 61% of the variance (see Table 5). Table 5 Factor analysis from sample 3 (AEQ Sections 3, 4 and 5 only) Factor Item Section 4 Quantity and timing of feedback On this course I get plenty of feedback about how I am doing The feedback comes very quickly -.82 There is hardly any feedback on my assignments when I get them back* When I get things wrong or misunderstand them I don t receive +.73 much guidance in what to do about it* I would learn more if I received more feedback* +.61 Whatever feedback I get comes too late to be useful* Section 5 Quality of feedback The feedback mainly tells me how well I am doing in relation to +.76 others* The feedback helps me to understand things better -.70 The feedback shows me how to do better next time -.77 Once I have read the feedback I understand why I got the marks I +.70 did I don t understand some of the feedback* +.71 I can seldom see from the feedback what I need to do to improve* +.74 Section 6 What you do with the feedback I read the feedback carefully and try and understand what the +.80 feedback is saying I use the feedback to go back over what I have done in the +.78 assignment The feedback does not help me with subsequent assignments* +.59 The feedback prompts me to go back over material covered earlier in the course I do not use the feedback for revising* -.42 I tend to only read the marks* +.73 % variance 27% 14% 12% 8% Items marked * are negatively scored The first two factors, together explaining 41% of the variance, were almost identical to Factors 1 and 2 identified above from the use of the complete AEQ (see Table 3)

8 These two factors concerned the quality of feedback and the use of feedback. These factors emerged in the same way despite the very different sample who were mainly not science students. The third factor was concerned with the timing of feedback (which did not emerge as a separate factor in Samples 1 and 2) and the fourth factor was less coherent. Conclusions 1. Students response to the quality and use of feedback emerged as the most important aspect of students response to assessment, accounting for 26% of the variance in Sample 1 and 2 combined, and 41% of the variance in Sample Students did not distinguish consistently between different characteristics of the feedback itself (its quantity and qualities). 3. Students did distinguish between the feedback itself and what they do with it. 4. Institutional contexts, that set the framework for the broad pattern of assessment students experience, had a very significant influence on students response to assessment, to a greater extent than did course contexts. 5. Despite these institutional differences, and the different types of students in the two institutional contexts studied, the same factors emerged from analysis of students responses to questionnaire items. 6. The same factors (concerning feedback) emerge from student responses in other discipline areas than science, in Institution A. 7. The prototype AEQ needs to be developed into a working version in which scale scores are more meaningful because they are supported by more coherent underlying factors. New scales and some new items are required. Despite the somewhat muddled relationship between the structure of Sections on the prototype AEQ and the structure of the factors that emerged from factor analysis, teachers have been able to interpret item and scale scores and use them to diagnose issues of concern that they were not previously aware of, that would benefit from attention. There are currently two follow-up studies under way at Institution A. The first study concerns what aspects of feedback do and do not make use of. Tutors are coding each others feedback on assignments and then interviewing each others students about what, exactly, they do with each category of feedback. Students use of model answers will also be explored. The intention is to revise tutor briefing and monitoring so as to orient tutor behaviour towards more useful forms of feedback. The second study concerns the impact on students studying, and on their revision, of the way they interpret the demands of different kinds of exam. Tutors are interviewing their own students and then an examinations questionnaire will be developed to measure to the extent to which different categories of response are evidence on courses with very different exam demands. The aim is to modify exam demands and to articulate their intended demands more clearly. In Institution B teachers are exploring the way students respond to and use all kinds of feedback, not just formal written feedback on assignments. They are developing a questionnaire which identifies both possible sources of feedback (such as question answering in lectures and guidance during laboratory sessions) and possible forms of impact (such as clarifying goals or correcting mistakes). The aim is to identify which sources of feedback are effective and ineffective at achieving various impacts, with the aim of changing the ways in which feedback is provided. All three of these studies have been prompted by teachers' interpretations of data from the prototype AEQ, though usually by identifying atypical patterns of response to individual items rather than atypical scale scores

9 The development of the AEQ On the basis of the above factor analyses the prototype AEQ will be developed into a working version which will contain seven scales in the following way: 1 Consolidating a single scale, concerned with the quality of feedback, combining the best discriminating items from Sections 3 and 4. 2 Consolidating a scale based on Section 5, on use of feedback. 3 Consolidating a scale on distribution of effort, using items from Section 2, and new items. 4 Developing a new scale, concerning focus on assignments, combining selected items from Section 1 and 2. 5 Developing a new scale on timing of feedback with items from Section 4 and additional items. 6 Developing a new scale on learning from the exam, with items from Section 7 and additional items. 7 Developing a new scale on approach to learning (in relation to assessment both exams and coursework) drawing on appropriate assessment items from the CEQ and items from Sections 2 and 7. 8 Developing a new scale on clear goals and standards, based on the CEQ. It is intended to make this revised AEQ freely available for use by teachers and researchers exploring assessment and learning. References Black, P. and Wiliam, D. (1998) Assessment and classroom learning. Assessment in Education. 5(1), Chickering, A. W. and Gamson, Z. F. (1991) Applying the Seven Principles to Good Practice in Undergraduate Education. San Francisco: Jossey-Bass. Gibbs, G. & Simpson, C. (in press) Does your assessment support your students learning? Journal of Teaching and learning in higher Education Gibbs, G., Simpson, C. & Macdonald, R. (2003) Improving student learning through changing assessment a conceptual and practical framework. European Association for Research into Learning and Instruction Conference, Padova, Italy, Aug Ramsden, P. (1991) A performance indicator of teaching quality in higher education: the Course Experience Questionnaire. Studies in Higher Education, 16,

10 Appendix 1 Assessment Experience Questionnaire (AEQ) Please answer every item quickly by giving your immediate response. Circle the appropriate code number to show your response to assessment. 1 Amount and distribution of study effort I do the same amount of study each week, regardless of whether an assignment is due or not. I can be quite selective about what I study and learn and still do well I only study things that are going to be covered in the assignments I have to study regularly if I want to do well on the course On this course, it is possible to do quite well without studying much In weeks when the assignments are due I put in many more hours Assignments and learning Tackling the assignments really makes me think I learn more from doing the assignments than from studying the course material. In completing the assignments you can get away with not understanding and still get high marks. The assignments give very clear instructions about what you are expected to do. When I tackle an assignment it is not at all clear what would count as a successful answer. The assignments are not very challenging Quantity and timing of feedback strongly agree agree? disagree strongly disagree On this course I get plenty of feedback on how I am doing The feedback comes back very quickly There is hardly any feedback on my assignments when I get them back When I get things wrong or misunderstand them I don t receive much guidance in what to do about it. I would learn more if I received more feedback Whatever feedback I get comes too late to be useful

11 4 Quality of feedback strongly agree agree? disagree strongly disagree The feedback mainly tells me how well I am doing in relation to others The feedback helps me to understand things better The feedback shows me how to do better next time Once I have read the feedback I understand why I got the mark I did I don t understand some of the feedback I can seldom see from the feedback what I need to do to improve What you do with the feedback I read the feedback carefully and try to understand what the feedback is saying. I use the feedback to go back over what I have done in the assignment The feedback does not help me with any subsequent assignments The feedback prompts me to go back over material covered earlier in the course. I do not use the feedback for revising I tend to only read the marks The examination and learning (only to be completed if there is an exam) Preparing for the exam was mainly a matter of memorising Doing the exam brought things together for me I learnt new things while preparing for the exam I understand things better as a result of the exam I ll probably forget most of it after the exam In the exam you can get away with not understanding and still get good marks. Comments you would like to make about the way the assessment affected your learning on the course

12 Appendix 2 Assessment Experience Questionnaire Scoring Sheet Write in the numbers circled for each question and add the scores for each scale. 1 Amount and distribution of study effort 2 Assignments and learning 3 Quantity and timing of feedback Total = Total = Total = 4 Quality of 5 What you do with 6 The examination feedback the feedback and learning Total = Total = Total = 1 Amount and distribution of study effort A high score indicates that students study evenly across weeks and across topics, and feel that they have to in order to do well. A low score indicates that students study effort is allocated narrowly to assessed topics and those weeks where assessment takes place, and feel they can get away with this and still do well. 2 Assignments and learning A high score indicates that students see assignment requirements as clear and challenging, requiring understanding. A low score indicates that assessment demands are perceived as unclear and that assignments are seen as unchallenging and as not requiring understanding. 3 Quantity and timing of feedback A high score indicates that students perceive that they get plenty of feedback fast enough. A low score indicates that students perceive the feedback to be insufficient to support their learning, and too late to be useful. 4 Quality of feedback A high score indicates that students find the feedback understandable and useful, explaining both grades, misunderstandings and how to improve. A low score indicates that the feedback is neither comprehensible nor useful, and only indicates how well the student is doing in relation to others. 5 What you do with the feedback A high score indicates that students use the feedback to guide follow-up learning, to tackle subsequent assignments differently, and to revise. A low score indicates that the feedback has little impact on subsequent studying and learning. 6 The examination and learning A high score indicates that the perceived exam demands had a positive influence on the quality of learning undertaken during the course and during revision and that the exam itself was a learning experience. A low score indicates that the perceived exam demands encouraged memorisation and subsequent forgetting