Writing Good Exam Questions

Transcription

1 Everyone deserves to be inspired! Writing Good Exam Questions A Self-study Workbook Written by Dr Kate Exley FOR TRAINING PURPOSES ONLY Produced by the Staff and Educational Development Unit, March 2010 (minor revisions made August 2012)

2 A Self-study Workbook Contents Page List of Figures and List of Tables 1 1. Introduction and Purposes of the workbook Intended learning Outcomes 2 2. LSHTM Exam requirements 4 3. The underlying principles of good question design 6 4. Aligning exam questions and specimen answers with intended learning outcomes What kinds of knowledge and skills can be tested in examinations? Reducing the impact of factors such as stress, interpretation, time Different styles and formats for exam questions Evaluating Draft Questions Marking Approaches : Using assessment criteria and marking schemes Ways of producing accurate and clear marking guidance for questions Ways of producing specimen answers suitable for distribution to students Exam question development, validation and approval processes Security issues and appeals Providing support and guidance for students Concluding Remarks 54 Further Reading suggestions 56 Appendices Appendix 1 Appendix 2 Appendix

3 List of Figures Figure 1 Diagram to illustrate the principles of Constructive Alignment in module design Page 10 Figure 2 Bloom s Taxonomy of Cognition Revisited by Anderson & Krathwohl (2001) 15 Figure 3 A Normal Distribution or bell-shaped curve 34 List of Tables Table 1 Table 2 A table of suggested verbs mapped against the Anderson and Krathwohl adapted levels of Bloom s Taxonomy of Cognition Ways in which intellectual skills can be tested through different question stems Page Table 3 Some Common Essay Style Questions used in Exams 25 Table 4 LSHTM Marking Gradepoints descriptions (Overarching criteria) 37 Table 5 Examples of grade conversions used at the School

4 1. Introduction and Purposes of the workbook 1.1 The intended reader This workbook is intended to support colleagues at The London School of Hygiene and Tropical Medicine as they seek to write appropriate Masters level examination questions and their accompanying assessment criteria and marking guidance. The booklet aims to provide clear guidance on what is expected and through the use of examples and exercises, enable colleagues to test out their own exam question writing. It is primarily intended for those who are new to writing examination questions although more experienced colleagues may find it useful as a reference or updating source. The workbook can be used in a number of ways. For those unable to attend the Writing Better Exam Questions staff development workshop, it can act as a distance learning resource that can be worked through systematically or it can be quickly consulted to check and review current practice. For those using the workshop in the distance learning mode the anticipated learning outcomes are To be familiar with the structure and format of LSHTM examinations To be able to apply the principles of constructive alignment To critique the different kinds of knowledge and skills that can be assessed through the examination format To consider how to ensure that questions are fair and equally accessible to all students. (For example the layout and design of the question on the page) To prepare appropriate marking guidance To be familiar with question approval and validation processes in the School 1.2 The underlying principle John Bigg s theory of Constructive Alignment is used to provide the underpinning framework for question design here. The different kinds of knowledge and skills that can be tested appropriately through a written examination method are discussed as is the need for questions to give equal and fair opportunity to all students. As many students at The School are from overseas and may not have English as a first language there are a number of factors to consider when writing clear and unambiguous questions that do not unintentionally favour particular student groups. A range of different question formats are reviewed and critiqued and a number of ways of quality assuring draft questions are suggested and explained. However, writing the exam question is only half the story producing the associated marking guidance is also considered here. Marking guidance can take a number of different forms ranging from specimen or model answers through to descriptive criteria and detailed marking schemes matched to necessary answer content. These too will be discussed with reference to examples from The School

5 Aims of assessment at LSHTM For all LSHTM courses, the overall aim of assessment is to facilitate the learning of important elements in the course and to test that the student has reached the minimum standard acceptable for the award. LSHTM Assessment Code of Practice (January 2012) - 3 -

6 2. LSHTM Exam requirements London-based MSc Course June exams: There are two three-hour written examination papers taken in June. Together these two papers contribute 30% to the final assessment (15% each). Paper 1 examines the content of Term 1 teaching. It usually comprises questions from each of the core/linear modules taken in Term 1. Thus the same questions for a particular module will appear on several MSc Course exam Paper 1s. Design and marking of these questions is co-ordinated by the teaching module organiser together with other teaching module staff. Paper 2 tests candidates ability to integrate the knowledge and skills acquired during the whole of the MSc course. Paper 2 was originally developed in the mid-1990s after full implementation of the present teaching module structure. As a whole, Paper 2 should be examining the key knowledge/skills which a candidate graduating with an MSc in X should have. In devising Paper 2, MSc Exam Boards should reflect on the intended learning outcomes for the MSc some of which are likely to require assessment in this exam (others might have been assessed in compulsory study modules the project etc). MSc intended learning outcomes can be found in the MSc Course Handbook, prospectus etc. Questions should require integration of knowledge/skills acquired in different parts of the MSc course they might use material from compulsory modules but not optional ones that only some of the class might take. Distance learning PGDip/MSc June exams: Most Distance Learning (DL) modules have a 2-hour exam covering the content of that module and this contributes % of the module s mark. MSc EPI and CT also have a 3-hour integrating paper (E400), akin to Paper 2 above, which candidates sit in their final year of the course. Exam questions are usually coordinated by the Module organiser or other designated members of staff. To provide guidance and formative support on the examination process DL exams are compiled each year into two 'examiners reports' for each MSc course - one for core modules and one for advanced modules. The reports give the complete exam papers together with a guide on how the questions should have been answered. DL students are sent reports from the 3 previous years. All module exams taken will count towards the degree, save only where a student has been assessed on more modules than are required in which instance the Exam Board will determine whether an award may be given, and which modules are counted towards it

7 The School s Assessment Code of Practice describes six assessment objectives that should be kept in mind when writing examination questions and designing assessments. These are Objectives of assessment at LSHTM Identify whether each student has attained a minimum level of achievement necessary to pass the course and identify those who fail to achieve that level. Note - intended learning outcomes should set out the minimum standard of learning required for the award and assessment should be designed to provide students with the opportunity to demonstrate that they have achieved or exceeded that standard (refer to Section 4 below). Focus learning on the important aspects of each course. Note In attempting to increase the difficulty of assessment tasks set it is important to aim to assess more deeply rather than more widely, ie to avoid focussing on peripheral and less consequential details (refer to Section 3 and 9 below). Provide feedback on performance so that learning may improve. Note The goal is that students are able to learn through the process of being assessed and view assessment as part of their learning process this requires that any feedback provided is designed to feedforward (refer to Section 12 below). Provide a means of encouragement. Note It is important to remember that nothing is more motivating than success and students need to be able to see the progress they are making and build their confidence as achievers (refer to Section 11 below). Interfere as little as possible with important, but ungraded, aspects of a student s educational experience. Note - We define learning outcomes but individuals will learn other things whilst studying and unintended outcomes can be equally valuable (refer to Section 5 below). Identify those students achieving the highest standards so that they can be considered for a Distinction. Note Designing exam questions, assessment criteria and marking guidance that allow differentiation and enable good students to get high marks (refer to Section 10 below)

8 3. The underlying principles of good question design The goal Test items should be really difficult for people who don't understand the subject material, but they should be straightforward for those who do. If an item is difficult because of complicated wording (e.g., double negatives) or vocabulary, you will end up testing language skills rather than ability in the discipline. The principles that underlie good question design are i. Clarity ii. Reliability, iii. Validity iv. Authenticity v. Fairness Thinking about each in turn i) Clarity Nothing in the content or structure of [a test] item should prevent an informed student from responding correctly. Gronlund (1998) The clarity of an exam question may be compromised by unclear test instructions, confusing and ambiguous terminology, overly verbose and complicated vocabulary and/or sentence structure plus unnecessary and distracting detail (Gay, L.R., & Airasian, P. 2000). The layout of a question is also very important in conveying clarity particularly in longer, multi-sectioned or data handling styled questions

9 Note some dyslexic students have a tendency to mis-read or miss completely a short second line of text or additional comment or second part of a question such as, e.g. What will be the outcome of adding further sodium chloride at this point? Explain your answer. In an interview, a dyslexic student spoke of this second line being hidden away and he had developed ways of re-reading questions to try and avoid this happening to him, however, the layout of a question may add to this problem, e.g. the indent here, on the second line may make it more hidden. EXERCISE Testing for clarity contrast the following versions of the same exam question (essay format answer required): Version A: Public health policy in the United Kingdom underwent a number of significant changes during the Twentieth Century that can be directly attributed to the needs and exigencies brought about by international conflict. Some of the changes and developments that resulted to health systems and service delivery are still with us today and it is important that we understand the background of circumstances that influenced the decisions that were made. Provide a short analysis charting what you consider to be the main transitions in public health policy brought about by the unique needs and challenges, both direct and indirect, of an environment of international conflict, within the UK health systems specifically, using the Second World War as an example. Version B: Compare the advances in UK public health policy pre- and post-second World War. Think about points such as: unclear test instructions, confusing and ambiguous terminology, being overly verbose, using complicated vocabulary, difficult or poor sentence structure, unnecessary and distracting detail

10 ii) Reliability Does the question allow markers to grade it consistently and reproducibly and does it allow markers to discriminate between different levels of performance? This frequently depends on the quality of the marking guidance and clarity of the assessment criteria. It may also be improved through providing markers with training and opportunities to learn from more experienced assessors. The likelihood of eliciting an accurate measure of a student s ability will be increased when students are provided with a variety of ways to demonstrate their knowledge and skills. For example, some students might generally do better on exams whilst other students do better in their coursework. Including both, in a course will accommodate those differences between students however as the DL courses are provided through the University of London s external programme, that restricts the mode of most assessment to examinations, this may not be an option for all Module leaders. However, even within a written examination we can include a variety of question formats that can help to triangulate and cater to a student s abilities and provide a more reliable measure of their attainments. iii) Validity A valid examination question measures achievement of the intended learning outcomes of the module/unit module (not just what is easy to measure!). The form of the examination question may also be of importance in ensuring validity. For example, examination questions that are short answer questions are a good way of assessing greater breadth of material covered in a course but tend to focus on testing attainment of knowledge and application of knowledge. Whilst longer essay style questions allow a more in depth exploration of subject material and require a candidate to build and structure an argument or explain a complex concept with wide reference to examples and readings. If these aspects are important they should be clearly described in the learning outcomes and be transparent in the assessment criteria for the assessment to be valid. iv) Authenticity Authenticity is the need to match the style and approach of question setting to the reality of practice. This is particularly important when considering the assessment of Masters level qualifications frequently taken by mature students who are accustomed to working within a professional context. A general example might be, rather than set an essay style question, ask students to present their understanding in the style of a professional, or industrial, or clinical report. This may be very important when considering the testing of procedural knowledge or functioning knowledge (please see 5.1). When the exam seeks to test a candidate s knowledge of how something works, the order or sequencing of events, the interplay between contributing factors etc it can be very important to ensure this is built into the question formatting and context setting to allow authenticity

11 Example A learning outcome for a module is..will be able to design survey questionnaires to gather quantitative and qualitative data in the field. An examination question to test this procedural (knowing how to do something) kind of knowledge (rather than memorisation of facts) could, for example, provide the students with a sample survey questionnaire and ask them to give feedback on it e.g. point out weaknesses in its design or suggest improvements and explain how it could be administered in the field. v) Fairness You need to give students a fair chance to demonstrate what they know and can do and to be able to succeed in examinations. Fairness can be facilitated by being very clear about expectations in student performance, providing examples of past examination papers, giving opportunities for students to practice and gain exam technique (through mocks for example), plus transparency in the processes and criteria that will be used to mark and grade their work. Students should know what is expected of them in order to obtain a particular grade and their marks should be a reflection of their abilities and not a reflection of extraneous and irrelevant factors such as gender, disability etc. Providing a level playing field is the aim and this is particularly important at The School when considering the different groups of students who come to study or embark upon DL courses, e.g. non-native English speakers, students who have previously experienced very different educational cultures, mature professionals etc

12 4. Aligning exam questions (and specimen answers) with intended learning outcomes* Constructive alignment is the term coined by John Biggs to describe a coherent approach to ensure that the learning outcomes, teaching and learning methods and the assessment for a unit of study are all directing student learning in the same direction. Figure 1. Diagram to illustrate the principles of Constructive Alignment in module design. What do you want your students to learn? Aims and Learning Outcomes How will you help your students to learn it? Teaching and Learning Methods Learner Support and Guidance How do you know any of it is working? Module Evaluation How will you know how well they have learnt it? Assessment Methods and Criteria * Learning Outcomes is the preferred terminology given in the QAA Codes of Practice however learning goals may also be described as learning objectives in some documentation

13 An excellent place to start when writing an exam question is to go back to the Learning Outcomes for the course or module. These should describe what it is that you want your students to know about or be able to do at the end of the course. Example At the end of the module students should be able to select an appropriate method and use it to test the significance of collected data. The learning outcome clarifies what opportunities need to be built into a test question and ensure that the test is valid. For the learning outcome given above students should be expected to select a method and have the scope to be able to apply the method to some data and finally to be able to comment on the significance or otherwise of the data. To further clarify it would be beneficial to demarcate these three different tasks within the question itself, Example selection of appropriate method, using the method and interpreting the significance of the findings perhaps as separate question sections a, b, and c and finally, for each to have a clear allocation of the total marks for the question. Considering scale and scope Examination questions should also aim to indicate to the students how much is required of them to achieve a good mark the scale and scope of their expected answers. One common way of doing this is to give a time limit (10 questions in 20 minutes) or you could limit the amount of space in the answer booklet or on-line proforma provided. Alternatively you can set a maximum word limit for responses. In addition to these structured ways of indicating the length of answers expected question setters can include Boundaries in their questions, such as Within the limits e.g. Between 2001 and

14 To what extent e.g. Using your knowledge of both prokaryotes and eukaryotes Quantities and amounts e.g. Provide 5 reasons why With reference to e.g. With reference to the published research from.. EXERCISE For the questions given below - Underline the verb and key elements of the question that give an indication of the extent (limits and boundaries) of the question. Do you feel these are appropriate for Masters level study? 1. Describe the three main methods of economic evaluation (40%). What are the main strengths and weaknesses of each method? (40%). Support your answer with examples of disease evaluation (20%) 2. A recent retrospective analysis of health records in the Gambia has suggested that the incidence of malaria has fallen dramatically in that country over the last 10 years. The elimination of the disease is beginning to be discussed. The National Malaria Control Programme has begun a surveillance system to detect future changes. What advice would you give the National Malaria Control Programme on how to organize a surveillance system for malaria. Give practical tips for ensuring its quality. 3. Write short notes on THREE of the following. In each case explain the importance of the infectious agent and the mode of transmission in its spread and control. a) rotavirus diarrhoea b) measles c) guinea worm d) dengue e) tuberculosis Please see Appendix 1. for some feedback comments on this exercise. You may also wish to refer directly to the learning outcomes of your modules and the Master s level descriptors in the Qualification Framework document

15 5. What kinds of knowledge and skills can be tested in examinations? It is possible to test a wide variety of different kinds of knowledge, skills and attitudes through the careful writing of examination questions. 5.1 kinds of knowledge that can be tested e.g. Knowledge domains 5.2 kinds of intellectual skills e.g. Analysing, Evaluating 5.3 kinds of transferable skills e.g. Writing skills, Time use 5.4 kinds of attitude e.g. Ethics, equality Again taking each of these elements in turn let us first consider the different kinds of Knowledge and ways of knowing that you may wish to test in your students. Exam questions should test a range of knowledge and skills at Masters level. They should test and reward critical appreciation and the ability to apply what has been learnt rather than the passive reproduction of memorised facts. Assessment Code of Practice, (2012) 5.1. The kinds of knowledge that can be tested knowledge domains Factual Knowledge Terminology, facts, figures Conceptual Knowledge Classification, Principles, Theories, Structures, Frameworks Procedural Knowledge Algorithms, Techniques and Methods and Knowing when and how to use them. Metacognitive Knowledge Strategy, Overview, Self Knowledge, Knowing how you know

16 EXERCISE Please consider the following four examination questions and decide what kind of knowledge you feel they would test? 1. What are the key steps and processes in bringing a new anti-cancer drug to market and introducing it for clinical use? 2. Write short notes on the following a. Bacterial pathogenicity b. Neisseria gonorrhoeae c. Group A streptococci 3. Using the tabulated data provided calculate the incidence risk of prostate cancer per 1000 men, per 5 years, at each of the given levels of alchohol consumption. 4. Why do malaria parasites persist in the human population. Explain the choice of drugs which could be used to prevent persistence of Plasmoduim falciparum and Plasmodium vivax The kinds of intellectual skills that can be tested At Masters level learning outcomes for/modules usually require students to demonstrate higher level cognitive and intellectual skills, ie it is not enough for students to demonstrate that they can remember facts and figures, names and dates; they need to show they are able to interpret the meaning of data and evaluate their significance. Several cognitive psychologists have been interested in categorising the different ways that we can learn and think about things the most famous of these being a group led by Benjamin Bloom in the mid 1950s. Bloom et al (1956) identified three different domains of learning, Cognitive (knowledge), Affective (attitudinal) and Psychomotor (manual skills). They went on to produce complex hierarchies of skills for the knowledge and attitudinal domains that have been re-visited and revised by many researchers since. When writing examination questions it can be extremely helpful to consider the Cognitive domain hierarchies in particular. Indeed, thinking carefully about the level of cognition that is to be tested will help to select the most appropriate verb to be used in the exam question

17 e.g. Do we want to test a candidate s ability to list important features, analyse the given findings? or critique the argument they give. Anderson et al s (2001) re-working of Bloom s taxonomy makes this easier as they chose to present the hierarchy of sub-categories as active verbs and it is their version particularly that has been widely used in course design and question design in more recent years. It is however important to remember that, Although Bloom's lends itself to wide application, each discipline must define the original classifications within the context of their field Crowe et al (2008) Figure 2. Bloom's Taxonomy of Cognition Revisited by Anderson & Krathwohl (2001) Evaluate Create Analyse Apply Understand Remember Note Some colleagues in the School may already be familiar with the original Bloom taxonomy that uses the terms Knowledge, Comprehension, Application, Analysis, Synthesis and Evaluation

18 Table 1 A table of suggested verbs mapped against the Anderson and Krathwohl adapted levels of Bloom s Taxonomy of Cognition Cognitive Level Verb Examples 1. Remember define, repeat, record, list, recall, name, relate, underline. 2. Understand translate, restate, discuss, describe, recognise, explain, express, identify, locate, report, review, tell. 3. Apply interpret, apply, employ, use, demonstrate, dramatise, practice, illustrate, operate, schedule, sketch. 4. Analyse distinguish, analyse, differentiate, appraise, calculate, experiment, test, compare, contrast, criticise, diagram, inspect, debate, question, relate, solve, examine, categorise. 5. Evaluate judge, appraise, evaluate, rate, compare, revise, assess, estimate 6. Create compose, plan, propose, design, formulate, arrange, assemble, collect, construct, create, set-up, organise, manage, prepare. It is easy to see how Bloom s very hierarchies become employed in different question stems, for example, see Table

19 Table 2. Ways in which intellectual skills can be tested through different question stems. Intellectual Skill Stem Comparing Describe the similarities and differences between... Compare the following two methods for... Relating & Effecting What are the major causes of... What would be the most likely effects of... Justifying Which of the following alternatives do you favor and why? Explain why you agree or disagree with the following statement. Summarising State the main points included in... Briefly summarize the contents of... Generalising Formulate several valid generalizations for the following data. State a set of principles that can explain the following events. Inferring In light of this information, what is most likely to happen when... How would person X be likely to react to the following issue? Classifying Group the following items according to... What do the following items have in common? Creating List as many ways as you can think of for/to... Describe what would happen if... Applying Using the principles of X describe how you would solve. Describe a situation that illustrates the principle of... Analysing Describe the reasoning errors in the following paragraph. List and describe the main characteristics of

20 Synthesising Describe a plan for providing that... Evaluating Describe the strengths and weaknesses of... (Adapted from Figure 7.11 of McMillan (2001) and Piontek, M.E. (2008)) Note you may like to compare these question stems with Bloom s taxonomy, given earlier and draw comparisons and to cross refer to the learning outcomes specified for your own Modules. EXERCISE Take a few moments to look down this list of question stems and select two that you feel could be used to test students on your module/course. Why have you selected these two? 5.3. The kinds of transferable skills that can be tested Short answer and essay styled questions do give an assessor the opportunity to judge a range of generic or transferable skills in the way students answer the questions or respond to the tasks set. The most obvious of these are to do with ability to write clearly and appropriately, to structure and organise answers so that most important points are prioritised and well made and the ability to cite and use source material effectively. If these skills are to be included and given value in the assessment this should be clearly stated in the assessment criteria used to make judgements and this fact should be made clear to students. At The School this is an important issue as many of the Masters students are non-native English speakers. What proportion of the marks for a test question are allocated to skills such as written English should be related to the Aims and Learning Outcomes for the course and context. In some cases accuracy and style may be considered important, e.g. to highlight professional skills and competencies, and be included in the assessment criteria, whilst in others such characteristics are not what is being taught and considered

21 5.4. The kinds of attitude that can be tested Attitudinal learning outcomes, such as equality, fairness, ethical considerations etc, may be important learning outcomes in School Masters programmes and as such are appropriate factors to be tested through the examinations. It is a complex area of assessment as one can argue that just because a candidate knows what they should be saying in response to an equality issue, this does not necessarily reflect what they really feel or how they would react. Examination answers may therefore only be considered as a partial reflection of a candidate s attitude. In some cases it may be that it is more straightforward to assume a student is adhering to the necessary programme attitudes but to penalise (through the grading structure) cases where such attitudes are transgressed. For example, when answers indicate important values are either not fully understood or are not being applied by a candidate, e.g. an answer is unacceptably gendered or racist etc. 6. Reducing the impact of extraneous factors such as stress, interpretation, time Ability to work under pressure or to demonstrate stress tolerance etc are unlikely to be valid learning outcomes for a Masters Programme at The School and therefore all attempts should be made to reduce the impact that stress and nerves may have on a student s performance in an examination. It is possible to set and run examinations in ways that limit the importance of stress induced factors (such as memory lapses) on success. Written examinations can be organised as open book exams* or question topics can be pre-seen by candidates. Such strategies reduce the need to question spot or the impact of luck in revising the right or wrong sub-selection of topics tested. They allow students to think more deeply about and possibly research, their views before attempting the questions (as with course work assessments) but do have the added advantage of avoiding some of the concerns of plagiarism in that candidates produce their individual answers under exam conditions. Those familiar with running these types of examination comment that the quality of student answers are frequently judged as a much higher standard (again as is the case with course work answers). * Open book examinations can allow students to take their own notes or choice of texts or previously specified items into the examination

22 If examinations are to be run traditionally as unseen, time constrained tests carried out by individuals in silence then there are a number of things that the question writer can consider to minimise the impact of such stress factors. For example, Check that the question does not assume a lot of background knowledge which may be culturally specific or introduce unnecessary bias; Provide any important (untested) background detail within the body of the question; Give mark or timing guides within the framing of the question that indicate the relative importance or attached weightings for each sub-section; Set multiple-part problem questions so that the parts are independent from each other. This means that if a student gets the first part wrong they don t automatically lose marks or subsequent sections and makes grading much quicker and more straightforward. E.g in the second part of a question, write something like In the next part of the calculation, assume that the answer to Part (a) was 25, regardless of what you actually got in Part (a). Note that 25 is NOT necessarily the correct answer to Part (a). EXERCISE Can you think of any additional aspects in the exam questions you will be writing that should be considered to reduce the impact of stress factors? Please list these here. An extended example (and exercise) is provided in Appendix

23 7. Different styles and formats for exam questions There are a number of ways in which examination questions can be written and structured that in turn require very different responses from students. Examination papers may consist of a variety of these formats. For example a paper may consist of an initial section of 10 compulsory, short answer questions followed by a second section in which the student is asked to attempt three from six longer questions which may be essay or case study or problem solving styled questions. Here are some examples of different ways questions are written at the School with a commentary highlighting important features (such as the need to avoid ambiguity, bias, inequality and yet be able to discriminate between different levels of attainment and achievement). 7.1 Objective Tests e.g. True-False, Matching Pairs and Multiple Choice Questions There are few examples of such question types being used extensively in summative assessments at the School and they are included here for completeness sake and an acknowledgement that some teachers may well be using these question formats as part of their class or on-line teaching, as self assessment or formative assessment opportunities for their students. Objective tests require a user to choose or provide a response to a question whose correct answer is pre-determined. Such a question might require a student to Select a solution from a set of choices (MCQ, true-false, matching, multiple response); Identify an object or position (graphical hotspot); Supply brief text responses (text input, word or phrase matching); Enter numeric text responses (number input); or Provide a mathematical formula (string evaluation or algebraic comparison). Pass-It, Good Practice Guide

24 True- False Used to test a breadth in knowledge of information but the problem of guessing is a major worry. Matching Pairs Used to assess knowledge of complex and inter-connecting relationships. Multiple Choice Questions - Different Formats There are many different types of MCQs. Some are especially well suited for certain types of content. Some are particularly good for testing higher-order learning. Some are inherently easier' or more difficult' than others. o One-Choice Completions - Best Answer The most commonly used MCQ format is simply a short-answer question with a number of alternatives to choose from. o Multiple-Choice Completions This MCQ format allows for more than one correct answer. Such questions are more difficult since the student is not just looking for one correct response among four incorrect responses. However, the intent of this format is not to test four separate points but rather to set up an interpretive exercise. o Quantitative and Functional Relationships An MCQ format that deals with quantitative and/or functional relationships. They are generally best for knowledge testing but can also be used to test higher-order learning outcomes. There is extensive guidance available on-line (particularly from Universities in the USA) on the construction of multiple-choice tests and some of these are listed in the references provided at the end of this workbook. Traditionally used to test lower order cognitive skills their use in assessing higher order, Masters skills, such as problem solving and analysis, is increasingly being explored. 7.2 Short answer questions can take many forms constructed-response or open-ended questions that require students to create an answer. These may be very short and of a fill in the blank nature or longer, a few sentences or a couple of paragraphs maximum. They can be used to test core knowledge from a module and check the student has the required breadth in understanding. Calculations and data manipulation questions

25 Example - The investigators want to perform a sample size calculation with 80% power and 5% 2-sided significance. They estimate that HIV-free survival at 7 months will be 60% in the control arm. (i) (ii) Calculate the sample size required to detect a 10% increase in HIV-free survival at 7 months in the intervention versus the control arm. (Hint: remember to identify your equation, define all your variables, show all your calculations and conclude appropriately) (10 marks) Assume that 5% of mother infant pairs are lost to follow up prior to the infant reaching 7 months and adjust your sample size calculation accordingly. (4 marks) EXERCISE How could you improve parts (i) and (ii) of the example question above? Please see the concerns that were raised by the Module team over the page

26 Improving the example question Here are the views of the Module leader who raised two questions relating to the clarity of the draft question Part (i) - It isn t clear whether the question is asking students to calculate an absolute or relative increase? This makes a big difference to the calculation (see below). This is an example how the omission of one word can have a significant difference on how the student answers!!!! It is therefore crucial to use technical terminology precisely and avoid expert shorthand that could be mis-leading to a new learner. i.e.) If we are asking about a Relative increase: n= F(,Β) x [ p 1 x (100 - p 1 ) + p 2 x (100 - p 2 )]/ (p 1 - p 2 ) 2 = type 1 error = 0.05 Β = type 2 error = 1- power = =0.2 F(,Β)= 7.85 p 1 = anticipated percentage of Infants in the control group HIV uninfected and alive by 7 months=60% p 2 = anticipated percentage of infants in intervention group HIV uninfected and alive by 7 months=66% n = sample size for each group n= 7.85 [(60x40) + (66x34)]/36 =7.85x4644/36=1013 Not accounting for loss to follow up, a sample size of 1013 women per study arm (2026 total) will give us 80% power and 5% significance to detect a 10% increase in HIV free survival in the intervention from 60% in the control arm. i.e.) And asking about an absolute increase: As before: p 1 = anticipated percentage of Infants in the control group HIV uninfected and alive by 7 months=60% BUT this time: p 2 = anticipated percentage of infants in intervention group HIV uninfected and alive by 7 months=70% (ie 60% plus 10%) n = sample size for each group n= 7.85 [(60x40) + (70x30)]/100 =7.85x 4500/100= Not accounting for loss to follow up, a sample size of 353 women per study arm (706 total) will give us 80% power and 5% significance to detect a 10% increase in HIV free survival in the intervention from 60% in the control arm Part (ii) - Will the formula be included in the question or the provided formulae sheet? This is a straightforward calculation for which there is a formula. Do you expect the students to memorise the formula or will they expect it to be provided? Being clear about what actually should be tested is the important factor here

27 7.3 Longer form Questions - Essay questions Longer format to allow students to respond to open ended questions at length. Used to test higher skills, writing and structuring skills, further reading and a deeper level of understanding. Assessors are frequently interested in a student s ability to organise and integrate a range of ideas and information and build an argument or make a case (the intellectual skills of synthesis and evaluation, going back to Bloom s taxonomy). Two types of essay questions can be readily identified, restricted-response and extended-response. Restricted-response essays focus on understanding of basic knowledge through relatively brief and confined written responses. e.g. Outline the morphology, genome organisation and replication of the human immunodeficiency virus (HIV). Extended-response essays allow student to construct a variety of interpretations and explanation and draw upon a wider and more flexibly defined set of information and sources e.g. The burden of disease caused by intestinal parasites in a community reflects the levels of personal and environmental hygiene. To what extent do you agree with this statement and what are its implications? Make reference to specific infections to support your conclusions. Table 3. Some Common Essay Style Questions used in Exams Question Stem Give a Quotation Discuss Make an Assertion Discuss Compare and Contrast Write-on Outline Describe Explain (with examples) Evaluate Analyse the advantages... Design a

28 EXERCISE Look back over recent examination papers set for your course or teaching module and add two more commonly used Question Stems to this list Longer form Questions Problem Solving / Data handling Here the students are provided with some data (this could be in written, tabulated, graphical form etc) and then asked a series of questions about it. The provided information may be some research findings or monitoring data. The questions usually begin with a couple of straightforward interpretative questions (e.g. Using the table of infection rates provided, which of the described drug therapies reduces the risk of infection the most?). They then move on to more complex questions of application and analysis that require the students to carry out standard manipulations or calculations of the data provided. The final questions are likely to be more evaluative and open-ended, requiring the students to predict likely impacts or suggest improvements etc. An Example On a hot summer day, children in three schools had a school outing to a playground where some of the children played in the recreational fountain. Two days later nearly half the children had symptoms of vomiting, diarrhoea, abdominal pain and headache. A retrospective cohort study was carried out to try to identify the source of the outbreak with the following results. Risk factor Exposed to risk factor Not exposed to risk factor Ill Not ill Ill Not ill Ate commercial ice-cream Drank water from taps near fountain Drank water from taps near sanitary facility Played in fountain Drank from fountain (a) Define what is meant by the risk and relative risk of becoming ill associated with each factor (10%)

29 (b) Calculate BOTH the risk and relative risk associated with each factor (30%). (c) Suggest possible interpretations of the results, and the implications for control recommendations (10%). The investigators wanted to identify the infectious agent involved. One possibility they considered was norovirus which is known to cause acute gastroenteritis. Although reverse transcription-pcr (RT-PCR) method is considered to be the gold standard for diagnosis of this viral infection, it requires skilful personnel and a wellequipped laboratory. A simpler diagnostic kit has been developed. The following table shows how the simpler diagnostic kit compares to the gold standard. Gold standard (RT-PCR) Diagnostic kit Norovirus present Norovirus absent Norovirus present 37 3 Norovirus absent (d) Would you advise the investigators to use the simpler diagnostic test in their epidemiological study? Would your recommendations change if the simpler diagnostic test was to be used in clinical practice? Justify your answer. (50%) [Note on norovirus: this highly infectious RNA virus causes a self-limited, mild to moderate disease that often occurs in outbreaks with clinical symptoms of nausea, vomiting, diarrhoea, abdominal pain, headache, low grade fever or combination of these symptoms. No treatment is indicated apart from rehydration in severe cases. ] EXERCISE In section 6. we discussed a number of ways that a question writer could minimise the impact extraneous factors, such as stress, interpretation, time-management etc in the way they set a question please look over the question above and identify at least three ways the question author has sought to do this

30 7.5 Longer form Questions Case study or Scenario based question In case study styled questions a context or situation is described in detail (e.g. this maybe a patient history or government strategy position etc). Such questions are often seen as being very authentic and ask students to apply their knowledge to a particular and novel, set of circumstances. They frequently take considerable work and effort to write well and usually involve a team of people who craft an idea into a realistic and challenging situation. Note - Some examples of this type of question are presented as examples in section 11. Giving Choice A common structure in examination papers is to have part of the paper core, to be attempted by everybody and other sections which provide a limited amount of choice, e.g. choose 2 from the following list of essay questions to complete. Whilst the structure of exam papers is set by the Board of Examiners and not by individual question setters, it is never-the-less interesting to consider the impact of providing question choice within an exam. Many people view the giving of choice as a way to increase fairness and reduce the affect of luck in question spotting. It allows students to address questions for which they feel most prepared and have been most interested in so seeing the best the student can produce. However, providing choice inherently reduces the validity and reliability of the test instrument because each student is in fact taking a different test and has been encouraged to sample from their learning in different ways. It is nearly impossible to create parallel exam questions that test achievement of the learning outcomes to the same extent, and it is equally difficult to grade two different essays absolutely comparably both factors making consistency very difficult (Piontek, 2008). EXERCISE Do you personally think that the giving of a choice in an examination, (e.g. choose 3 from the following 6 questions) is fair?

31 8. Evaluating Draft Questions It is very difficult to write a question and then immediately see the ambiguities or errors that it contains. Separating the creating from the evaluating roles in time can help. Write a question and then come back to it the following day and re-read with fresh eyes. When you have a draft question, next write a model/specimen answer and/or some marking guidance. As you do this come to a decision about the appropriate break down of marks and try to estimate how long it will take to tackle the question, part by part. In coming up with the marking scheme for your question you might find it helpful to have the learning outcomes for the module or course in sight to refer to so that you can check that you are valuing the right things and giving credit to Master s level criteria. Below is a checklist of questions to use once you have a draft question (doing some of this in a group with questions on overheads can work well):

32 Checklist for reviewing draft exam questions 1. What is the question intended to measure? (eg factual recall, data processing/analysis skills, problem-solving skills, policy analysis skills, critical analysis skills) 2. What else does it actually measure? (eg does it rely too much on factual recall?) 3. Does it measure what we said we would measure? (Is it aligned with the teaching on the course, the content covered and emphasised and the intended learning outcomes?) 4. How well does the question relate to intended learning outcomes (of the teaching module or MSc)? 5. Is the language simple, clear, unambiguous and straightforward? 6. What are the key words describing the task? Are they clear?(eg: list, define, suggest reasons behind the effect are better than interpret, discuss, evaluate) 7. Is the language used easy to understand, including by candidates for whom English is not their first language (eg does it use colloquial phrases)? 8. Check punctuation and grammar as this can markedly change the meaning of sentences (eg panda eats, leaves and shoots ). 9. Does the question give an advantage or disadvantage to those candidates with particular professional backgrounds (eg medics)? 10. How reliably can the answers be marked? 11. If the question is in sections, is the division of percent of marks between sections appropriate? Are there consequences for later sections if a candidate makes an error in an early section? If yes, how will the marking cope with this possibility? 12. Can the question be completed in the time available (including reading, thinking and reviewing time), including those for whom English is not their first language? 13. Does the question lead to answers which will distinguish between weak and strong candidates, eg are there elements for candidates to demonstrate distinction-level skills/knowledge?

33 Question Validation The Masters programme that you contribute too is likely to have its own process of question validation and process of compiling the examination paper. It is important that you ascertain this from the module leader and adhere to it. In general terms, however, once you have the question, model/specimen answer and marking scheme written ask someone else to answer it (do not give them the model/specimen answer), timing each part of the question. It allows you to check that your calculated time it takes to complete estimates were about right. Modify the question, and timings and marking scheme based on any misunderstanding made clear by their answer. It can be helpful to agree a question swap with a colleague and undertake an informal peer review of the questions you have both written. This frequently happens across a course team. At this stage you will be ready to submit your question to the module leader and they too will scrutinise your question and may get back to you with further suggested improvements (please see the extended case study in the Appendix for further detail about the way The School conducts examination question approval processes.) EXERCISE Evaluating a question Please read the following draft question and suggest improvements When you have had a go turn the page and you will see the changes that the examiners team finally made to the question. Question X Draft Describe the structure of the cell plasma membrane and its principal components. How and where are plasma membranes usually made in the eukaryotic cell. How are molecules transferred across the membrane into and out of the cell : Water Ethanol Sodium and Potassium ions Sugars. What other functions in the cell may lipids serve? Over the page you will find the edited version of Question X that was eventually accepted and used in the examination