Module Evaluation Framework

Transcription

1 Module Evaluation Framework Introduction 1. The Quality Assurance Agency states, in Chapter B5 of the UK Quality Code for Higher Education, that it is an expectation that: Higher education providers take deliberate steps to engage all students, individually and collectively, as partners in the assurance and enhancement of their educational experience It is a clear requirement that providers seek the views of students on all aspects of their provision, working with the student body to obtain, analyse and act upon feedback from students on their modules. 2. The College Education Committee has adopted, for this purpose, a Module Evaluation Framework, designed to facilitate existing good practice, and give ownership to departments and faculties of the evaluation processes, while establishing baselines which all modules must comply with. 3. This Framework consists of regulations for module evaluation, which all programmes must abide by, and guidance on the operation of module evaluation which is designed to support module convenors / organisers, departments and schools in developing their own module evaluation policies and practice. The regulations establish a standard scale for questions, and includes a small number of compulsory questions to be used for all module evaluations; it is anticipated that questionnaires will be established that supplement these compulsory questions with other questions tailored to local needs. The guidance includes a question bank which includes questions that may be used, guidance on the delivery and analysis of the survey and technical guidance for methods such as online evaluation. 4. This paper outlines that Framework in detail. Module Evaluation Regulations 5. The following regulations apply in respect of module evaluation: (a) (b) (c) Programmes at King s College London are normally modular in structure. Where programmes are offered that are not modular then a module as referred to in this section of the Regulations may include any discrete block of learning up to and including a full year of study on the programme in question. Each module must offer a minimum of one opportunity for formal evaluation by students enrolled on that module each year. Each module evaluation must include the following statements, which students must be asked to evaluate on the scale 1-5, where 5= strongly agree and 1= strongly disagree. (i) (ii) (iii) (iv) The module was intellectually stimulating; The module was well organised; Feedback on my work 1 has helped me clarify things I did not understand The criteria used in marking were made clear in advance (where applicable); 1 This question need only be asked where students have submitted work for formal assessment four weeks or more before they are in a position to complete the evaluation

2 (v) (vi) (vii) The teacher / lecturer(s) was good at explaining things; The teacher / lecturer(s) taught at an appropriate pace; Overall I am satisfied with the quality of the module Additional questions may be asked at the discretion of the relevant Faculty Education Committee or its delegated committee. (d) Each module that is in part formally assessed no later than four weeks before the final teaching session for the module should normally include the following statement, which students must be asked to provide a yes/no answer to: I received feedback on the coursework that I submitted for this module within 4 weeks. (e) (f) Module evaluation must normally be anonymous and undertaken either in writing or online. All reasonable steps must be taken to enable respondents to respond without revealing their identity. Module convenors / organisers must analyse responses. Each Departmental Education Committee (or equivalent) must outline a process for consideration of module evaluation for each module. This must include: a systematic process for identifying actions to be taken as a result of module evaluation for each module. A systematic process for ensuring that Heads of Department,Faculty Education Leads and Deans of Faculty receive an analysis of each module evaluation This process must be approved by the appropriate Faculty Education Committee. (g) Outcomes from module evaluation must be fed back to students, via Staff-Student Liaison Committees as well as published on Departmental websites which are accessible to enrolled students. Published information may include the analysis reports prepared by module convenors, outcome reports produced and approved by Departmental Education Committees (or equivalent) or some other output as approved by the Faculty Education Committee. Care must be taken to ensure that respondents cannot be identified by any published information. Guidance Introduction What do we mean by module evaluation? 1. At King s College London programmes of study are normally made up of discrete blocks of learning, consisting of a defined number of credits (normally, 15, 20, 30, 40 or 60). These blocks are known as modules. The aim of module evaluation is to obtain systematic written feedback from students on the design, delivery and assessment (apart from final examination) of these modules. Module evaluation normally consists of a questionnaire completed by the students, either in paper or online form, at any point at or before the final teaching session of the module. 2. Where programmes do not consist of discrete credit-rated modules, module evaluation principles can be applied to any coherent block of learning that exists within the programme.

3 What is it that we are evaluating? 3. The objectives of module evaluation are to obtain feedback on; (a) The design of the module and its coherence within the overall programme of study; (b) the delivery of the module, including the quality of teaching and learning support, the organisation of the module and the quality of learning resources used to support the module; (c) the relevance of the assessment methods used (other than the final examination) to the material taught in the module; (d) the feedback given to the students on their performance in any assessment undertaken as part of the module (e) any other aspect of the design and delivery of the module that has a direct impact upon the student experience. What is the purpose of this evaluation? 4. The purposes of module evaluation are to: (a) Provide module convenors / organisers with insight into the quality of teaching on the module; (b) Provide module convenors / organisers with feedback and evidence to support teaching enhancement and excellence in teaching, including the development of the curriculum, teaching methods, learning resources and assessment of modules and their parent programmes; (c) Provide departments with feedback and evidence to support the monitoring of the design and delivery of modules, supporting the development and delivery of modules and programmes and to take immediate action where necessary; (d) Enable students to contribute to the design and development of modules and programmes, by providing a systematic opportunity for their opinions to be considered as part of regular review processes; (e) Provide module convenors / organisers and departments with feedback and evidence on what works, enabling good practice to be shared across departments and faculties. 5. It should be noted that the purpose of module evaluation is to gather evidence to inform academic judgements about all aspects of the design, delivery and assessment of modules, but that the College reserves the right to use its academic judgement to decide when to implement any suggestion resulting from module evaluation. The College regards the development of the curriculum, teaching methods, learning resources and assessment of modules and their parent programmes as academic judgements the Office of the Adjudicator for Higher Education defines academic judgement as judgements where only the opinion of an academic expert will suffice. There will be occasions where module convenors / organisers consider individual or collective feedback to be misguided, and that implementation of given suggestions would be likely to compromise the standards or quality of the module. In such circumstances it is good practice, particularly where feedback or suggestions have been provided from more than one source, to provide a written response explaining why the suggestions have not been followed. The College should treat all feedback seriously, but this should not be misinterpreted as a commitment to implement any or all suggestions for amendment to existing practice. 6. It should also be noted that module evaluation feedback is most helpful when it can be considered as representative of as wide a group of students as possible, and that its robustness is compromised by low response rates. Every effort should be made to ensure that evaluations are completed by as high a number of students as possible.

4 7. The collection of module feedback is frequently an activity that occurs at the end of a module. Changes in response to feedback will therefore largely benefit future cohorts of students. Module organisers are encouraged to develop mechanisms that enable current students to feedback comments such that changes can be made to benefit them directly. Such mechanisms would complement the retrospective collection of feedback Relationship with QA processes and surveys 8. The primary intersection point between module evaluation and other College QA process is via annual monitoring. Annual monitoring is the opportunity for programme leads to reflect on the design and delivery of their programmes, utilising various data and evidence. Module evaluation should form part of the evidence used to gauge the success of programmes in meeting their aims and objectives, and a systematic process should be established for each programme for the analysis of module evaluation responses. In particular, module leads should consider module evaluation feedback in conjunction with data on assessment results and progression for the module. 9. Programme / Quinquennial review also has an important relationship with module evaluation; the focus of these reviews, which take place every five / six years will be more about the effective operation of module evaluation processes, rather than detailed scrutiny of evaluations themselves. 10. The College operates a number of different processes for obtaining the views of students, and for engaging students in quality assurance and enhancement of its programmes and modules. These include Staff-Student Liaison Committees, where programme representatives will routinely raise issues of interest to the student body for discussion. It is good practice for summary reports of module evaluation to be discussed with these Committees, where the timing of meetings permits this. (see Module Evaluation Reporting and Follow Up ). 11. The College also has an established complaints policy, and students should be informed that, should they wish to make a formal complaint about any aspect of the delivery of their programme, then this should be done in accordance with the complaints policy and not via their module evaluation. The module evaluation process should be anonymous and may also take several months to complete, which would prevent timely intervention relating to a complaint. 12. Other key evaluations undertaken by and for the College include the NSS, King s Student Survey and PTES, and care should be taken as far as possible to avoid conflicting messages and survey fatigue for students being asked to complete these surveys. Module Evaluation Design 13. Module evaluation is carried out by questionnaire, either online or in hard copy. Questionnaires will normally be scale questions, asking students to rate their level of satisfaction in each area on a scale of 5 (very high / good) to 1 (very low / poor). 14. The Module Evaluation regulations state that the following questions should form part of all module evaluations: Student Engagement (i) The module was intellectually stimulating; Organisation and Resources (ii) The module was well organised;

5 Assessment and Feedback (iii) Feedback on my work has helped me clarify things I did not understand (where applicable) (iv) The criteria used in marking were made clear in advance (where applicable); Teaching Quality (v) The teacher / lecturer(s) was good at explaining things; (vi) The teacher / lecturer(s) taught at an appropriate pace; Module Quality (vii) Overall I am satisfied with the quality of the module. 15. The regulations require that these questions be asked for all modules, but the format of the questionnaire is at the discretion of the department / faculty. In particular, where modules are team-taught programmes may choose to provide questionnaires relating to each teacher, or alternatively relate the questions to the team, rather than individuals. Faculties are encouraged, where possible, to facilitate the use of module evaluation in providing evidence to support promotion applications on the basis of excellent teaching. 16. Other possible questions that could be included are attached as Appendix A; questions can be added to this question bank at any time. Module Evaluation Delivery 17. Module evaluation can be delivered in a number of different ways, each of which has advantages and drawbacks. In considering the delivery method, programmes should consider: How best to maximise response rates. It is important to have relatively high response rates to ensure that firm conclusions can be drawn. Module convenors / organisers should aim for at least 25% of students completing the module to complete the evaluation. Where a response rate of 25% is not achieved then results should be treated with caution and consideration should be given in subsequent years on positive steps that can be taken to increase response rates; The resource burden of the analysis of results, taking into account realistic actions that could follow the evaluation and the timescale in which analysis must be performed; The College s commitment to technology-enhanced learning and innovative solutions that enhance quality and efficiency. 18. Possible methods for delivery include: (i) (ii) Paper-based delivery, with students asked to complete paper questionnaires. This will normally be the method that ensures the highest response rates, as time can be taken during lectures or seminars to ask students to complete the survey and hand-in. However, analysis of these questionnaires is usually time consuming, and may require an individual to input questionnaire responses into a spreadsheet or similar; On line questionnaires (often delivered by web-based software such as Bristol Online Surveys or Survey Monkey); these often have functions attached which will deliver analysis of quantitative responses which can cut down on analysis time, but response rates are often very low as students are normally requested to complete the surveys in their own time, unless appropriate steps are taken to facilitate and monitor responses (including regular reviews of response rates, frequent requests

6 to students to complete surveys from senior members of staff etc). These can also be moderately time-consuming to set up, and some online survey companies will charge for their services. Recommended websites / companies are: Bristol Online Surveys (contact Paula Webster, paula.webster@kcl.ac.uk) Survey Monkey (iii) (iv) (v) College websites including KEATS and Sharepoint can also facilitate module evaluation. Quizzes can be developed on KEATS, and it is possible to make these anonymous (so that reports are produced from the quizzes that do not identify respondents by name). A mixed approach using paper-based questionnaires with Optical Character Recognition software to assist with quantitative analysis. IT Services may be able to assist with the development of this type of approach, but will need sufficient time to develop; please contact them on 8888@kcl.ac.uk to discuss. Electronic voting systems. These are managed at School level and often use clickers which enable students to vote on a short set of questions in real time (for example the Florence Nightingale Faculty of Nursing & Midwifery and Dickson Poon School of Law use these sorts of systems). Please contact the Learning Technology team for your school for further information 19. Module convenors / organisers should consider ways of maximizing response rates, particularly when developing evaluations that students will complete outside of contact time. Analysis of previous module evaluation responses has shown that low response rates are likely to lead to unrepresentatively low evaluations. It is likely that extra effort is needed to ensure high response rates for large modules; analysis shows that larger student groups tend to have lower response rates than smaller groups. A target of 25% / 10 respondents (which ever is the higher) should be sufficient to draw reasonably safe conclusions from evaluations. Methods such as the enabling of smart phone access is good practice; it may help to discuss how this can be achieved with IT Services. It is anticipated that a pilot project adopting this approach will be taken forward during 2014/ In addition to questionnaires there are a variety of methods in which to gauge student feedback. These can include real-time voting (as used by various departments and Faculties including Informatics and the Nursing & Midwifery), where students are supplied with clickers and asked to respond to questions which can range from questions on the curriculum / assessment to logistical questions about the lecture environment. IT Services can provide further support to set such systems up, and similar systems can be supported by external websites such as Powervote and Socrativ. Module Evaluation Analysis Support for development of analytical methods for module evaluation, including identification of areas for improvement, use of qualitative evaluation, identifying and sharing good practice and escalation methods; 21. The methodology for the analysis of module evaluations will depend upon the methodology used for conducting the questionnaire, and in particular whether online methods are used to collect results, which will usually enable speedy analysis of quantitative data via the software package used to conduct the survey. 22. The National Student Survey identifies any student who indicates that they are not satisfied or very satisfied with their overall experience (ie who grades the overall

7 satisfaction question at 3 or below) as not satisfied when publishing NSS figures. It is therefore recommended that programmes establish an average rating of 3.5 or higher as a broad target for satisfaction questions, with an expectation of follow up action to be taken by the module convenors / organisers to address issues where the satisfaction rating does not reach this score (although see paragraph 23). 23. It is important, when analysing data, to establish norms for individual questions, taking contextual information into account. For instance it is normal for students studying quantitative subjects to report lower satisfaction ratings for those modules than for other modules. Where possible programme teams should use time series data for individual modules and questions to assess where evaluation grades are significantly higher, or lower, than might otherwise be expected. Follow up actions may then be focused upon significant outliers. 24. Sharing of good practice is an important part of module evaluation and equal weight should be given to identifying good and insufficient practice in the analysis of module evaluation. 25. Qualitative feedback is important in analysing issues highlighted by quantitative data, and should be collated and provided to the module convenors / organisers in a format that enables them to analyse it against the questions that the feedback informs. Convenors / organisers should not attempt to answer each point made in qualitative feedback; occasionally critical feedback will be received which will contradict the overall satisfaction ratings. 26. It is good practice to involve students in the analysis of module evaluations. This may be by discussing reports at Staff-Student Liaison Committees prior to consideration at Departmental Education Committees, or establishment of focus groups to discuss evaluations. 27. Where serious allegations are made in module evaluation then they should be referred to the appropriate Head of Department; it should be noted however that a formal complaint cannot be considered by the College unless the identity of the complainant is clear. Module Evaluation Reporting and Follow Up 28. Once surveys have been analysed reports should be reported and appropriate follow-up actions taken. Departmental Teaching Committees should establish a formal reporting and follow up process for their module evaluation, which should be approved by the relevant Faculty Education Committee. 29. Module convenors / organisers should review the qualitative and quantitative data for their modules and consider any changes that they wish to put in place for the following academic year. 30. All analyses should be made available to Heads of Department (or their delegated nominee responsible for teaching quality). Analyses should be available upon request to Deans of Faculty, Faculty Teaching leads, the Sectoral Vice-Principal and Principal, as well as to internal and external reviewers (for instance panel members for Programme Review / Quinquennial Review, or QAA Higher Education Review). 31. The process approved by the Department must include a process for feeding outcomes back to students, either via Staff-Student Liaison Committees or other appropriate fora. Use of innovative technologies to communicate outcomes is encouraged, but care should be taken to ensure an audit trail is available to demonstrate that students have a reasonable opportunity to see the outcomes of module evaluation.

8 32. Module Evaluation should feed into the College s annual monitoring and programme review processes. Annual monitoring reports should include a commentary on module evaluation, with details of actions to be taken resulting wholly or in part from feedback received via module evaluation. Programme reviews may take a more detailed look at module evaluations, and examples of evaluations together with reports and details of follow up actions should be made available to panels. 33. Where analysis reveals comments relating to services provided outside of the relevant department (such as library services or IT) then this aspect should be fed back to the appropriate office for information. 34. Academic staff may use module evaluations in support of promotion applications or other processes to evidence high quality teaching. Module evaluation data may also be used to inform professional development or peer review mechanisms College Education Committee July 2014

9 Appendix One Module Evaluation Framework Question Bank This Question Bank includes an inexhaustive list of questions that may be included in module evaluation questionnaires, at the discretion of the Faculty Education Committee (or its delegated committee). It is intended to assist module convenors and programme leads in designing module evaluation questionnaires that meet their local needs; questions from the question bank must supplement the module evaluation questions required for each module evaluation under the King s academic regulations. The Question Bank is intended to be a dynamic document; additional suggested questions may be added by contacting the Quality & Academic Support Unit, trevor.pearce@kcl.ac.uk. Lecture / Seminar Leader The lecturer or seminar leader was well prepared; The lecturer or seminar leader gave interesting classes; The lecturer or seminar leader gave informative classes; The lecturer or seminar leader explained the subject clearly; The lecturer or seminar leader was effective in leading the class; The lecturer or seminar leader was receptive to students questions; The lecturer or seminar leader stimulated interest in the subject; The lecturer or seminar leader stimulated discussion on the subject; The lecturer or seminar leader stimulated me to think and learn; The lecturer or seminar leader had a good rapport with the class; The lecturer or seminar leader was available to answer questions in office hours; The lecturer or seminar leader was approachable; The lecturer or seminar leader was keen to help; The standard of teaching on this module is high; The teacher / lecturer stimulates my interest in the subject; Module design The learning outcomes were clear and relevant; The module design was clear and coherent; The learning and teaching methods used were effective; The workload for this module was reasonable; The requirements for me to complete this module satisfactorily were clear; Module Information This module corresponded to the description provided, in terms of stated aims and objectives This module had helpful reading lists and handouts This module had helpful material provided electronically Assessment & Feedback For this module you received constructive comments on your contributions to in-class discussion; For this module you received feedback which enables you to see how you can improve; For this module you were proactive in seeking further guidance / feedback on assessment; For this module you understand what will be required of you in the assessment; For this module you understand the marking criteria used to mark your work; The assessment for this module allowed me to demonstrate what I understand;

10 Appendix One Learning Resources For this module the library resources were adequate; For this module key texts were readily available in the library; For this module teaching aids were adequate (for example audio-visual equipment, slide projectors, whiteboards); The learning resources (eg reading lists, online resources) were valuable for my understanding of the module s content For this module the learning material provided has been helpful (e.g. reading lists); Technology Enhanced Learning For this module the e-learning platform met your learning needs. The online space for the module was well-organized The online space for the module was easy to navigate For this module the online learning content provided has been helpful. This module contained useful information about being an online learner The module offered good opportunities for interaction and communication (to your peers and the tutor); Regular feedback about your performance was provided in a timely manner throughout the module via the online space for the module. General Questions If this was an optional module, please say what motivated you to choose it; What were the positive features of this module? Were there any negative features of this module? Did you raise any concerns during the module and did you feel these were addressed? Please estimate the amount of preparation you did for this module on a weekly basis (fewer than 3 hours, 3-5 hours, more than 5 hours) Please estimate when you began using library resources for your marked coursework (fewer than two weeks before the deadline, 2-5 weeks before the deadline / more than 5 weeks before the deadline) Please estimate your commitment to this module (Strong / Average / The minimum necessary) Please give us an estimate of your attendance on the module. If less than 75% please indicate reasons (100% - 75% / less than 75%) The module was well organised; The module ran smoothly; Overall my expectations of this module were fulfilled; Clinical / Other Placements Practical classes complemented the content of module Practical classes were well organised Practical classes provided good opportunity to interact with staff Student engagement This module challenged me to do my best work This module contributed to me becoming an independent learner This module encouraged me to take responsibility for my learning This module encouraged me to feel part of a community committed to learning This module encouraged me to make active decisions about how I study. This module stretched me intellectually

11 This module met my expectations Appendix One