Assessment. Leeds School of Medicine. Date 18 October Leeds School of Medicine (the School), and Bradford Royal Infirmary.

Transcription

1 Check Leeds School of Medicine Date 18 October 2012 Locations Visited Programme Team Leader Visitors GMC staff Purpose of the check Summary Leeds School of Medicine (the School), and Bradford Royal Infirmary. MB ChB Professor Sean Hilton Dr Oona Stannard Abdul Ahad Wahid Samara Zinzan Jennifer Barron The School was last visited by the GMC in Since then Tomorrow s Doctors (2009) has been introduced. Every medical school in the UK has been asked to self-assess its compliance with Tomorrow's Doctors (2009). The GMC has identified assessment, quality management of clinical placements and the introduction of student assistantships as key themes of challenge, therefore we focused the check on these three main areas to test the accuracy of the School s self reporting to the GMC. The School appears to be doing well overall. Assessment The school is compliant with our standards except in respect of blueprinting assessment to the outcomes in Tomorrow s Doctors and providing feedback to students. Sampling did however indicate that the outcomes were bring assessed and the assessment reflects the content of the curriculum, which is blueprinted. Providing feedback to students is an area many schools continue to struggle with.

2 Quality Management The school is compliant with our standards and quality management of clinical placements is working well. It has good systems for identifying problems, acting on them and feeding that back to the students. The School is working to involve students more and has recently recruited students to take part in its quality management visits Student Assistantship The school is meeting our standards in terms of the content and purpose of the student assistantships however staff and students did not all understand the intent of the assistantship. Students are satisfied with the teaching they receive at Bradford Royal Infirmary. There needs to be greater clarity around the School s plans for student assistantships as we found some confusion amongst students and staff with regards to their timing and purpose. Our findings are in line with those expected from the School s self assessed reporting to the GMC and the School was aware of the areas we highlighted as both good and requiring action to ensure the standards in Tomorrow s Doctors 2009 are met. Concerns Good practice No serious concerns were found. 1. The School has an innovative approach to assistantships. It is developing a super-assistantship and conducting its own research into the student experience of assistantships. For those students remaining in the Yorkshire & Humber Deanery this will take place in their first foundation post. (TD84) 2. At Bradford Royal Infirmary the students have a varied experience of working with Foundation Year 1 doctors (F1s) on placements, and we heard how the students work with an F1 for a week in a surgical block. Although this is not the scheduled student assistantship it is an additional opportunity for students to act as an assistant to an F1. Students were very complimentary about this model and its benefit for preparing them for foundation training. (TD84) 3. The students we met with were very impressed by the designated Clinical Educators and the teaching they provide. (TD122) 4. The School s responsiveness to student evaluation is good. The problems F1 doctors related back to their experience as medical students were not reported by the Year 5 students we met. For example, Year 5 students had been allocated a new timetabled

3 session for Tuesday afternoons in response to previous student evaluation that there was nothing to do during this time. (TD41) 5. The School has started to post student comments onto a shared website after each placement, although organisers can see the student evaluation relating only to their organisation. (TD41) 6. The overall scheme of assessment is comprehensive and appears to be working well. The students we met were well prepared and understood what was required of them in each assessment and to progress at each stage of the programme. The students receive guidance on their assessments in multiple ways including lecture weeks, handbooks, student selected components (SSC) mark schemes, assessment study guide by year, and foundation doctors talking about their experience. (TD123) 7. The School provides handbooks for students but has also developed orientation videos for each placement in which they can find information such as how to get to their placement, where to park, and who they will meet when they arrive for their placement. (TD123) Requirements 1. Feedback for all students following written and practical assessments must be improved. Students are unable to compare their progress with students in their cohort as a result of the limited feedback given on their written assessments. Students also spoke of being unclear as to their areas of strength and relative weakness and knowing where they most need to improve. (TD85) 2. The communication around student assistantships to the students, and to some extent staff, needs to be clearer. Year 5 students were not familiar with the concept or timing of the student assistantship. (TD123) 3. The School has blueprinted the curriculum to Tomorrow s Doctors outcomes and blueprint assessments to the curriculum. The School must add the outcomes to the assessment blueprint. (TD112) Recommendations 1. The School should consider introducing a way of discussing comments and feedback through the personal tutor system. Year 5 students and F1 doctors were unaware of the significance and weight of the end of attachment assessments submitted through the sharepoint system. They found that the grades or comments were not followed up in the medical school unless they had failed to meet requirements. Therefore the students questioned whether it would make any difference if they scored well above average, or

4 simply met requirements. (TD122) Additional Findings 1. The students in Year 4 complete five types of workplace based assessments (WPBA) in each attachment using iphones provided by the School. Whoever examines the student enters the results into the iphone. Some educational supervisors and clinical educators were not confident in using the iphone, with the risk that these individuals would not be involved in WPBAs. However we heard from the School that it recognises this and will be running appropriate training sessions. 2. We heard from the senior management team at the School that the introduction of the iphone has reduced a lot of paperwork. There seemed still to be some duplication of work as students needed some paperwork signed off as well as information being entered via iphone. However, we heard from the School that more is being brought in to the app but this is an iterative process. 3. Students reported that the iphone initiative was particularly useful for quick referencing of conditions and medication names on the wards. Some of the F1s also suggested the addition of other apps such as the British National Formulary (BNF) and National Institute for Health and Clinical Excellence (NICE) guidelines. 4. An issue arose previously with Year 4 written papers where errors with the questions were identified by students on the day of the exam. The School explored the issue and analysed whether the performance in those questions affected students scores. Due to the small number of items with errors, including or removing the items did not make a difference to who passed or failed. Nevertheless, it was decided that the questions should be taken out of the final results. The process for coming to this decision was shared with the students. The School has improved its quality control of exam papers and anticipates that errors of this kind are less likely to occur. 5. Student professionalism is assessed at the end of clinical placements with a professional attribute form completed by the educational supervisor, who must be a consultant. The students can fail these, so professionalism and its assessment is given appropriate weight. The amount of detail provided varies between educational supervisors. The form asks the educational supervisor to assess and provide feedback on communication skills, multi disciplinary team working, attendance, and attitude towards staff and patients. The senior educational team at Bradford Royal Infirmary informed us they would usually hear about any

5 professionalism issues from the School prior to the student starting their placement. 6. The local education providers receive student evaluation of clinical placements from the School which we heard was useful and received quickly. The senior educational team at Bradford Royal Infirmary informed us they had made changes to placements as a result of these evaluations. For example, Bradford Royal Infirmary was able to recognise clinical teachers strengths and weaknesses using student evaluation and matched year groups to the clinical teachers that would best meet the students needs. 7. Students found that assessments were generally fair and representative of the curriculum. Both written examinations and Objective Structure Clinical Examination (OSCE) stations tested material within the core curriculum. 8. The School has developed a sequential OSCE which runs three weeks after the final Year 5 OSCE for those who did not meet the required threshold. This means students do not have to wait until November to re-sit their exam and avoid delaying the beginning of F1. 9. The clinical teachers and educational supervisors we met informed us that a minimum of 25 characters has to be entered in to the iphone app for feedback on workplace based assessments. It appeared this was as a result of student evaluation that the feedback previously might be as short as one word. 10. Feedback to students on clinical placements appeared to be working well for the Year 3 and 4 students we met. The students were observed on examinations and history-taking and then received both written and verbal feedback. However, the Year 3 and 4 students felt that the feedback from the mini Clinical Evaluation Exercise (mini-cex) was more of a tick box exercise and not very well regulated. The Year 5 students felt that the feedback varied depending on the educational supervisor. 11. Feedback on written assessments was given in a list containing the student numbers and the grade they had received. A percentage mark is not given, and students are unable to see where they are within the grade they have received, nor are they able to compare their results with peers. 12. Students may give evaluation of their placements through the School s sharepoint system by completing a form online or on the iphone provided for Year 4 and 5 students. The students could see

6 where issues raised by previous students had been resolved. 13. There are good internal quality control systems in place at Bradford Royal Infirmary before issues are escalated to the School for quality management. Bradford Royal Infirmary has a year rep system where a student from each year attends meetings and feeds back to the students in their year by or on Facebook. 14. A new curriculum is in the process of introduction by the School. The current Year 3 students at Bradford Royal Infirmary are the first cohort of this new curriculum and are being asked to fill in a lot of evaluation forms as a consequence. They are aware this is for quality assessment purposes. 15. Engagement between staff at the Bradford Royal Infirmary and the School is open and clear. The School is working with all local education providers to address any inconsistencies between sites. We heard how local education providers had worked together to remedy previous inconsistency in in-course assessments. 16. Bradford Teaching Hospitals NHS Trust has an F1 mentor system which links Year 5 students to an F1 and supplements the student assistantship, helping the students become more familiar with the F1 role. 17. Bradford Royal Infirmary is planning to pilot a prescribing test in which the Year 5 students will have a yellow sticker for the week they are prescribing to highlight to the nursing staff the need for sign off by a doctor % of the Consultants at Bradford Royal Infirmary are paid a teaching supplement, but it is not in their job plan. They are paid for delivery as well as preparation time of teaching. Consultants are appraised on their educational role despite it not being in their job plan. There was good Service Increment for Teaching (SIFT) accountability for clinical placements. 19. In the 2011 Medical School Annual Return completed by the School, they self assessed full compliance with Domain 2: Quality Assurance, review and evaluation in Tomorrow s Doctors (2009) for the first time. We heard from the School that it goes out to local education providers and encourages them to submit one form. Students in Years 1 to 3 use online data to submit evaluations, and Years 4 and 5 students use an app on their iphone. This information goes on to the School s sharepoint system. The School thinks processes are at their best now and the sharepoint system is

7 helping it to enable this. Monitoring The School will need to report on what actions it is taking regarding the requirements listed above in the 2013 Medical Schools Annual Return. Response to findings Good practice Professor David Cottrell, Dean of Medicine The School welcomes the very positive summary report in respect of the GMC check visit to the Leeds MBChB programme, and its particular focus on Assessment, Quality Management and Assistantship. We are very pleased to see so much excellent practice recognised within the report, both in the summary and the additional findings section. We have been pleased to share this with our students, staff and NHS partners. We were very pleased to see the report acknowledge the quality of clinical teaching and positive student views about the LEP reviewed in the visit (Bradford Teaching Hospitals). The comprehensive nature of our assessment provision and dedicated quality loop continues to be developed as one of the Leeds Institute of Medical Education s particular research strengths, translated back into good practice in Learning & Teaching. The highlighting of innovative work around student assistantships is welcomed, as are the wider comments about support for students around assessment, clinical placement and placement evaluation. Requirements The clarity of the requirements from the GMC check visit is appreciated, and provides an additional lever to accelerate our development plans. We recognise that feedback is a challenging area for all HEIs, but work continues with our new MBChB curriculum with the provision of feedback rich, non graded passes for in course assessments. Graded, high stakes assessments will benefit from the roll out of new, domain based feedback mechanisms in OSCE and written tests from Summer This will be supported by the final stages of implementation of a new format of student support that will allow more monitoring of progression at individual student levels As part of curriculum change, clearer communication will allow the recognition of our post finals assistantship period, in addition to the assistantship ethos that is being inculcated across the entirety of Year 5 of the Leeds MBChB. This has previously been described/regarded as workplace shadowing, and the introduction of the national preemployment shadowing week will allow us to more clearly delineate

8 assistantship for staff and students. The growth of our research led super assistantships is already widely and highly regarded by students. We welcome the encouragement to complete the loop by tagging TD domains into our assessment blueprints, and are exploring ways to do this in 2013 assessments. We feel this will further strengthen Leeds programmatic assessment. Recommendations As part of the final implementation of our new student support system, all final year students will receive a specific appraisal in December 2012-January 2013 to review progress, overview final year in-course assessments and learning goals and careers/next phase of Foundation applications. This mirrors the approach taken in years 1-4 of the course (for which the new system is fully operational), which has received widespread accolade by the student body.