3 June 2014 Education and Training Advisory Board 5 To consider Education surveys plan 2014-17 Issue 1 With the success of the National Training (NTS) there have been calls to expand the use of surveys as an engagement tool to better understand the experience of other cohorts of doctors and of medical students. 2 We have proposed a programme of education surveys work linked to key policy milestones to inform the Corporate Strategy for 2014-17 Recommendations 3 The Education and Training Board is asked to provide advice on the surveys work programme over 2014-17, taking into account the following: a Considering the proposed programme of Education surveys work at Annex A: i ii Is the sequencing right? Are there any gaps in cohorts of doctors that we should be engaging with through surveys? b Considering how the GMC should engage with Medical Students: i ii What might be the benefit of a GMC survey of medical students? What would the survey cover? iii Which students should we be engaging with eg all years, only clinical years, only final year? c What engagement tools might help us understand the diversity and concerns of staff and associate grade doctors?
Education surveys plan 2014-17 Issue 4 This paper outlines a four-year programme of education surveys work which is summarised at Annex A. The programme has two key aims: a To ensure the NTS (which is a core element of our Quality Improvement Framework) remains relevant. b To explore whether learning from the NTS can be transferred and whether surveys (not necessarily the same technical tool as the NTS) could provide valuable insights into the experience of other cohorts of doctors and medical students. 5 The NTS is an important engagement tool that helps us understand the views of doctors in training and take action based on evidence of their concerns. There is an opportunity for us to explore whether surveys (or opinion polls) could similarly supplement our face to face engagement, enabling us to hear the views of a broader range of doctors and future doctors and for them to contribute to the GMC in a systematic way. This kind of approach which is about engagement and communication - would be different in kind to the NTS which is a function of our statutory duty to maintain standards of medical education. We would need to keep these purposes distinct, though they are clearly related. 6 We are aware that any collection of information places a burden to some extent on the person or organisation giving the information. The survey work programme would therefore need to be developed in a measured way and each possible new survey would need to be considered in terms of the impact on others of both supporting survey delivery and responding to survey results. The National Trainee 7 The NTS is an important quality assurance tool that measures the extent to which doctors in training consider our training standards are being met in their case. As we begin to implement the recommendations of the review of our Quality Assurance (QA) function and begin to review our education standards we will also need fundamentally to review the NTS to ensure alignment against our new standards as they are developed. 8 The survey plays a (perhaps sometimes overly) central role in postgraduate deans local quality management systems. It is used locally to: allocate finite quality team resources, monitor sites the dean s team does not have the resources to visit during that year, and monitor the effectiveness of action plans where visits are undertaken. 2
9 We have invested in improving the technology and systems underlying the survey to provide results to the field faster and this has been successful. There may however be more that we could do to make the survey reports simpler and more accessible to a wider range of audiences. The Trainer 10 The trainer survey inherited from the Postgraduate Medical Education and Training Board (PMETB) had a poor response rate (in 2011 it was 44.6%). There was no clear definition of who we regarded as a trainer and deans did not always hold lists of non-general Practice (GP) trainers because there was no requirement to do so. 11 We decided to pause the trainer survey in 2012 so we could focus on the of the NTS and allow time to develop a framework for recognising and approving trainers across all specialties, including GP programmes. With that now in place, by this summer deans should hold lists of recognised trainers across all specialties. 12 A pilot survey of trainers will run in: the three London Local Education and Training Boards, Health Education West Midlands and the Wales Deanery during October-November this year. 13 These pilots will help us test our approach to delivering and reporting on the survey as well as the questions which we are developing together with the pilot sites and other groups such as the National Association of Clinical Tutors. The question areas include appraisal and time for training and opportunities as a trainer. Reports will be provided for each LETB/Deanery and an overall evaluation report on the pilot including recommendations for implementation early in 2015. ing medical students 14 In 2012 the NHS Future Forum recommended that we should consider surveying medical students. In 2013 the Francis Report on the Mid-Staffordshire NHS Foundation Trust included at recommendation 159 that s of medical students and trainees should be developed to optimise them as a source of feedback of perceptions of the standards of care provided to patients and enable them to raise any concerns they had about the clinical environments in which they are learning. 15 We currently use evaluative feedback from students from a number of different sources in our quality assurance processes: a The National Student run by Ipsos Mori. We use it for horizon scanning and to identify potential areas of investigation during a visit, but it is light on specific data about medical courses. 3
b Prior to a regional review we conduct a structured online survey of medical students to help the team ensure they are investigating the areas of most concern. However this is voluntary and we do not know how representative the responses are. c During the inspection the teams of Associates audit student evaluations and the medical school s response over a period of years. d We have also supported the Medical Schools Council to build on the work undertaken in Scotland to ask students to evaluate their clinical placements. The methodology is different to the Trainee in that we do not ask students directly for their feedback but collect aggregated results according to pre-set criteria from the medical schools. Approximately half of the medical schools participate in this project. 16 Given the range of different survey interventions currently used in undergraduate quality assurance and the call from the Francis Report for a further survey enabling students to raise patient safety concerns, there is a clear opportunity and support for us to review our approach to ensure it is proportionate and effective with clear rationale regarding the use of the information. 17 We propose that we begin to review our approach to medical student surveys from 2015. This will allow us to respond to the findings emerging from the review of Tomorrow s Doctors 2009 and to take into account progress on the case for a national exam and moving full registration. ing recently qualified consultants and GPs 18 During GMC roundtable discussions, recently qualified GPs and consultants told us that a better understanding of the management responsibilities of the consultant/gp role and a clearer understanding of the NHS structures would have helped make the transition to their new roles easier. 19 A survey of recently qualified consultants and GPs may help us understand whether training programmes in particular parts of the UK better prepare doctors in training for their next career step. It may also help us understand whether some specialty programmes or medical royal colleges/faculties better prepare doctors for their next step. 20 In considering the feasibility and desirability of such a survey, we would first need to consider whether we were best placed to undertake it. Some colleges already run such a survey, though others would not have the resources, and it is not possible to do any benchmarking across specialty programmes. Specialist and Associate Specialist grade doctors 4
21 Our 2010-13 Education Strategy also undertook to consider whether a survey could be used to engage with Specialist and Associate Specialist (SAS) grade doctors about their experience. Our roundtable discussions with SAS doctors indicate that while they generally have an increased focus on meeting service requirements, they undertake varied roles depending on the nature and team arrangements of the hospital. 22 A structured survey questionnaire may therefore not be able to capture the extent of variation in these doctors working practices and experiences. Qualitative research methodologies such as focus groups would provide more opportunity to explore new information. Alternatively, blind survey methods with faster results such as online polls could enable the outcomes of focus groups to be tested more widely within the SAS population without the need for significant information systems or complex statistical modelling. 23 We are therefore proposing to explore whether a surveys work programme would be useful for this cohort of doctors in 2016-17. Over the next few years we will consider how we can use face to face engagement tools such as roundtable and focus groups combined with social media and online tools to better understand their experiences and training and needs. s as a broader engagement tool 24 We have discussed in this paper groups of doctors and future doctors for whom we have a need to engage more widely on their training and but clearly it is important for us to be listening to all doctors on our register about the issues that matter most to them and to receive feedback on our work. 25 It may be that we can make more use of surveys or opinion polls as a way of listening to and supporting the profession; making sure that our agenda reflects the things that will make the most difference to patients and doctors, but recognising that this is different from quality assurance of education. Resources 26 If four new surveys were implemented (trainer, student, SAS, new GP/consultant), the scale of the surveys programme resources would significantly increase for the GMC. The resource impact on medical schools, LETBs/deaneries, the NHS and doctors completing the survey is equally a key consideration in developing the use of surveys. 27 The trainer survey has been designed to minimise the burden on deans teams however there is likely to be some impact as LETBs/deaneries promote the survey and encourage trainers to participate. The pilots this autumn will help us test the resources required locally against the benefits of the information yielded. 5
Supporting information How this issue relates to the corporate strategy and business plan 28 Strategic aim 4: Work more closely with doctors, medical students and patients on the frontline of care. Developing a stronger data analysis and insight capability to better understand the profession. 29 The NTS is the main way we communicate with all doctors in GMC-approved training programmes and it is vital in understanding the experience of junior doctors at this early point in their career. 30 The scale of the Education surveys work programme will affect the capacity and opportunity of teams across the GMC. It is therefore important to understand the implications of a potential work programme before committing to further work within our new Corporate Strategy to cover 2014-17. How the issues differ across the four UK countries 31 Our standards for education and Quality Improvement Framework apply consistently across the UK and doctors training in each of the four countries participate in the same way. The online survey reporting tool allows us to investigate results by geographic area and deanery/local Education and Training Board thereby helping us to understand if there are particular concerns or good practice in each of the four countries. What equality and diversity considerations relate to this issue 32 The survey asks questions about ethnicity and disability, helping us to progress our policy work in these areas and to understand whether different groups of doctors experience training differently. We have also shared non-identifiable survey data to support research into whether there are patterns in doctors experience of training that relate to age and gender. 33 We would ask the same or similar demographic questions for each new cohort of doctors we engage with using surveys so that we can better understand whether the experiences they share with us are affected by equality and diversity characteristics. If you have any questions about this paper please contact: Kirsty White, Head of Planning, Research and Development, kwhite@gmc-uk.org, 020 7189 5308 6
5 Education surveys plan 2014-17 Summary of proposed work programme Annex A Summary of proposed work programme and key milestones 1. This table provides an overview of proposed project phases for each proposed survey over the lifetime of the GMC s corporate strategy. 2. To demonstrate how this programme of work might expand if all projects were successful we have assumed that: a. We will continue to deliver the Trainee. b. We will honour our commitment to pilot a Trainer and this will result in an annual cycle of surveys. c. Each feasibility study for a new survey will conclude that the survey should proceed. A1
Trainee Trainer Medical Students Recently qualified consultants and GPs Policy milestones Policy milestones Policy milestones Policy milestones 2014 2015 2016 2017 *Revised Education Standards consultation begins *Existing survey continues with incremental refinement. * Postgraduate Deans and Medical Schools hold a list of local trainers in the four roles identified by the GMC framework *Scoping work begins, including an initial pilot of a selection of the trainer population *Timing of full registration is considered *Tomorrow's Doctors standards are reviewed *Government response to Shape of Training *New approach to standards is confirmed *Existing survey continues with incremental refinement* *Fundamental review of purpose and format of survey; what is needed to test the new standards? Publication of GP Trainers *Refinement and extension of pilot *The case for a national exam is considered *Consider the case for a Student, may involve pilot *Identifies current state of gaps in training as perceived by new consultants/gps Date standards come into effect to be confirmed *Existing survey continues with incremental refinement *Piloting of new survey proposals *Postgraduate Deans hold the list of trainers, all of whom have been recognised as meeting the criteria within the framework * becomes business as usual - perhaps surveying a different cohort of trainers each year * pilot /refinement *Revision based on pilot results and second phase of piloting or roll out begins depending on pilot results Publication of GPs and other trainers *Trainer continues as business as usual. Longitudinal functionality is added. *Student, depending on pilot * Pilot benchmarking survey * refined and continues perhaps surveying doctors from different programmes over different years A2
SAS Doctors 2014 2015 2016 2017 *Consider how we engage with SAS doctors using engagement Tools (Blogs, tailored communications and the link to CPD which may lead to a future survey) * Trial engagement tools and identify any segmentation requirements * Refine engagement tools * Pilot of broad benchmarking survey A3