Joint response from the UK Foundation Programme Office and the Medical Schools Council to the Independent Review of the SJT May 2013

Transcription

1 National Director Professor Derek Gallen FRCGP MMEd ILTM Deputy National Director Professor Stuart Carney MPH MRCPsych FAcadMEd Chair Professor Tony Weetman MD DSc FRCP FMedSci Executive Director Dr Katie Petty-Saphon Joint response from the UK Foundation Programme Office and the Medical Schools Council to the Independent Review of the SJT May 2013 The UK Foundation Programme Office (UKFPO) and the Medical Schools Council (MSC) deeply regret the scanning error, and the subsequent re-running of the allocation algorithm in February/ March this year. We welcome the Independent Review, and the speed at which Professor John McLachlan was able to undertake it, to enable all organisations involved in running the national application process to move forward. We do not underestimate the anxiety caused to applicants by the scanning errors, and we are truly grateful for the extra effort and hard work undertaken by all involved to enable resolution within nine working days. We especially thank medical school colleagues across the country who manually verified the answer sheets; staff at Konetic who re-developed FPAS to allow for re-running the algorithm; and staff at the Work Psychology Group who repeated the test equating and scoring of the SJT. We should especially like to thank students for their patience and understanding in those difficult circumstances. When the initial allocations were announced, we had been reassured with absolute confidence that the data were accurate. A single error was identified in the 24 hours following the allocation, and the Review commends the decision to communicate to all applicants as early on in the process as possible, and also to provide daily updates with the new information. The Review also confirms that it was the best resolution to involve the medical schools in the manual verification of every single answer sheets and scanned data files. Over the course of these days, four types of error, affecting a total of 353 applicants, were identified. Corrections were made, and the testequating between the different SJT papers was repeated, with the resultant changes to 1,531 SJT point scores. A total of 148 applicants were subsequently allocated to a different foundation school. In addition to the specific recommendations of the review, which we had already integrated into the process for next year, additional improvements will be incorporated into the FP 2014 application process following feedback from applicants, foundation school staff, medical school staff, as well as the companies involved in delivering the computer system and developing the SJT intellectual property. The FP/ AFP 2014 Applicant s Handbook, which sets out the application timeline and the detailed processes for the Foundation Programme and Academic Programme commencing in August 2014, will be published no later than 21 June The UKFPO and the MSC wish to reassure all involved that additional steps are in place ahead of FP 2014 to ensure that the situation does not happen again. We address each of the recommendations from the Independent Review below.

2 Recommendation 1.1: that the use of SJTs in selection for Foundation be continued while evidence is gathered as indicated below. The UKFPO and MSC are confident that the SJT and EPM remain the most appropriate methods for selection to the Foundation Programme HEE has confirmed that it is very supportive of SJTs. Recommendation 1.2: that evidence for the validity of SJTs in selection for Foundation context be gathered as a matter of high priority. Evaluation of the SJT for FP 2013 confirmed from previous research findings that the SJT is a face valid and content valid, reliable and robust measure of the person specification. From the outset of the SJT development work, plans have been in place to ensure that research and evaluation into the criterion-related validity (which may be concurrent or predictive) of the SJT is undertaken and that this in turn informs Foundation Programme recruitment. As part of the stakeholder engagement and research with the Improving Selection to the Foundation Programme project (please see a Project Group with stakeholder representatives including medical students, employers, foundation schools, medical schools, the UKFPO, the MSC and the four UK health departments, was established, and contributed considerable expertise to the evaluation of different selection methods. This group re-formed as an Advisory Group to the UKFPO Rules Group, named the ISFP Research, Consultation and Evaluation Group. Although the group does not hold a budget, it can bring forward ideas for research questions, and consider the viability of research proposals. Applicants to FP 2013 will take up their posts in August 2013, and it will take several years before there is a large enough evidence base to truly evaluate the predictive validity of the SJT and/or EPM. One of the research proposals recommended by the ISFP Research, Consultation and Evaluation Group looks at the incremental evidence during the FY1 year in order to see how applicants go on to perform on the job. Recommendation 1.3: that UKFPO and MSC explore with HEE, DH, GMC and other stakeholders the possibilities of longitudinal tracking of students and doctors through their subsequent careers. The UKFPO and the MSC have been exploring the establishment of a UK-wide Medical Selection Outcomes Research Database (MSORD) with HEE, the GMC and UKCAT amongst other stakeholders, for around 12 months prior to the SJT for FP Preliminary pilot work into the methodologies to link performance data from the SJT and EPM with other performance data is underway, while the governance and data protection arrangements are being finalised. The intention is for the database to be extended to include postgraduate performance measures for example selection into specialty training, as well as entry into medical school. This would provide an extraordinarily valuable tool in evaluating selection methods. Recommendation 1.4: that an independent psychometrician with relevant health care experience be appointed to the Rules group. The UKFPO will liaise with Professor John McLachlan (the author of the independent review report) to identify a suitable independent psychometrician, with the aim of having a member of Rules Group appointed by 25 July Recommendation 3.1: that the RULES group specifies unequivocally the nature of the process intended. If it is a selection for employment process, then the criteria by which candidates are deemed unappointable should be made explicit beforehand. The Rules Group agrees with this recommendation and will state clearly the criteria for withdrawing applicants from the process due to a low score in advance. The UKFPO will revise guidance for applicants accordingly. If the full details of the criteria are not known when the FP/AFP 2014 Applicant s Handbook is published in June,

3 the UKFPO will indicate clearly in the handbook when the details will be made available, and this will be before the first SJT date of 6 December Recommendation 3.2: that student members of the Rules Group be invited to contribute to all strategic discussions relating to the use of SJTs. BMA Medical Student Committee representatives are active members of the Rules Group. The group has agreed that the student representatives will only be excluded when individual SJT items or individual applicants are discussed. Recommendation 3.3: that further research is carried out into the nature of very low scores on the SJT. We agree with this recommendation, and this is a significant part of the work overseen by the ISFP Research, Consultation and Evaluation Group. Recommendation 3.4: that subsequent to the research described in Recommendation 3.3, the Rules Group consider if a remediation programme might be developed for failing candidates. Remediation for medical students is a matter for medical schools and is not within the remit of the Rules Group. Equally, it would not be possible to influence a remediation programme for failing applicants from non-uk medical schools. For the FP 2014 application process, it is intended that agreement from applicants will be sought for their scores to be shared with their medical schools. In future years, for the applicants who give this consent, the UKFPO will be able to provide medical schools with an analysis of scores to assist with the development of a remediation programme for students who score low on the SJT should they wish to introduce such a programme. Recommendation 3.5: that candidates who receive very low scores but are still deemed appointable, should be able to avail themselves of any remediation programmes available for failing candidates, with a view to addressing issues on commencing employment. As noted for recommendation 3.4, the issue of a remediation programme for students is one for medical schools to consider. More work is needed to understand how an undergraduate remediation programme could be shared with employing organisations after graduation. Recommendation 4.1: that the UKFPO should review sympathetically the cases of candidates affected in this first year of operation of the process, and that these candidates should be able to apply for vacancies that become available through the reserve process. The UKFPO is sympathetic to the plight of applicants withdrawn from the FP 2013 national process due to a low SJT score. However, the national application process is a competitive process for employment and all fully eligible applicants, regardless of their country of qualification, must be treated in the same way. The criteria for withdrawing applicants due a low SJT score were applied consistently across all applicants and this must remain the case. Applicants who were withdrawn from the FP 2013 national process due to a low SJT score may apply for any vacancies remaining at the end of the national process provided they meet the eligibility criteria for the vacancy. These vacancies are advertised by individual employing organisations and recruited locally; the UKFPO is not involved in this process.

4 Recommendation 6.1: that irrespective of whichever company is contracted to carry out printing and scanning in the future, the Medical Schools Council should brief them beforehand on the nature and purposes of the SJT and should remain in close liaison throughout the entire process, not just the end stages of reporting. Please see response to Recommendation 7.3 below. Recommendation 7.1: multi and missing mark procedures should be established as a matter of urgency by MSC with the provider companies for the next round of selection. Extensive enhanced and additional quality checks are to be used at the point of scanning, with MSC being directly involved in decisions taken, for example around ambiguous marks/rubbings out, the management of multi-marks and missing marks, and the physical checking of answer sheets to ensure absolute confidence in the accuracy of the scanned data. Recommendation 7.2: realistic timescales and deadlines should be agreed with the commercial providers, taking into account the experiences gathered this year. The timelines for scanning and verification of data have been extended, with advice from several scanning houses and medical school scanning teams, to ensure that all quality checks, including extensive manual checks of the original answer sheets against the scanned data file, are undertaken and that these in turn are audited. Recommendation 7.3: that irrespective of whichever company is contracted to carry out scanning in the future, the Medical Schools Council should brief them beforehand on the nature and purposes of the SJT and should remain in close liaison throughout the entire process. At all times, MSC will liaise directly with the scanning company chosen. Meetings have already taken place with the company used for FP 2013, and with other interested scanning houses, and the detailed brief around the high-stakes nature and purpose of the SJT is undoubtedly clear. MSC intends to work closely with FP2014 subcontractors over the coming months to optimise the entire process. Recommendation 7.4: that the scanning company be sent both attendance and absence lists to ease the task of checking candidate forms. Attendance and absence reports will be provided by the MSC for FP 2014, as they were for FP Recommendation 7.5: MSC should generate a separate.csv ( comma-separated values file storing tabular data as plain text) file for each individual exam date showing only the candidates taking the exam on that date, again to ease checking. The scanning of the original answer sheets will be undertaken in batches by medical school and date. We agree that separate.csv files should be generated by date for FP 2014, as they were for FP Indeed, every batch of answer sheets (for a given school and date) will be used to generate a single.csv file at the point of scanning. Recommendation 7.6: that adequate time is allowed, not just for scanning, but also for checking the results prior to allocation and informing the candidates of the outcomes. Quality checks were already in place for FP 2013 to ensure the accuracy of the data at all stages following scanning prior to informing applicants. These checks are also to be enhanced for FP 2014, and the extended timeline allows for this.

5 Recommendation 8.1: Written Risk Analysis and Mitigation policies should be developed to cover the handling of results, including discussions with experienced OMR systems operators. Risk management policies are being reviewed in partnership with all organisations involved in the delivery of the SJT, EPM, FPAS and those affected by the timeline of the recruitment process. Meetings have already taken place with four groups of experienced OMR systems operators. Recommendation 11.1: the validity review should consider all the components of selection for Foundation, including the decile ranking, and additional points, separately and in combination We agree with this recommendation, and this is part of the work overseen by the ISFP Research, Consultation and Evaluation Group. Pyschometric advice has been sought regarding the weighting of the SJT and EPM. For FP 2014, the SJT and the EPM will continue to be equally weighted, but this will continue to be reviewed year to year in light of new research evidence. Recommendation 11.2: no transcription of responses should be permitted after the expiry of the set time. Consistency of practice across sites is essential. The time allowed for the SJT is 2 hours 20 minutes, and answers must be recorded onto the answer sheet during this time. No points can be awarded for answers which are not recorded on the answer sheet, and no answers may be added after the end of the assessment. We will make absolutely clear that there is to be consistent practice across all sites.