Review of the GMC s PLAB test: final report

Transcription

1 Review of the GMC s PLAB test: fial report

2 Cotets Executive summary 01 What is the PLAB test? 08 Why is this review eeded? 09 How we carried out the review 10 Our fidigs from the review 12 Maitaiig cofidece i the PLAB test 44 Aex 1: Terms of membership ad referece 45 Aex 2: List of recommedatios 49 Appedix: Glossary of terms 52

3 Executive summary The Professioal ad Liguistic Assessmets Board (PLAB) test is desiged to test cadidates ability to practise medicie at the level expected at the ed of the first year of the Foudatio Programme (F1). This is whe graduates of UK medical schools are grated full registratio. The Geeral Medical Coucil (GMC) reviews the PLAB test from time to time to make sure that it remais fit for purpose. These reviews aim to make sure that the test cotiues to commad the cofidece of those who have a vested iterest i it, such as employers, educators ad traiers, ad patiets ad the public. They eed the PLAB test to be a objective, fair, o-discrimiatory ad fit-for-purpose method of assessig the kowledge ad skills of iteratioal medical graduates who apply for registratio with the GMC. Our terms of referece for this review fell uder four broad themes: esurig stadards cotet cofidece outcomes. Geeral Medical Coucil 01

4 We are cocered that employers do t fully uderstad the purpose of the test Esurig stadards ad cotet Extedig the scope of the test The PLAB test s blueprit, which is i effect the test s curriculum, is mapped appropriately agaist: This could be achieved, for example, through a situatioal judgemet test. But the GMC will eed to explore the most appropriate way of itroducig this exteded assessmet. the GMC s core guidace for doctors, Good medical practice the outcomes expected at the ed of F1 traiig i the GMC s guidace, The Traiee Doctor We are cocered that employers do t fully uderstad the purpose of the test, what it ca ad ca t assess, ad the level at which it is set. The GMC should therefore give greater promiece to the blueprit ad the overarchig statemet. the Foudatio Programme curriculum. However, the two-part format of the test limits the rage of values ad priciples i Good medical practice that ca be tested. We therefore recommed that the GMC exted the scope of the PLAB test to iclude a assessmet of the values ad priciples that ca t curretly be tested. It should also chage the ame of the test to more accurately reflect its purpose, ad promote the test with employers to icrease their uderstadig of it. 02 Geeral Medical Coucil

5 Limit test attempts ad reduce the time for which passes remai curret Cadidates are allowed to take Part 1 of the PLAB test as ofte as they eed to pass, but they must pass Part 1 agai if they do ot pass Part 2 withi four attempts. We recommed that the GMC impose a limit of four attempts at both Part 1 ad Part 2 to maitai cofidece i the test. Further attempts should oly be allowed if circumstaces beyod the cotrol of a cadidate affect their performace, or if they first demostrate remediatio. The GMC should give cadidates more-iformative feedback to help them prepare better for resits, ad should develop a framework for cadidates to demostrate remediatio. Some cadidates will be workig as doctors outside the UK while they progress through the PLAB process ad register with the GMC. Some, however, will ot give the risk to patiet safety of skills decay, we coclude that the GMC should reduce the timeframe for the currecy of PLAB passes from three to two years. This two-year timeframe mirrors the UK Foudatio Programme Office s requiremet that applicats for F1 traiig take a cliical assessmet if they graduated more tha two years previously. It also mirrors the requiremet for GPs who wat to joi a performers list but have bee out of geeral practice for more tha two years to take a cliical assessmet. Geeral Medical Coucil 03

6 The results of a geeralisability study will help the GMC to decide how best to icrease the reliability of Part 2 Could the PLAB test deliver a atioal licesig exam? The GMC s Coucil asked us to cosider the feasibility of the PLAB test deliverig a atioal licesig examiatio for UK graduates towards the ed of F1, i the evet that they decide that a atioal assessmet is eeded at that stage of traiig. We coclude that it was premature to decide whether the PLAB test is the appropriate way of deliverig a future atioal licesig examiatio. Durig our review, brigig forward full registratio for UK graduates was proposed. Furthermore, those resposible for medical educatio ad traiig i the UK have yet to agree the purpose ad objectives of a atioal licesig examiatio. Both issues must be resolved before a decisio o the most appropriate way of deliverig a atioal assessmet ca be agreed. Cofidece The methodologies used to set the stadard ad pass marks for the PLAB test are widely used i medical educatio ad traiig aroud the world ad are supported by evidece. The GMC should cotiue to use these methodologies, but review evidece of other methodologies used i a medical cotext as it emerges. The GMC should cosider usig item respose theory ad statistical equatig to support the curret stadard settig ad scorig methodologies. Improvig the reliability of the Part 2 exam The Part 1 examiatio has a cosistetly high reliability ad works well. The GMC should, however, work to improve the reliability of Part 2. We have suggested a rage of optios, such as icreasig the umber ad/or legth of the objective structured cliical examiatio (OSCE) statios. The results of a geeralisability study will help the GMC to decide how best to icrease the reliability of Part 2. The GMC will, however, have to balace the competig iterests of icreasig reliability ad the logistical feasibility of, for example, icreasig the umber ad/or legth of statios. 04 Geeral Medical Coucil

7 Sice our review bega, the GMC has itroduced a formal appeal procedure I the iterests of patiet safety, the GMC should cotiue to apply oe stadard error of measuremet to the Part 2 examiatio, give the risk to patiets of false positive cadidates. These are cadidates who do ot have the basic theoretical kowledge ad practical cliical skills to practise safely but pass the test by virtue of repeated attempts either fortuitously or due to familiarity with the examiatio material, for example. The GMC should, however, explore optios for reducig the possibility of cadidates compesatig betwee practical ad examiatio skills statios o the oe had ad commuicatio ad history takig statios o the other had. Reasoable adjustmets ad complaits are hadled well The GMC has a robust, objective ad fair procedure for dealig with requests for reasoable adjustmets to support cadidates to take the PLAB test. The GMC should review the procedure ad its guidace for cadidates to make sure it reflects relevat learig from its 2012 health ad disability review. The GMC also has procedures ad processes i place to deal with cadidates complaits about the PLAB test. Sice our review bega, the GMC has itroduced a formal appeal procedure. We suggest that the GMC regularly review appeal outcomes to idetify ad dissemiate treds, lessos leart ad developmet work required. The GMC gives cadidates sufficiet iformatio about the PLAB test to help them apply ad prepare for it. It also directs cadidates to the websites of other orgaisatios, where they ca fid iformatio about job opportuities, the cost of livig i the UK ad immigratio requiremets. We suggest that the GMC regularly review the iformatio ad guidace it provides i the light of cadidates ogoig feedback. Geeral Medical Coucil 05

8 More female examiers should be recruited We were cocered by the uderrepresetatio of wome i the Part 2 examier pool. The GMC has 135 examiers, but oly 20 are wome. The GMC has already doe work to uderstad the barriers that female doctors face i becomig examiers. We recommed that the GMC try to icrease the umber of female examiers through the cotiued use of targeted recruitmet campaigs. Outcomes Research commissioed for the review foud that doctors who pass the PLAB test uderperform i the membership examiatios of the Royal College of Physicias (MRCP (UK)) ad the Royal College of Geeral Practitioers (MRCGP) relative to UK qualified doctors. They also uderperform i the Aual Review of Competece Progressio (ARCP) process. However, the purpose of the PLAB test is ot to idetify whether cadidates have the potetial to achieve equivalet outcomes as UK graduates i postgraduate medical educatio ad traiig or through medical career pathways. It is desiged to test cadidates ability to practise medicie at the level expected at the ed of F1 traiig. 06 Geeral Medical Coucil

9 Our recommedatios should make sure that cofidece i the PLAB test remais Why do iteratioal medical graduates uderperform i these postgraduate examiatios? The reasos for the differetial outcomes are ot kow. They might reflect the fact that iteratioal medical graduates face barriers that UK graduates do t for example, they may take loger to fid employmet ad ted to follow less structured career pathways. Furthermore, the issue is ot uique to the UK there is evidece of uderperformace i postgraduate examiatios of a similar magitude for iteratioal medical graduates elsewhere (for example, i Caada We were cocered by the evidece of differetial outcomes. However, without a explaatio for the reasos, we were uable to recommed a defiitive, appropriate ad proportioate course of actio. We therefore recommed that the GMC ad other iterested parties ivestigate further to idetify why PLAB cadidates uderperform i these postgraduate examiatios ad i the ARCP. If ad whe the reasos are idetified, the GMC should cosider ay chages that are eeded to the purpose ad stadard of the PLAB test. Before cadidates ca sit the PLAB test, they must demostrate their kowledge of Eglish. They ca do this by achievig scores i the academic versio of the Iteratioal Eglish Laguage Testig System (IELTS) required by the GMC. The research commissioed for the review foud that IELTS scores have predictive validity for ARCP outcomes. We therefore recommed that the GMC cosider icreasig its IELTS requiremets. Our fial recommedatios Take together, our recommedatios o extedig the scope of the PLAB test, improvig the reliability of Part 2, limitig attempts ad reducig the currecy of passes i the PLAB test should make sure that cofidece i the PLAB test remais util a decisio is made o whether to itroduce a UK atioal licesig examiatio. We suggest that the GMC publish a progress report o takig forward our recommedatios i moths. Geeral Medical Coucil 07

10 What is the PLAB test? Why cadidates take the PLAB test All doctors must be registered ad licesed with the GMC if they wat to work as doctors i the UK. How doctors are registered ad licesed depeds, geerally, o their atioality ad where they qualified. Doctors who graduate outside the UK ad the EEA are kow as iteratioal medical graduates. Before the GMC will register ad licese them, iteratioal medical graduates must show that: they have the ecessary kowledge of the Eglish laguage * Most do this by achievig the GMC s required scores i the academic versio of the IELTS. These are at least 7.0 i each domai (readig, writig, speakig ad listeig) with a overall score of at least 7.5. What does the PLAB test do? The PLAB test is desiged to test the ability of iteratioal medical graduates to practise medicie at the level expected at the ed of F1 traiig. This is whe graduates of UK medical schools are grated full registratio. I other words, the GMC expects PLAB cadidates to show that they have the same level of theoretical medical kowledge ad practical cliical skills at the same stage of traiig as UK medical graduates. The test s format There are two parts to the PLAB test. The first kow as Part 1 is a writte test of kowledge applied to the care ad treatmet of patiets. The secod kow as Part 2 is a assessmet of cliical ad commuicatio skills. Oly cadidates who have passed Part 1 ca take Part 2. they have the kowledge ad skills required for medical practice i the UK Most do this by passig the GMC s PLAB test. * Sectio 21B(1)(d) of the Medical Act I Jue 2014 the GMC icreased the overall score required to 7.5 from 7.0. Sectio 21B(1)(b) of the Medical Act Geeral Medical Coucil

11 Why is this review eeded? The GMC reviews the PLAB test from time to time to make sure that it remais fit for purpose. The previous review took place i This review was commissioed to make sure that the test cotiues to commad the cofidece of those who have a vested iterest i it, such as employers, educators ad traiers, ad patiets ad the public. They eed the PLAB test to be a objective, fair, o-discrimiatory ad fit-for-purpose method of assessig the kowledge ad skills of iteratioal medical graduates who apply for registratio with the GMC. The review s terms of referece * are at aex 1 o page 45 * I April 2012 the GMC s Coucil ameded the terms of referece to ask us to iclude cosideratio of the feasibility of the PLAB test deliverig a atioal licesig examiatio if oe is itroduced. Geeral Medical Coucil 09

12 How we carried out the review I April 2011, the GMC established a idepedetly-led workig group to udertake the review (its membership is set out i aex 1). Membership icluded a medical director, a doctor ivolved i educatio ad traiig, a patiet/public represetative, a licesed doctor who has take the PLAB test, ad two experts to advise the workig group. The workig group met ie times betwee November 2011 ad March Our discussios were iformed by a rage of evidece, icludig: a literature review * This examied evidece o the umber of times cadidates are allowed to sit professioal examiatios (UK, Europe ad elsewhere) ad assessmets ad the periods of validity (i other words the currecy) of passes i these tests. It also reviewed examiatio ad assessmet methodologies ad best practice. a review of the PLAB test s blueprit This is, effectively, the PLAB test s curriculum. a aalysis by Professor Chris McMaus ad Mr Richard Wakeford This covered performace i the PLAB test ad the membership examiatios of the Royal College of Physicias (UK) the MRCP (UK) ad the Royal College of Geeral Practitioers the MRCGP. a primary research project * This aalysed performace i the PLAB test, the ARCP ad IELTS. It also examied GMC fitess to practise allegatios ad outcomes for these doctors. We refer to this as the Durham report. * McLachla JC, Illig J, Rothwell C, Margetts JK, Archer J, Shrewsbury D (2014) Developig a evidece base for the Professioal ad Liguistic Assessmets Board (PLAB) Test Lodo, GMC. McMaus C, Wakeford R (2014) Assessig the equivalece of PLAB graduates to UK graduates Lodo, GMC. Professor McMaus is educatioal advisor to the MRCP (UK) ad a member of the PLAB review workig group. Richard Wakeford is psychometric/assessmet cosultat to the MRCGP. 10 Geeral Medical Coucil

13 The GMC s mai statutory objective is to protect, promote ad maitai the health ad safety of the public a aalysis of aoymised Part 2 data a GMC ihouse literature review This looked at the challeges faced by iteratioal medical graduates ad juior doctors trasitioig to the UK workplace. feedback o key tasks i our terms of referece. We received this through: a call for writte evidece roudtable discussio groups i Norther Irelad, Scotlad ad Wales with people with a iterest i the PLAB test, such as educators ad traiers, ad the British Medical Associatio two roudtable discussio groups with members of the public (Cardiff ad Lodo) a roudtable discussio group with urses ad midwives (Norther Irelad) a meetig with a group of refugee doctors We have referred to feedback relevat to our coclusios ad recommedatios throughout this report. A presetatio by a expert o the theory ad developmet of situatioal judgemet testig. A presetatio o the use of situatioal judgemet tests as part of the GP specialty traiig selectio process. The GMC s mai statutory objective is to protect, promote ad maitai the health ad safety of the public. Our overridig cocer has therefore bee to make sure that the PLAB test cotiues to assess the kowledge ad skills of iteratioal medical graduates to the stadard ecessary for practisig as a doctor i the UK. We have also kept i mid at all times our public sector equality duty uder sectio 149 of the Equality Act a survey of PLAB test cadidates. * Tiffi PA, Illig J, Webster LAD, McLachla JC (2014) The Validity of the Professioal ad Liguistic Assessmets Board (PLAB) Exam: Research Report Lodo, GMC. The ARCP is a process that provides a formal ad structured review of evidece to moitor doctors progress throughout each stage of medical traiig. Geeral Medical Coucil 11

14 Our fidigs from the review This sectio sets out our coclusios ad recommedatios uder the four themes i our terms of referece. A full list of our recommedatios is at aex 2 (pages 49 51). A glossary of terms used is i the appedix (pages 52 56). 12 Geeral Medical Coucil

15 The PLAB test is set at the level of successful completio of F1 traiig Themes 1 ad 2 Esurig stadards ad cotet What the PLAB test ca ad ca t examie the blueprit Iteratioal medical graduates who take the PLAB test have o cotact with the GMC i a educatioal sese. Their performace i the test is a sapshot of their performace o the day they take it, rather tha a measure of how well they have mastered learig objectives. The blueprit is i effect the PLAB test s curriculum. It sets out the scope ad cotet of the test i terms of the topics, skills ad procedures that a doctor who passes the test eeds to kow ad be able to do. It icludes what the PLAB test ca ad ca t examie. The blueprit, as well as iformatio for cadidates o how to iterpret ad use it, ad liks to source documets ad referece material are published o the GMC s website. You ca fid these at The PLAB test is set at the level of successful completio of F1 traiig. The blueprit is therefore mapped agaist the Foudatio Programme curriculum, the outcomes expected at the ed of F1 traiig i The Traiee Doctor, * ad the GMC s core guidace, Good medical practice. It also takes ito accout UK hospital episode statistics ad the NHS Read Codes (so that it accurately reflects the workload of a doctor i the UK) ad the Royal College of Geeral Practitioers traiig curriculum (so that testig o commo, importat or acute coditios ad the maagemet of log-term coditios i primary care are icluded ). Questios for each Part 1 ad Part 2 examiatio are chose usig a samplig grid to make sure that the cotet is cosistet. * Geeral Medical Coucil (2011) The Traiee Doctor Lodo, GMC. Geeral Medical Coucil (2013) Good medical practice Lodo, GMC. The blueprit does ot iclude the advaced duties of a geeral practitioer. Geeral Medical Coucil 13

16 Good medical practice ad F1 traiig We approached reviewig the cotet of the PLAB test i two ways. We cosidered whether the blueprit is mapped appropriately agaist Good medical practice ad is cosistet with the outcomes at the ed of F1 traiig set out i The Traiee Doctor. We also asked people with a iterest i the test whether they thik the PLAB test adequately assesses the values ad priciples i Good medical practice. The blueprit s overarchig statemet * does, however, idetify these attributes. It makes it clear that although they might ot be assessed durig the PLAB test, these qualities are expected of doctors who attempt ad pass the test, ad will be appraised i the workplace. Professioal stadards ad ethics Feedback we received was cosistet the PLAB test eeds greater emphasis o the wider ethical values ad priciples that uderpi good medical practice i the UK. For example: We re satisfied that the blueprit is mapped appropriately agaist Good medical practice ad the Foudatio Programme curriculum, that it is cosistet with the outcomes expected at the ed of F1 traiig, ad that it idetifies the broader professioal attributes that successful cadidates are expected to demostrate i cliical practice i the UK. We coclude that the blueprit is therefore fit for purpose. That said, we recogise that the PLAB test ca t examie some of the professioal ad ethical attributes idetified i the blueprit because of the limitatios of its curret two-part format. cadidates attitudial attributes patiet autoomy workig withi limits of competece ad seekig advice workig uder pressure workig i multi-discipliary healthcare teams raisig cocers probity how cadidates deal with their ow health. Assessig these kids of values ad attributes is, however, a challege for ay medical assessmet it is ot uique to the PLAB test. * Geeral Medical Coucil (2014) GMC PLAB test blueprit Lodo, GMC. Available at Blueprit_overarchig_statemet DC2714.pdf_ pdf (accessed 20 March 2014). 14 Geeral Medical Coucil

17 Assessig these kids of values ad attributes is a challege for ay medical assessmet Revalidatio will make sure that all doctors workig i the UK egage i a regular aual appraisal ad develop ad maitai their kowledge ad skills throughout their career. Also, as our review was takig place, the GMC developed the Welcome to UK practice programme for doctors ew to UK practice. This aims to raise awareess of the ethical ad professioal stadards expected of doctors who practise i the UK through participative evets ad olie tools (ethical scearios ad a self-assessmet tool * ). We welcome the fact that the GMC has made these olie resources available to PLAB cadidates to help them prepare for the PLAB test. However, the GMC ca t make it madatory for cadidates to participate i the Welcome to UK practice programme i order to book a place o the PLAB test or to joi the medical register. Give the barriers that iteratioal medical graduates face whe trasitioig to the UK workplace, it would be ufair to expect them to have the same i-depth kowledge ad uderstadig of the professioal values ad priciples set out i Good medical practice as UK graduates. However, these values ad priciples (which are ot curretly examied through the PLAB test because of the limitatios of its curret format) are fudametal for safe, effective ad compassioate healthcare i the UK. We therefore coclude that the PLAB test does ot adequately assess the full rage of the professioal values ad priciples i Good medical practice. Recommedatio: The GMC should exted the scope of the PLAB test to iclude a assessmet of the wider ethical values ad priciples i Good medical practice that the curret format ca t test. * You ca fid the Welcome to UK practice web pages ad tools at: (accessed 20 March 2014). Research ito these barriers is available at: (accessed 20 March 2014). Geeral Medical Coucil 15

18 We welcome the work the GMC has already carried out to itroduce more real-life scearios i the Part 2 examiatio How ca the GMC bridge this gap? The literature review also idetified this weakess i the PLAB test s curret format. It suggested that the GMC cosider itroducig a situatioal judgemet test to bridge the gap. The Royal College of Geeral Practitioers (RCGP), the Coferece of Postgraduate Medical Deas of the Uited Kigdom (COPMeD) ad the Committee of Geeral Practice Educatio Directors (COGPED) also suggested this. Situatioal judgemet tests are used i the UK as part of the selectio process to Foudatio Programme ad specialty GP traiig to test whether cadidates have the professioal attributes they eed at the relevat stage of traiig. But some people believe that situatioal judgemet tests oly show whether cadidates kow what they should do, rather tha what they would do if faced with the scearios i real practice. Does the PLAB test reflect real-life practice? Iterested parties have told us that, i their view, the PLAB test assesses the ability to recall kowledge rather tha to apply it. They said it should test ability to apply kowledge i cliical settigs more robustly. We therefore welcome the work the GMC has already carried out to itroduce more real-life scearios i the Part 2 examiatio, such as: couplet statios These use iformatio idetified at oe statio to iform actio at the ext statio. high-fidelity simulator statios These test cadidates ability to deal with abormal sigs ad acutely ill patiets. professioalism statios These test cadidates ethical priciples. hadover statios These test cadidates ability to assimilate iformatio ad prioritise patiets eeds. Recommedatio: The GMC should explore how best to exted the scope of the PLAB test to iclude a assessmet of the professioal values ad priciples i Good medical practice that are t curretly tested. For example, this could be doe by icludig a situatioal judgemet test or aother mechaism. 16 Geeral Medical Coucil

19 We were cocered that the ame of the test does t reflect its purpose Do employers uderstad the test s purpose? Whe we discussed the blueprit we were worried about whether employers uderstood what the PLAB test does ad does t assess, ad that it is set at the level of ed of F1 traiig. We were also cocered that the ame of the test does t reflect its purpose. This could lead to misuderstadig, particularly amog employers, about what the PLAB test is desiged to do. Employers are, of course, resposible for makig sure that the doctors they employ have the appropriate kowledge, skills ad experiece for particular posts. They must ot rely o a pass i the PLAB test as evidece of this. Recommedatio: The GMC should give greater promiece to the PLAB test s blueprit ad its overarchig statemet to improve geeral uderstadig of the curret scope of the PLAB test. Recommedatio: The GMC should chage the ame of the PLAB test to reflect more accurately its purpose. For example, it could be called the GMC s Kowledge ad Cliical Skills Tests. Recommedatio: The GMC should promote the PLAB test s purpose with employers to icrease their uderstadig that the test is set at the level of etry to F2 traiig (the secod year of Foudatio Programme traiig). Geeral Medical Coucil 17

20 There was cocer that cadidates who eed multiple attempts are less competet tha those who pass first time The GMC should limit the umber of times a cadidate ca attempt the test Cadidates ca sit Part 1 as may times as they eed to pass, provided their evidece of proficiecy i Eglish remais curret. They are the allowed four attempts to pass Part 2, withi three years of the date they passed Part 1. If they do t pass Part 2 o their fourth attempt, or withi three years, they must pass Part 1 agai (ad give further evidece of their proficiecy i Eglish, if their curret evidece is more tha two years old) before beig allowed further attempts at Part 2. We received cosistet feedback that there should be a limit o the umber of resits allowed. Opiios varied o what the limit should be, but there was cocer that cadidates who eed multiple attempts are less competet tha those who pass first time. Ad ulimited attempts therefore udermie cofidece i the test. Survey respodets also supported a limit agai, with differet views o what the limit should be. However, there was also support to maitai the curret situatio, particularly amog cadidates who had ot passed the test ad those who eeded more tha oe attempt to pass. 18 Geeral Medical Coucil

21 There is a risk to patiet safety whe doctors pass the test by virtue of repeated attempts What is a appropriate limit for the umber of resits? Through the literature review, we tried to fid objective evidece about the umber of times cadidates ca sit comparable professioal examiatios ad assessmets. The literature review * foud that: the evidece base for the umber of resits allowed i comparable examiatios ad assessmets was at best weak ad geerally abset the umber of attempts allowed is part of a examiatio s desig, ad time limits iteract with traiig programmes, idividual specialty requiremets ad differet atioal cultures The literature review also highlighted the risig chace of false positives as the umber of resits icreases. I other words, there is a risk to patiet safety whe doctors pass the test by virtue of repeated attempts, either fortuitously or due to familiarity with the examiatio material, ad do ot i fact have the basic theoretical kowledge ad practical cliical skills to practise safely. There are also costs for cadidates retakig examiatios, such as time ad fiacial ivestmet. The literature review suggested that a limit of four attempts at Part 1 ad four attempts at Part 2 is a reasoable compromise betwee these competig iterests. resittig examiatios beefits some cadidates, particularly cadidates who oly just fail there is moderately strog evidece to idicate that there is o further beefit after four attempts. I other words, performace teds to plateau after the fourth attempt. * McLachla JC, Illig J, Rothwell C, Margetts JK, Archer J, Shrewsbury D (2014) Developig a evidece base for the Professioal ad Liguistic Assessmets Board (PLAB) Test Lodo, GMC. Geeral Medical Coucil 19

22 The GMC should impose a limit of four attempts at both Part 1 ad Part 2 of the PLAB test Resit attempts relate to future fitess to practise cocers The Durham report idetified treds relevat to whether to limit resits. For example, eve after cotrollig for age ad geder, performace at the first attempt i both Part 1 ad Part 2 predicted a reduced likelihood of cesure for fitess to practise cocers. The report also idetified possible associatios betwee multiple attempts at both parts of the PLAB test ad the likelihood of cesure for fitess to practise cocers. It foud: a slight differece i the likelihood of cesure betwee cadidates who took Part 1 three times ad those who passed o first attempt a large differece betwee cadidates who passed Part 1 o their first attempt ad those who had four or more attempts cadidates who took Part 2 three times or more were far more likely to be cesured tha those who passed o first sittig. There is a argumet that a cadidate meets the stadard by achievig the required scores o the day whether at first or subsequet attempts. However, i the light of the evidece idetified i the literature review ad the fidigs i the Durham report, we believe that the GMC should impose a limit of four attempts at both Part 1 ad Part 2 of the PLAB test. I our view, these limits strike the right balace betwee esurig patiet safety, maitaiig cofidece i the test ad maagig cadidates expectatios. A limit of four attempts is cosistet ad fair Medical royal colleges impose limits o the umber of attempts at their atioal professioal examiatios. I lie with guidace edorsed by the GMC, cadidates are allowed a maximum of six attempts but there is flexibility for idividual colleges to set more striget limits. Limitig the umber of attempts at Part 1 or Part 2 of the PLAB test to four would therefore be cosistet with the spirit of the guidelies o limitig attempts at UK postgraduate examiatios. We also agree, however, that there might be exceptioal circumstaces beyod cadidates cotrol that affect performace o the day. I the iterests of fairess, we recommed that the GMC take ito accout exceptioal circumstaces if a limit o attempts at the PLAB test is itroduced. 20 Geeral Medical Coucil

23 We also coclude that, i the iterests of fairess, cadidates who do ot pass withi the limits should ot automatically be precluded from takig the test agai. However, we agree that the GMC should oly allow a further attempt if a cadidate ca demostrate that they have addressed the gaps i their performace i the PLAB test over a appropriate period. This is cosistet with the GMC s guidace to the medical royal colleges where cadidates have exceeded the maximum umber of attempts at ay compoet of atioal professioal examiatios. We ackowledge that may of these usuccessful cadidates will be workig or udertakig postgraduate traiig i healthcare systems elsewhere i the world. They might ot, therefore, have access to the same level of educatioal support available to doctors i either professioal practice or postgraduate medical educatio ad traiig i the UK. However, we agree that they should demostrate that they have egaged i additioal learig ad professioal developmet over a acceptable period of time before beig allowed a further attempt at the PLAB test. This will help to improve the chaces of success for cadidates who have t passed withi the limits we propose ad for whom there must ievitably be a questio about their capability of practisig safely at the level of etry to F2 traiig. The literature review suggested two years as a appropriate timeframe for a period of remediatio before allowig a further attempt, although there is o evidece to support this timeframe. We ackowledge the challege for the GMC of assessig the objectivity ad robustess of evidece that cadidates might submit. We also ackowledge that the opportuity to give meaigful feedback i the cotext of a summative examiatio where cadidates do t have educatioal supervisors to help them reflect will ievitably be limited. We suggest that the GMC develop a framework withi which cadidates ca realistically, but objectively ad robustly, demostrate sufficiet remediatio to allow a further attempt at Part 1 or Part 2. Recommedatio: The GMC should impose a limit of four attempts at both Part 1 ad Part 2 of the PLAB test. Further attempts should oly be allowed if circumstaces beyod cadidates cotrol have affected performace or o the basis of demostrable remediatio over a period acceptable to the GMC. Geeral Medical Coucil 21

24 More meaigful feedback would help cadidates to idetify future learig eeds Feedback o the performace of cadidates Givig feedback to cadidates o their examiatio or assessmet results (both successful ad usuccessful) helps them to reflect o their performace ad idetify future learig eeds. The GMC curretly gives PLAB cadidates their Part 1 scores. It also gives usuccessful Part 2 cadidates a breakdow of the grade they were give i each OSCE statio. However, most survey respodets ad a group of refugee doctors told us that they would like better feedback o their performace i the PLAB test. This is because it would help them to prepare better for resits ad to idetify areas for cotiuig professioal developmet i the workplace. There is therefore a desire for a greater level of feedback amog cadidates. More meaigful feedback would help cadidates to idetify future learig eeds ad could help mitigate the impact of a limit o the umber of attempts for cadidates who eed more tha four attempts to pass either part of the PLAB test. Recommedatio: The GMC should provide more meaigful feedback o performace to PLAB test cadidates. Are the curret timeframes too geerous? Successful Part 1 cadidates must pass Part 2 withi three years of passig Part 1. If they do t, they must resit ad pass Part 1 agai, as well as givig further evidece of their Eglish skills if their curret evidece has expired. Geerally speakig, successful Part 2 cadidates must also apply for registratio ad a licece to practise withi three years of passig Part 2. It s ot clear why some cadidates do t apply for registratio ad a licece to practise i the UK immediately, or relatively soo, after passig Part 2, give that the PLAB test is desiged to assess kowledge ad skills for the purposes of registratio. Feedback we received was geerally cosistet the curret timeframes are too log because the kowledge ad skills of some PLAB cadidates could have deteriorated by the time they eter the UK workplace. There was, however, o cosesus o what the timeframes should be. Some survey respodets also supported shorter timeframes but, agai, there was o cosesus o appropriate timeframes. A sigificat miority said, however, that there should be o time restrictios. Respodets who were ot registered ad licesed with the GMC expressed a stroger preferece for o time limit tha those who were. 22 Geeral Medical Coucil

25 What is a fair timeframe? The literature review sought to idetify evidece about how log passes i comparable examiatios ad assessmets remai curret before cadidates have to retake them. It foud that currecy is most ofte tied to idividual examiatio structures ad desig, ad ca also be liked to traiig structures. This makes it difficult to geeralise o good practice. It also foud extesive evidece o skills decay, which depeds o a rage of variables. For example: skills required for complex tasks decay quicker tha those required for simpler tasks theoretical kowledge might decay more slowly tha practical skills But the issues are clearly complex the literature review cocluded that, other tha the evidece o skills decay, there is a lack of sigificat evidece for decidig appropriate periods of currecy of passes i the PLAB test. There is, however, a precedet for assessig the cliical competece of doctors who have take time away from medical educatio ad professioal practice i the UK. The UK Foudatio Programme Office requires applicats for F1 traiig to take a cliical assessmet if they graduated more tha two years previously. Ad GPs who wat to joi a performers list but have bee out of cliical geeral practice for more tha two years must also take a cliical assessmet. the cofidece ad experiece of the idividual the ature of iitial traiig ad feedback Give the risk to patiet safety of skills decay, it is urealistic for cadidates to expect that passes i either part of the PLAB test will ot be time-boud. acquirig ew kowledge ad reliquishig previously leared kowledge. Some cadidates will be egaged i medical practice while progressig through the PLAB process ad durig the time betwee passig Part 2 ad registerig with the GMC. Some, however, will ot ad we are particularly cocered about the risk of the currecy of their kowledge ad skills for patiet safety. Recommedatio: The GMC should require cadidates to pass Part 2 of the PLAB test withi two years of passig Part 1. It should also require cadidates to apply for GMC registratio ad a licece to practise withi two years of passig Part 2. Geeral Medical Coucil 23

26 It is premature to decide how to deliver a future licesig examiatio util its purpose ad objectives have bee agreed Could the PLAB test deliver a atioal licesig examiatio? The prospect of usig existig examiatios ad assessmets to deliver ay future atioal licesig examiatio is attractive. Usig the PLAB test would obviate the eed to develop ew examiatios ad assessmets, which would have cosiderable developmet ad implemetatio costs. But there are sigificat questios that eed to be aswered before decidig whether it would be appropriate to use the PLAB test to deliver a future atioal licesig examiatio. It is also premature to decide how to deliver a future licesig examiatio util its purpose ad objectives have bee agreed by those resposible for medical educatio ad traiig i the UK. For example, whether the examiatio is a summative assessmet of educatioal achievemet followig a udergraduate medical programme; or whether it should assess suitability for etry to postgraduate educatio ad traiig. Discussios o deliverig a possible licesig examiatio ca oly begi oce these priciples have bee agreed. The most fudametal questio is: what will be the future shape of educatio ad traiig for UK medical studets? Proposals to brig forward the poit of full registratio for UK graduates are uder cosideratio, most recetly i the Shape of Traiig review. * If full registratio is brought forward i this way, the outcomes that the GMC will require medical studets to demostrate at the ed of their udergraduate educatio ad traiig might chage. Brigig forward full registratio without a period of provisioal registratio might therefore have implicatios for the blueprit of a atioal licesig examiatio ad the stadard at which it is set. * Shape of Traiig (2013) Securig the future of excellet patiet care. Fial report of the idepedet review. Led by Professor David Greeaway Lodo, GMC. 24 Geeral Medical Coucil

27 The PLAB test could, i theory at least, deliver a atioal licesig examiatio Would the PLAB test remai? It is also importat to ote that the PLAB test will o loger exist if a atioal licesig examiatio is itroduced for UK graduates. The PLAB test goverace framework would ot therefore be appropriate for a UK atioal licesig examiatio, although the GMC s experiece of ruig the PLAB test could iform the developmet of a atioal licesig examiatio. Medical schools, foudatio schools, local educatio ad traiig boards, deaeries, ad employers will eed to be joitly resposible, ad accoutable, for ay future atioal licesig examiatio. The experiece of professioal regulators ad licesig boards i other jurisdictios (for example, i the US ad Caada) would also helpfully iform the developmet phase. examiatio for UK graduates towards the ed of F1 traiig. The blueprit is fit for purpose i terms of stadard ad cotet. It is mapped agaist Good medical practice, the Foudatio Programme curriculum ad the outcomes expected at the ed of F1 traiig. The stadard settig methods for both parts of the PLAB test are well studied i the literature ad widely used to set the stadards of medical examiatios ad assessmets aroud the world. The methods used to deliver the test (multiple choice examiatio ad OSCE) are widely used i examiatios i medical educatio ad traiig. Havig said that ad o the assumptio that o chage is made to the poit whe UK graduates are grated full registratio the PLAB test could, i theory at least, deliver a atioal licesig Geeral Medical Coucil 25

28 The impact of a atioal licesig examiatio o GMC resources A detailed capacity study would be eeded to help uderstad the pressure o existig GMC resources ad to model proposals (ad the likely capital ad operatioal costs) of the GMC deliverig the OSCE compoet of a atioal licesig examiatio. I 2013, just over 7,500 medical studets graduated from UK medical schools. It would ot be difficult for the GMC to deliver a multiple choice examiatio to such a large cohort locally through computerbased testig. However, a iitial review of the GMC s Cliical Assessmet Cetre s capacity to meet the additioal demad of aroud 7,500 UK graduates each year suggests that there would be a shortfall i capacity. Furthermore, if demad is compressed ito a short period (for example, a hypothetical two-moth period), logistical problems ievitably become greater. Releasig some 7,500 doctors from their posts over a short period could affect employers ability to deliver high-quality healthcare, if a atioal licesig examiatio is take towards the ed of F1 traiig. Other safety ad fairess issues to cosider I additio, adequate safeguards would have to be developed to make sure that stadards remai cosistet ad to mitigate ay risk to patiet safety. Ruig the OSCE compoet of a atioal licesig examiatio locally across the UK would itroduce variables that could adversely impact stadards, such as icosistet markig by examiers across differet sites, ad quality of veue ad equipmet. The GMC ad parter orgaisatios would eed to develop, expad ad maitai the existig Part 1 ad Part 2 questio baks to make sure that the itegrity of both examiatios is ot compromised. While challegig, this is ot isurmoutable ad could be achieved, at least partly, through sharig ad adaptig other orgaisatios questio baks. The challeges of recruitig ad traiig sufficiet OSCE examiers, ad maitaiig reliability, should ot be uderestimated. 26 Geeral Medical Coucil

29 The law would also eed to chage if full registratio for UK graduates is depedet o success i a atioal licesig examiatio There will also be priciples of fairess ad equality of opportuity to cosider. If a atioal licesig examiatio took place towards the ed of F1 traiig, cadidates who take it too early durig F1 would be disadvataged they will ot have bee give the same opportuities for learig ad gaiig experiece compared with cadidates who take it at a much later stage. This could be avoided if all cadidates take the assessmet o the same day, but this would affect delivery of healthcare. What are the implicatios for F1 traiees who fail? The GMC ad those resposible for medical educatio ad traiig would eed to cosider very carefully the scope for resits ad a appeal procedure. The law would also eed to chage if full registratio for UK graduates is depedet o success i a atioal licesig examiatio. There is curretly o statutory provisio to require UK graduates to pass a atioal licesig examiatio before the GMC registers ad liceses them. We agree that it is too early to determie whether the PLAB test could, or should, deliver ay future atioal licesig examiatio i the UK. This ca oly be decided oce both the future shape of educatio ad traiig for medical studets ad ewly qualified doctors ad the purpose ad objectives of ay future licesig examiatio are settled. If a decisio is the made that the PLAB test is the appropriate mechaism to deliver a atioal licesig examiatio, the GMC ad those resposible for medical educatio ad traiig will eed to resolve the caveats we have idetified with regard to capacity, goverace, maitaiig stadards, ad equality ad fairess. Recommedatio: The GMC ad those resposible for medical educatio ad traiig i the UK should agree the most appropriate way of deliverig a atioal licesig examiatio whe the future shape of educatio ad traiig for medical studets ad ewly qualified doctors ad the purpose ad objectives of the examiatio are settled. Geeral Medical Coucil 27

30 Theme 3 Cofidece Should the method of stadard settig chage? The GMC curretly uses the Agoff method to set the stadard of the Part 1 examiatio, ad borderlie group scorig methods to set the stadard of the Part 2 assessmet. The literature review cocluded that there is o sigle uiversally optimum method of stadard settig The curret methods used i PLAB are well studied i the literature, show acceptable properties, ad there is o cosistetly better method uder all circumstaces. * The literature review highlighted other stadard settig methods. It also idetified other statistical approaches, such as item respose theory, that ca support stadard setters ad help develop a deeper uderstadig of how items ad cadidates perform. Recommedatio: The GMC should retai the Agoff ad borderlie group scorig methods for settig the stadards of the Part 1 examiatio ad the Part 2 assessmet. Recommedatio: The GMC should cosider usig item respose theory ad statistical equatig to support the curret stadard settig ad scorig methodologies for the PLAB test. Recommedatio: The GMC should regularly review the PLAB test s stadard settig ad scorig methodologies to esure that the test remais up to date ad i lie with evidece of other methods i a medical cotext. * McLachla JC, Illig J, Rothwell C, Margetts JK, Archer J, Shrewsbury D (2014) Developig a evidece base for the Professioal ad Liguistic Assessmets Board (PLAB) Test Lodo, GMC (page 56). 28 Geeral Medical Coucil

31 The reliability of the Part 1 examiatio is cosistetly high Reliability of the Part 1 examiatio Part 1 is a writte test of medical kowledge desiged to assess the applicatio of cliical kowledge to the care of patiets. It is a three-hour multiple choice examiatio that comprises 200 sigle best aswer questios. It ca be take i the UK or overseas. This examiatio method is very widely used i udergraduate ad postgraduate medical educatio ad traiig aroud the world. The reliability of the Part 1 examiatio is cosistetly high at 0.9 ad above. (A perfectly reproducible test would have a reliability coefficiet of 1.0.) The Part 1 examiatio therefore works well. Computer-based support Part 1 cadidates mark their aswers o aswer sheets, which are the scaed usig a optical mark reader. The questios i ay Part 1 examiatio may already iclude the use of images, electrocardiograms ad x-rays. However, the literature review oted that computer-based testig eables a wider rage of assessmet approaches 1 such as the use of video or audio clips. The literature review also oted that computerbased testig eables much quicker feedback for cadidates ad a greater rage of data aalysis. Agaist these beefits, however, the literature review highlighted that it might ot always be feasible to use computer-based testig if large umbers of computer statios are eeded i a wide variety of locatios. Recommedatio: The GMC should explore the feasibility of itroducig computer-based testig for the Part 1 examiatio ad electroic markig for both parts of the PLAB test. This would eable the GMC to provide quicker feedback for cadidates, use a wider ad eve more realistic rage of assessmet techiques, ad gather additioal itelligece for aalysis. * McLachla JC, Illig J, Rothwell C, Margetts JK, Archer J, Shrewsbury D (2014) Developig a evidece base for the Professioal ad Liguistic Assessmets Board (PLAB) Test Lodo, GMC (page 56). Geeral Medical Coucil 29

32 This examiatio format is well recogised ad widely used i both udergraduate ad postgraduate medical educatio ad traiig The Part 2 examiatio Part 2 is a OSCE that takes cadidates through 14 five-miute statios. It is a assessmet of practical, cliical ad commuicatio skills i four domais: commuicatio, examiatio, history takig ad practical skills. It takes place i the GMC s Cliical Assessmet Cetre ad oly cadidates who have passed Part 1 ca take Part 2. This examiatio format is well recogised ad widely used i both udergraduate ad postgraduate medical educatio ad traiig aroud the world. We support its use for the Part 2 examiatio. Examiers are carefully selected ad well traied ad are committed to the examiatio. The facilities at the GMC s purpose-built Cliical Assessmet Cetre are excellet. Settig the pass mark for Part 2 Each statio i each Part 2 examiatio has betwee three ad six objectives. Each objective is weighted, with the total weightigs for each statio addig up to 100%. Examiers award cadidates a grade betwee A ad E for each objective but are ot aware of the objectives weightigs. Grades awarded are the coverted to a scale of 0 4 ad calculated to give a overall statio score. The examier also gives a separate overall judgemet of each cadidate s overall performace (pass, borderlie or fail). They base this o their professioal experiece usig the otioal miimally competet doctor at the level of etry to F2 traiig as a bechmark. The passig score for each statio is the calculated from the mea scores of previous cadidates to whom examiers have give borderlie judgemets. The scores for each statio are added, plus oe stadard error of measuremet (SEM), to determie the total score for each Part 2 examiatio. 30 Geeral Medical Coucil