Specialty Selection Test (SST) Pilot INFORMATION PACK

Specialty Selection Test (SST) Pilot INFORMATION PACK Page 1 of 18

Contents Section 1: Introduction... 2 Section 2: One-page overview... 2 Section 3: FAQs for Applicants... 3 Section 2: FAQs for LETBs/Deaneries/Specialties... 10 Section 1: Introduction As part of the Medical and Dental Recruitment and Selection (MDRS) Programme, Health Education England (HEE) is working with the devolved nations, the Work Psychology Group and academics in the University College of London Medical School to pilot a Specialty Selection Test (SST). The SST will be piloted across a number of different medical specialties, across the United Kingdom, and includes a Clinical Problem Solving Test (CPS) and a Situational Judgement Test (SJT). The purpose of this pilot is to evaluate whether these tests enhance the current methods of selection used for ST1/CT1 level recruitment. This information pack has been developed to assist with raising awareness about the SST pilot; the objectives, the approach, and how you will be involved. The pack contains a one-page overview document and Frequently Asked Questions (FAQs), which aim to answer any questions you and your colleagues might have, and to further assist with any applicant queries that may arise over the course of the coming months. Your feedback is important to us, so please don t hesitate to contact the team via email at: hee.mdrs@nhs.net if you have further questions or any suggestions for improving this document. Section 2: One-page overview The one-page overview document (appendix 1) is a high-level brief that covers the implementation approach, who s involved and why we are doing this. This document was distributed to all audience groups during August 2013. Page 2 of 18

Section 3: FAQs for Applicants 1. What is the SST pilot and why are we doing this? As part of the Medical and Dental Recruitment and Selection (MDRS) Programme, Health Education England (HEE) is working with the devolved nations, the Work Psychology Group and academics in the University College of London Medical School to pilot a Specialty Selection Test (SST). The SST will be piloted across a number of different medical specialties, across the United Kingdom, and includes a Clinical Problem Solving Test (CPS) and a Situational Judgement Test (SJT). The purpose of this pilot is to evaluate whether these tests enhance the current methods of selection used for ST1/CT1 level recruitment. We are seeking doctors help to participate and provide feedback about the pilot test. The results from this pilot will enable a group of leading selection researchers to consider whether there are any advantages with introducing this test and how the results correlate with current selection methods. We will also continue to follow applicant progression for a number of years, to ensure a long-term robust evaluation. Ultimately, we are assessing whether the SST is a good predictor of a doctor s performance throughout their training and to ensure the selection process is reliable, valid, fair and cost-effective. 2. How will the SST pilot work? We plan to pilot the SST between 13-25 January 2014 for applicants to specialty training at ST1/CT1 level. On 11 December 2013, all applicants for ST1/CT1 will receive an email invitation to take part in the Specialty Selection Test pilot. The test will be delivered online through Pearson VUE test centres in the same way the General Practice (GP) test is run. You will be asked to choose a time from a range of time-slots and a venue from one of many Pearson VUE driving test centres across the United Kingdom (UK). These venues are located within close proximity to hospitals across the UK. The test itself will consist of a two hour and fifteen minute online test. It will be completely computer based and the questions will be developed by leading specialists according to person specifications and foundation programme curriculum. If you are already doing the Page 3 of 18

GP test as part of your assessment for GP or Broad Based Training (BBT) you will not be asked to do the additional test. If you are applying to a number of different specialities you will only be required to sit the test once. 3. Who is eligible to sit the SST pilot? Any doctor applying for ANY specialty training post at ST1/CT1 level is eligible to sit the test for the Specialty Selection Test pilot. All applicants will be asked to take part - especially those in smaller specialties with fewer applicants - as we need to ensure that enough trainees sit the pilot to provide enough data for a thorough assessment. 4. What will the SST pilot evaluate? The pilot will evaluate the correlations of performance in the selection test to the selection decision, and where applicable, to scores from currently used assessment tests as well. It is important to note that the test results will not form part of the actual selection process. We aim to introduce a testing method that is fairer, more consistent, offers more transparency and is less labour intensive, and so we will be assessing these components too. By taking part in the pilot, your results and feedback will enable the team to evaluate the selection process and introduce improvements to the way in which doctors are selected for specialty training posts. Comparisons to achievements in on-job performance and in professional exams will also be made at a later date as ultimately, we want to assess whether the multispecialty test results can predict how you perform during specialty training. Page 4 of 18

5. Will the results contribute to recruitment scores or be passed on to regulating bodies? Your results will not form part of the recruitment process nor will your results impact on your fitness to practise. This is a pilot evaluation study only and as such, individual results will not be shared with regulatory bodies. Your results will only be seen by the research team who will evaluate the correlations of performance in the selection test to the selection decision, and where applicable, to scores from currently used assessment tests as well. It is important to note that the test results will not form part of the actual selection process. 6. Which specialties are involved in the SST pilot? A detailed list will be made available shortly. Discussions have taken place with Medical Royal Colleges to be involved in and support the SST pilot, and most have confirmed their support and willingness for their applicants to be involved in this pilot evaluation. 7. Why are we piloting a Specialty Selection Test hasn t this been piloted already? Similar tests and pilots have been used before in many specialties, but not across specialties currently 7,000 out of the 12,000 doctors applying for specialty training take a Clinical Problem Solving (CPS) and a Situational Judgement Test (SJT). The test we are piloting will be similar to that used in General Practice (GP) and Public Health and similar to the specialty specific tests that have been piloted in surgery, acute specialties, medicine and other specialties. The test in GP was successfully implemented in 2007 and used to date as part of the selction process, constituting 30% of a trainee doctor s total recruitment score. In comparison with other methods of selection, these tests have been shown to be the best predictor of performance in the work place and likelihood of successful completion of specialty training. Applicants consistently rate these tests as fair and appropriate to use as sub-components of speciality selection. Each of the devolved nations is focused on the National Health Services (NHS) objective to recruit staff into the NHS with appropriate values and behaviours, as well as knowledge and skills. The Specialty Selection Test will support the need to incorporate values-based selection methods and is a possible way of meeting these requirements. Page 5 of 18

This pilot is an evaluation of selection processes for the benefit of our patients and you, as an NHS doctor. 8. How will I benefit from taking part in the SST pilot what s in it for me? This pilot aims to improve the way doctors are selected in the future and this is your chance to give your views on this selection process and make your voice heard! By participating in the pilot, you will also receive feedback on how you performed and go into the draw to win one of five prizes, which includes a 32GB ipad and Kindle Fire HD. As an applicant, your feedback will be viewed as very important and you will be able to have your say on how appropriate you found the selection test to be in relation to selection into specialty training, and whether you think it allowed you to demonstrate your aptitude for specialty training. This information will then be used to inform the Specialties and other stakeholders on whether these selection methods add additional value to the selection processes currently used. 9. Why will the pilot test be across multi-specialties and not specific to the specialty that I'm applying for? The pilot of the SST is designed to be broad-based due to the nature of the overlap of person specifications required by all specialties. A good example of this is the fact that all specialties require trained doctors not just to be clinically competent, but also empathic and sensitive with patients and colleagues, behaving with professional integrity, with the ability to prioritise and cope well under pressure and work effectively in a team. All doctors who have undertaken Foundation will be familiar with the Foundation Curriculum by which these skills, values and behaviours are embedded. The Specialty Selection Test does not seek to replace individual specialty specifications. Each speciality will still have its own speciality specific elements within the selection process. Page 6 of 18

10. Have CPS and SJT tests been used before and if so, by whom? Yes. The CPS and SJT test is successfully being used currently for applicants to General Practice (GP) (since 2007) and Public Health (since 2009). Approximately 7000 out of 12,000 applicants to ST1/CT1 training already sit the test each year; though it is currently specific to the specialty they are applying for. A multispecialty pilot (of 800 applicants) was undertaken in 2010 across several ST1/CT1 specialties. The pilot test performed well, with the results correlating well with the selection scores, but there wasn t enough involvement across all specialties to determine whether the test would be beneficial for all specialties. This is one of the reasons why we are undertaking a Specialty Selection Test pilot across a wider range of specialties. 11. Is it mandatory to take part in the SST pilot? If you are not sitting the General Practice (GP) test, you will be invited to take part. The option to make participation mandatory will be at the discretion of the specialty you apply to. If you are applying for GP specialty training or Broad Based Training (BBT) then you will be required to sit the GP selection test as it is part of the assessment centre for entry into GP training. Questions from the SST pilot will be embedded in the GP test, but will not count towards your score in the selection process. This will allow us to study how well the multispecialty questions perform compared to items in the GP test. 12. Will travel expenses be reimbursed? Yes, standard class travel expenses will be reimbursed for the SST pilot project up to the value of 10.00. There is a large selection of Pearson VUE test centres, which are located within close proximity to hospitals across the UK and should be easily accessible. Page 7 of 18

13. Do I need to do the test if I m applying for General Practice (GP) or any other specialty? As indicated above, those applying to General Practice (GP) training and Broad Based Training (BBT) who already undertake an online test will not need to sit an additional test for the SST pilot. If you are applying to specialties other than General practice, we are seeking your support to undertake the test and we will request your participation. As such, your involvement in the pilot would be greatly appreciated. 14. Will the mark I achieve be known to anyone involved in selection into my chosen specialties? Absolutely not! Your individual results will only be seen by the research team who will be evaluating results to determine whether the SST pilot test adds value to the specialty recruitment selection process. 15. How will the security of the data about me and my results be ensured? The results and information obtained from this pilot will be anonymised and secured using the same measures that are currently in place to protect selection results for General Practice (GP). We will use secure file transfer protocols to manage your information, ensuring that your data is kept completely secure. 16. Will the SST be used in the selection process for shortlisting during 2014 specialty training recruitment? Your performance in the piloting of the SST will not in any way affect your selection scores for entry into specialty training in 2014 at ST1/CT1 level. It will not be used for shortlisting or at any other stage of the specialty training process. A poor performance in the pilot will not have any impact on your application to a specialty training post at ST1/CT1 level. Page 8 of 18

17. Where can I find out more information? You can contact us at hee.mdrs@nhs.net for more information. The Specialty Training website will be updated regularly and the applicant guide is another useful source where you can find more information about the pilot and what is involved. Page 9 of 18

Section 2: FAQs for LETBs/Deaneries/Specialties 1. What is the SST pilot and why are we doing this? As part of the Medical and Dental Recruitment and Selection (MDRS) Programme, Health Education England (HEE) is working with the devolved nations, the Work Psychology Group and academics in the University College of London Medical School to pilot a Specialty Selection Test (SST). The SST will be piloted across a number of different medical specialties, across the United Kingdom, and includes a Clinical Problem Solving Test (CPS) and a Situational Judgement Test (SJT). The purpose of this pilot is to evaluate whether these tests enhance the current methods of selection used for ST1/CT1 level recruitment. We are seeking doctors help to participate and provide feedback about the pilot test. The results from this pilot will enable a group of leading selection researchers to consider whether there are any advantages with introducing this test and how the results correlate with the current selection methods used. We will also continue to follow applicant progression for a number of years, to ensure a long-term robust evaluation. Ultimately, we are assessing whether the SST is a good predictor of a doctors performance throughout their training and to ensure the selection process is reliable, valid, fair and cost-effective. 2. What is a CPS and SJT? The Specialty Selection Test pilot consists of two components: a Clinical Problem Solving test (CPS), which is an applied medical knowledge test based on the Foundation Programme curriculum; and a Situational Judgement Test (SJT), which assesses behaviours and values in a series of workplace based scenarios. Page 10 of 18

3. What is the timeframe for the SST pilot? We plan to pilot the SST between 13-25 January 2014 for applicants to specialty training at ST1/CT1 level. We will ask for applicant details to be emailed to our SST support person when applications close and on 11 December 2013 all applicants for ST1/CT1 will receive an email invitation to take part in the Specialty Selection Test pilot. The test will be delivered online through Pearson VUE test centres in the same way the General Practice (GP) test is run. Applicants will be asked to choose a time from a range of time-slots and a venue from one of many Pearson VUE driving test centres across the United Kingdom (UK). 4. How will I be involved what does this mean to me? As a member of a recruiting organisation, a list of the applicants who have applied and their contact details will need to be sent to a central location. These details will be confirmed shortly. Participants will then receive an email invitation to take part in the SST pilot and will choose from a range of time-slots and Pearson VUE test centres across the United Kingdom. Health Education England (HEE) will offer support during the SST pilot by assigning a staff member centrally within GPNRO to respond to applicant queries specifically related to the pilot and to manage the reimbursement of travel costs. This frequently asked questions document will also be updated regularly to support any queries, and the applicant guide will include relevant information about the pilot. 5. How does this approach differ to the Machine Marked Test project that was piloted in 2010? In 2010 a Machine Marked Test was piloted where the majority of applicants were from General Practice (GP) and other applicants were mainly from the larger specialties. For the pilot in 2014, we will NOT manually machine mark the tests and we aim to collaborate with all specialties entering specialty training at ST1/CT1 level. SST CPS and SJT questions will be also piloted into the GP test as a proof of concept, which will be marked, but will not contribute to the selection score. Page 11 of 18

We also aim to increase the number of applicants sitting the SST pilot in order to make useful analysis of data obtained across the specialties and to develop a test which has face validity for applicants from all specialties. 6. What are the projected benefits associated with implementing this SST pilot? By implementing this SST pilot for ST1/CT1 applicants we aim to further ensure that specialty trainees have the knowledge required to apply clinical practice and to respond appropriately to patient needs. Without a coherent approach, there is likely to be a plethora of small costly alternative approaches developed in every speciality. The pilot aims to add a valid, reliable, cost-effective selection tool to national recruitment and selection. Each of the devolved nations is focused on a National Health Services (NHS) objective to recruit staff into the NHS with appropriate values and behaviours, as well as knowledge and skills. The Specialty Selection Test will support the need to incorporate values-based selection methods and is a possible way of meeting these requirements. 7. If the SST pilot is deemed successful will shortlisting become a mandatory part of the process for all specialties? The results of the pilot will guide specialties on whether or how the SST may be used to improve the current selection methods used. All specialties and recruiting organisations will have the opportunity to consider the results obtained from the pilot and will discuss the implications with respect to shortlisting and the selection process, with the possibility of introducing this selection method into Specialty Training. Page 12 of 18

8. How will the lessons learnt from running the SJT for selection into the 2012 Foundation Programme be incorporated into this pilot? A full analysis of the Situational Judgement Test (SJT) for selection into the 2012 foundation programme found that the issues were related to the scanning of the test mark sheets, rather than the test itself. The pilot in 2014 will not use this manual approach to machine mark the SST, but will use the same tried and tested methods of computer-based delivery that have been used for General Practice (GP) recruitment since 2007 and by many royal colleges for their speciality examinations. 9. Which specialties are involved in the SST pilot? A detailed list will be made available shortly. Discussions have taken place with Medical Royal Colleges to be involved in and support the SST pilot, and most have confirmed their support and willingness for their applicants to be involved in this pilot evaluation. 10. Will the SST pilot be mandatory for those specialties who agree to participate? Each specialty will decide whether or not participation in the SST pilot will be mandatory, based on the likely number of applications for the specialty and the number of applications required to obtain meaningful data. Within General Practice (GP), the necessary test items will be embedded within the existing computer delivered test, so 7000 of the 12,000 applicants to ST1/CT1 will already be enrolled in the SST pilot. 11. Why is the SST pilot across specialties and not specialty specific? In specialties that have piloted or implemented similar tests there are generally good correlations between scores and interview scores and also the likelihood of success in professional exams (where this has been studied). The aim is to explore whether a standardised tool that would complement speciality specific recruitment can add value to Page 13 of 18

current shortlisting and selection processes. It would complement, not replace, the assessment of individual suitability for a given specialty or group of specialties. The pilot will look at the evidence that the test can predict performance in each speciality, and explore the face validity of the questions. 12. How will the SST pilot be run and how will the test be marked? We plan to pilot the SST during 13-25 January 2014, at Pearson VUE test centres across the UK. The actual assessment will take approximately two hours and fifteen minutes and the computer based questions will be coordinated and marked by Pearson VUE, the company who currently manages the General Practice (GP) selection test. 13. Who will write the questions for the SST pilot? There is already a substantial item bank of questions from which the Specialty Selection Tests will be selected. The blueprint for the CPS test will be based on Foundation Programme Curriculum. The SJT test will comprise of a bank of questions that relate to specific behaviours and values common to all doctors, across all specialties and an engagement. 14. How many questions will be developed for the SST pilot? A paper is currently being developed to assess this requirement. There is already a substantial pool of tested questions that can be used for the SST pilot that will be reviewed in order to improve and subsequently determine their suitability. New items will be generated to fill gaps identified in the assessment blueprint. Page 14 of 18

15. Who will be eligible to sit the SST pilot? Doctors applying for any specialty training post at ST1/CT1 level are eligible to sit the SST pilot and we will encourage as many applicants as possible to take part. Applicants who apply only to academic posts will not be involved at the pilot at this stage. Applicants who apply to GP specialty training and Broad Based Training (BBT) will sit SST test items as part of the existing stage 2 assessment for entry into GP training. The questions from the SST pilot will be embedded into the existing GP test, but will not count towards the applicants score in the selection process. This will allow us to study how well the multispecialty questions perform compared to items in the GP test. 16. Will applicants who apply for both GP and another specialty be required to sit both tests? Those applicants who apply to General Practice (GP) training and Broad Based Training (BBT) will undertake an online CPS and SJT test items as part of the GP selection process and as such, will not need to sit an additional test for the Specialty Selection Test pilot. Applicants only applying to specialties other than GP will be encouraged to sit the test and therefore we will be requesting their participation. Applicants will only need to sit the test once, irrespective of the number of specialties they apply to. 17. Will the SST pilot test results form part of the scoring for 2014? The SST results from the pilot project will NOT form part of the scoring for applicants in 2014. This will be made clear in all communications. The results will only be used by the investigators who will compare the scores achieved during the selection process for the applicant s chosen training programme. Every applicant involved will receive feedback on how they performed in the SST. Page 15 of 18

18. Who is leading on the pilot? The pilot is part of the overarching Medical and Dental Recruitment and Selection (MDRS) Programme, which is governed by Health Education England, on behalf of our wider stakeholders across the United Kingdom. The quality and standards workstream, which sits under the governance structure of the Medical and Dental Recruitment and Selection (MDRS) Programme, has developed the business case and the procurement strategy for the pilot. This group will oversee the development and review the final results before a paper is submitted to the executive, programme, and steering board/s on the project's final outcome. 19. Is the quality and standards workstream still working on other objectives? Yes. The quality and standards workstream will continue to work on the other remaining objectives that have been tasked to this group. These objectives include: assessing the ways in which raters are trained across the recruiting organisations; and improving the quality of selection processes in all specialities. 20. What framework is being applied to the SST pilot? The SST pilot will be overseen by a sub-group of the quality and standards group, which comprises of a group of stakeholders across the Royal Colleges, Local Educational Training Boards, British Medical Association, Health Education England, the Work Psychology Group and University College London. Project tasks and actions will be driven by this working group and they will ensure that the pilot is on track to achieve the projected outcomes. Page 16 of 18

21. Will travel costs be reimbursed? Yes, travel costs will be reimbursed for applicants by Health Education England as part of the SST pilot project. This cost has been capped at an amount of 10.00 due to the wide range of Pearson VUE centre locations and an assessment of the travel reimbursement costs for GPNRO. Recruiting organisations will be asked to submit any reimbursement claims received to the dedicated support person, who will then manage the process. 22. Our organisation doesn't shortlist; will we be expected to for the SST pilot? You do not need to shortlist in order for applicants to take part in the SST pilot. If you are a recruiting organisation, we will ask you to send a list of the candidates who have applied to a central email address so that we can arrange for the applicants to attend the assessment centre and sit the test. 23. Will support be available to respond to applicant queries about the SST pilot? Health Education England (HEE) will offer support during the SST pilot by assigning a staff member centrally within the Directorate of Education and Quality (DEQ) to respond to applicant queries specifically related to the pilot. This member of staff will sit within the GPNRO team and will manage travel reimbursement for applicants. Frequently Asked Questions will also be made available to support any queries on the Specialty Training website, and the applicant guide will include relevant information about the pilot. 24. Who can I contact for more information? You can contact the MDRS team at hee.mdrs@nhs.net for any queries relating to the SST pilot and they will redirect you to a member of the SST working group. We will aim to respond to your query as soon as possible. Page 17 of 18

Appendix 1 A Specialty Selection Test is being piloted for entry into Specialty Training As part of the Medical and Dental Recruitment and Selection (MDRS) Programme, Health Education England (HEE) is working with the devolved nations, the Work Psychology Group and academics in the University College of London Medical School to pilot a selection test that includes a Clinical Problem Solving Test (CPS) and a Situational Judgement Test (SJT), for entry into Specialty Training. Known as the Specialty Selection Test (SST) pilot, this will be piloted between 13-25 January 2014 for applicants to specialty training at ST1/CT1 level. The results from this pilot will enable a group of leading selection researchers to consider any advantages of introducing this test and how the results correlate with the current selection methods used. We will share these results widely with stakeholders and continue to follow applicant progression for a number of years for a long-term evaluation. Ultimately, we are assessing whether the SST is a good predictor of a doctors' performance throughout their training and to ensure the selection process is reliable, valid, fair and cost-effective. Similar tests and pilots have been used before - currently 7,000 out of the 12,000 doctors applying for specialty training take a CPS and a SJT 1. Compared to other selection methods, these tests have been shown to be the best predictor of performance in the work place and likelihood of successful completion of specialty training. The National Health Service (NHS) needs to recruit staff into the NHS with appropriate values and behaviours, as well as knowledge and skills. This principle has been incorporated into NHS mandates and organisation objectives. The Specialty Selection Test is a values-based selection method and is a possible way of meeting these requirements. Applicants will be asked for their views after completing the test on how appropriate they find the selection test in relation to selection into specialty training. Discussions have taken place with Medical Royal Colleges and most have confirmed they wish their applicants to be involved in this SST pilot. Applicants taking part in the pilot will sit the test only once, irrespective of the number of specialties they apply to. All identifying information will be kept confidential. The results will not form part of the selection process and will not be made available to selectors - it is a pilot only. Input from applicants is crucial and results will inform us about whether these selection methods add additional value to the selection processes currently used. All eligible applicants will be contacted by a company called Pearson VUE, once they have applied to the specialty, and will be able to choose a test centre that is most convenient to them. This will be managed separately to the online application process for recruitment into ST1/CT1 level. As a member of a Local Education Training Board (LETB), Deanery or Royal College recruiting organisation, you will receive further, specific, information regarding how this process will be managed. HEE will offer support during the SST pilot from a staff member centrally within the Directorate of Education and Quality (DEQ) who will respond to applicant queries specifically related to the pilot. Frequently Asked Questions will also be available and the applicant guide will include relevant information about the pilot. For more information, please contact the team via email at: hee.mdrs@nhs.net. 1 General Practice (GP) and Public Health use a CPS and a SJT test for specialty selection. Specialty specific tests have also been piloted in surgery, acute specialties, medicine and other specialties. The test in GP was successfully implemented in 2007 and used to date as part of the selection process; constituting 30% of a trainee doctor's total recruitment score. Page 18 of 18