Reporting tool user guide

Transcription

1 Outcome reports: Pass rates in postgraduate examinations Reporting tool user guide March 2015 Contents Introduction... 2 Using the reports... 3 Entering the online data tool: Understanding the reports... 3 Postgraduate progression reports... 3 Undergraduate progression reports... 3 Using the reports... 4 Understanding the charts... 5 Understanding Pass rate... 5 Interpreting the reports... 5 Annex: Additional Notes and Data Definitions... 7 Who is represented within this data set?... 7 What examinations are covered by the data set?... 7 Is any data suppressed in the reporting tool?... 7 Definitions of data items and source... 8

2 Introduction Throughout their training doctors undertake a wide variety of assessments including examinations which test their knowledge, skills, attitudes and behaviours against the requirements set out in the training programme curricula. Medical Royal Colleges and Faculties (colleges and faculties) design the curricula and assessments systems which doctors must achieve during their specialty training. These are approved by the GMC. This document provides an overview of the reporting tool, developed in collaboration with colleges and faculties, through which we have published data on doctors exam pass rates. It can be used in conjunction with other progression reports we have published looking at how successful doctors are in their annual appraisal and in which area of medicine doctors enter the GP or specialist register. These reports highlight variation in the progress of different cohorts of doctors through training and are intended to drive improvement in the quality of medical education and training. 2

3 Using the reports The reports can be accessed through the GMC website There are two sets of reports showing exam outcomes; one reporting the pass rates by where the doctor completed their undergraduate medical degree, known as their Primary Medical Qualification (PMQ), and the second split by the doctor s postgraduate body which includes by Deanery as well as the college or faculty setting their exam. Within the postgraduate report, performance is also shown by demographic factors, including age group, gender and ethnic group. Entering the online data tool: Understanding the reports When you enter the reporting tool from the GMC website you are shown a page called Understanding the Reports. The purpose of this page is to provide basic definitions and links to further documentation. You can return to this page easily from either reporting tool: From the postgraduate reporting tool this is done via the Main Menu button which is accessible from all reports in this reporting tool. From the undergraduate reporting tool select Understanding the reports from the report. Postgraduate progression reports From the Understanding the Reports page in the postgraduate reporting tool select Main menu for a list of the different reports within the tool and select the one you wish to view. The options are: Pass rate by deanery Pass rate by deanery by diet Pass rates by place of primary medical qualification and ethnic group Pass rates by gender and age group Undergraduate progression reports From the Understanding the Reports page in the undergraduate reporting tool select continue to report to view exam pass rates by primary medical qualification awarding body. 3

4 Using the reports The filtering and navigation buttons are similar across all the reports within this tool. In our example in Figure 1 the location of the filter options, navigation buttons and benchmark have been highlighted. The filter options change according to the report selected; in this example you can filter by College / Faculty, Exam Name, PMQ World Region and the year of the exam sitting. Navigation buttons in the bottom left of each report page allow users to return to the main menu or view the data in a tabular format Figure 1 Pass rates by deanery 4

5 Understanding the charts In our Figure 1 example the pass rate for each deanery is shown against the pass rate for candidates across all deaneries (the benchmark pass rate). The deanery pass rate and its associated confidence interval are shown as a dot with a horizontal line cutting through it. The benchmark pass rate and its associated confidence interval 1 are represented by the vertical purple line and band. Understanding Pass rate The pass rate for a cohort represents the proportion of passes for all attempts within the specified period. The number of attempts can exceed the number of candidates for two reasons. Firstly, an individual may take several attempts to pass the same exam within a reporting period. Secondly a candidate could sit more than one exam during a year. If, for example, a candidate had taken MRCP Part 1 and Part 2 in the same year and you had set the filter options to show all exams from the Royal Colleges of Physicians then both attempts by the one candidate would be included in the results. Interpreting the reports The reports compare the pass rates of a reporting group, such as the pass rate for candidates from a single deanery, against the benchmark group, the pass rate for candidates across all deaneries. In addition to the display of pass rates for individual groups, a confidence interval is displayed to assist contextual interpretation. The 95% confidence interval of the reporting group is displayed visually in Figure 2 as a horizontal band extending either side of the pass rate. Because many users will wish to use the reports as a measure of the underlying success rate of different groups in passing exams, confidence intervals have been used to account for sampling uncertainty across time. Groups with a small number of cases will have wider confidence intervals due to a greater level of uncertainty. An outlier is defined when the 95% confidence interval of reporting group does not intersect with 95% confidence interval of the benchmark group. Figure 2 An example showing the display of outliers 5

6 Above outlier (1): An above outlier is any set of values where the lower 95% confidence limit for the given cohort is above the upper 95% confidence limit of the benchmark group s mean. They are coloured purple. Non-outlier (2): Although the mean score of a non-outlier may be above or below the benchmark mean, the confidence intervals of these two groups intersect. For these groups, it is not clear if the difference between the cohort pass rate and that of the benchmark is generally representative of differing performance. They are coloured teal. Below outlier (3): A below outlier is any set of values where the upper 95% confidence limit for the given cohort is below the lower 95% confidence limit of the benchmark group s mean. They are coloured navy The confidence intervals have been calculated using the recommended method for proportions in Altman et al. (2000) Statistics with Confidence. This approximates a 95 per cent confidence interval based on a Binomial distribution. Altman, DG, Machin, D, Bryant, TN, et al. (2000), Statistics with Confidence, 2nd ed., BMJ Books, London. 6

7 Annex: Additional Notes and Data Definitions Who is represented within this data set? These reports only contain outcomes for candidates registered with the GMC. Only doctors registered with the GMC are able to practice in the UK and will include doctors in a training programme in the UK. GMC registration is not a requirement for all royal college and faculty exams. Some colleges such as the Royal Colleges of Physicians, allow doctors practicing overseas to take their exams, and others, for example the Faculty of Public Health examinations, are open to candidates from a range of healthcare disciplines, not limited to doctors. As a result the pass rates in our reports are for doctors registered to practice in the UK and the average pass rate for the exam as a whole may be different. What examinations are covered by the data set? These reports cover examinations taken by doctors in specialty training programmes including core and higher specialty training as well as run-through training programmes. The examinations are complex often with multiple components within an exam and multiple papers within each component. This report does not attempt to compare pass rates across different examinations. Different colleges and faculties report pass rates at different levels; some have provided pass rates for the overall exam only whilst others have provided pass rates for each individual component. For example: The Royal College of Obstetricians and Gynaecologists Part 2 MRCOG exam consists of a written examination followed by an oral assessment for those candidates who are successful in the written examination. Pass rates in the reporting tool are based on candidates who passed both parts of the Part 2 MRCOG exam. The structure of examinations varies significantly dependent on the knowledge, skills, behaviours or attitudes it is testing and the time at which it is taken during the training programme. Some exams can be taken only by doctors in a training programme and others can be taken by any doctor in UK practice. Each will be subject to bespoke requirements around the number of attempts allowed, maximum or minimum time periods for passing multiple components, and standard setting methodology. Is any data suppressed in the reporting tool? Cohorts made up of only a small number of distinct individuals are not displayed in the reports. This can mean that detail can be limited for small specialties or when multiple filters are applied in combination. Cohorts with a pass rate of zero percent are not displayed. 7

8 Definitions of data items and source Variable Source Notes Pass rate Medical Royal Colleges and Faculties The pass rates for cohorts are calculated using row level candidate data submitted by the colleges and faculties. Exam year Medical Royal Colleges and Faculties The report presents exam pass rates for all attempts within an academic year for example August 2013 July In the filter this is labelled as the year in which the academic year ends, i.e in our example above. Deanery / LETB National Training Survey (NTS) A candidate s deanery is taken from the annual National Training Survey census*; in 2014 the census date was 26 March This is correct at a point in time. There may be a small proportion of cases in which this does not represent the deanery the candidate was in at the time of sitting an exam. PMQ Awarding Body *Please note: the 2014 NTS records Deanery rather than LETB for doctors in training. This represents the awarding body of the medical degree held by a doctor. In most cases this is the same as the medical school attended by the doctor, but as a result of changes over time to institutions being granted degree-awarding powers it may be different. The University of London for example, represents students from different medical schools across London. Therefore, where London schools are shown, the data is not necessarily representative of their entire population. Students have the opportunity to self-select University of London or their respective school as their PMQ awarding body. 8

9 PMQ World Region Ethnic Group Diet Gender Age Group NTS This is a grouping to show in which part of the world a doctor gained their undergraduate medical degree (also known as Primary Medical Qualification or PMQ). It is based on broad geographical regions: UK (United Kingdom), EEA (European Economic Area excluding UK) and IMG (International Medical Graduates). Ethnicity is primarily taken from the National Training Survey. Where values are not declared in the NTS, the Medical is used as an alternative source. Where values remain undefined, they are grouped into the category of Not known. Diet is a term used to describe an exam sitting within the year. Some exams have multiple diets or exam sittings within a single year. Others have only one. Diets have been grouped into Autumn, Spring and Summer. We have assigned the following dates to each time period: Autumn 1st August to 31st December Spring 1 st January to 31 st March Summer 1 st April to 31 st July The age of an individual represents their calendar age on the date of sitting an exam. This is calculated using the date of birth taken from the medical register. For reporting purposes we have grouped age into broader categories. 9