N a t i o n a l B o a r d o f M e d i c a l E x a m i n e r s Characteristics of Clinical Clerkships Fall 2014
Table of Contents Summary... 3 Table 1. Unweighted and Weighted Clerkship Responses... 5 Chart 1. US LCME Medical Schools Use of the NBME Clinical Subject Examinations for Years 2011-2012 and 2012-2013... 6 Table 2. Count of Type of Medical Schools Using the NBME Clinical Subject Examinations for Years 2011-2012 and 2012-2013... 7 Chart 2. Clerkships with Traditional Curricula... 8 Table 3. Clerkship Lengths... 9 Table 4. Number of Scheduled Rotations per Year... 10 Table 5. Start Months for Rotations... 11 Table 6. Academic Purposes for Using the NBME Subject Exam... 12 Chart 3. Match Between NBME Subject Examination and School Curriculum... 13 Chart 4. Clerkships Requiring Students to Pass NBME Subject Exam... 14 Table 7. Methods for Determining Passing Score... 15 Chart 5. Average Minimum Scores for Passing During the 2011-2012 Academic Year... 16 Table 8. Weight Given to the Subject Test Score for Final Course/Clerkship Grade... 17 Chart 6. How Subject Test Scores are used to Determine Honors-level Performance... 18 Chart 7. Average Minimum Scores for Honors during the 2011-2012 Academic Year... 19 Chart 8. Use of Computer Simulations in Curriculum... 20 Copyright 2015, National Board of Medical Examiners (NBME ). All rights reserved. Characteristics of Clinical Clerkships 2
Summary The NBME offers standardized, objective, discipline-based, and comprehensive (inter-disciplinary) basic and clinical science examinations for use in assessment throughout the medical school curriculum. Examinations are developed and reviewed by nationally representative committees and content experts following the same rigorous protocol used to build Step 1 and Step 2 of the United States Medical Licensing Examination (USMLE). Subject examinations are designed to provide institutions with effective evaluation tools and useful examinee performance data that can be compared with nationally representative groups of students at the same stage of training. The NBME Subject Examination Program currently provides basic and clinical science subject examinations to 138 US medical schools accredited by the Liaison Committee on Medical Education (LCME ), 12 Canadian medical schools, 14 osteopathic medical schools, and 39 international medical schools. In the Fall of 2012, the National Board of Medical Examiners emailed Executive Chief Proctors at 222 medical schools and satellite campuses asking them to forward an invitation to complete an on-line questionnaire to the clerkship directors in Clinical Neurology, Family Medicine, Medicine, Obstetrics and Gynecology, Pediatrics, Psychiatry and Surgery. The purpose of the survey was to gather descriptive information about clerkship characteristics and use of computer case simulations. Data gathered for each clerkship included type and length of clerkship rotations, curriculum type for the 2012-2013 academic year, purpose of the exam, and for the 2011-2012 academic year, procedures used to determine minimum passing scores, minimum passing score and the score required to achieve Honors. For the Family Medicine Clerkships, these questions related to the use of the Family Medicine Modular examination only. Ultimately, 675 clerkships across the disciplines completed the questionnaire for an approximate response rate of 43% overall. Responses were weighted to compensate for nonresponse and to allow for generalization to a larger population of clerkships (Table 1). For example, Clinical Neurology clerkships accounted for 10.8% of all respondents. Without weighting the data, Clinical Neurology clerkships would be underrepresented in any statistics that were computed using the entire dataset. Weighting the responses allows for each clerkship to have equal representation in summary statistics. The population of clerkships described in the following pages consists of Clinical Neurology, Family Medicine, Medicine, Obstetrics and Gynecology, Pediatrics, Psychiatry and Surgery clerkships at medical schools and satellite campuses that currently have at least one clerkship which uses an NBME Subject Examination. The 222 respondents in the population of generalization are a mix of US LCME-accredited, allopathic medical schools and their satellite campuses (70%), Osteopathic medical schools and their satellite campuses (17%), international medical schools (9%) and Canadian medical schools (5%). Chart 1 shows the distribution of US LCME medical schools that used the NBME Clinical Subject Examinations in years 2011-2012 and 2012-2013 and Table 2 shows the distribution by categories of medical schools (US LCME, international, osteopathic and Canadian) using the Clinical Subject Examinations for those same time periods. Overall, 88% of clerkships follow a traditional curriculum (Chart 2). Obstetrics/Gynecology and Pediatrics clerkships were most likely to have a traditional curriculum (95%) and Clinical Neurology clerkships were the Characteristics of Clinical Clerkships 3
least likely (81%). The length of the clerkships for the 2012-2013 academic year varied greatly (Table 3). Almost all Clinical Neurology clerkships (97%) reported clerkship lengths of 3 4 weeks. Except for Medicine clerkships (32% indicating 12-week clerkship length), most rotation lengths were between 4 and 8 weeks in length (Table 3). Unsurprisingly, those clerkships which had shorter clerkship lengths were likely to have more scheduled rotations in a given year than those clerkships with longer clerkship lengths (Table 4). Slightly less than half (48%) of all clerkships began the first rotation of the academic year in July 2013 (Table 5). June 2013 and May 2013 were the next most common months for beginning rotations. The most common academic reason (76%) for using the NBME Clinical Subject Examinations is for course or clerkship assessment (Table 6). Other common reasons include practice for the USMLE (37%) and evaluation of mastery or minimal competence (37%). Respondents were asked to rate how well the discipline-specific NBME Subject Examination matched the school curriculum using a scale of 1 (matches poorly) to 5 (matches very well). As shown in Chart 3, the overall average match was 3.6 (95% CI: 3.54-3.64). Approximately 95% of clerkships require students to pass the NBME Subject Exam (Chart 4). The method for determining the passing score varies (Table 7), but is most likely to be via norms (based on academic year or reference group) using percentile ranks (31%) and quarterly norms using percentile ranks (28%). The average of the 2011-2012 passing scores used by the clerkships ranged from 59 (95%CI: 58.44-59.55) (Surgery) to 65 (95% CI: 63.57-66.70) (Psychiatry) with an overall average minimum passing score of 61.9 (95% CI: 61.6 62.3). See Chart 5. The Subject Exam Score counted for 20 30% of the final course or clerkship grade for about one-third of all clerkships (Table 8). Use of the NBME Subject Exam scores for determining honors-level performance ranged from a low of 22 % for Family Medicine Clerkships using the Family Medicine Modular exam to a high of 46% for Psychiatry Clerkships (Chart 6). With the exception of Family Medicine clerkships, clerkships were about twice as likely to use student scores than student ranks in determining honors-level performance (Chart 6). Family Medicine Clerkships giving the Family Medicine modular exam were about equally likely to use either method. Family Medicine Clerkships that used the Family Medicine modular exam also reported the lowest average score for awarding honors-level performance of 74 (95% CI: 72.70-75.20) while Psychiatry Clerkships reported the highest average score of 85 (95%CI: 83.93-85.88) (Chart 7). A section of the questionnaire asked about computer simulation use in the curriculum (Chart 8). Pediatric clerkships were the most likely to use computer simulations (56%) and Psychiatry was the least likely (12%). Additional analyses on computer simulation were presented in a poster at the 2013 Research in Medical Education Conference (RIME). Future NBME research endeavors of data collection of Clerkship characteristics will facilitate the continuous improvement of the NBME Subject Examination Program. Characteristics of Clinical Clerkships 4
Table 1 Unweighted and Weighted Clerkship Responses Unweighted Responses Clerkship Frequency Percent Response Rate Weighted Responses Frequency Percent Clinical Neurology 73 10.8 32.9 222 14.3 Family Medicine 98 14.5 44.1 222 14.3 Medicine 102 15.1 45.9 222 14.3 Ob/Gyn 95 14.1 42.8 222 14.3 Pediatrics 105 15.6 47.3 222 14.3 Psychiatry 97 14.4 43.7 222 14.3 Surgery 105 15.6 47.3 222 14.3 Total 675 100.0 43.4 1554 100.0 Characteristics of Clinical Clerkships 5
Examinations Chart 1 - US LCME Medical Schools Use of the NBME Clinical Subject Examinations for Years 2011-2012 and 2012-2013 Clinical Neurology 79 88 Family Medicine Family Medicine Modular 38 42 58 57 Medicine OB/Gyn Pediatrics Psychiatry Surgery 110 114 125 126 127 130 125 127 125 126 2011-2012 2012-2013 0 20 40 60 80 100 120 140 Number of Schools Characteristics of Clinical Clerkships 6
Table 2 Count of Type of Medical Schools Using the NBME Clinical Subject Examinations for Years 2011-2012 and 2012-2013 NBME Subject Exam Years Medical School Type US LCME Canadian Osteopathic International Total Number of Schools Clinical Neurology Family Medicine Family Medicine Modular Medicine Ob/Gyn Pediatrics Psychiatry Surgery 2011-2012 79 0 1 6 86 2012-2013 88 0 1 6 95 2011-2012 58 1 9 5 73 2012-2013 42 1 3 4 50 2011-2012 38 0 3 2 43 2012-2013 57 0 4 7 68 2011-2012 125 8 9 16 158 2012-2013 126 8 5 19 158 2011-2012 127 7 8 14 156 2012-2013 130 8 5 19 162 2011-2012 110 4 8 15 137 2012-2013 114 5 5 19 143 2011-2012 125 7 9 11 152 2012-2013 127 7 5 15 154 2011-2012 125 8 9 14 156 2012-2013 126 9 5 18 158 Characteristics of Clinical Clerkships 7
Chart 2 - Clerkships with Traditional Curricula Note: Standard errors range from 1.3% to 3.2%. Characteristics of Clinical Clerkships 8
Table 3 Clerkship Lengths Clerkship (percent of clerkships) Length Clinical Neurology Family Medicine Medicine Ob/ Gyn Pediatrics Psychiatry Surgery 3 Weeks 17 2 0 0 0 1 0 4 Weeks 80 31 0 23 17 41 8 5 Weeks 2 4 2 3 4 5 1 6 Weeks 2 30 2 58 39 45 10 7 Weeks 0 0 0 0 3 2 4 8 Weeks 0 32 49 15 35 7 54 9 Weeks 0 0 1 0 1 0 2 10 Weeks 0 0 9 0 0 0 3 11 Weeks 0 0 4 0 0 0 2 12 Weeks 0 1 32 0 0 0 17 13+ Weeks 0 1 1 1 1 0 1 Note: Standard errors range from 0.5% to 7.6%. Characteristics of Clinical Clerkships 9
Table 4 Number of Scheduled Rotations per Year Clerkship (percent of clerkships) Number of Rotations Clinical Neurology Family Medicine Medicine Ob/ Gyn Pediatrics Psychiatry Surgery 2 or fewer 4 14 1 0 1 0 8 3 8 4 4 0 1 2 0 4 0 3 34 4 4 0 14 5 1 4 5 2 2 3 2 6 10 18 27 18 32 16 47 7 0 4 3 4 4 3 1 8 6 23 6 48 33 36 9 9 or more 72 30 20 24 24 40 19 Note: Standard errors range from 0.9% to 7.5%. Characteristics of Clinical Clerkships 10
Table 5 Start Months for Rotations Clerkship (percent of clerkships) Start Month Clinical Neurology Family Medicine Medicine Ob/ Gyn Pediatrics Psychiatry Surgery Mean Jan 1 1 2 1 1 0 2 1 Mar 2 2 4 5 4 2 4 3 Apr 2 2 2 2 5 3 3 3 May 13 13 11 12 11 11 11 12 Jun 14 16 17 19 17 16 17 17 Jul 53 47 52 48 50 52 53 51 Aug 7 8 5 6 5 7 6 6 Sep 0 6 3 4 5 2 3 3 Oct 1 0 0 1 1 1 0 1 Nov 2 0 0 0 0 1 0 0 Dec 0 0 0 0 0 0 0 0 Note: Standard errors range from 2.1% to 8.6%. Characteristics of Clinical Clerkships 11
Table 6 Academic Purposes for Using the NBME Subject Exam Clerkship (percent of clerkships) Purpose Course /clerkship assessment Practice for USMLE Mastery/ minimal competency Student Self- Assessment Identifying at-risk students Promotion requirement End-of-year assessment Make-up/ remediation Part of review course Advanced placement Clinical Neurology Family Medicine Medicine Ob/ Gyn Pediatrics Psychiatry Surgery Mean 56 44 87 87 86 90 84 76 26 19 44 47 45 44 35 37 26 15 44 46 28 40 23 32 15 10 38 40 30 32 28 28 8 13 45 27 36 28 30 27 13 13 24 29 39 27 18 23 4 3 25 16 20 14 13 13 6 3 16 20 10 12 9 11 1 4 2 11 2 1 1 3 0 4 2 11 2 1 3 3 Other 2 6 2 3 1 1 1 2 Note: Standard errors range from 0.5% to 8.2%. Characteristics of Clinical Clerkships 12
Chart 3 - Match Between NBME Subject Examination and School Curriculum Rating Scale: 1=Match Poorly 3=Matched Somewhat 5=Matched Very Well Note: Standard errors range from.04 to.11. Characteristics of Clinical Clerkships 13
Chart 4 - Clerkships Requiring Students to Pass NBME Subject Exam Note: Standard errors range from 1.2% to 2.7%. Characteristics of Clinical Clerkships 14
Table 7 Methods for Determining Passing Score Clerkship (percent of clerkships) Method Clinical Neurology Family Medicine Medicine Ob/ Gyn Pediatrics Psychiatry Surgery Norms (based on academic year or ref group), using percentile rank 42 29 30 31 29 36 22 Quarterly Norms, using percentile rank 25 15 40 22 39 20 26 Grading Guidelines provided by NBME Sub. Exam Prog Norms (based on academic year or ref group), (e.g., 1 std dev below nat'l avg score) 28 9 11 15 11 13 10 0 16 8 12 13 19 17 Scaled Scores 3 18 5 3 5 6 14 Other 3 4 5 13 4 5 8 Quarterly Norms using means (e.g., 1 std dev below the quarterly average score) 0 9 2 3 0 1 3 Note: Standard errors range from 0.7% to 9.9%. Characteristics of Clinical Clerkships 15
Chart 5 - Average Minimum Scores for Passing During the 2011-2012 Academic Year Scaled scores have a mean of 70 and a standard deviation of 8. Note: Standard errors range from 0.20% to.80%. Characteristics of Clinical Clerkships 16
Table 8 Weight Given to the Subject Test Score for Final Course/Clerkship Grade Clerkship (percent of clerkships) Weight Clinical Neurology Family Medicine Medicine Ob/ Gyn Pediatrics Psychiatry Surgery 1-10% 6 5 7 7 4 3 4 11-20% 30 24 18 11 20 12 24 21-30% 32 38 33 28 31 27 31 31-40% 17 11 17 25 14 19 17 41-50% 7 7 4 6 4 10 10 51-60% 7 6 2 2 3 10 4 61-70% 0 0 0 11 1 3 0 71% or more 0 9 20 10 23 15 11 Note: Standard errors range from 0.6% to 9.5%. Characteristics of Clinical Clerkships 17
Chart 6 - How Subject Test Scores are used to Determine Honors-level Performance Note: Standard errors range from 2.1% to 7.1%. Characteristics of Clinical Clerkships 18
Chart 7 - Average Minimum Scores for Honors during the 2011-2012 Academic Year Scaled scores have a mean of 70 and a standard deviation of 8. Note: Standard errors range from 4.6% to 12.4%. Characteristics of Clinical Clerkships 19
Chart 8 - Use of Computer Simulations in Curriculum Note: Standard errors range from 2.3% to 7.6%. Characteristics of Clinical Clerkships 20