Characteristics of Clinical Clerkships

Transcription

1 N a t i o n a l B o a r d o f M e d i c a l E x a m i n e r s Characteristics of Clinical Clerkships Fall 2014

2 Table of Contents Summary... 3 Table 1. Unweighted and Weighted Clerkship Responses... 5 Chart 1. US LCME Medical Schools Use of the NBME Clinical Subject Examinations for Years and Table 2. Count of Type of Medical Schools Using the NBME Clinical Subject Examinations for Years and Chart 2. Clerkships with Traditional Curricula... 8 Table 3. Clerkship Lengths... 9 Table 4. Number of Scheduled Rotations per Year Table 5. Start Months for Rotations Table 6. Academic Purposes for Using the NBME Subject Exam Chart 3. Match Between NBME Subject Examination and School Curriculum Chart 4. Clerkships Requiring Students to Pass NBME Subject Exam Table 7. Methods for Determining Passing Score Chart 5. Average Minimum Scores for Passing During the Academic Year Table 8. Weight Given to the Subject Test Score for Final Course/Clerkship Grade Chart 6. How Subject Test Scores are used to Determine Honors-level Performance Chart 7. Average Minimum Scores for Honors during the Academic Year Chart 8. Use of Computer Simulations in Curriculum Copyright 2015, National Board of Medical Examiners (NBME ). All rights reserved. Characteristics of Clinical Clerkships 2

3 Summary The NBME offers standardized, objective, discipline-based, and comprehensive (inter-disciplinary) basic and clinical science examinations for use in assessment throughout the medical school curriculum. Examinations are developed and reviewed by nationally representative committees and content experts following the same rigorous protocol used to build Step 1 and Step 2 of the United States Medical Licensing Examination (USMLE). Subject examinations are designed to provide institutions with effective evaluation tools and useful examinee performance data that can be compared with nationally representative groups of students at the same stage of training. The NBME Subject Examination Program currently provides basic and clinical science subject examinations to 138 US medical schools accredited by the Liaison Committee on Medical Education (LCME ), 12 Canadian medical schools, 14 osteopathic medical schools, and 39 international medical schools. In the Fall of 2012, the National Board of Medical Examiners ed Executive Chief Proctors at 222 medical schools and satellite campuses asking them to forward an invitation to complete an on-line questionnaire to the clerkship directors in Clinical Neurology, Family Medicine, Medicine, Obstetrics and Gynecology, Pediatrics, Psychiatry and Surgery. The purpose of the survey was to gather descriptive information about clerkship characteristics and use of computer case simulations. Data gathered for each clerkship included type and length of clerkship rotations, curriculum type for the academic year, purpose of the exam, and for the academic year, procedures used to determine minimum passing scores, minimum passing score and the score required to achieve Honors. For the Family Medicine Clerkships, these questions related to the use of the Family Medicine Modular examination only. Ultimately, 675 clerkships across the disciplines completed the questionnaire for an approximate response rate of 43% overall. Responses were weighted to compensate for nonresponse and to allow for generalization to a larger population of clerkships (Table 1). For example, Clinical Neurology clerkships accounted for 10.8% of all respondents. Without weighting the data, Clinical Neurology clerkships would be underrepresented in any statistics that were computed using the entire dataset. Weighting the responses allows for each clerkship to have equal representation in summary statistics. The population of clerkships described in the following pages consists of Clinical Neurology, Family Medicine, Medicine, Obstetrics and Gynecology, Pediatrics, Psychiatry and Surgery clerkships at medical schools and satellite campuses that currently have at least one clerkship which uses an NBME Subject Examination. The 222 respondents in the population of generalization are a mix of US LCME-accredited, allopathic medical schools and their satellite campuses (70%), Osteopathic medical schools and their satellite campuses (17%), international medical schools (9%) and Canadian medical schools (5%). Chart 1 shows the distribution of US LCME medical schools that used the NBME Clinical Subject Examinations in years and and Table 2 shows the distribution by categories of medical schools (US LCME, international, osteopathic and Canadian) using the Clinical Subject Examinations for those same time periods. Overall, 88% of clerkships follow a traditional curriculum (Chart 2). Obstetrics/Gynecology and Pediatrics clerkships were most likely to have a traditional curriculum (95%) and Clinical Neurology clerkships were the Characteristics of Clinical Clerkships 3

4 least likely (81%). The length of the clerkships for the academic year varied greatly (Table 3). Almost all Clinical Neurology clerkships (97%) reported clerkship lengths of 3 4 weeks. Except for Medicine clerkships (32% indicating 12-week clerkship length), most rotation lengths were between 4 and 8 weeks in length (Table 3). Unsurprisingly, those clerkships which had shorter clerkship lengths were likely to have more scheduled rotations in a given year than those clerkships with longer clerkship lengths (Table 4). Slightly less than half (48%) of all clerkships began the first rotation of the academic year in July 2013 (Table 5). June 2013 and May 2013 were the next most common months for beginning rotations. The most common academic reason (76%) for using the NBME Clinical Subject Examinations is for course or clerkship assessment (Table 6). Other common reasons include practice for the USMLE (37%) and evaluation of mastery or minimal competence (37%). Respondents were asked to rate how well the discipline-specific NBME Subject Examination matched the school curriculum using a scale of 1 (matches poorly) to 5 (matches very well). As shown in Chart 3, the overall average match was 3.6 (95% CI: ). Approximately 95% of clerkships require students to pass the NBME Subject Exam (Chart 4). The method for determining the passing score varies (Table 7), but is most likely to be via norms (based on academic year or reference group) using percentile ranks (31%) and quarterly norms using percentile ranks (28%). The average of the passing scores used by the clerkships ranged from 59 (95%CI: ) (Surgery) to 65 (95% CI: ) (Psychiatry) with an overall average minimum passing score of 61.9 (95% CI: ). See Chart 5. The Subject Exam Score counted for 20 30% of the final course or clerkship grade for about one-third of all clerkships (Table 8). Use of the NBME Subject Exam scores for determining honors-level performance ranged from a low of 22 % for Family Medicine Clerkships using the Family Medicine Modular exam to a high of 46% for Psychiatry Clerkships (Chart 6). With the exception of Family Medicine clerkships, clerkships were about twice as likely to use student scores than student ranks in determining honors-level performance (Chart 6). Family Medicine Clerkships giving the Family Medicine modular exam were about equally likely to use either method. Family Medicine Clerkships that used the Family Medicine modular exam also reported the lowest average score for awarding honors-level performance of 74 (95% CI: ) while Psychiatry Clerkships reported the highest average score of 85 (95%CI: ) (Chart 7). A section of the questionnaire asked about computer simulation use in the curriculum (Chart 8). Pediatric clerkships were the most likely to use computer simulations (56%) and Psychiatry was the least likely (12%). Additional analyses on computer simulation were presented in a poster at the 2013 Research in Medical Education Conference (RIME). Future NBME research endeavors of data collection of Clerkship characteristics will facilitate the continuous improvement of the NBME Subject Examination Program. Characteristics of Clinical Clerkships 4

5 Table 1 Unweighted and Weighted Clerkship Responses Unweighted Responses Clerkship Frequency Percent Response Rate Weighted Responses Frequency Percent Clinical Neurology Family Medicine Medicine Ob/Gyn Pediatrics Psychiatry Surgery Total Characteristics of Clinical Clerkships 5

6 Examinations Chart 1 - US LCME Medical Schools Use of the NBME Clinical Subject Examinations for Years and Clinical Neurology Family Medicine Family Medicine Modular Medicine OB/Gyn Pediatrics Psychiatry Surgery Number of Schools Characteristics of Clinical Clerkships 6

7 Table 2 Count of Type of Medical Schools Using the NBME Clinical Subject Examinations for Years and NBME Subject Exam Years Medical School Type US LCME Canadian Osteopathic International Total Number of Schools Clinical Neurology Family Medicine Family Medicine Modular Medicine Ob/Gyn Pediatrics Psychiatry Surgery Characteristics of Clinical Clerkships 7

8 Chart 2 - Clerkships with Traditional Curricula Note: Standard errors range from 1.3% to 3.2%. Characteristics of Clinical Clerkships 8

9 Table 3 Clerkship Lengths Clerkship (percent of clerkships) Length Clinical Neurology Family Medicine Medicine Ob/ Gyn Pediatrics Psychiatry Surgery 3 Weeks Weeks Weeks Weeks Weeks Weeks Weeks Weeks Weeks Weeks Weeks Note: Standard errors range from 0.5% to 7.6%. Characteristics of Clinical Clerkships 9

10 Table 4 Number of Scheduled Rotations per Year Clerkship (percent of clerkships) Number of Rotations Clinical Neurology Family Medicine Medicine Ob/ Gyn Pediatrics Psychiatry Surgery 2 or fewer or more Note: Standard errors range from 0.9% to 7.5%. Characteristics of Clinical Clerkships 10

11 Table 5 Start Months for Rotations Clerkship (percent of clerkships) Start Month Clinical Neurology Family Medicine Medicine Ob/ Gyn Pediatrics Psychiatry Surgery Mean Jan Mar Apr May Jun Jul Aug Sep Oct Nov Dec Note: Standard errors range from 2.1% to 8.6%. Characteristics of Clinical Clerkships 11

12 Table 6 Academic Purposes for Using the NBME Subject Exam Clerkship (percent of clerkships) Purpose Course /clerkship assessment Practice for USMLE Mastery/ minimal competency Student Self- Assessment Identifying at-risk students Promotion requirement End-of-year assessment Make-up/ remediation Part of review course Advanced placement Clinical Neurology Family Medicine Medicine Ob/ Gyn Pediatrics Psychiatry Surgery Mean Other Note: Standard errors range from 0.5% to 8.2%. Characteristics of Clinical Clerkships 12

13 Chart 3 - Match Between NBME Subject Examination and School Curriculum Rating Scale: 1=Match Poorly 3=Matched Somewhat 5=Matched Very Well Note: Standard errors range from.04 to.11. Characteristics of Clinical Clerkships 13

14 Chart 4 - Clerkships Requiring Students to Pass NBME Subject Exam Note: Standard errors range from 1.2% to 2.7%. Characteristics of Clinical Clerkships 14

15 Table 7 Methods for Determining Passing Score Clerkship (percent of clerkships) Method Clinical Neurology Family Medicine Medicine Ob/ Gyn Pediatrics Psychiatry Surgery Norms (based on academic year or ref group), using percentile rank Quarterly Norms, using percentile rank Grading Guidelines provided by NBME Sub. Exam Prog Norms (based on academic year or ref group), (e.g., 1 std dev below nat'l avg score) Scaled Scores Other Quarterly Norms using means (e.g., 1 std dev below the quarterly average score) Note: Standard errors range from 0.7% to 9.9%. Characteristics of Clinical Clerkships 15

16 Chart 5 - Average Minimum Scores for Passing During the Academic Year Scaled scores have a mean of 70 and a standard deviation of 8. Note: Standard errors range from 0.20% to.80%. Characteristics of Clinical Clerkships 16

17 Table 8 Weight Given to the Subject Test Score for Final Course/Clerkship Grade Clerkship (percent of clerkships) Weight Clinical Neurology Family Medicine Medicine Ob/ Gyn Pediatrics Psychiatry Surgery 1-10% % % % % % % % or more Note: Standard errors range from 0.6% to 9.5%. Characteristics of Clinical Clerkships 17

18 Chart 6 - How Subject Test Scores are used to Determine Honors-level Performance Note: Standard errors range from 2.1% to 7.1%. Characteristics of Clinical Clerkships 18

19 Chart 7 - Average Minimum Scores for Honors during the Academic Year Scaled scores have a mean of 70 and a standard deviation of 8. Note: Standard errors range from 4.6% to 12.4%. Characteristics of Clinical Clerkships 19

20 Chart 8 - Use of Computer Simulations in Curriculum Note: Standard errors range from 2.3% to 7.6%. Characteristics of Clinical Clerkships 20