Development and Validation of a Web-Based Assessment Tool for the Extended Focused Assessment With Sonography in Trauma Examination

ORIGINAL RESEARCH Development and Validation of a Web-Based Assessment Tool for the Extended Focused Assessment With Sonography in Trauma Examination Joshua E. Markowitz, MD, RDMS, James Q. Hwang, MD, RDMS, RDCS, Christopher L. Moore, MD, RDMS, RDCS Article includes CME test Received July 29, 2010, from the Albert Einstein Medical Center, Philadelphia, Pennsylvania USA (J.E.M.); Brigham and Women s Hospital, Boston, Massachusetts USA (J.Q.H.); Yale University School of Medicine, New Haven, Connecticut USA (C.L.M.). Revision requested August 13, 2010. Revised manuscript accepted for publication September 26, 2010. We thank our panel of ultrasound experts who assisted with our initial question development; ultrasound directors Anthony J. Dean, MD, at the University of Pennsylvania and John P. Fojtik, MD, at the Drexel University College of Medicine; and Gregory Luke Larkin, MD, MS, MSPH, at the Yale University School of Medicine and Sharon Griswold- Theodorson, MD, at the Drexel University College of Medicine for assistance with statistical analysis. This study was supported in part by a research grant from the Connecticut College of Emergency Physicians (Dr Markowitz, primary investigator). Portions of this work were previously presented at the Connecticut College of Emergency Physicians Annual Meeting, November 14, 2007, Rocky Hill, Connecticut (research award for best poster presentation); American Institute of Ultrasound in Medicine Annual Convention, March 15, 2008, San Diego, California; Society for Academic Emergency Medicine Annual New England Regional Meeting, April 30, 2008, Shrewsbury, Massachusetts; Drexel Discovery Day, October 15, 2008, Philadelphia, Pennsylvania; and Connecticut College of Emergency Physicians Annual Meeting, October 22, 2008, Rocky Hill, Connecticut. Address correspondence to Joshua E. Markowitz, MD, RDMS, Department of Emergency Medicine, Albert Einstein Medical Center, 5501 Old York St, Korman Building, Suite 104, Philadelphia, PA 19141 USA. E-mail: markowij@einstein.edu Abbreviations ACEP, American College of Emergency Physicians; EFAST, extended focused assessment with sonography in trauma; NBME, National Board of Medical Examiners Objectives Extended focused assessment with sonography in trauma (EFAST) examinations are routinely performed by emergency physicians and general surgeons as an integral part of trauma care. Although guidelines for competency in the EFAST examination exist, tools to assess competency are lacking. Our goal was to develop and validate a Web-based competency assessment tool to evaluate providers who perform the EFAST examination. Methods A multiple-choice test regarding the performance and interpretation of the EFAST examination was developed following National Board of Medical Examiner guidelines. Five emergency physician experts with fellowship training in emergency ultrasound established face and content validity. The test was administered to emergency medicine residents and ultrasound fellows. Concurrent validity was evaluated by assessing the correlation of scores on our test with guidelines set by the American College of Emergency Physicians (ACEP) for emergency ultrasound. Scores were analyzed, and statistical analysis was performed. Results Sixty-three emergency medicine residents and 2 ultrasound fellows from 2 residency programs completed the assessment tool. Examinees who met ACEP guidelines scored significantly higher than those who had not: 70.4% (95% confidence interval, 67.3% 73.4%) versus 48.3% (43.2% 53.5%). Evaluation of scores showed a significant correlation between an increased level of training, participation in an ultrasound rotation, and the number of EFAST and total ultrasound examinations performed with higher test scores. However, overall test scores were lower than initially anticipated. Conclusions Use of this assessment tool for interpretation of EFAST images showed face and content validity. Score trends showed a significant correlation with existing ACEP guidelines, ultrasound experience, and the training level. Scores continued to improve with experience beyond ACEP recommended guidelines. Key Words education; emergency medicine; examination; extended focused assessment with sonography in trauma; ultrasound; validity E mergency ultrasound imaging, performed at the bedside by emergency physicians, is now considered standard practice, is endorsed by all major emergency medicine organizations, and is a required element of residency training. 1 5 The extended focused assessment with sonography in trauma (EFAST) examination is one of the most widely used applications of bedside emergency ultrasound and has expanded over the years to include views of the abdomen, heart, and thorax. 6 The EFAST examination 2011 by the American Institute of Ultrasound in Medicine J Ultrasound Med 2011; 30:371 375 0278-4297/11/$3.50 www.aium.org

is noninvasive, is repeatable, and does not involve ionizing radiation. Use of the EFAST examination has been shown to decrease the time to definitive treatment and improve the efficiency of care. 7 9 Research has shown that it can be performed with limited training, and it has been incorporated into advanced trauma life support guidelines. 10 12 Although the FAST examination as a diagnostic tool is well accepted and used widely, controversy has existed among medical specialties regarding the ensurance of competency to perform the examination, suggesting the need for a validated assessment tool. In 2007, the American Institute of Ultrasound in Medicine and the American College of Emergency Physicians (ACEP) adopted joint guidelines for the performance of the FAST examination, including accepting ACEP s recommendation for training of 25 to 50 FAST examinations. 13,14 Recently, the American Council of Graduate Medical Education removed the number of examinations required for proficiency from residency requirements in favor of requiring the subjective assessment of competency. In February 2008, the Council of Residency Directors in Emergency Medicine devoted a day-long session to developing recommendations on the education and competency assessment in bedside emergency ultrasound. A validated test would provide one standardized method for assessing resident competency. 15 Although there is some research regarding methods of training and the number of examinations required to achieve competency, 16,17 we are not aware of any published objective measures of competency for the EFAST examination. We sought to develop and validate a Web-based multiple-choice test for standardizing the assessment of competency in interpretation and clinical correlation of images from the EFAST examination in emergency situations. Materials and Methods was developed following National Board of Medical Examiners (NBME) guidelines. 18 Areas of the EFAST examination were divided into the 6 major categories: perihepatic, perisplenic, pelvic, pericardial, and thoracic for pneumothorax and pleural fluid. Within each category, questions were developed with the goal of examining competency in assessing normal ultrasound images, images with positive findings, and common pitfalls. On the basis of expert opinions, a mix of questions that included easy, moderate, and difficult were included. Questions included image interpretation as well as incorporation of management decisions. For purposes of data analysis, an absolute pass/fail standard cutoff of 70% was established using the modified Angoff method as described by the NBME. 18 The experts involved in question development established this level as the percentage of borderline examinees who would answer the questions correctly. Still images and video clips were used in most questions. Vignettes were designed to be straightforward without double negatives. Questions could be answered without the need to review the answer choices. All answers were written with a single correct answer choice and 3 or 4 distracters. The online test was designed using Quizmaker 2.1 (Articulate, Inc, New York, NY; Figure 1). Dynamic images were rendered to Shockwave format using Sorensen Squeeze (http://www.sorenson.com). Using a Delphi process, 19 face and content validity was established by submitting the test to a panel of 5 experts with fellowship training in emergency ultrasound to check for overall quality, accuracy, and completeness. Figure 1. Sample extended focused assessment with sonography in trauma (FAST) question from the online test. During the test, the video in this question would play in a continuous loop. Study Design A prospective cross-sectional study was designed to develop and assess the validity of a multiple-choice test compared to ACEP guidelines and other measures of competency currently in use. The study was determined to be exempt from informed consent by the Yale University School of Medicine Human Investigation Committee (Institutional Review Board) and the Brigham and Women s Hospital Human Investigation Committee. Test Development A bank of 41 multiple-choice questions regarding the performance and interpretation of the EFAST examination 372 J Ultrasound Med 2011; 30:371 375

Study Group and Population The test was developed by the primary investigators and administered at the 4-year emergency medicine residency programs at Yale New Haven Hospital and Massachusetts General Hospital/Brigham and Women s Hospital. Residents complete an ultrasound rotation during their first year at Yale New Haven Hospital and in their second year at Massachusetts General Hospital/Brigham and Women s Hospital. Yale New Haven Hospital also offers a 1-year emergency ultrasound fellowship, accepting 2 fellows per year. All residents and fellows were asked to participate on a voluntary basis. Researchers were blinded to the participant s personal information. Study Protocol Participants included postgraduate year 1 through 4 emergency medicine residents and 2 emergency ultrasound fellows in training. Participants were provided with a randomly assigned personal record number by a third party to keep scores anonymous. Ultrasound directors at each institution provided the participants with the actual number of total ultrasound and EFAST examinations performed. Participants then provided data on prior ultrasound experience as part of the test. The test was taken on a computer without assistance. Results were returned to the primary investigator for analysis, who was blinded to the identity of the examinees. Data Analysis Scores were tabulated using Access (Microsoft Corporation, Redmond, WA) and Excel (Microsoft Corporation), and all tests were analyzed for mean scores with 95% confidence intervals. Construct validity was measured by correlating scores with current ACEP guidelines in SPSS (SPSS Inc, Chicago, IL). Results There were 65 participants who completed the test during September and October 2007. Among the participants, there were 12 postgraduate year 4, 11 postgraduate year 3, 16 postgraduate year 2, and 24 postgraduate year 1 residents as well as 2 emergency ultrasound fellows who participated in this project. Results are shown in Table 1. Mean scores (95% confidence intervals) were significantly higher among those who had met the ACEP guidelines (completion of an ultrasound course and at least 25 EFAST examinations) versus those who had not: 70.4% (67.3% 73.4%) versus 48.3% (43.2% 53.5%). Discussion Proficiency in the EFAST examination has typically been measured by the number of examinations performed, with the minimum required numbers varying among different specialty societies. However, numbers alone do not account for variations in individual ability or for differences in the quality and variety of the scanning experiences (eg, number of positive findings, false-positive findings, and abnormal anatomy). This study is the first attempt that we are aware of to create a valid test that assesses competency in understanding and interpreting the EFAST examination and then compares the results to ACEP guidelines as well as the training level, prior ultrasound education, number of EFAST examinations performed, and total number of ultrasound examinations performed. Table 1. Test Results Parameter Mean Score 95% CI n ACEP guidelines No 48.3 43.2 53.5 33 Yes 70.4 67.3 73.4 32 Level of training PGY 1 48.4 42.0 54.8 24 PGY 2 55.6 48.9 62.3 16 PGY 3 67.6 63.0 72.0 11 PGY 4 73.2 68.8 77.5 12 Fellow 86.6 84.2 89.0 2 Rotation (PGY 1 and 2) No rotation 44.0 38.6 49.4 25 Rotation 63.4 59.1 67.7 15 No rotation PGY 1 43.5 36.6 50.4 18 PGY 2 45.3 36.5 54.1 7 Rotation PGY 1 63.0 56.3 69.7 6 PGY 2 63.7 57.9 69.5 9 Total ultrasound examinations performed <100 47.8 42.9 52.8 33 100 200 67.5 64.2 70.8 15 201 300 72.1 66.9 77.2 11 >300 77.2 70.2 84.3 6 Total EFAST examinations performed <25 47.7 42.2 53.3 30 25 50 65.5 61.1 70.0 16 51 75 66.6 62.4 70.8 10 76 100 77.0 69.9 84.1 7 >100 80.5 70.9 90.1 2 ACEP indicates American College of Emergency Physicians; CI, confidence interval; EFAST, extended focused assessment with sonography in trauma; and PGY, postgraduate year. J Ultrasound Med 2011; 30:371 375 373

Simply stated, validity is the ability of a test to measure what it intends to measure. In practice, validity is typically assessed using face validity, content validity, and criterionrelated validity. We established face and content validity through expert review of the questions. By asking emergency ultrasound experts to provide feedback on the questions for the 6 areas of the EFAST examination, we were able to assess the appropriateness and completeness of the content. Criterion-related validity measures the correlation of a test with another external measure. In this case, we measured criterion-related validity by correlating test scores with current accepted measures of competency as outlined by ACEP. Although scores were lower than anticipated, score trends for this test showed a significant correlation with ACEP guidelines as well as with the other criteria. Although the use of ultrasound by physicians at the bedside continues to increase in both the academic and community settings, evaluating the achievement of competency remains challenging and controversial. 20,21 Most guidelines for ensuring competency center around either residency training or performing a certain number of examinations. 1 The assessment tool we have designed showed a significant increase in competency in image interpretation between those who had performed fewer than 25 examinations and those who had performed more. The results also suggest further improvement as the number of ultrasound examinations performed increases. As residencies move away from number requirements and toward competency assessments, a validated test will provide a better method for measuring competency. 4 Although the trends of the scores reflected our expectations, the overall scores were significantly lower than anticipated. This finding may mean that the difficulty level of this test was higher than initially anticipated, but because we are using ACEP guidelines for our reference standard, it is also possible that the guidelines may not be stringent enough. Scores for participants who met ACEP guidelines were significantly higher than those who did not meet the guidelines; however, there was a significant percentage of participants who met ACEP guidelines but did not achieve a passing score of higher than 70%. In addition, the scores continued to improve even beyond 100 EFAST examinations. Further evaluation of these scores showed mean passing rates for only fellows and postgraduate year 4 residents and those who had performed more than 75 EFAST examinations. In 2008, ACEP revised its guidelines for emergency ultrasound, stating that whereas a minimum number of examinations are recommended, proficiency may not always be defined by numerical goals. 1 We agree with this statement, and these results further advocate that some form of testing may be necessary to assess proficiency. The development of a validated test will provide a method for assessing the adequacy of interventions and guidelines designed to improve physician education in this important and controversial area. Limitations Ultrasound examinations performed by physicians at the bedside require skills in both obtaining and optimizing the images and interpreting the images. This test was not designed to evaluate the ability of emergency physicians to obtain ultrasound images. The actual number of EFAST examinations and total ultrasound examinations performed were based on the ultrasound logs and may have been greater than reported secondary to underreporting by clinicians. Although participation in both residencies was high, participants were asked to volunteer and were not required to take this test. As such, it is possible that there was a bias toward volunteering by participants with better test-taking skills or greater knowledge in ultrasound. Although participation was high in both programs, the sample size for participants who had performed a greater number of ultrasound examinations (76 100 EFAST examinations, >100 EFAST examinations, and >300 total ultrasound examinations) was low. Although we were primarily evaluating scores of participants with more than or fewer than 25 EFAST examinations in correlation with the ACEP guidelines, test results for these participants help show that scores continued to improve with the increasing number of ultrasound examinations performed. Although these scores appeared to follow trends set by the other groups, a greater number of participants might have improved the power of these results. The NBME recommends the establishment of an absolute rather than a relative passing standard. The NBME acknowledges that this is by definition somewhat arbitrary. We established a passing cutoff of 70%, which given the performance on the test may have been too high for competent interpretation of a basic EFAST examination. Although deployment of this test on the Internet provided a method for increased availability of this assessment tool, the quality of the videos due to size constraints was not as high as in a non Web-based ultrasound test. Although higher-quality images may have led to higher scores, we had designed the images to be consistent with average emergency department ultrasound examinations, providing a more accurate assessment of competency. Finally, whereas the initial validity of this test has been assessed, the test-retest reliability of the assessment tool has yet to be evaluated. 374 J Ultrasound Med 2011; 30:371 375

Conclusions Initial use of this new Internet-based assessment tool for the interpretation of EFAST images showed face and content validity. Scores improved with increasing experience in both EFAST and the overall number of ultrasound examinations performed, including experience beyond the ACEP recommended guidelines of 25 examinations. References 1. American College of Emergency Physicians, Section of Ultrasound. Emergency ultrasound guidelines. Approved October 2008. American College of Emergency Physicians website. http://www.acep.org/ WorkArea/DownloadAsset.aspx?id=32878. Accessed June 13, 2009. 2. Mateer J, Plummer D, Heller M, et al. Model curriculum for physician training in emergency ultrasonography. Ann Emerg Med 1994; 23:95 102. 3. American Academy of Emergency Medicine. Position statement: performance of emergency screening ultrasound examinations. Approved February 1, 1999. American Academy of Emergency Medicine website. http://www.aaem.org/positionstatements/ultra.php. Accessed June 13, 2009. 4. Accreditation Council for Graduate Medical Education, Residency Review Committee. Emergency medicine guidelines. Accreditation Council for Graduate Medical Education website. http://www.acgme.org/ acwebsite/navpages/nav_110.asp. Accessed June 13, 2009. 5. Heller MB, Mandavia D, Tayal VS, et al. Residency training in emergency ultrasound: fulfilling the mandate. Acad Emerg Med 2002; 9:835 839. 6. Cosby KS, Kendall JL. Practical Guide to Emergency Ultrasound. Philadelphia, PA: Lippincott Williams & Wilkins; 2006. 7. Ma OJ, Gaddis G, Steele MT, Cowan D, Kaltenbronn K. Prospective analysis of the effect of physician experience with the FAST examination in reducing the use of CT scans. Emerg Med Australas 2005; 17:24 30. 8. Ma OJ, Mateer JR, Ogata M, Kefer MP, Wittmann D, Aprahamian C. Prospective analysis of a rapid trauma ultrasound examination performed by emergency physicians. J Trauma 1995; 38:879 885. 9. Rozycki GS, Ochsner MG, Jaffin JH, Champion HR. Prospective evaluation of surgeons use of ultrasound in the evaluation of trauma patients. J Trauma 1993; 34:516 526. 10. Jehle D, Heller M. Ultrasonography in Trauma: The FAST Exam. Dallas, TX: American College of Emergency Physicians; 2003. 11. Corbett SW, Andrew HG, Baker EM, Jones WG. ED evaluation of pediatric trauma patient by ultrasonography. Am J Emerg Med 2000; 18:244 249. 12. American College of Surgeons, Committee on Trauma. Advanced Trauma Life Support for Doctors. 8th ed. Chicago, IL: American College of Surgeons; 2008. 13. American Institute of Ultrasound in Medicine. AIUM practice guideline for the performance of the focused assessment with sonography for trauma (FAST) examination. J Ultrasound Med 2008; 27:313 318. 14. American Institute of Ultrasound in Medicine. Training guidelines for physicians who evaluate and interpret diagnostic ultrasound examinations. Approved March 16, 2008. American Institute of Ultrasound in Medicine website. http://www.aium.org/publications/statements.aspx. Accessed June 13, 2009. 15. Akhtar S, Theodoro D, Gaspari R, et al. Resident training in emergency ultrasound: consensus recommendations from the 2008 Council of Emergency Medicine Residency Directors Conference. Acad Emerg Med 2009; 16(suppl 2):S32 S36. 16. Mandavia DP, Aragona J, Chan L, Chan D, Henderson SO. Ultrasound training for emergency physicians: a prospective study. Acad Emerg Med 2000; 7:1008 1014. 17. Hertzberg BS, Kliewer MA, Bowie JD, et al. Physician training requirements in sonography: how many cases are needed for competence? AJR Am J Roentgenol 2000; 174:1221 1227. 18. Case SM, Swanson DB. Constructing Written Test Questions for the Basic and Clinical Sciences. Philadelphia, PA: National Board of Medical Examiners; 2002. 19. Illinois Institute of Technology. The Delphi method. Illinois Institute of Technology website. http://www.iit.edu/~it/delphi.html. Accessed June 13, 2009. 20. Moore CL, Gregg S, Lambert M. Performance, training, quality assurance, and reimbursement of emergency physician performed ultrasonography at academic medical centers. J Ultrasound Med 2004; 23:459 466. 21. Moore CL, Molina AA, Lin H. Ultrasonography in community emergency departments in the United States: access to ultrasonography performed by consultants and status of emergency physician performed ultrasonography. Ann Emerg Med 2006; 47:147 153. J Ultrasound Med 2011; 30:371 375 375