Assessment of Learners and Evaluation of Programs

Transcription

1 Assessment of Learners and Evaluation of Programs November 6, 2012 Jongyeol Kim, MD, RPVI, RVT Associate Professor Neurology Co-Chair, Ultrasound Steering Committee Faculty Coordinator for School of Medicine Programs F. Marie Hall SimLife Center Texas Tech University Health Sciences Center

2 Content Definition of assessment and evaluation Principles of assessment and outcome-based assessment Commonly used assessment methods Review on existing curriculum and programs Action items Reference

3 Assessment and Evaluation * The use of the terms assess and assessment is consistently reserved for description of activities related to the assessment of medical student performance, while the use of the terms evaluate and evaluation is consistently reserved for faculty, resident, course and clerkship/clerkship rotation, and program evaluation. * Introduction FUNCTIONS AND STRUCTURE OF A MEDICAL SCHOOL. LCME

4 LCME Standards Related to Assessment of Learning Outcomes ED-1-A. The objectives of a medical education program must be stated in outcome-based terms that allow assessment of student progress in developing the competencies that the profession and the public expect of a physician. The objectives of the medical education program are statements of the items of knowledge, skills, behaviors, and attitudes that medical students are expected to exhibit as evidence of their achievement. The educational objectives, along with their associated outcome measures, should reflect whether and how well graduates are developing these competencies as a basis for the next stage of their training.

5 LCME Standards Related to Assessment of Learning Outcomes ED-33. There must be integrated institutional responsibility in a medical education program for the overall design, management, and evaluation of a coherent and coordinated curriculum. Curriculum management signifies leading, directing, coordinating, controlling, planning, evaluating, and reporting. Evidence of effective curriculum management includes the following characteristics: Evaluation of program effectiveness by outcomes analysis, using national norms of accomplishment as a frame of reference. Monitoring of content and workload in each discipline, including the identification of omissions and unplanned redundancies. Review of the stated objectives of each individual course and clerkship (or, in Canada, clerkship rotation), as well as the methods of pedagogy and medical student assessment, to ensure congruence with programmatic educational objectives.

6 LCME Standards Related to Assessment of Learning Outcomes ED-46. A medical education program must collect and use a variety of outcome data, including national norms of accomplishment, to demonstrate the extent to which its educational objectives are being met. The medical education program should collect outcome data on medical student performance, both during program enrollment and after program completion, appropriate to document the achievement of the program s educational objectives. The kinds of outcome data that could serve this purpose include 1. performance on national licensure examinations, 2. performance in courses and clerkships (or, in Canada, clerkship rotations) and other internal measures related to educational program objectives, 3. academic progress and program completion rates, 4. acceptance into residency programs, and 5. assessments by graduates and residency directors of graduates' preparation in areas related to medical education program objectives, including the professional behavior of its graduates.

7 LCME Standards Related to Assessment of Learning Outcomes ED-47. In evaluating program quality, a medical education program must consider medical student evaluations of their courses, clerkships (or, in Canada, clerkship rotations), and teachers, as well as a variety of other measures. It is expected that the medical education program will have a formal process to collect and use information from medical students on the quality of courses and clerkships/clerkship rotations. The process could include such measures as 1. questionnaires (written or online), 2. other structured data collection tools, 3. focus groups, 4. peer review, and 5. external evaluation.

8 Multidimensional Aspect of Assessment Appropriate Assessment Process Valid Reliable Impact on learning Practicality Use of appropriate assessment tool Written assessment Clinical/Practical assessment Observation portfolio Assessment of full range of learning outcome Miller s Learning Pyramid

9 Assessment: Type Formative Guiding future learning, providing reassurance, promoting reflection, shaping values Provide benchmarks to orient the learner who is approaching a relatively unstructured body of knowledge Reinforce students intrinsic motivation to learn and inspire them to set higher standards for themselves Summative Making an overall judgment about competence, fitness to practice, or qualification for advancement to higher levels of responsibility Provide professional self-regulation and accountability * Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA 2002;287:

10 Commonly Used Assessment Methods Assessment category Written assessments Clinical practice assessments Observation Portfolios and Other Records of Performance Representative instruments Multiple Choice Questions (MCQs) Extended Matching Items (EMIs) Patient Management Problems (PMPs) Short Answer Questions Completion Questions Objective Structured Clinical Examination (OSCE) Objective Structured Practical Examination (OSPE) Objective Structured Long Examination Record (OSLER) Group Objective Structured Clinical Examination (GOSCE) Tutor s report Checklists Logbooks Portfolios Essay Progress Test Dissertation Report Long cases Practical Examination Spot Examination Rating scales Patient report Procedural Logs Peer and Self-Assessment Peer report Self-report

11 Commonly Used Assessment Methods RM Epstein, Assessment in Medical Education N Engl J Med 2007;356:

12 Commonly Used Assessment Methods RM Epstein, Assessment in Medical Education N Engl J Med 2007;356:

13 Assessment Pyramids Does (Action) Shows How (Performance) Knows How (Competence) Know (Knowledge) attitudes/ethics, decision making, role of doctor, personal development clinical skills, practical procedures, health promotion, communication, information handling investigation, management medical sciences Shumway JM, Harden RM; Association for Medical Education in Europe. AMEE Guide No. 25: The assessment of learning outcomes for the competent and reflective physician. Med Teach. 2003;25:569-84

14 Assessment Pyramids Does (Action) Shows How (Performance) Observation, logs, portfolios peer observation Clinical and practical assessment e.g. OSCE Knows How (Competence) Written assessment Know (Knowledge) Written assessment Shumway JM, Harden RM; Association for Medical Education in Europe. AMEE Guide No. 25: The assessment of learning outcomes for the competent and reflective physician. Med Teach. 2003;25:569-84

15 USC Assessment Goals: to use utlrasonography to enhance the acquisition of factual knowledge in the basic sciences such as anatomy, physiology, and pathology; to equip the students with a physical diagnosis and bedside diagnostic skills to positively impact the care they provide to their patients, and to educate them on the advantages as well as the limitations of ultrasonography in clinical decision making and performing ultrasound-directed procedures M1 and M2 OSCE at the end of the 1 st or 2 nd semester; based on the ultrasound instruction during the semester Ultrasound questions in the MCQs or on-line leering module with self assessment; knowledge of ultrasound and its clinical applications M3 clerkship IM: two separate OSCE: thyroid cyst; central-line placement FM; OSCE; AAA screening OB/GYN: OSCE of normal pregnancy Surgery: OSCE of FAST Pediatrics: OSCE: volume status in child ICU: ultrasound related to critical care Hoppmann et al. An integrated ultrasound curriculum (iusc) for medical students: 4-year experience Crit Ultrasound J April; 3(1): 1 12

16 Hoppmann et al. An integrated ultrasound curriculum (iusc) for medical students: 4-year experience Crit Ultrasound J April; 3(1): 1 12

17 USC Evaluation Course evaluations M1 and M2: anonymous on-line course evaluation at the end of each semester M3: an ultrasound curriculum evaluation at the end of the academic year M4: completion of electives and special programs Hoppmann et al. An integrated ultrasound curriculum (iusc) for medical students: 4-year experience Crit Ultrasound J April; 3(1): 1 12

18 USC Evaluation Hoppmann et al. An integrated ultrasound curriculum (iusc) for medical students: 4-year experience Crit Ultrasound J April; 3(1): 1 12

19 Emergency Medicine Residency The performance of at least 150 US examinations in critical or lifesaving situations promotes a minimum acceptable level of exposure Goal: ensuring all EM residents have a basic set of skills to allow for interpretation of US into their daily clinical practice after residency training is completed. Assessment of US Technique 1. FAST examination 2. Emergent cardiac imaging 3. Evaluation of the aorta for AAA 4. Identification of early intrauterine pregnancy via transabdominal and transvaginal sonography 5. Procedural guidance for ED procedures 6. Identification of gallbladder pathology 7. Detection of deep venous thrombosis 8. Recognition of hydronephrosis and determination of bladder volume 9. Detection of subcutaneous fluid collection, abscess, and foreign body 10. Identification of pneumothorax 11. Identification of ocular pathology Resident Training in Emergency Ultrasound: Consensus Recommendations from the 2008 Council of Emergency Medicine Residency Directors Conference Acad Emerg Med Dec;16 Suppl 2:S32-6.

20 Emergency Medicine Residency Assessment of proper machine settings, probe positioning, image acquisition, and documentation Technical aspects, including but not limited to image quality, image framing, identification of landmarks, and completeness of imaging protocol Real patients or simulator/standardized patients OSCE, SDOT (standardized Direct Observation Tool), videotape of person performing US examination for later review Assessment of Image Interpretation Assessment of competency in performing and interpreting US via review of static or dynamic video Real world Evaluate US technique and image interpretation Medical decision makingassessment of proper machine settings, probe positioning, image acquisition, and documentation Standardized MCQ examination; 35-question exam, covering the primary application of EU (FAST, US in pregnancy, AAA, EM Echo, Biliary US, Renal US, Procedural US), Ocular US, Soft-tissue Abscess, DVT 13 physics and image-acquisition techniques 13 image interpretation with clinical integration 9 pure image interpretation Only image interpretation and integration into patient management, but not actual image acquisition Resident Training in Emergency Ultrasound: Consensus Recommendations from the 2008 Council of Emergency Medicine Residency Directors Conference Acad Emerg Med Dec;16 Suppl 2:S32-6.

21 Emergency Medicine Residency Goal Ensuring all emergency medicine residents have a basic set of skills to allow for interpretation of US into their daily clinical practice after residency training is completed Does (Action) Shows How (Performance) Knows How (Competence) Observation Logs Portfolios Peer observation Clinical and practical assessment OSCE SDOT Videotaping of exam MCQs: physics, techniques, interpretation Hands-on assessment Know (Knowledge) MCQs Review of static/dynamic video Shumway JM, Harden RM; Association for Medical Education in Europe. AMEE Guide No. 25: The assessment of learning outcomes for the competent and reflective physician. Med Teach. 2003;25:569-84

22 Assessment Methods Method Domain Type of Use Ultrasound Ed. Written Exercises Multiple-choice questions Key-feature and script-concordance questions Short-answer questions Structured essays Knowledge, ability to solve problems Clinical reasoning, problemsolving ability, ability to apply knowledge Ability to interpret diagnostic tests, problem-solving ability, clinical reasoning skills Synthesis of information, interpretation of medical literature Summative assessments within courses or clerkships; national in-service, licensing, and certification examination National licensing and certification examinations Summative and formative assessments in courses and clerkships Preclinical courses, limited use in clerkships Physics including artifacts Knobology Interpretation of imaging Cases Clinical cases Physics including artifacts Knobology Interpretation of imaging Cases Clinical cases RM Epstein, Assessment in Medical Education N Engl J Med 2007;356:

23 Assessment Methods Method Domain Type of Use Ultrasound Ed. Assessments by supervising clinicians Global ratings with comments at end of rotation Structured direct observation with checklists for rating (e.g. mini-clinical evaluation exercise or video review Clinical skills, communication, teamwork, presentation skills, organization, work habits Communication skills, clinical skills Global summative and sometimes formative assessments in clinical rotations Limited use in clerkships and residencies, a few boardcertification examinations Oral examinations Knowledge, clinical reasoning Limited use in clerkships and comprehensive medical school assessments, some board-certification examinations Clinical rotation with ultrasound in various specialties OSCE SDOT Video Clinical rotation RM Epstein, Assessment in Medical Education N Engl J Med 2007;356:

24 Assessment Methods Method Domain Type of Use Ultrasound Ed. Clinical simulations Standardized patients and objective structured clinical examinations Some clinical skills, interpersonal behavior, communication skills Formative and summative assessments in courses, clerkships, medial schools, national license examinations, board certification in Canada OSCE Simulated clinical cases with Standardized patients Incognito standardized patients Actual practice habits Primarily used in research some courses, clerkships, and residencies use for formative feedback SP High-technology simulations Procedural skills, teamwork, simulated clinical dilemmas Formative and summative assessment OSCE, simulated clinical cases using Simulators RM Epstein, Assessment in Medical Education N Engl J Med 2007;356:

25 Assessment Methods Method Domain Type of Use Ultrasound Ed. Multisource ( 360-degree) assessments Peer assessments Professional demeanor, work habits, interpersonal behavior, teamwork Formative feedback in courses and comprehensive medical school assessments, formative assessments for board certification Peer review forms Patient assessments Ability to gain patients trust; patient satisfaction, communication skills Formative and summative, board certification, use by insurers to determine bonuses Feedback by patient or SP Self-assessments Knowledge, skills, attitudes, beliefs, behaviors Formative Self-report Portfolios All aspects of competence, especially appropriate for practice-based learning and improvement and systemsbased practice Formative and summative uses across curriculum and with-in clerkships and residency programs, used by some U.K. medical schools and specialty boards Portfolios: archives of imaging, documents of all ultrasound activities, etc RM Epstein, Assessment in Medical Education N Engl J Med 2007;356:

26 Future Plans and Action Items General consensus on objectives, goals, and outcomes of ultrasound curriculum in medial school curriculum National norms of accomplishment Collect a variety of outcome data for assessment Consensus or standardized medical student evaluation of courses and curriculum Organized activities by various organizations including SUSME, AAMC, and other specialty associations

27 Integration of Ultrasound in Medical Education

28 TTUHSC SOM Dr. Mitchell: President of Texas Tech University Health Sciences Center Dr. Berk: Dean of the School of Medicine and Executive Vice President and Provost, TTUHSC Key Faculty of Ultrasound Steering Committee Dr. Blackwell OB & GYN Dr. Casanova OB & GYN Dr. Kim Neurology Drs. Raj and Alalawi Internal Medicine and Critical Care Drs. Sasin and Piel Emergency Medicine Dr. Arvandi - Cardiology Dr. Santana Vascular Surgery Dr. Arentz - Surgery Dr. Mitchell Sports Medicine Dr. Lee Cell Biology and Chemistry 28

29 References 1. FUNCTIONS AND STRUCTURE OF A MEDICAL SCHOOL, Standards for Accreditation of Medical Education Programs Leading to the M.D. Degree, May Goldie J. AMEE Education Guide no. 29: evaluating educational programmes. Med Teach. 2006; 28: Shumway JM, Harden RM; Association for Medical Education in Europe. AMEE Guide No. 25: The assessment of learning outcomes for the competent and reflective physician. Med Teach. 2003;25: Hoppmann R. et al. An integrated ultrasound curriculum (iusc) for medical students: 4-year experience. Crit Ultrasound J. 2011;3: Epstein RM. Assessment in Medical Education. N Engl J Med 2007; 356: Resident Training in Emergency Ultrasound: Consensus Recommendations from the 2008 Council of Emergency Medicine Residency Directors Conference Acad Emerg Med. 2009;16 Suppl 2:S Society of Ultrasound in Medical Education: 8. International expert statement on training standards for critical care ultrasonography. Expert Round Table on Ultrasound in ICU. Intensive Care Med Jul;37(7):