Competency Based Graduate Medical Education Moving from process to outcomes February 4, 2014 Andy Varney, MD
Competenglish Competency based medical education An outcomes based approach to the design, implementation, assessment, and evaluation of a medical education program using an organizing framework of competencies. Competency An observable ability of a health professional related to a specific activity that integrates knowledge, skills, values, and attitudes. Since competencies are observable, they can be measured and assessed to ensure their acquisition. Competencies can be assembled like building blocks to facilitate progressive development. Competence Possessing the array of abilities (knowledge, skills, and attitudes) across multiple domains or aspects of performance in a certain context. Statements about competence require descriptive qualifiers to define the relevant abilities, context, and stage of training. Competence is multi dimensional and dynamic. It changes with time, experience, and setting. Milestone A significant point in development that helps to define the appropriate developmental trajectory of a trainee. They identify the discrete knowledge, skills, and attitudes expected of learners as they progress through training. Narrative Streams Developmental milestones based descriptions of trainee competence in the six ACGME general competency domains. Each Narrative Stream corresponds to a competency domain within each of the six General Competencies and describes developmental progress across that competency domain in a horizontal fashion from left to right.
No one dangerouslyirrelevant.org jumps a 20 foot chasm in two 10 foot jumps. Miguel Guhlin http://www.flickr.com/photos/midnight digital/3086238863/in/pool jumping O U T C O M E S P R O C E S S
COGNITIVE PERCEPTUAL MODEL OF CLINICIAN EDUCATORS 1 30% of IM PGY 2 & 3 Residents NEVER observed. 2 <50% report being observed taking Hx/PEx Reconstructing behaviors into competencies is difficult 3 inference 4. Rarely reported as received Often reported as given SO WHAT S THE PROBLEM? Frame of Reference Validity 5 Reliability 5 Generalisability
1. ABIM 2009 Fast Track Data 2. JGIM 2008: 23(7); 1010-1015 3. Opening the black box of clinical skills assessment Kogan, JR et al, Med Educ 2011; 45; 1048-1060 4. Opening the black box of clinical skills assessment Kogan, JR et al, Med Educ 2011;45; 1048-1060 5. Factor Analysis Methods and Validity Evidence: Wetzel, A, Acad Med 2012; 87; 1060-1069.
Competency assessment is a contextual, synthetic process.
MEDICAL KNOWLEDGE: Learner Level: Check the boxes if the resident meets the criteria. Biological Sciences: The resident can recall the basic anatomy, physiology, microbiology, pathology and pharmacology to care for patients with respiratory and critical illness. Clinical Sciences: The resident is able to recognize common clinical presentations of critically ill patients admitted to the ICU Procedural Knowledge: The resident understands the indications and contraindications for invasive procedures in the ICU. Manager Level: The resident must meet all of the Learner Level before they can meet any of the Manager Level. Biological Sciences: The resident can manage patients with critical illness that requires the integration of anatomy, physiology, microbiology, pathology, pharmacology and neuroscience principles, including common diagnostic test interpretation. Clinical Sciences: The resident demonstrates sufficient knowledge to stabilize critical ill patients and utilize technology and therapeutics to diagnose and treat patients with critical illnesses. Procedural Knowledge: The resident demonstrates the manual dexterity and procedural knowledge required to safely and successfully perform invasive procedures in the ICU. Teacher/Leader Level: The resident must meet all of the Learner and Manager levels before they can meet this criteria. The resident is proficient in the competency of medical knowledge and functions as a team leader while teaching these skills to the medical team. Medical Knowledge On the grading scale choose the selection that corresponds to the boxes you have checked. If you did not directly observe the resident as a Learner, Manager or Teacher use "Unable to Assess" and move on to the next competency. Fails Achieves Achieves two Achieves Achieves Achieves all of Achieves all of Achieves all Unable all one of the of the Learner all of the all of Learner the Learner and the Learner levels and is to Learner level. Learner and one of the two of and Manager a Teacher Assess Level. Level. Manager the Manager levels. and Leader Level. Criteria. Comments
Windows to Competence Caverzagie and Iobst
EPA - Lead and work within interprofessional teams Maintain climate of mutual respect and shared values 10
EPA - Lead and work within interprofessional teams Engage in collaborative communication 11
EPA - Lead and work within interprofessional teams Identify and understand roles of team members 12
EPA - Lead and work within interprofessional teams Manage diverse opinions with goal optimizing patient care 13
Accept feedback EPA - Lead and work within interprofessional teams 14
EPA - Lead and work within interprofessional teams
Accept feedback EPA - Lead and work within interprofessional teams Maintain climate of mutual respect and shared values Identify and understand roles of team members Manage diverse opinions with goal optimizing patient care Engage in collaborative communication 16
COMPETENCE 17
Standard of Measurement: Entrustment I. Resident has knowledge and some skill, but is not allowed to perform the EPA independently. II. III. IV. Resident may act under proactive, ongoing, full supervision. Resident may act under reactive supervision, i.e., supervision is readily available on request. Resident may act independently. V. Resident may act as a supervisor and instructor Ten Cate O, Snell L, Carraccio C. Medical competence: the interplay between individual ability and the health care environment. Med Teach. 2010;32 8):669 75 Crossley J, Johnson G, Booth J, Wade W. Good questions, good answers: construct alignment improves the performance of Workplace based assessment scales. Med Educ. 2011 Jun;45(6):560 9
Entrustment = Trustworthiness 1,2 Grounded in 4 attributes of Learners Knowledge and skill Discernment insight/awareness limitations 2 Conscientiousness Truthfulness Entrustment implies a level of competence. 1) Kennedy et al ACAD Med; 83 (lo suppl): 589 92 2) Eva, KW, Regher ACAD Med 2005; 80 reflect in action > reflect on action
Entrustment: Construct Alignment 1 2 3 4 5 Resident requires complete direction to accomplish the task Resident may initiate a task, but requires proactive, ongoing supervision Resident may act under reactive supervision, supervision is available upon request Resident may act independently Resident may act as a supervisor and instructor ATTENDING ENTRUSTMENT ASSESSMENT
NAS Reporting Milestones: aka, The Educational Milestones NAS Milestones ACGME NAS-FAQ http://www.acgme-nas.org Accessed: April 21, 2013
The Educational Milestones University of Nebraska Medical Center N Engl J Med 2012; 366: 1051-6
Specialty Milestones Snapshot Sullivan G, Simpson D, Cooney T, Beresin E. A Milestone in the Milestones Movement: the JGME Milestones Supplement. JGME: 2013, 5(1):1 4
Internal Medicine Milestones
How will the Milestones be used? In the NAS each program is required to form a Clinical Competency Committee (CCC) CCC will use data from assessment tools and faculty observations to evaluate the resident s progress and competency toward achieving the Milestones CCC makes a consensus decision on the progress of each resident in the program and provides recommendation to the program director University of Nebraska Medical Center FAQ s about the NAS December 2012
Things to know about Milestone based evaluations: Expected benefits Behaviorally anchored Potentially more objective Standardize expectations for teachers and learners Accelerate change for learners Increase dispersion and information content of scores with reduced common cognitive errors: Central tendency giving the similar score to disparate individuals across different traits Halo or horn effect allowing past experience or reputation to influence scores Compensation usually balancing a low score by exaggerating performance in another area Expected limitations Assessment spread too thin: too many behaviors unevenly assessed undermining fairness and standardization, and difficult to validate. More subtle skills may not be effectively captured. Severity error the tendency for behavioral evaluations to accentuate specific failures over general competency. Questions answered Mechanics: Reporting Milestones for each discipline developed or in final phase Construct development and alignment can help Rotational assessments are formative and CCC makes summative judgments
Central Tendency in evaluations Impact: Residents receive false reassurance Information regarding specific strengths and weaknesses is lost Negative promotion decisions have inadequate support Source: Grouping too many behaviors in a single question Use of a calendar relative standard. High stakes impact and relationship issues.social pressures may push faculty to cushion negative evaluations. Lack of clarity on how to weigh a single failure in relation to an overall pattern of behavior, natural ability, and likely future performance.
Determining how to evaluate a milestone What is the Goal of assessment? Capacity Has necessary knowledge, skills, and attitude to perform successfully Usually measured in recall based tests, verbal questioning or in simulations. Often thought of as a medical student or internship level measurement, but could apply in any setting where a learner is about to start a new activity. Ability Performs skill successfully under observation (at least once) Measured formally on CEX s, procedure observations, OSCE s, and intermittently through observation on rounds. As the observations are anecdotal they often captures extremes of performance failures and extraordinary success. They should be treated as formative data. Learners can be incorrectly labeled Frequency On going measurement of the successful performance of a skill. Per human resources literature, frequency tests are best predictors of future performance. Usually measured through frequent simple measures as a percent of events above a threshold. Clinical outcome data from electronic data can be presented this way. Supervisors have more inter-rater agreement when they are asked to estimate frequency of good performance, then when asked to rate an employee s ability based on specific successes and failures. I.e., they should count all failures equally, discounting the severity of failure.
Role of CCC Members of CCC make a consensus decision on the progress of each resident Offer a group perspective to program director Serve as an early warning system for residents failing to progress University of Nebraska Medical Center FAQ s about the NAS December 2012
A Paradigm Shift From Process to Outcomes Assessment Curricular Milestones Entrustable Professional Activities Milestones Next Accredit ation System How Milestones..Can be used by And allow.to report faculty to programs. outcomes via the NAS. assess resident progress Clinical Competency Evaluation Committee
Medical Knowledge Attending Termination Patient Care Peer Interpersonal And Communication Skills Self Trainee Clinical Competency Committee Promotion Systems Based Practice Nurses Case Managers Practice Based Learning Patients Performance Below Standards Professionalism Office Staff Remediation Root Cause Analysis 8 D s Probation
Learners Assessments within Program: Direct observations Audit and performance data Multi source FB Simulation ITExam Institution and Program CCC: Judgment and Synthesis Reporting Milestones Accreditation: ACGME/RRC Program Aggregation Faculty, PDs and others
Value of Group Discussion / Decisions More likely to uncover deficiencies in knowledge and professionalism (UME) Hemmer; Acad Med 1997, 2001 Better predicts poor internship performance Lavin; Acad Med 1999 Society benefit of doubt Gaglione; Acad Med, 2005 Case based faculty development Hemmer, Acad Med, 2000
Value of Group Discussion (2) Improved inter rater reliability, reduced range restriction in multiple domains Thomas; JGIM 2011 Detected additional 18% of resident deficiencies requiring remediation Schwind; Acad Med 2004 No individual dominates discussions Williams, TLM 2005; Gaglione 2005
Assessments and Milestones The CCC will review and use assessment data including faculty member assessments of residents on rotations, self evaluations, peer evaluations, and evaluations by nurses and other staff members. Each program may continue to use its current resident assessment tools and phase in tools developed specifically for the milestones when these become available. University of Nebraska Medical Center FAQ s about the NAS December 2012
Competency Based Assessment Effect of Construct Alignment RESULTS AND OUTCOMES CoBRA - Impact on grade inflation % of time an intern received an 8 ABIM 7/1/07 2/1/09 2/1/09 10/31/10 11/1/10 7/1/11 7/1/11 6/30/12 40.0% 35.0% 33.6% 30.0% 25.0% 20.0% 23.0% 19.0% 21.2% 25.0% 17.1% 23.1% 15.0% 10.0% 5.0% 9.7% 6.0% 4.0% 2.2% 11.1% 7.0% 3.9% 3.2% 12.0% 10.8% 10.0% 10.3% 8.2% 6.9% 6.9% 5.1% 4.9% 3.8% 3.0% 1.6% 9.9% 5.7% 3.8% 1.6% 10.5% 6.8% 5.1% 2.9% 0.0% Patient Care Medical Knowledge Practice Based Learning & Improvement Interpersonal & Communication Skills Professionalism Systems Based Practice OVERALL
Competency Based Assessment Effect of Construct Alignment Result and Outcomes CoBRA Impact on use of left side of scale % of time a PGY1 resident received a 3 ABIM 7/1/07 2/1/09 2/1/09 10/31/10 11/1/10 7/1/11 7/1/11 6/30/12 12.0% 10.8% 10.9% 10.0% 8.0% 6.5% 6.0% 5.4% 5.4% 4.3% 4.0% 2.0% 0.0% 0.6% 3.3% 2.3% 2.2% 1.1% 1.3% 0.0% 0.0% 0.0% 2.5% 3.5% 3.2% 3.3% Patient Care Medical Knowledge Practice Based Learning & Improvement 0.7% 1.5% 3.0% 1.3% Interpersonal & Communication Skills 1.9% 2.2% 1.7% 0.6% 0.0% Professionalism 3.0% 1.5% 3.0% Systems Based Practice 3.6% 2.6% 1.3% 0.2% OVERALL
Competency Based Assessment Effect of Construct Alignment DIFFERENCE BETWEEN PGY 1 AND 3 RESIDENT ACROSS ACGME Competencies Increased Assessor discrimination 6/1/04 6/30/07 (ABIM) 7/1/07 10/31/10 (CoBRA) 11/1/10 7/1/11 (CoBRA) 7/1/11 6/30/12 (CoBRA) 2.00 1.80 1.60 1.40 1.20 1.00 1.38 1.26 1.05 1.24 1.18 1.10 1.09 1.04 1.03 1.04 1.00 1.03 1.03 0.99 1.46 1.30 1.15 0.80 0.60 0.74 0.75 0.62 0.63 0.81 0.46 0.73 0.40 0.20 0.00 Patient Care Medical Knowledge Practice Based Learning & Improvement Interpersonal & Communication Skills Professionalism Systems Based Practice
SO ARE THERE ANY ANSWERS? Setting based educators: intensivists, hospitalists, nocturnists, ambulists Embed workplace assessments in essential activities of clinical practice. Ability + Autonomy = Entrustment Constant Formative Feedback Essential Summative via PD/Advisers Construct Alignment Group Meetings Criterion Referenced
Faculty Development Awakening Creating Change: Advancing Competency Based Assessments Your clinician educator faculty are the people who must generate and sustain the commitment for authentic workplace assessments.
But First: Have the faculty agreed. 1) There is a problem that needs attention? 2) On the definition of the problem/issue? 3) To work together on the problem/issue? 4) How to work together on the problem/issue? 5) On the solution(s) to the problem/issue? 6) On any implementation plan and action steps?
Goal: Facilitate your clinician educators to assess what really matters. What outcomes, behaviors, skills, attitudes are important? What can be observed or inferred within our current workflow? What could we change in our workflow to improve direct observation of what matters? Ask the faculty to frame across ACGME competencies using criterion behaviors or discipline milestones. Start organically/synthetically creating champions for direct observation and CBA.
Starting Over Understand the context for change before you act Define shared values and engage clinician educators in positive action Create transparency and safety for developing each contribution to the whole. Remember it is an iterative process. Share Power and Influence to create a shared mental model. Coaching/Mentoring demonstrates commitment to your faculty Help your faculty create better tools to measure what matters.
Opportunities for CBA GME collaboration SIU GME Toolkit: articles, Q sorts, curricular milestones, tools to assist, samples. SIU GME: Work together to advance? Procedural competency Entrustable Professional Activities Focused workshops to sharpen skills
Thanks for your attention! Questions? Suggestions? Comments?