Program Evaluation for Competency-based medical Education: Are we making a difference? Elaine Van Melle, Leslie Flynn, Alyshah Kaba, Walter Tavares, Tanya Horsley Thursday September 29, 2016 12:00 3:30 Room 208 Scotiabank Convention Centre
We do not have an affiliation (financial or otherwise) with a pharmaceutical, medical device or communications organization. Je n ai aucune affiliation (financière ou autre) avec une entreprise pharmaceutique, un fabricant d appareils médicaux ou un cabinet de communication. 2
Workshop Objectives 1. Use a logic model to focus your CBME program evaluation 2. Consider how to make a program evaluation scholarly 3. Consider how to build capacity for program evaluation
Our Assumption CBME is... an approach to preparing physicians for practice that is fundamentally oriented to graduate outcome abilities and organized around competencies derived from an analysis of societal and patient needs. It deemphasizes time-based training and promises greater accountability, flexibility, and learner centredness. A+ Frank JR, Mungroo R, Ahmad Y, Wang M, de Rossi S, Horsley T. Toward a definition of competency-based education in medicine: a systematic review of published definitions. Med Teacher. 2010;32: 631-7.
CBME... Start with the end in mind.... We will prepare physicians... Who are able to meet population health needs
CBME... Start with the end in mind.... We will prepare physicians... Who are able to meet population health needs BETTER?
Our Approach Please may I be excused? My brain is full!
Logic Model - defined Inputs Activities Outputs Outcomes Shortterm Mediumterm Longterm Tool to build common understanding of how a program is supposed to work Creates a focus for evaluation activities Build in scholarship McLauglin & Jordan, 2004
Logic Model Evaluating Educational Innovations Purpose What is the problem? Inputs Activities Outputs What resources are required? What activities are critical and/or unique? What products or behaviours arise as a result of this program? Short term Outcomes Medium term Long term What do we anticipate will happen as a result of this program? Needs Assessment Conceptual Framework(s) External Factors Unintended Outcomes What assumptions or conceptual frameworks shape the design of the activities? What conditions exist that influence how the program is delivered? What positive or negative unanticipated outcomes might arise or arise as a result of this program? PROGRAM DELIVERED PROGRAM RESULTS Van Melle, E. 2016 AM last Page Academic Medicine, Vol. 91, No. 10.
What is Program Evaluation? The systematic investigation of the quality of programs, projects, subprograms, subprojects, and/or any of their components or elements, together or singly p. xxv
Multiple activities combine in various ways to contribute to the achievement of outcomes over time CBME Complex Innovation
CBME Critical Activities FRAMEWORK Competencies required for practice are clearly articulated PROGRESSION Competencies and their developmental markers are sequenced progressively TAILORED EXPERIENCES Authentic, work-based learning environments organized to facilitate the developmental acquisition of competencies COMPETENCY- FOCUSED INSTRUCTION Teachers who act as coaches in a way that promotes the developmental acquisition of competencies PROGRAMMATIC ASSESSMENT Assessment practices support & document the developmental acquisition of competencies
CBME & the Cycle of Program Design & Implementation Pre-implementation LT Outcomes ST Outcomes Cycle of CBME Program Design & Implementation Accountability Design Clarification & Improvement
Where are you at? Pre-implementation LT Outcomes ST Outcomes Cycle of CBME Program Design & Implementation Accountability Design Clarification & Improvement
Typically we think of program evaluation as the end Pre-implementation LT Outcomes Program Evaluation ST Outcomes Cycle of CBME Program Design & Implementation Accountability Design Have we achieved what we intended? Clarification & Improvement
Program Evaluation can occur at any stage Pre-implementation Why should we implement CBME? LT Outcomes Are we achieving system-wide impacts? ST Outcomes Are we achieving our intended outcomes? Cycle of CBME Program Design & Implementation Accountability What will it take to implement? Design What are we implementing? Clarification & Improvement Are we implementing as intended?
Logic Model and the Cycle of Program Development Pre-implementation Needs Assessment Purpose LT Outcomes System-wide Impacts ST Outcomes Program Outcomes Cycle of CBME Program Design & Implementation Accountability Inputs Design Key Activities Clarification & Improvement Outputs
Logic Model Needs Assessment & Purpose Statement Purpose What is the problem? Inputs Activities Outputs What resources are required? What activities are critical and/or unique? What products or behaviours arise as a result of this program? Short term Outcomes Medium term Long term What do we anticipate will happen as a result of this program? Needs Assessment Conceptual Framework(s) External Factors Unintended Outcomes What assumptions or conceptual frameworks shape the design of the activities? What conditions exist that influence how the program is delivered? What positive or negative unanticipated outcomes might arise or arise as a result of this program? PROGRAM DELIVERED PROGRAM RESULTS
Purpose Statement Why CBME? Because we have to
Why CBME - The Big Picture Frenk at al, 2010 To ensure physicians are well prepared to provide quality patient care in a rapidly changing heath care environment
Why CBME? Shorter term Outcomes Enhance quality of feedback Earlier identification of residents who are not advancing as expected Clear evidence (multiple sources) to inform progression decisions
Why CBME? What problem do you have that CBME is intended to address? Can you create one compelling sentence that describes the purpose of CBME in your context?
Importance of Purpose Faculty buy-in CBME is not something someone else can make you do!
Needs Assessment What did your literature review reveal? What did your data collection reveal? (observations, surveys etc.)
Importance of Purpose to Program Evaluation Lewis Carroll
Importance of Purpose to program Evaluation how will you know you have arrived? Lewis Carroll
Logic Model Inputs and Activities Purpose What is the problem? Inputs Activities Outputs What resources are required? What activities are critical and/or unique? What products or behaviours arise as a result of this program? Short term Outcomes Medium term Long term What do we anticipate will happen as a result of this program? Needs Assessment Conceptual Framework(s) External Factors Unintended Outcomes What assumptions or conceptual frameworks shape the design of the activities? What conditions exist that influence how the program is delivered? What positive or negative unanticipated outcomes might arise or arise as a result of this program? PROGRAM DELIVERED PROGRAM RESULTS
Input What Resources are Required to Implement CBME? Administrative support Program leadership Curriculum resources & tools Faculty Development Change management Program Evaluation
Activities What activities are critical for CBME? FRAMEWORK Competencies required for practice are clearly articulated PROGRESSION Competencies and their developmental markers are sequenced progressively TAILORED EXPERIENCES Authentic, work-based learning environments organized to facilitate the developmental acquisition of competencies COMPETENCY- FOCUSED INSTRUCTION Teachers who act as coaches in a way that promotes the developmental acquisition of competencies PROGRAMMATIC ASSESSMENT Assessment practices support & document the developmental acquisition of competencies
Black box Evaluation Does it Work? I think you should be more explicit here in Step 2
Activities - Examples of sub components FRAMEWORK Determining which competencies are required for practice Creating Entrustable Professional Activities PROGRESSION Sequencing EPAs and milestones into a continuum TAILORED EXPERIENCES Implementing Longitudinal versus block experiences Creating simulated learning experiences Facilitating residents to be self-regulated learners COMPETENCY- FOCUSED INSTRUCTION Equipping your teachers to be coaches Enhancing the quality of feedback PROGRAMMATIC ASSESSMENT Establishing a Competence Committee Implementing new assessment tools
Determine your priorities Where do we start... What is next... Then what...
Activities example Queen s University Department of Family Medicine Implementing an electronic Portfolio Assessment System (PASS) Academic Advisor meets quarterly with resident - reviews portfolio, provides feedback, sets goals
Activities example Queen s University Department of Family Medicine Implementing an electronic Portfolio Assessment System (PASS) Academic Advisor meets quarterly with resident - reviews portfolio, provides feedback, sets goals Core Component Programmatic Assessment Sub component Academic Advisors ACTIVITIES Meet AA meets with resident 1X every 3 months Prepare AA reviews portfolio Discuss Provides feedback Makes recommendation Resident Promotion Committee
Draft a list of your activities What is your core component? What is your sub-component or project? What are the key activities?
In Summary: Outcomes are necessary but insufficient onto themselves There is nothing in a caterpillar that tells you it s going to be a butterfly Buckminster Fuller
Logic Model Needs Assessment & Purpose Statement Purpose What is the problem? Inputs Activities Outputs What resources are required? What activities are critical and/or unique? What products or behaviours arise as a result of this program? Short term Outcomes Medium term Long term What do we anticipate will happen as a result of this program? Needs Assessment Conceptual Framework(s) External Factors Unintended Outcomes What assumptions or conceptual frameworks shape the design of the activities? What conditions exist that influence how the program is delivered? What positive or negative unanticipated outcomes might arise or arise as a result of this program? PROGRAM DELIVERED PROGRAM RESULTS
Output versus Outcome? Output Products that arise from the activities e.g., Description of competencies, # faculty development workshops, # resident learning plans Quality of new assessment tools - important because they can be readily captured and counted to show the level of activity
Output versus Outcome? Outcomes Changes that happen as a result of the program over time What are we doing differently as a result of CBME?
Determining Outcomes Outcomes Changes that happen as a result of the program over time Outcomes Proximal Distal Enhanced readiness for practice Enhanced patient care outcomes More Degree of program influence Less
Describe a proximal outcome for your CBME program What changes will be brought about as result of your CBME program? With learners... Faculty... Other...
What about Kirkpatrick? Impact Longterm First published 1959 Purpose: To show the business value & worth of training Kirkpatrick & Kirkpatrick, 2009 Behaviour Learning Reaction 4 levels of outcomes Short term
Logic Model Evaluating Educational Innovations Purpose What is the problem? Inputs Activities Outputs What resources are required? What activities are critical and/or unique? Outcomes Short term Medium term Long term Needs Assessment Conceptual Framework(s) What assumptions or conceptual frameworks shape the design of the activities? 1 2 3 4 PROGRAM DELIVERED PROGRAM RESULTS
Outcome measures - What kind of evidence do we need? Longterm Short term Impact Behaviour Learning Reaction Quality Outcome indicators (e.g. improve pt experience, decrease falls, LOS) Skill Testing Performance during patient care Adherence to protocols Attitudinal Survey Knowledge tests MCQs, Short answer Learner Feedback Satisfaction Survey
So far... CBME is a complex innovation It is important to describe the links between CBME delivery and outcomes Now you can determine the focus of your evaluation
CBME Program Evaluation Example Queen s University Department of Family Medicine Implementing an electronic Portfolio Assessment System (PASS) Academic Advisor meets quarterly with resident - reviews portfolio, provides feedback, sets goals Core Component Programmatic Assessment Sub component Academic Advisors ACTIVITIES Meet AA meets with resident 1X every 3 months Prepare AA reviews portfolio Discuss Provides feedback Makes recommendation Resident Promotion Committee
Jane Griffiths Director of Assessment I think that faculty are more engaged - I wonder Are we changing the culture of assessment?
Queen s Family Medicine Logic Model Purpose To ensure assessment practices are aligned with CBME Inputs PASS System Faculty Development 1:1 User Guides To transform Resident residency Orientation education programs from time-based to Competency-based Activities Meet AA meets with resident 1X3 months Prepare AA reviews portfolio Discuss Provides feedback Makes recommendation Resident Promotion Committee Outputs Resident learning plans Completed portfolios Proximal: Program Shift in the culture of assessment Outcomes Distal: System
Queen s Family Medicine Focus made explicit Purpose To ensure assessment practices are aligned with CBME Inputs PASS System Faculty Development 1:1 User Guides To transform Resident residency Orientation education programs from time-based to Competency-based Activities Meet AA meets with resident 1X3 months Prepare AA reviews portfolio Discuss Provides feedback Makes recommendation Resident Promotion Committee Outputs Resident learning plans Completed portfolios Proximal: Program Shift in the culture of assessment Outcomes Distal: System Are we changing the culture of assessment?
Queen s Family Medicine Logic Model Purpose To ensure assessment practices are aligned with CBME Inputs PASS System Faculty Development 1:1 User Guides To transform Resident residency Orientation education programs from time-based to Competency-based Activities Meet AA meets with resident 1X3 months Prepare AA reviews portfolio Discuss Provides feedback Makes recommendation Resident Promotion Committee Outputs Resident learning plans Completed portfolios Proximal: Program Shift in the culture of assessment Outcomes Distal: System Are we changing the culture of assessment?
Queen s Family Medicine Scholarly Component
Queen s Family Medicine Logic Model Purpose Inputs Activities Outputs Outcomes Meet AA meets with resident 1X3 months Proximal: Program Distal: System If Prepare AA reviews portfolio Discuss Provides feedback Makes recommendation Resident Promotion Committee Shift in the culture of assessment Then Are we changing the culture of assessment?
Creating an Evaluation Question What problem do we have that CBME will address? Pre-implementation Are we having an impact on patient care? LT Outcomes ST Outcomes Are we identifying residents earlier who may need additional learning experiences? Cycle of CBME Program Design & Implementation Clarification & Improvement Did we use our resources efficiently? Accountability Design How is our competence committee functioning? Is our competence committee functioning as we planned?
Creating an Evaluation Question 1. Can be answered in a reasonable timeframe and at a reasonable cost 2. Data can be collected: They are not philosophical, moral or religious questions 3. The answer is not predetermined by the phrasing of the question 4. It is an important question and you can say why 5. The answer is actionable Patton, 2012 Specific Measureable Attainable Realistic Timely
Logic Model Evaluating Educational Innovations Purpose What is the problem? Inputs Activities Outputs What resources are required? What activities are critical and/or unique? What products or behaviours arise as a result of this program? Short term Outcomes Medium term Long term What do we anticipate will happen as a result of this program? Needs Assessment Conceptual Framework(s) External Factors Unintended Outcomes What assumptions or conceptual frameworks shape the design of the activities? What conditions exist that influence how the program is delivered? What positive or negative unanticipated outcomes might arise or arise as a result of this program? PROGRAM DELIVERED Formative Process Questions PROGRAM RESULTS Summative Outcome Questions
Your Turn What is your question?
Making your CBME Program Evaluation Scholarly Purpose Inputs Activities Outputs Outcomes Proximal: Program Distal: System Needs Assessment Conceptual Framework(s) External Factors Unintended Outcomes What aspect of CBME do I want to illuminate? Bordage, 2009, Conceptual frameworks to illuminate and magnify, Medical Education
Using Conceptual Frameworks... Allows for predictions, identification of variables Informs how programs lead to outcomes Haji et al., 2013
We all use frameworks (whether we know it or not) Educators and researchers constantly use conceptual frameworks to guide their work, even if they themselves are not consciously aware of the frameworks. Bordage 2009
Conceptual Frameworks What are your assumptions about how CBME works?
Difference between Program Evaluation and Research
Difference between Program Evaluation and Research
Building Capacity for Program Evaluation What is the scope of our CBME evaluation? - one component, entire program?? Who are the key stakeholders? What expertise do we require? Do we have access to the required expertise? How supportive is our leadership? How will our results be used?
Identifying key stakeholders People who can affect or be affected by the program... Decision-makers, Implementers, Managers, Program Users/participants...
Using a Logic Model to Build Capacity Involve all key stakeholders Create a draft logic model Determine the focus for your evaluation Revisit the draft Conduct the evaluation Revisit the draft...
Putting it all together Please may I be excused? My brain is full!
CBME... Start with the end in mind.... We will prepare physicians... Who are able to meet population health needs BETTER?
CBME... Purpose What is the problem? Inputs Activities Outputs What resources are required? What activities are critical and/or unique? What products or behaviours arise as a result of this program? Short term Outcomes Medium term Long term What do we anticipate will happen as a result of this program? Needs Assessment Conceptual Framework(s) External Factors Unintended Outcomes To ensure physicians are well prepared to provide quality patient care in a rapidly changing healthcare environment Enhanced patient care outcomes
CBME... Purpose What is the problem? Inputs Activities Outputs What resources are required? What activities are critical and/or unique? What products or behaviours arise as a result of this program? Short term Outcomes Medium term Long term What do we anticipate will happen as a result of this program? Needs Assessment Conceptual Framework(s) External Factors Unintended Outcomes To ensure physicians are well prepared to provide quality patient care in a rapidly changing healthcare environment If... then If... then If... then If... then Chain of Evidence Enhanced patient care outcomes
A final word of caution...
Workshop Objectives 1. Use a logic model to focus your CBME program evaluation 2. Consider how to make a program evaluation scholarly 3. Consider how to build capacity for program evaluation
Workshop Objectives 1. Use a logic model to focus your CBME program evaluation 2. Consider how to make a program evaluation scholarly 3. Consider how to build capacity for program evaluation Thank you!
CBME Assumptions... Examples The conditions needed for learning are: Exposure to models and exemplars that illustrate the to-be-learned processes and outcomes The opportunity to perform focused practice and rehearsal skills Immediate informative feedback while performing Skills performed in meaningful and diverse settings The opportunity to reflect on performance Swing, S. 2010. Medical Teacher, 32: 663-668.