Assessment-Driven Continuing Professional Development

Transcription

1 Assessment-Driven Continuing Professional Development Implications for Physicians, Assessors, Regulators and Certifying Colleges Author: Dr. Kevin Imrie Date: April 7, 2014

2 Conflict of Interest Declaration v I have been a CPSO peer-assessor v I was Vice-President, Education for the Royal College of Physicians and Surgeons and am their president-elect v I have no financial relationships with members of pharmaceutical or medical supply companies. v I do not hold any research grants funded by industry. v I do not serve on an advisory board of any forprofit industry.

3 Learning Objectives At the end of the presentation, you will: Be knowledgeable about the role of CPD in ensuring ongoing competence Be able to reflect on the role the assessor plays in contributing to CPD Be familiar with the movement towards competency-based education in training and in practice

4 Competency / Competencies An observable ability of a health professional Reflects a spectrum 2009 Royal College and The International CBME Collaborators

5 Competent Possessing the required abilities at a specified stage of medical education Is always qualified by a frame of reference 2009 Royal College and The International CBME Collaborators

6 Competency-Based Medical Education is an outcomes-based approach to the design, implementation, assessment and evaluation of a medical education program using an organizing framework of competencies 2009 Royal College and The International CBME Collaborators

7 IOM report, 2001 Health care harms patients too frequently and routinely fails to deliver its potential benefits. 7

8 Persistent Gap in Performance Consistent evidence of failure to translate research findings into clinical practice 30-40% patients do not get treatments of proven effectiveness % patients get care that is not needed or potentially harmful. Grol R (2001). Med Care 8

9 Persistent Gap in Quality of Care Adherence to recommended health care indicators provided to adults Content Areas Preventative care 54.9% Acute care 53.5% Chronic care 56.1% McGlynn A (2003). NEJM 9

10 Persistent Gap in Patient Safety Quality of care concerns in hospitals Adverse events occur in % of all hospital admissions At the Ottawa Hospital adverse event rate was 12.7% with 38% deemed preventable* - 61% of the events occurred prior to hospital Forster et al CMAJ April 13, 2004

11 Traditional CME 11

12 Our data show some evidence that interactive CME sessions that enhance participant activity and provide the opportunity to practice skills can effect change in professional practice and, on occasion, health care outcomes. Based on a small number of wellconducted trials, didactic sessions do not appear to be effective in changing physician performance. Davis D, JAMA

13 Informs: Role for Assessment in CPD Assessment is an educational imperative Individuals, groups/teams with data or information to identify unperceived needs Guides: Identification and development of learning plans through providing credible feedback 13

14 Assessment pivotal to training 14

15 Assessment in Practice Two Key Questions 1. Is there are role for self-assessment within the spectrum of assessment strategies or options? 2. What does the literature say about the efficacy of formal assessment strategies in promoting learning, enhancing competence or improving performance? 15

16 Accuracy of Self-Assessment Results 17 of 725 articles met inclusion criteria 20 comparisons between self and external measures 13 demonstrated little, no, or an inverse relationship 7 demonstrated a positive association 16

17 Davis D, JAMA While suboptimal in quality, the preponderance of evidence suggests that physicians have a limited ability to accurately self-assess. The processes currently used to undertake professional development and evaluate competence may need to focus more on external assessment. 17

18 Self-Assessment in Practice Key Messages Personal, unguided reflection or any global judgment of one s ability or performance in a particular domain is 1. Poorly performed 2. Conceptually flawed 3. Unlikely to be enhanced through training or education! (Read: it s a waste of time! ) 18

19 Reflecting in action 19

20 Reflecting on action 20

21 Self-Assessment in Practice How accurate are decisions to pause and learn before acting? Knowing when to look it up: a new conception of self-assessment ability. Focus on: Eva, Regehr, Academic Medicine 2007 Situational awareness when at the limits of knowledge, ability, experience OR When confidence in ability is lacking. 21

22 Self-Monitoring in Practice Definition of Self-Monitoring the moment-by-moment awareness of the likelihood that one has the requisite knowledge / skills to act in a particular situation. Conclusion: Greater accuracy in self-monitoring than any global aggregation of performance. Eva and Regehr Adv Health Sci Educ Theory Pract

23 Self-Monitoring of Physicians Focus for Future Research 1. Exploring the concept of self-guided learning. Brydges: Academic Medicine Oct The quality of questions and use of resources in self-directed learning: personal learning projects in the maintenance of certification program. T Horsley: JCEHP

24 Assessment for Learning What is the role for peer or personal assessments of competence or performance within mandatory systems of continuing professional development? 24

25 Physician Assessment Royal College Strategic Plan Included two important goals: 1. Promote competence and performance assessment through engaging in lifelong learning 2. Develop standards for the assessment of performance of physicians in practice. 25

26 Assessment in CPD: Some Key Assumptions Assessment strategies and processes must be: 1. Embedded within the practice context and supported by the health system. 2. Relevant to every dimension of professional practice and across all content domains. 3. Supported within environments that are safe and divorced from threats of litigation or fear of failure. 26

27 Assessment in CPD: Some Key Assumptions Assessment strategies and processes must be: 4. Applicable for individuals, groups or inter-professional health teams. 5. Able to generate data and provide constructive feedback. 6. Able to facilitate the identification of areas where further learning should be focused. 27

28 Categories of Assessment 1. Multi-Source Feedback 2. Simulation 3. Audit and Feedback 4. Learning Portfolios 5. Information Systems 28

29 Multi-Source Feedback Research literature has established 1. Reliability 2. Feasibility and cost effectiveness 3. Educational impact if feedback was Credible, specific and accurate and/or When coaching was provided Sargeant JCEHP 2011 Miller BMJ 2010 Violata C. BMJ

30 PAR: Physician Achievement Review 30

31 Conclusions Multi-Source Feedback Valuable formative assessment strategy for individual physicians. Empiric evidence to establish reliability in content areas applicable to all physicians. Relevant to observable behaviors in three CanMEDS Roles: Communicator, Collaborator, Professional. Quality of the data / feedback provided influence and guide physician response. 31

32 Audit and feedback Impact on performance and health outcomes is small to moderate Relative effectiveness is enhanced when: 1. Baseline compliance with recommended practice was low! 2. Feedback is provided: By a colleague or supervisor More than once Delivered in both written/oral formats Identifies targets with an action plan 32

33 Centrality of Feedback Impact of workplace based assessment on doctor s education and performance: a systematic review Miller, BMJ 2012 Performance changes were more likely to occur when feedback was credible and accurate or when coaching was provided to help subjects identify their strengths and weaknesses. 33

34 Audit and Feedback Conclusions Valuable assessment strategy for individual physicians in their Medical Expert roles. Validity (face and content) based on trustworthy data that is patient specific. Multiple systematic reviews have established the conditions that influence significant behavior change. Feedback is central and critical. Applicable to multiple sources of data. 34

35 Our Current Vision Definition Informed Self-Assessment A set of processes through which individuals use external and internal data to generate an appraisal of their own ability. Mann K, Sargeant J. Acad Med

36 Informed Self-Assessment Conceptual Model 1. Sources of information 2. Interpretation of information 3. Response to information 4. External and internal conditions that influence each of these steps 5. Tensions arising from competing data and external influences Sargeant J. Acad Med

37 Informed Self-Assessment Some key lessons for the future 1. Work place or work environment must support and enable greater access to data with feedback 2. Data and feedback will be essential to maintaining and improving performance and contribute to better health outcomes. 37

38 Some Conclusions Assessment for Learning in CPD Requires the intentional integration of multiple strategies that: 1. Support self-monitoring and the competencies of self-directed learning. 2. Provide credible data with trusted feedback to identify the path to improvement. 38

39 Current status Our Conclusions and Next Steps 1. Assessment must become a MANDATORY COMPONENT of any lifelong learning strategy for practice. 2. We are developing a scoping review to inform a set of recommendations for our Council on strategies and options to affirm the continued competence of Fellows of the Royal College. 39

40 New MOC Framework Section 1 Group Learning Accredited Group Learning Unaccredited Group Learning Section 2 Self-Learning Planned Learning Scanning Activities Systems Learning Section 3 Assessment Knowledge Assessment Performance Assessment

41 MOC Program Changes 1 of 2 (to be reflected in MAINPORT in 2014) Policy Change Fellows and MOC Program participants with new MOC cycles starting on or after January 1, 2014 will be required to complete a minimum of 25 credits in each section of the MOC Program over a 5-year cycle. i.e. min 25 credits in Group Learning, min 25 credits in Self-Learning, and min 25 credits in Assessment across a cycle Rationale: Promotes assessment as a key learning strategy Less restrictive than maximums Removes the ceiling effect of recording credits 41

42 MOC Program Changes - further information To learn more, please visit: members/moc/moc_program For support, please contact the Royal College Services Centre: cpd@royalcollege.ca

43 Competence by Design (CBD) A Vision for Competency-based Education across the continuum Click to edit Master subtitle style

44 CanMEDS 2015: Refresh of our competency framework Practical framework to support competence across the continuum of a physician s career. Robust implementation plan to support Fellows and medical educators with the roll-out of CanMEDS

45 Competence by Design: Vision for Competency-based education across the continuum 45

46 Conclusions Health system and physician performance needs to improve Education alone is insufficient Self-assessment can be effective, but needs to be evidence-informed, guided and credible A competency-based model of medical education across the continuum is needed (and on its way) You as peer-assessors will play a vital role

47 Many Thanks to Dr Craig Campbell Director of CPD Dr Ken Harris Executive Director, Office of Education 47