The Clinical Competency Committee: Data Driven Education & Assessment

Transcription

1 The Clinical Competency Committee: Data Driven Education & Assessment Nathan R. Selden, MD, PhD Campagna Chair of Pediatric Neurosurgery Residency Program Director Chair, Milestones Group for Neurosurgery

2 Trends in Residency Training Driven by two factors A perceived problem: Public, regulatory pressure A potential tool: Enhanced access to granular practice and educational data

3 Role of the ACGME Commitment to reform that is: Reasonable At least potentially effective in improving educational and patient outcomes Not excessively burdensome where possible The influence of neurosurgery has been critical Duty hours, case categories, supervision

4 Concerns Nevertheless Change is happening fast No new resources provided in support

5 Role of the SNS Represent and advocate for PDs and PCs Facilitate collaboration between programs Provide tools to facilitate progress Maintain our historical focus on the importance of mentorship and coaching in surgical education

6 NAS: Resident Evaluation New Common Program Requirements for Resident Evaluation (V.A.1) The program director must appoint the Clinical Competency Committee. CCC must have at least three program faculty CCC members may also include non-physician members of the health care team and residents in their final year 2013 Accreditation Council for Graduate Medical Education (ACGME)

7 NAS: Resident Evaluation New Common Program Requirements for Resident Evaluation (V.A.1) CCC activities include: reviewing all resident evaluations completed by all evaluators semi-annually preparing and ensuring the reporting of Milestones evaluations of each resident semi-annually to the ACGME making recommendations to the program director for resident progress, including promotion, remediation, and dismissal 2013 Accreditation Council for Graduate Medical Education (ACGME)

8 Now: RAC Residency Advisory Committee Program Director, Chair, Program faculty Resident representation Functions Discuss individual resident progress and problems Review and improve curriculum, rotations

9 July: CCC Clinical competency committee Program Director, Chair, Program faculty Resident representation, Non-MD representation Functions Discuss individual resident progress and problems Review and improve curriculum, rotations

10 July: CCC Clinical competency committee Program Director, Chair, Program faculty Resident representation, Non-MD representation Functions Discuss individual resident progress and problems Review and improve curriculum, rotations

11 July: CCC Clinical competency committee Program Director, Chair, Program faculty Resident representation, Non-MD representation Functions Discuss individual resident progress and problems Formulate milestones reports to ACGME

12 July: CCC Clinical competency committee Program Director, Chair, Program faculty Resident representation, Non-MD representation Functions Discuss individual resident progress and problems Formulate milestones reports to ACGME

13 NAS: Resident Evaluation New Common Program Requirements for Resident Evaluation (V.A.1) CCC activities include: reviewing all resident evaluations completed by all evaluators semi-annually preparing and ensuring the reporting of Milestones evaluations of each resident semi-annually to the ACGME making recommendations to the program director for resident progress, including promotion, remediation, and dismissal? 2013 Accreditation Council for Graduate Medical Education (ACGME)

14 Milestones Key features Minimal standards of achievement based on Specific competencies Developmental stage in learning Use objective and subjective, expert information Reproducible Believable

15 Milestones Published by the ACGME 24 one page milestones 16 Medical Knowledge and Patient Care for subspecialties (including Critical Care) 8 General Competencies: Professionalism, Communications, PBLI, SBP Journal of Graduate Medical Education, March 2013

16 Journal of Graduate Medical Education March 2013

17

18 Sub-Competency General Competency Developmental Progression Or Milestone Set Milestone

19

20

21

22

23

24

25

26

27

28

29

30

31

32 Sources of data I. Evaluations II. Portfolio III. Examinations IV. Case data

33 Sources of data I. Evaluations II. Portfolio III. Examinations IV. Case data

34 NAS: Resident Evaluation New Common Program Requirements for Resident Evaluation (V.A.1) CCC activities include: reviewing all resident evaluations completed by all evaluators semi-annually preparing and ensuring the reporting of Milestones evaluations of each resident semi-annually to the ACGME making recommendations to the program director for resident progress, including promotion, remediation, and dismissal 2013 Accreditation Council for Graduate Medical Education (ACGME)

35 Evaluation Types Summative ( rotation ) evaluations General competencies Faculty 360 degree (Self, Nurse, Peer) Patient care Subspecialty specific (8) Formative ( on the fly ) evaluations Clinical encounter Surgical procedure

36 What do we have now? Program Coordinators Survey 88 responses 87% of 101 programs

37 Resident Evaluations 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% 67.0% 85.2% 94.3% Peer 360 Evaluation Self 360 Evaluation Health Professional or Nurse 360 Evaluation Patient 360 Evaluation Faculty General Evaluation Faculty Procedure-based Evaluation Faculty Clinical Encounter Evaluation Before Milestones, what resident evaluations did you generate for your program?

38 Resident Evaluations 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Most programs assign: HP 360s Summative evaluation by faculty Less than half assign: Procedural-based evaluation Clinical encounter evaluation Before Milestones, what resident evaluations did you generate for your program? Peer 360 Evaluation Self 360 Evaluation Health Professional or Nurse 360 Evaluation Patient 360 Evaluation Faculty General Evaluation Faculty Procedure-based Evaluation Faculty Clinical Encounter Evaluation

39 90% of programs use an electronic evaluation system What system does your program currently use in order to evaluate residents? 10.2% 9.1% 40.9% 22.7% 17.0% E-Value MedHub New Innovations Paper Forms Other

40 Evaluations for Milestones Goals Make our existing assessment tools specific to the milestones developmental level descriptions Assist the PD and PC in drafting milestone levels prior to CCC meeting Provide consistency in evaluation of residents between programs

41 Evaluations for Milestones Method Avoid numerical scales Utilize the specific narrative language of the milestone developmental attributes Include the opportunity for narrative comment This helps residents learn how to improve

42

43 Boot Camp

44 Evaluation question Breaking bad news Not observed Unsatisfactory Participates breaking bad news Leads breaking bad news Manages communication of unexpected outcome

45 Evaluation question Breaking bad news Not observed Unsatisfactory Participates breaking bad news Leads breaking bad news Manages communication of unexpected outcome

46

47

48 Program Director

49 Evaluation question Accountability Not observed Unsatisfactory Recognizes personal limits Assumes ownership Leads team

50 Evaluation question Accountability Not observed Unsatisfactory Recognizes personal limits Assumes ownership Leads team

51

52 Patient Care Milestones 8 subspecialty milestone sets Evaluations divided into phases of care Clinical evaluation and work-up PARQ Technical skills Peri-operative care

53

54

55 Evaluation question Work-up and treatment plan Not observed Unsatisfactory Initiates work-up Formulates work-up and treatment plan Formulates plan for patient with comorbidities

56 Evaluation question Work-up and treatment plan Not observed Unsatisfactory Initiates work-up Formulates work-up and treatment plan Formulates plan for patient with comorbidities

57

58 Technical Skill Milestones For each specialty specific procedure type (routine or complex), skill evaluated for 4 components: Positioning, set-up, prep & drape Approach Key portion Closure and transfer to care setting

59 OHSU Evaluation Process Multiple observers complete milestone compatible end of rotation evaluations PC translates evaluation data to the milestone form levels PD reviews these draft milestone assignments The Clinical Competency Committee (CCC) can focus efficiently Discrepancies between different observers Residents failing to progress appropriately

60 OHSU Evaluation Process Final step does not change Direct feedback to individual residents Program director Faculty mentor

61 Sources of data I. Evaluations II. Portfolio III. Examinations IV. Case data

62 Portfolio Some milestones fulfillment material is part of the resident portfolio

63

64 Portfolio PC and PD portfolio review Contributes peer reviewed literature Accurate/timely ACGME case log Accurate/timely duty hours log Receives patient praise notices Punctual for conferences Organizes educational activities Prepares for transition to practice

65 Resident Requirements 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% ABNS Board Exam for Self- Assessment ACGME Case Log entries Boot Camp - Interns Boot Camp - Juniors Courses/Didactics: Off- Campus Courses/Didactics: On- Campus Peer-reviewed Publications Quality Committee Involvement / Other Committee Involvement Quality Improvement Project SANS - Self-assessment in Neurological Surgery Simulations Written reports (unpublished, for portfolio only) What portfolio requirements or training tools do you currently require your residents to complete?

66 Resident Requirements 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 75.0% 44.3% Milestones: Publications required in PBLI: Research QI assessment in SPB: Safety & Systems What portfolio requirements or training tools do you currently require your residents to complete? 51.1% ABNS Board Exam for Self- Assessment ACGME Case Log entries Boot Camp - Interns Boot Camp - Juniors Courses/Didactics: Off- Campus Courses/Didactics: On- Campus Peer-reviewed Publications Quality Committee Involvement / Other Committee Involvement Quality Improvement Project SANS - Self-assessment in Neurological Surgery Simulations Written reports (unpublished, for portfolio only)

67 Portfolio Educational resources Lists E&M code elements Implements EMR template Creates/updates order set Participates in QI Basic clinical epidemiology Study design and quality Utilizes registry data

68 Portfolio Educational resources Lists E&M code elements Coding module Implements EMR template EMR project Creates/updates order set EMR project Participates in QI QI project Basic clinical epidemiology HIP course Study design and quality HIP course Utilizes registry data Registry module

69 Portfolio SNS help with fundamentals ICP monitor placement EVD placement Central line placement Breaking bad news Informed consent Hand offs Critical event management

70 Portfolio SNS help with fundamentals ICP monitor placement SNS Boot Camp EVD placement SNS Boot Camp Central line placement SNS Boot Camp Breaking bad news SNS JR Course Informed consent SNS JR Course Hand offs SNS JR Course Critical event management SNS BC & JRC

71 Sources of data I. Evaluations II. Portfolio III. Examinations IV. Case data

72 Examinations MK milestones require input from knowledge based examinations ABNS Primary Examination SANS Program based testing

73 Sources of data I. Evaluations II. Portfolio III. Examinations IV. Case data

74 New Case Categories: Adult Cranial CRANIAL # DC1 Craniotomy for brain tumors 60 DC2 Craniotomy for trauma 40 DC3a Craniotomy for intracranial vascular lesion 40 DC3b Endovascular surgery for tumors or vascular lesions 10 DC4 Craniotomy for pain 5 DC5 Transsphenoidal sellar/parasellar tumors (endoscopic and 15 microsurgical) DC6 Extracranial vascular procedures 5 DC7 Radiosurgery 10 DC8 Functional procedures 10 DC9 VP shunt 10 SUBTOTAL 205

75 Evaluations Exams Part I Boards SANS Portfolio Case Data CCC Consensus

76 Analysis The CCC provides consensus from both Objective data Subjective data Providing Reproducibility Believability

77 SNS Resources SNS Program Director Tool Kit Online repository for curriculum and administration Distribute evaluations Share resources Discuss methods

78 SNS Resources Neurosurgical Portal Comprehensive online learning tool Based on Matrix Curriculum with didactics Radiology, Pathology, Anatomy Operative videos Lectures: CNS University, SNS-AANS Modules Focused assessment: SANS Linked to Milestones Automatic reporting to PDs

79 Evaluations Exams Part I Boards SANS Portfolio Case Data CCC Consensus

80 Resident Promotion Aided by milestone comparisons Initially: to peers in program Eventually: to national specialty benchmarks Tempo of individual resident development Can vary within limits Endpoint for safe independent practice Does not vary Level 4 is required for graduation

81 Resident Promotion Failure to progress Remediation Assign mentor Require additional readings, SANS, testing Assign skills lab and/or simulator practice Add or modify rotations Repurposing or separation

82 NAS: Annual Program Evaluation New Common Program Requirements for Annual Program Evaluation (V.C.1) Program director must appoint Program Evaluation Committee (PEC) PEC members: at least 3 program faculty; representation from residents Written description of PEC responsibilities PEC plans, develops implements evaluates program activities, develops competency-based goals and objectives, conducts annual program review, ensures areas of non-compliance are corrected 2013 Accreditation Council for Graduate Medical Education (ACGME)

83 NAS: Annual Program Evaluation New Common Program Requirements for Annual Program Evaluation (V.C.2) The program, through the PEC, must document formal, systematic evaluation of the curriculum at least annually, and is responsible for rendering a full, written annual program evaluation (APE) Accreditation Council for Graduate Medical Education (ACGME)

84 Now: RAC Residency Advisory Committee Program Director, Chair, Program faculty Resident representation Functions Discuss individual resident progress and problems Review and improve curriculum, rotations

85 July: PEC Program evaluation committee Program Director, Chair, Program faculty Resident representation Functions Discuss individual resident progress and problems Review and improve curriculum, rotations

86 July: CCC-PEC (OHSU) Program evaluation committee Program Director, Chair, Program faculty Resident representation, Non-MD representation Functions Discuss individual resident progress and problems Formulate milestones reports to the ACGME Review and improve curriculum, rotations

87 Program Director Concerns Faculty Burden Time One every 6 months Combine CCC and PEC Individual evaluation burden about the same Benefits CCC service is mark of seniority Counts for P&T under Education mission Ability to favorably influence residents and the residency

88 Program Director Concerns PC Burden Unfunded mandate

89 About two thirds of PCs hold an A.A. or coordinator job title Which selection best describes your official position title at your institution? 6.8% 25.0% Administrative Assistant Administrative Coordinator 1.1% 9.1% 58.0% Manager Director Other

90 Program Director Concerns PC Burden Emerging duties may be more suited to manager or director level

91 About two-thirds have additional non-nsg residency duties What percentage of your position's FTE is dedicated to Neurological Surgery Residency tasks? 1.1% 10.2% 29.5% 34.1% 100% 76% - 99% 51% - 75% 26% - 50% 25% or less 25.0%

92 Program Director Concerns PC Burden Emerging duties may be more suited to manager or director level Milestones (and NAS in general) will likely require some increase in FTE

93 Over four-fifths do not have additional residency support staff Is there another person who works with you on Neurological Surgery Residency tasks? 6.8% 11.4% No Yes - Part-time Yes - Full-time 81.8%

94 About three fifths predict no increase in residency program FTE Will your department increase the FTE dedicated to Neurological Surgery Residency tasks due to NAS and the Milestones? 9.1% 33.0% Yes No I Don't Know 58.0%

95 Program Director Concerns Readiness Have you held a "mock" Clinical Competency Committee meeting with your program faculty? 24.1% Not yet Yes 75.9%

96 Program Director Concerns Timing Important to appoint CCC now Pilot CCC meeting to trouble shoot before December

97 Challenge What the biggest challenge to these educational reforms? On Sunday, at this meeting, we heard concern that scientists and physicians are collecting huge data sets they are not adequately analyzing and using for change RESIDENCY TRAINING FACES THE SAME PROBLEM

98 Imperatives Avoid focus on data collection and churning Let your PC run these systems Elevate their skills and responsibility Focus faculty on data analysis and overview Utilize their expertise

99 Imperatives Emphasize the wisdom and and judgment of faculty educators about trainees Informed by the assessment, milestones and other data (not replaced) Pause activity for a focused CCC and PEC mtg A half dozen key faculty with full attention to creative thinking about residents and the residency program

100 Timeline June Appoint CCC members and prepare for milestones evaluations July to December First evaluation period December First milestones evaluations submitted to ACGME

101 PD-PC Workshop Saturday October 19 th, San Francisco, CA In conjunction with the CNS Annual Meeting PC13 Free registration for PDs, PCs and other departmental educators PCs can access the meeting space without registration What to do with the data? Identifying problems Helping residents succeed

102 Thanks Milestones Group Aviva Abosch (Functional) Richard Byrne (Tumor) Robert Harbaugh (Critical Care) William Krauss (Spine) Timothy Mapstone (Pediatrics, Matrix Curriculum) Oren Sagher (Pain & Peripheral Nerve) Nathan Selden (General Competencies) - Chair Gregory Zipfel (Vascular)

103 Thanks Advisory Group Dan Barrow ABNS Past Chair Hunt Batjer Chair, RRC Kim Burchiel President-elect, SNS Ralph Dacey President, SNS Arthur Day SNS Past President Allan Friedman SNS Past President Fred Meyer ABNS Secretary ACGME Pam Derstine Exec Dir, Neurosurgery RRC Laura Edgar Milestones Project Lead

104 Thanks SNS CoRE Subcommittee Chairs Rich Byrne SNS Courses Sander Connolly Modules Tim Mapstone Curriculum Warren Selman Portal Greg Zipfel Tool Kit