The Clinical Competency Committee: Data Driven Education & Assessment Nathan R. Selden, MD, PhD Campagna Chair of Pediatric Neurosurgery Residency Program Director Chair, Milestones Group for Neurosurgery
Trends in Residency Training Driven by two factors A perceived problem: Public, regulatory pressure A potential tool: Enhanced access to granular practice and educational data
Role of the ACGME Commitment to reform that is: Reasonable At least potentially effective in improving educational and patient outcomes Not excessively burdensome where possible The influence of neurosurgery has been critical Duty hours, case categories, supervision
Concerns Nevertheless Change is happening fast No new resources provided in support
Role of the SNS Represent and advocate for PDs and PCs Facilitate collaboration between programs Provide tools to facilitate progress Maintain our historical focus on the importance of mentorship and coaching in surgical education
NAS: Resident Evaluation New Common Program Requirements for Resident Evaluation (V.A.1) The program director must appoint the Clinical Competency Committee. CCC must have at least three program faculty CCC members may also include non-physician members of the health care team and residents in their final year 2013 Accreditation Council for Graduate Medical Education (ACGME)
NAS: Resident Evaluation New Common Program Requirements for Resident Evaluation (V.A.1) CCC activities include: reviewing all resident evaluations completed by all evaluators semi-annually preparing and ensuring the reporting of Milestones evaluations of each resident semi-annually to the ACGME making recommendations to the program director for resident progress, including promotion, remediation, and dismissal 2013 Accreditation Council for Graduate Medical Education (ACGME)
Now: RAC Residency Advisory Committee Program Director, Chair, Program faculty Resident representation Functions Discuss individual resident progress and problems Review and improve curriculum, rotations
July: CCC Clinical competency committee Program Director, Chair, Program faculty Resident representation, Non-MD representation Functions Discuss individual resident progress and problems Review and improve curriculum, rotations
July: CCC Clinical competency committee Program Director, Chair, Program faculty Resident representation, Non-MD representation Functions Discuss individual resident progress and problems Review and improve curriculum, rotations
July: CCC Clinical competency committee Program Director, Chair, Program faculty Resident representation, Non-MD representation Functions Discuss individual resident progress and problems Formulate milestones reports to ACGME
July: CCC Clinical competency committee Program Director, Chair, Program faculty Resident representation, Non-MD representation Functions Discuss individual resident progress and problems Formulate milestones reports to ACGME
NAS: Resident Evaluation New Common Program Requirements for Resident Evaluation (V.A.1) CCC activities include: reviewing all resident evaluations completed by all evaluators semi-annually preparing and ensuring the reporting of Milestones evaluations of each resident semi-annually to the ACGME making recommendations to the program director for resident progress, including promotion, remediation, and dismissal? 2013 Accreditation Council for Graduate Medical Education (ACGME)
Milestones Key features Minimal standards of achievement based on Specific competencies Developmental stage in learning Use objective and subjective, expert information Reproducible Believable
Milestones Published by the ACGME 24 one page milestones 16 Medical Knowledge and Patient Care for subspecialties (including Critical Care) 8 General Competencies: Professionalism, Communications, PBLI, SBP Journal of Graduate Medical Education, March 2013
Journal of Graduate Medical Education March 2013
Sub-Competency General Competency Developmental Progression Or Milestone Set Milestone
Sources of data I. Evaluations II. Portfolio III. Examinations IV. Case data
Sources of data I. Evaluations II. Portfolio III. Examinations IV. Case data
NAS: Resident Evaluation New Common Program Requirements for Resident Evaluation (V.A.1) CCC activities include: reviewing all resident evaluations completed by all evaluators semi-annually preparing and ensuring the reporting of Milestones evaluations of each resident semi-annually to the ACGME making recommendations to the program director for resident progress, including promotion, remediation, and dismissal 2013 Accreditation Council for Graduate Medical Education (ACGME)
Evaluation Types Summative ( rotation ) evaluations General competencies Faculty 360 degree (Self, Nurse, Peer) Patient care Subspecialty specific (8) Formative ( on the fly ) evaluations Clinical encounter Surgical procedure
What do we have now? Program Coordinators Survey 88 responses 87% of 101 programs
Resident Evaluations 100.0% 90.0% 80.0% 70.0% 60.0% 50.0% 40.0% 30.0% 20.0% 10.0% 0.0% 67.0% 85.2% 94.3% Peer 360 Evaluation Self 360 Evaluation Health Professional or Nurse 360 Evaluation Patient 360 Evaluation Faculty General Evaluation Faculty Procedure-based Evaluation Faculty Clinical Encounter Evaluation Before Milestones, what resident evaluations did you generate for your program?
Resident Evaluations 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Most programs assign: HP 360s Summative evaluation by faculty Less than half assign: Procedural-based evaluation Clinical encounter evaluation Before Milestones, what resident evaluations did you generate for your program? Peer 360 Evaluation Self 360 Evaluation Health Professional or Nurse 360 Evaluation Patient 360 Evaluation Faculty General Evaluation Faculty Procedure-based Evaluation Faculty Clinical Encounter Evaluation
90% of programs use an electronic evaluation system What system does your program currently use in order to evaluate residents? 10.2% 9.1% 40.9% 22.7% 17.0% E-Value MedHub New Innovations Paper Forms Other
Evaluations for Milestones Goals Make our existing assessment tools specific to the milestones developmental level descriptions Assist the PD and PC in drafting milestone levels prior to CCC meeting Provide consistency in evaluation of residents between programs
Evaluations for Milestones Method Avoid numerical scales Utilize the specific narrative language of the milestone developmental attributes Include the opportunity for narrative comment This helps residents learn how to improve
Boot Camp
Evaluation question Breaking bad news Not observed Unsatisfactory Participates breaking bad news Leads breaking bad news Manages communication of unexpected outcome
Evaluation question Breaking bad news Not observed Unsatisfactory Participates breaking bad news Leads breaking bad news Manages communication of unexpected outcome
Program Director
Evaluation question Accountability Not observed Unsatisfactory Recognizes personal limits Assumes ownership Leads team
Evaluation question Accountability Not observed Unsatisfactory Recognizes personal limits Assumes ownership Leads team
Patient Care Milestones 8 subspecialty milestone sets Evaluations divided into phases of care Clinical evaluation and work-up PARQ Technical skills Peri-operative care
Evaluation question Work-up and treatment plan Not observed Unsatisfactory Initiates work-up Formulates work-up and treatment plan Formulates plan for patient with comorbidities
Evaluation question Work-up and treatment plan Not observed Unsatisfactory Initiates work-up Formulates work-up and treatment plan Formulates plan for patient with comorbidities
Technical Skill Milestones For each specialty specific procedure type (routine or complex), skill evaluated for 4 components: Positioning, set-up, prep & drape Approach Key portion Closure and transfer to care setting
OHSU Evaluation Process Multiple observers complete milestone compatible end of rotation evaluations PC translates evaluation data to the milestone form levels PD reviews these draft milestone assignments The Clinical Competency Committee (CCC) can focus efficiently Discrepancies between different observers Residents failing to progress appropriately
OHSU Evaluation Process Final step does not change Direct feedback to individual residents Program director Faculty mentor
Sources of data I. Evaluations II. Portfolio III. Examinations IV. Case data
Portfolio Some milestones fulfillment material is part of the resident portfolio
Portfolio PC and PD portfolio review Contributes peer reviewed literature Accurate/timely ACGME case log Accurate/timely duty hours log Receives patient praise notices Punctual for conferences Organizes educational activities Prepares for transition to practice
Resident Requirements 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% ABNS Board Exam for Self- Assessment ACGME Case Log entries Boot Camp - Interns Boot Camp - Juniors Courses/Didactics: Off- Campus Courses/Didactics: On- Campus Peer-reviewed Publications Quality Committee Involvement / Other Committee Involvement Quality Improvement Project SANS - Self-assessment in Neurological Surgery Simulations Written reports (unpublished, for portfolio only) What portfolio requirements or training tools do you currently require your residents to complete?
Resident Requirements 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 75.0% 44.3% Milestones: Publications required in PBLI: Research QI assessment in SPB: Safety & Systems What portfolio requirements or training tools do you currently require your residents to complete? 51.1% ABNS Board Exam for Self- Assessment ACGME Case Log entries Boot Camp - Interns Boot Camp - Juniors Courses/Didactics: Off- Campus Courses/Didactics: On- Campus Peer-reviewed Publications Quality Committee Involvement / Other Committee Involvement Quality Improvement Project SANS - Self-assessment in Neurological Surgery Simulations Written reports (unpublished, for portfolio only)
Portfolio Educational resources Lists E&M code elements Implements EMR template Creates/updates order set Participates in QI Basic clinical epidemiology Study design and quality Utilizes registry data
Portfolio Educational resources Lists E&M code elements Coding module Implements EMR template EMR project Creates/updates order set EMR project Participates in QI QI project Basic clinical epidemiology HIP course Study design and quality HIP course Utilizes registry data Registry module
Portfolio SNS help with fundamentals ICP monitor placement EVD placement Central line placement Breaking bad news Informed consent Hand offs Critical event management
Portfolio SNS help with fundamentals ICP monitor placement SNS Boot Camp EVD placement SNS Boot Camp Central line placement SNS Boot Camp Breaking bad news SNS JR Course Informed consent SNS JR Course Hand offs SNS JR Course Critical event management SNS BC & JRC
Sources of data I. Evaluations II. Portfolio III. Examinations IV. Case data
Examinations MK milestones require input from knowledge based examinations ABNS Primary Examination SANS Program based testing
Sources of data I. Evaluations II. Portfolio III. Examinations IV. Case data
New Case Categories: Adult Cranial CRANIAL # DC1 Craniotomy for brain tumors 60 DC2 Craniotomy for trauma 40 DC3a Craniotomy for intracranial vascular lesion 40 DC3b Endovascular surgery for tumors or vascular lesions 10 DC4 Craniotomy for pain 5 DC5 Transsphenoidal sellar/parasellar tumors (endoscopic and 15 microsurgical) DC6 Extracranial vascular procedures 5 DC7 Radiosurgery 10 DC8 Functional procedures 10 DC9 VP shunt 10 SUBTOTAL 205
Evaluations Exams Part I Boards SANS Portfolio Case Data CCC Consensus
Analysis The CCC provides consensus from both Objective data Subjective data Providing Reproducibility Believability
SNS Resources SNS Program Director Tool Kit Online repository for curriculum and administration Distribute evaluations Share resources Discuss methods
SNS Resources Neurosurgical Portal Comprehensive online learning tool Based on Matrix Curriculum with didactics Radiology, Pathology, Anatomy Operative videos Lectures: CNS University, SNS-AANS Modules Focused assessment: SANS Linked to Milestones Automatic reporting to PDs
Evaluations Exams Part I Boards SANS Portfolio Case Data CCC Consensus
Resident Promotion Aided by milestone comparisons Initially: to peers in program Eventually: to national specialty benchmarks Tempo of individual resident development Can vary within limits Endpoint for safe independent practice Does not vary Level 4 is required for graduation
Resident Promotion Failure to progress Remediation Assign mentor Require additional readings, SANS, testing Assign skills lab and/or simulator practice Add or modify rotations Repurposing or separation
NAS: Annual Program Evaluation New Common Program Requirements for Annual Program Evaluation (V.C.1) Program director must appoint Program Evaluation Committee (PEC) PEC members: at least 3 program faculty; representation from residents Written description of PEC responsibilities PEC plans, develops implements evaluates program activities, develops competency-based goals and objectives, conducts annual program review, ensures areas of non-compliance are corrected 2013 Accreditation Council for Graduate Medical Education (ACGME)
NAS: Annual Program Evaluation New Common Program Requirements for Annual Program Evaluation (V.C.2) The program, through the PEC, must document formal, systematic evaluation of the curriculum at least annually, and is responsible for rendering a full, written annual program evaluation (APE). 2013 Accreditation Council for Graduate Medical Education (ACGME)
Now: RAC Residency Advisory Committee Program Director, Chair, Program faculty Resident representation Functions Discuss individual resident progress and problems Review and improve curriculum, rotations
July: PEC Program evaluation committee Program Director, Chair, Program faculty Resident representation Functions Discuss individual resident progress and problems Review and improve curriculum, rotations
July: CCC-PEC (OHSU) Program evaluation committee Program Director, Chair, Program faculty Resident representation, Non-MD representation Functions Discuss individual resident progress and problems Formulate milestones reports to the ACGME Review and improve curriculum, rotations
Program Director Concerns Faculty Burden Time One every 6 months Combine CCC and PEC Individual evaluation burden about the same Benefits CCC service is mark of seniority Counts for P&T under Education mission Ability to favorably influence residents and the residency
Program Director Concerns PC Burden Unfunded mandate
About two thirds of PCs hold an A.A. or coordinator job title Which selection best describes your official position title at your institution? 6.8% 25.0% Administrative Assistant Administrative Coordinator 1.1% 9.1% 58.0% Manager Director Other
Program Director Concerns PC Burden Emerging duties may be more suited to manager or director level
About two-thirds have additional non-nsg residency duties What percentage of your position's FTE is dedicated to Neurological Surgery Residency tasks? 1.1% 10.2% 29.5% 34.1% 100% 76% - 99% 51% - 75% 26% - 50% 25% or less 25.0%
Program Director Concerns PC Burden Emerging duties may be more suited to manager or director level Milestones (and NAS in general) will likely require some increase in FTE
Over four-fifths do not have additional residency support staff Is there another person who works with you on Neurological Surgery Residency tasks? 6.8% 11.4% No Yes - Part-time Yes - Full-time 81.8%
About three fifths predict no increase in residency program FTE Will your department increase the FTE dedicated to Neurological Surgery Residency tasks due to NAS and the Milestones? 9.1% 33.0% Yes No I Don't Know 58.0%
Program Director Concerns Readiness Have you held a "mock" Clinical Competency Committee meeting with your program faculty? 24.1% Not yet Yes 75.9%
Program Director Concerns Timing Important to appoint CCC now Pilot CCC meeting to trouble shoot before December
Challenge What the biggest challenge to these educational reforms? On Sunday, at this meeting, we heard concern that scientists and physicians are collecting huge data sets they are not adequately analyzing and using for change RESIDENCY TRAINING FACES THE SAME PROBLEM
Imperatives Avoid focus on data collection and churning Let your PC run these systems Elevate their skills and responsibility Focus faculty on data analysis and overview Utilize their expertise
Imperatives Emphasize the wisdom and and judgment of faculty educators about trainees Informed by the assessment, milestones and other data (not replaced) Pause activity for a focused CCC and PEC mtg A half dozen key faculty with full attention to creative thinking about residents and the residency program
Timeline June 2013 - Appoint CCC members and prepare for milestones evaluations July to December 2013 - First evaluation period December 2013 - First milestones evaluations submitted to ACGME
PD-PC Workshop Saturday October 19 th, San Francisco, CA In conjunction with the CNS Annual Meeting PC13 Free registration for PDs, PCs and other departmental educators PCs can access the meeting space without registration What to do with the data? Identifying problems Helping residents succeed
Thanks Milestones Group Aviva Abosch (Functional) Richard Byrne (Tumor) Robert Harbaugh (Critical Care) William Krauss (Spine) Timothy Mapstone (Pediatrics, Matrix Curriculum) Oren Sagher (Pain & Peripheral Nerve) Nathan Selden (General Competencies) - Chair Gregory Zipfel (Vascular)
Thanks Advisory Group Dan Barrow ABNS Past Chair Hunt Batjer Chair, RRC Kim Burchiel President-elect, SNS Ralph Dacey President, SNS Arthur Day SNS Past President Allan Friedman SNS Past President Fred Meyer ABNS Secretary ACGME Pam Derstine Exec Dir, Neurosurgery RRC Laura Edgar Milestones Project Lead
Thanks SNS CoRE Subcommittee Chairs Rich Byrne SNS Courses Sander Connolly Modules Tim Mapstone Curriculum Warren Selman Portal Greg Zipfel Tool Kit