The ACGME Self-Study: Key Roles for Program Coordinators

Transcription

1 Accreditation Council for Graduate Medical Education The ACGME Self-Study: Key Roles for Program Coordinators Ingrid Philibert, PhD, MBA, Senior Vice President for Field Activities 2014 Accreditation Council for Graduate Medical Education

2 Disclosures Employed by the ACGME; no other items requiring disclosure Objectives Disclosures and Objectives Describe what is new about the self-study and the 10-year site visit Discuss the elements of the self-study Explore the concept of continuous improvement Describe the role of the program coordinator Offer practical suggestions for program coordinator involvement in self-study preparation and process

3 The Next Accreditation System Annual data collection and review A Self-Study and a site visit every 10 years Increased focus on continuous improvement Institutional oversight Ongoing assessment and improvement using the Annual Program Evaluation Programs with a status of Continued Accreditation free to innovate

4 The Program Self-Study A comprehensive review of the program Information on how the program creates an effective learning and working environment and how this leads to desired educational outcomes Analysis of strengths, weaknesses, opportunities and threats, and ongoing plans for improvement months later: the 10-year site visit Time lag is by design to give programs time to make improvements

5 Rationale A self-study without a concurrent site visit allows for a frank and forthright review of the program 12 to 18-month time lag between self-study and 10- year visit allows programs to make improvements Program communicates improvements to Review Committee before 10-year site visit, paired with observation on program aims and context from self-study summary Program can provide addendum to summary document as needed Planned: Comprehensive assessment of the utility and benefits of the self-study approach

6 The Scoop : A Pilot of an Added Voluntary Visit after Completing the Self-Study The Self-Study Pilot Visit Field staff with special added training review and offer feedback on the self-study to further progress toward improvement and meeting aspirational goals Not an accreditation visit Program volunteers for the visit Programs Eligible for the Pilot Phase I programs with an initial 10-year site visit between April 2015 and July 2016 ACGME will await results from initial pilot before extending to other Phase I programs and/or Phase II programs

7 Aims A Pilot: An Added Voluntary Visit After Completion of the Self-Study (cont). Learn if a non-accreditation site visit with feedback accelerates program self-improvement Learn about effective approaches for conducting the self-study Participation Process Completely voluntary; ACGME contacts eligible programs and asks if interested in participating Concurrent Small delay for programs not identified as alpha programs (alpha programs have been notified) Other programs with an early 10-year visit also get a small delay in the start date for their self-study ACGME will communicate self-study start date

8 A Time Line, Program with July Year Visit (Voluntary segments in shown in red) Time ACGME Program March 2015 July 2015 (12 days pilot visit) July 2015 August 2015 July 2016 October 2016 (12 days before 10-year visit) October 2016 Winter 2016/17 meeting DFA sends notice to begin self study Asks program to volunteer for self-study pilot visit If Yes: Self-study pilot visit Team sends report to program Sends notice of 10-year site visit 10-year site visit Review Committee reviews 10-year visit and self-study Conducts Self Study Uploads self-study summary to ADS Program can update selfstudy summary if desired Updates ADS data, uploads summary of improvements

9 After the Self Study: Program Prepares Self-Study Summary (All Programs) Brief (4 to 5 pages, ~ 2300 word) summary of key dimensions of the Self-Study Aims External environmental assessment (Opportunities and Threats) Process of the Annual Program Evaluation and the Self-Study Learning that occurred during the self-study (Optional!) Information on areas for improvement identified in the self-study not included in the Summary Summary is uploaded into ADS

10 The Self-Study Pilot Visit (Only Programs that Volunteer) Visit is based on the program s self-study summary Information on areas for improvement shared verbally only by program leaders during the site visit A specially trained team of 2 site visitors Different team from the 10-year site visit Team offers verbal feedback Dialogue on strengths and areas/suggestions for improvement the program identified in its self-study Team prepares written report and shares with program Report is NOT shared with the Review Committee Program may update its self-study summary in response to the feedback

11 The 10-Year Accreditation Site Visit (All Programs) The 12- to 18-month period is by design to allow programs implement improvements Different team of site visitors A PIF-Less Visit Program update their self-study summary and provides information ONLY on the improvements that were realized from their self-study No request for information on areas that have not been resolved Team provides verbal feedback Key strengths and suggestions for improvement Team prepares a written report for the Review Committee

12 Review Committee Review of the 10-Year Visit (All Programs) Available to the Review Committee ADS Data The program s summary from the self-study 12 to 18 months earlier The program s summary of improvements achieved as a result of the self-study No data collected on areas still in need of improvement The site visitors report from the 10-year site visit (a full accreditation visit) Review of program aims, context and the improvements from the self-study allows the RC to assess the effectiveness of the self-study, with data on the improvements achieved as 1 measure of effectiveness

13 Review Committee Actions (All Programs) Review Committee provides a Letter of Notification from the Full Accreditation Site Visit Citations Areas for improvement Review Committee provides feedback on the Self- Study taking into consideration Program aims and context Improvements reported and verified during the 10-year visit Effectiveness of the self-study, based on the improvements the program reported it made as a result of its self-study No accreditation impact for initial feedback on the self-study

14 Self-Study Elements

15 Self-Study Scope Assesses current performance and ongoing improvement effort Initial period: since the program s last accreditation review Ultimately, a 10-year interval Reviews improvement activities, successes achieved, and areas in need of improvement Uses data from successive Annual Program Evaluations, ACGME data, other relevant information

16 Self-Study Objectives Assess compliance and improvement using data from prior Annual Program Evaluations and data collected/aggregated for the self-study Focus on Program Strengths Program Areas for Improvement Consider Program Aims The program s external environment Environmental Opportunities Environmental Threats Track ongoing improvements and the success of actions taken

17 Program aims New Areas: Program Aims and Opportunities and Threats Should be realistic Evaluate effectiveness in meeting aims Assess relevant initiatives and their outcomes Opportunities and Threats Assess how factors and contexts external to the program (eg, institutional, local, regional and national) that affect the program Opportunities: Factors that favor the program, that the program may take advantage of Threats: Factors that pose risks

18 Defining Program Aims Set aims as part of the annual program evaluation Relevant considerations Who are our residents/fellows? What do we prepare them for? Fellowship Academic practice Leadership and other roles Who are the patients/populations we care for?

19 Strengths and Areas for Improvements Strengths and Areas for Improvements identified by: Citations, areas for improvement and other information from ACGME The Annual Program Evaluation Other program/institutional data sources Data on improvements should pertain to the period since the program s last accreditation review

20 Enhancing the Definition of Areas for Improvement Negative aspects that detract from the value of the program or place it at a disadvantage. What factors within your control detract from the ability to maintain a high-performing program? What areas need improvement to accomplish objectives or enhance or supplement your existing strengths? What does your program lack (expertise in a certain subspecialty, a type of technology, access to a particular patient population, faculty with interest and skill in research)? Is there a lack of some types of resources; is the institution constrained in its capacity to provide support?

21 Defining Opportunities Opportunities are external attractive factors that, if acted upon, will contribute to the program flourishing. What are your capabilities for further evolving the program; how can you capitalize on them? Has there been recent change in your immediate context that that creates an opportunity for your program? Are these opportunities ongoing, or is there a narrow window for them? How critical is the timing?

22 Defining Threats Threats include external factors that affect the program. While the program cannot control them, beneficial to have plans to address them if they occur. What factors beyond your control place your program at risk? What are changes in residents specialty choice, regulation, or other factors that may affect the future success of your program? Are there challenges or unfavorable trends in your immediate context that may affect your program? E.g. faculty burdened with heavy clinical load that prevents effective teaching and mentorship

23 Benefits of a Focus on Program Aims Suggests a relevant dimension of the program: What kinds of graduates do we produce for what kinds of practice settings and roles? Allows for a more tailored approach to creating a learning environment Focus on specific aims can produce highly desirable graduates that match patient and healthcare system needs (1) Enhances the focus on functional capabilities of graduating residents Fits with a milestones-based approach to assessment Hodges BD. A Tea-Steeping or i-doc Model for Medical Education?, Academic Medicine, vol. 85, No. 9/September Supplement, 2010, pp. S34-S44.

24 Benefit of a Focus on Environmental Context Facilitates assessment of the program s performance in its local environment What are program strengths? What should definitely be continued (important question in an environment of limited resources) What are areas for improvement? Prioritize by relevance to program aims, compliance, importance to stakeholders Useful for all programs, particularly highperforming programs: What will take our program to the next level?

25 The Self-Study Process

26 Who Should Organize and Conduct the Self-Study? Not defined by ACGME Members of the Program Evaluation Committee are the logical choice Natural extension of improvement process through the Annual Program Evaluation PEC requirements 1 The PEC must be composed of at least 2 faculty members and at least 1 resident (core) ; must have a written description of responsibilities (core) PEC Membership may be expanded for the Self- Study 1 ACGME Common Program Requirements, Effective July 2013

27 The Program Evaluation Committee a) must be composed of at least two program faculty members and should include at least one resident; (Core) b) have a written description.; and c) participate actively in: should participate actively in: planning, developing, implementing, and evaluating educational activities of the program; (Detail) reviewing and making recommendations for revision of competency-based curriculum goals and objectives; (Detail) addressing areas of non-compliance with ACGME standards; and, (Detail) reviewing the program annually using evaluations of faculty, residents, and others, as specified below. (Detail) Yes resident members Interface with CCC Citations and actions to address

28 Self-Study Data Gathering Annual Program Evaluation data, ACGME Resident and Faculty Survey data, other program and institutional data Focus on data gathering as a learning exercise Evaluate strengths and areas for improvement Explore opportunities and threats Reflect stakeholder (residents, faculty, and relevant others) participation, input and perspective Data provide evidence to support conclusions

29 Self-Study Data Gathering (2) Interviews Verify and validate data Identify areas that have been resolved and areas and priorities for improvement Identify program strengths Review and revise program aims Assess and validate strengths, weaknesses, opportunities and threats

30 Range of Potential Annual Evaluation and Self- Study Inputs Narrative input from interviews, focus groups Case Logs End of rotation evaluations Faculty Survey Curriculum Resident Survey ITE Milestone Data Board Performance Citations and Responses If done, Special Review and other GMEC Reviews Program Evaluation Committee Program Performance and Improvement CCC Feedback on Assessment System

31 The Shewhart PDSA Cycle PLAN prepare the change DO implement the change STUDY monitor and analyze impact of change ACT revise and standardize the change

32 Identifying Areas for Improvement Based on data and facts Focus on learners, patients and other stakeholders Prioritize (cannot improve everything at once) Systems Thinking Program and institutional systems relevant to resident education and the area needing improvement Process Thinking Processes: sets of related tasks used to accomplish something Processes are focal areas for improvement

33 Tracking Improvements Design and Implement solutions Identify individual or group that will be responsible Identify and secure resources Timeline Follow-up is key: ensure all issues addressed Documentation to facilitate ongoing tracking Example: A simple spreadsheet recording improvements achieved and ongoing priorities Record over multiple years of improvement

34 Sample Improvement Plan Area for Improvement Issue(s) Improvement Plan Group Responsible Target Completion Date Follow-up Dissemination of Goals and Objectives Posted on Intranet (5 clicks to reach) Not accessed Not known how or how much this is used by residents and faculty Educate residents and faculty More prominent placement on Intranet (1-click) Make accessible/ viewable in every setting Integrate with resident formative evaluations 2 residents and 1 faculty member (names) give them credit for work June XXXX for implementatio n at start of new academic year Quarterly survey regarding effectivenes s of new approach Spot check

35 Components of an Effective Self-Study Fits the nature of the program and its aims Ensures effective evaluation of entire program with positive impact Engages program leaders and others Faculty, residents, fellows, coordinators, staff Potentially: graduates, institutions hiring them Is efficient in its execution Reporting focused on Improvements achieved Tracking of action items for future improvement

36 Resident Participation in the Self-Study Resident participation critical: They are the beneficiaries of the educational program They have first hand knowledge of areas that need improvement Double benefit: Residents help improve their own education Resident participation in educational QI effort can be used to meet the requirement for resident involvement in quality and safety improvement

37 Coordinated Self-Study for Core and Subspecialty Programs Coordination of curriculum and program resources Needs of core and subspecialty programs taken into account Subspecialties can access to core resources Core oversight of fellowships Assess common strengths, areas for improvement Action plans for areas for improvement Increase efficiency Less time and resources spent, coordinated collection and review of data

38 Organizing the Self-Study for a Core Program and its Dependent Subspecialty Programs Effective: Individuals with interest and the most knowledge about improvement efforts Efficient: Linking the Self-Study to existing structure for identifying and prioritizing areas for improvement, and tracking action plans and success Coordinated: Identifying common areas for improvement across programs that can be considered and addressed collectively to conserve resources and maximize impact

39 A brief document prepared by the program ~5-7 pages for core program, less for subspecialties Focus: key Self-Study dimensions Aims The Self-Study Summary Opportunities and Threats Self-study process Who was involved, how were date collected and interpreted Evidence of ongoing improvement through sequential Annual Program Evaluations

40 Key Roles for the Program Coordinator

41 Key Roles for the Program Coordinator Ensure regular, accurate ADS Updates Record/aggregate data and improvement realized via the Annual Program Evaluation Participate in the annual program evaluation Provide input from the coordinator s perspective Track action plans for areas for improvement Maintain a multi-year record of improvements and areas still being worked on Coordinate self-study data collection processes, including surveys, interview, focus groups

42 Key Roles for the Program Coordinator (con t) Maintain self-study data Provide input into self-study Coordinate self-study Coordinate planning the 10-year site visit with the assigned lead field representative Coordinate activities on the site visit day Provide input on the site visit day

43 Information and Education Plan Self-Study Sessions at AEC Appointment time at AEC for programs with an early self-study (IP and Field Staff with Self-Study pilot experience) Planned Webinars: New approach to Self-Study and 10-Year Site Visit, Self-Study Basics, PDSA, program evaluation Article on rationale for approach in June issue of JGME Self-study web page to go live in March Self-Study mailbox for questions and feedback (ACGME monitors)

44 In Closing If you are interested in more information/q and A Contact us for questions about the site visit self-study@acgme.org Ingrid Philibert, PhD, MBA, Sr. VP, Field Activities: iphilbert@acgme.org, 312/ ACGME Field Staff Topical Sessions in Hyatt La Jolla Room A Thursday, 2/26 5:00 pm 6:00 pm Friday, 2/27 6:45 7:45 am Saturday, 2/28 6:45 7:45 am