A strong evaluation process is essential

Transcription

1 A Method for Defining Competency-Based Promotion Criteria for Family Medicine Residents Laura Torbeck, PhD, MS, and Alan Stevens Wrightson, MD Abstract The Accreditation Council for Graduate Medical Education (ACGME) has mandated a shift from a structure- and process-based educational system to a competency-based system. The ACGME has not provided criteria (standards), preferring to leave that to the discretion of the individual training programs. Such criteria and an overall strong evaluation process are essential for residents to attain the appropriate knowledge, skills, and attitudes. With this need in mind, the authors describe an evaluation process in which they developed ACGME-competency-based promotion criteria for family medicine residents at the University of Kentucky College of Medicine in The authors thought that by providing residents and faculty with minimum criteria that residents must demonstrate at each level, the residency program could address the ACGME mandate to provide competency-based training and improve residents progress toward promotion. Along with the promotion criteria, the method of instruction and the setting for each criterion were identified. Tools were developed to assess the criteria, including a computerbased same day preceptor evaluation, a resident portfolio, and multisource feedback instruments. This information was formatted into a matrix. Making the task and criteria clearer to learners allows them to better demonstrate what is expected of them. Residency educators can target remediation in those residents failing to meet the criteria and improve faculty skills, especially in terms of how to train for and assess competence. The authors describe the initial use of the promotion criteria, including how the faculty and residents responded to it. Acad Med. 2005; 80: A strong evaluation process is essential for residents to attain the appropriate knowledge, skills, and attitudes. In this article, we describe a process in which competency-based promotion criteria were developed for family practice residents at the University of Kentucky College of Medicine. Residency programs often struggle with the performance of some of their residents. The developmental process that residents undergo may not be clear to program directors, especially when residents early in their careers rotate through other specialties and are not continuously under the evaluative eye of their home department. This is an inherent flaw in the traditional structure-and-process-based educational system of 20th-century graduate medical education (GME), that Dr. Torbeck is assistant professor of surgery and surgical educator, Indiana University School of Medicine, Indianapolis, Indiana. Dr. Wrightson is assistant professor of family medicine and program director of the Family Medicine Residency, University of Kentucky College of Medicine, Lexington, Kentucky. Correspondence should be addressed to Dr. Torbeck, Department of Surgery, Indiana University School of Medicine, 244 Emerson Hall, Indianapolis, IN is, a system based on exposure to specific content for a prescribed period of time (i.e., block rotations). 1 When the faculty are not instructed to look for common performance criteria, residents may be judged satisfactory based on unstructured observations, 2 comparisons with other residents (normreferenced assessment), 3 or simply whether their team functioned well. When promotion time approaches, it is not uncommon for residents to be promoted year after year, even when their performance is at times substandard or not fully apparent from global rotation evaluations. The Accreditation Council for Graduate Medical Education (ACGME) has called for the shift from a structure-and process-based educational system to a competency-based system for two main reasons. First, in light of public demand, GME must be accountable for quality and safety. 1 Second, the few studies that have addressed outcomes show superior results with the competency-based system. 4,5 The ACGME has attempted to standardize GME by mandating training based on acquiring competence in six core areas: patient care, medical knowledge, professionalism, systemsbased practice, practice-based learning and improvement, and interpersonal and communication skills. 6 The ACGME has not provided criteria (standards) that indicate competence in any of these six areas, preferring to leave that to the discretion of the individual training programs. For this reason, developing curricula to foster the ACGME competencies has become important. In reviewing the existing medical literature, Carracio et al. 1 summarized a stepwise approach to developing such curricula. The four steps are (1) competency identification, (2) determination of competency components and performance levels, (3) competency evaluation, and (4) overall assessment of the process. In the rest of this article, we address the process by which we and our colleagues at one medical school carried out the first three steps, with the goal of developing measurable criteria that can be used to set promotion standards for residents. Specifically, the residency division at the University of Kentucky College of Medicine s Department of Family Practice and Community Medicine set out to develop promotion criteria based on the ACGME core competencies. We 832 Academic Medicine, Vol. 80, No. 9 / September 2005

2 and our colleagues in the residency division felt that by providing residents and faculty with minimum criteria that residents must demonstrate at each level, we could address the ACGME mandate to provide competency-based training and improve resident progress toward promotion. Developing the Criteria Starting the process In 2003, both of us began the task of developing promotion criteria by searching the literature for studies identifying standards or criteria for resident promotion. Few studies were found related to residents progress and promotion. 2,7 We then reviewed the information contained within the last three Residency Assistance Program workshops of the American Academy of Family Physicians, Frey et al. s outpatient core competency presentation notes were earmarked as having specific core competencies that could easily be rewritten as promotion criteria. Next, we reviewed the residency curriculum of the Department of Family Practice and Community Medicine at the University of Kentucky College of Medicine and listed promotion criteria that we thought were important for Family Medicine residents at each level of that curriculum to achieve. Taking this list and extrapolating from Frey et al. s work, we composed a list of working promotion criteria. The next steps At the Department of Family Practice and Community Medicine s annual faculty retreat in the fall of 2003, we were given a large portion of the morning to present the working promotion criteria, discuss the importance and feasibility of proceeding with this initiative, and receive feedback from the faculty on how to modify and clarify the information. In their feedback, the faculty suggested only minor modifications to the promotion criteria, which we then made. Keeping in mind the ACGME s mandate, we thought it was important to link each promotion criteria with a competency. We reworked the competency and subcompetency information available on the ACGME Outcome Project s Web site 6 to include a label for each competency and subcompetency. For example, the label MK1 stands for one of the subcompetencies of the medical knowledge competency (see List 1 and Table 1 for other examples of these labels). Creating labels was done for ease in recognizing more readily where each subcompetency is being addressed within our curriculum. In addition to linking a label with each competency criterion, we also considered what the method of instruction would be for each criterion, how each criterion would be evaluated, and within what setting each criterion would be assessed. Tools were developed to address these competencies, including a Web-based, competency-based faculty evaluation, a resident portfolio, and multisource feedback instruments. We adopted key concepts from Bope et al. s 9 speech on competency-based residency education presented at the 2004 Workshop of Directors of Family Practice Residencies and decided upon a set number of competencies to address. We then formatted the information described above into a matrix. A later version of that matrix is presented in Table 1. The end result The final steps in the process entailed presenting the improved promotion criteria, in the context of the matrix, to the faculty for their approval. Faculty development occurred in the spring of 2004, during which the authors presented the working promotion criteria, discussed the importance and feasibility of proceeding with this initiative, and received feedback from the faculty on how to modify and clarify the information contained within the matrix. Feedback from the faculty was received and only minor modifications to the matrix were offered. The modifications were then made and an improved version of the matrix was created, described below and shown in Table 1. Description of the Matrix Sixty-four promotion criteria were developed. Twenty-nine of them were specific to postgraduate-year 1 () residents, 18 were specific to residents, and 17 were specific to residents. Each ACGME subcompetency was addressed at least once. The method of instruction for each criterion varied but mainly focused on preceptor observation and advisor mentoring. The setting for addressing the criteria most often occurred in both the and outpatient venue but occasionally was specific to one or the List 1 The ACGME s Project Outcome Document for Two of the Six General Competencies, with Competency and Subcompetency Labels Added* ACGME General Competencies The residency program must require its residents to develop the competencies in the six areas below to the level expected of a new practitioner. [This list shows only two of those areas.] Toward this end, programs must define the specific knowledge, skills, and attitudes required and provide educational experiences as needed in order for their residents to demonstrate the competencies. Medical Knowledge (MK) Residents must demonstrate knowledge about established and evolving biomedical, clinical, and cognate (e.g., epidemiological and social behavioral) sciences and the application of this knowledge to patient care. Residents are expected to: MK1: Demonstrate an investigatory and analytic-thinking approach to clinical situations MK2: Know and apply the basic and clinically supportive sciences that are appropriate to their discipline Interpersonal and communication skills (ICS) Residents must be able to demonstrate interpersonal and communication skills that result in effective information exchange and teaming with patients, their patients families, and professional associates. Residents are expected to: ICS1: Create and sustain a therapeutic and ethically sound relationship with patients ICS2: Use effective listening skills and elicit and provide information using effective nonverbal, explanatory, questioning, and writing skills ICS3: Work effectively with others as a member or leader of a health care team or other professional group * ACGME means Accreditation Council for Graduate Medical Education. The full Project Outcome Document may be found at Academic Medicine, Vol. 80, No. 9 / September

3 Table 1 A Matrix Illustrating the Promotion Criteria Developed by the Authors for Family Medicine Residents; Each Criterion is Linked to Relevant Information, Including How the Skill Defined by the Criterion is Taught and Evaluated and How It Relates to the ACGME s Subcompetencies Resident s year Promotion criteria ACGME subcompetency label* Method of instruction and feedback Evaluation method Setting 1. Identifies purpose of visit PC2, ICS2 Preceptor observes, asks, and 2. Gathers complete and reliable PC2 Preceptor observes and reviews history H&P 3. Develops an appropriately ordered, reasonable differential PC3, MK1, PBL3 Preceptor asks and reviews diagnosis (3 or more) for presenting problem 4. Orders appropriate labs/tests for MK2, PBL5, PC3 Preceptor reviews and questions the presenting problem resident choice(s) 5. Presents working diagnosis to patient ICS2, PC5 Preceptor observes in room and reviews on videotape with resident 6. Discusses appropriate follow-up SBP3, SBP4, SBP5, Preceptor observes, asks, and and/or discharge planning PC4 7. Prescribes medications PBL5, PC6, PC4, Preceptor observes, asks, and appropriately MK2 8. Considers the ramifications of treatment (medications, IV, PC7, PC3, MK2 Preceptor observes, asks, and fluids, radiologic studies, etc.) including interactions, side effects, and potential complications 9. Educates patient about P3, ICS2, ICS1, Preceptor observes, asks, and prescribed medications PC5 10. Documentation is legible, PC9, MK2, MK1, Preceptor reviews chart Inpatient concise, in SOAP format for each problem and with a completed problem list for each ICS2 documentation and patient 11. Health maintenance SPP3, ICS3 Preceptor reviews information, Outpatient information is updated, including medicine/allergy list, and signs the notes and problem list 12. Demonstrates a commitment to P1, P2, P3 Preceptor models this and ; MSF All rotations carrying out professional coaches residents responsibilities 13. Patient is billed appropriately P2, SBP2 Coding class and preceptor reviews billing 14. Able to take in-house call P1-3, PC2-7, PC9 Preceptors observe and coach ; MSF Inpatient independently 15. Recognizes limitations and PC3, P1 Orientation role-playing ; MSF seeks help appropriately 16. Accepts feedback well P1, P2, PBL1 Orientation role playing; preceptors emphasize when they are giving feedback 17. Uses instructional technology to determine best medical PC3, PBL5, PC6 Preceptor asks; noon conference presentations evidence 18. Introduces self to patient and P3, PC1, ICS1 Preceptor observes Portfolio; OSCE addresses patient with OSCE appropriate title 19. Becomes competent in all First PC2, PC1, MK2 Preceptors observe and instruct Portfolio; OSCE Year Resident Physical Exam OSCE skills 20. Demonstrates sensitivity to a diverse patient population P1, P3 Advisor reviews portfolio entries Portfolio COPC, HV (Table continues) 834 Academic Medicine, Vol. 80, No. 9 / September 2005

4 Table 1 (Continued) Resident s year Promotion criteria ACGME subcompetency label* Method of instruction and feedback Evaluation method Setting 21. Learns from experience PBL 1, PBL5 Advisor reviews portfolio entries Portfolio 22. Recognizes that patient s needs P3 Advisor reviews portfolio entries Portfolio HV supersedes resident s needs 23. Document appropriately using SBP4, SBP5 Advisor reviews portfolio entries Portfolio HV the FMC Home Visit documentation form 24. Complete and assist in billing SBP2, SBP4-5 Advisor reviews portfolio entries Portfolio HV for eight home visits 25. Documents all procedures PC7 Advisor reviews portfolio log Portfolio All rotations performed during 26. Begin to evaluate the literature PBL2 Advisor reviews portfolio entries Portfolio FPC for presentations 27. Receives an evaluation and ICS3, PBL1 Advisor reviews evaluations in IGE All rotations feedback for each outside rotation during portfolio 28. Attends 50% of all noon P1, MK2 P& P Committee reviews sign-ins Portfolio FPC conferences 29. Meets expected behaviors and P1 PD reviews resident knowledge Portfolio FPC knows content identified in Resident Manual and clarifies 30. Implements a negotiated ICS2, ICS1, PC1, Preceptor observes, asks, and management plan with patient PC3-6, P1, P3 31. Addresses sensitive issues appropriately such as mental PC1, PC2, PC5, P2, P3, ICS1, ICS2 Preceptor observes and ; behavioral science curriculum ; MSF health or risk behaviors 32. Tends to chronic problems P1, PC4, PC5, PC8 Preceptor asks and when appropriate 33. Incorporates health PC8, ICS1, P2 Preceptor asks and reviews health Outpatient maintenance and preventative care where appropriate maintenance information 34. Arranges appropriate medical PC9, SBP1, SBP4-5, Social worker observes and ; MSF and ancillary referrals ICS3 advises 35. Manages clinic duties efficiently ICS3, PBL1, SBP3 One-on-one feedback at midclinic ; MSF Outpatient period and at end of clinic 36. Discusses with patients end-oflife issues appropriately and in a PC1, ICS2-3, P2 Preceptors model this and step back to allow resident to sensitive way discuss with patient 37. Acts as chief on service PC2-7 PC9, PBL2, Preceptors observe and coach ; MSF Inpatient 4-6, P1-2, SBP Responds appropriately in emergent/urgent situations ICS3, PC3, MK2 Preceptor goes with resident to patient emergency to observe Faculty evaluation; and instruct MSF 39. Can teach students and interns in clinic and on hospital service PBL6 Preceptors observe, chief resident encouraged to observe and ; MSF coach 40. Complete and assist in billing SBP2, SBP4-5 Advisor reviews portfolio entries Portfolio HV for 12 home visits 41. Works with nonphysician professionals in a way that ICS3 Multidisciplinary team observes, asks and advises ; MSF COPC and FMC garners mutual respect and excellent patient care 42. Documents all procedures performed during PC7 Advisor reviews portfolio log Portfolio All rotations (Table continues) Academic Medicine, Vol. 80, No. 9 / September

5 Table 1 (Continued) Resident s year Promotion criteria 43. Receives an evaluation and feedback for each outside rotation during 44. Attends 75% of all noon conferences 45. Meets expected behaviors and knows content identified in updated/revised Resident Manual 46. Critically evaluates the literature during presentations 47. Passes USMLE Step III examination 48. Works with patient and family to develop a collaborative relationship and management plan that includes care of acute and chronic issues, health maintenance, disease prevention, and continuity of care 49. Works with and motivates all staff in a way that garners mutual respect and efficient patient care ACGME subcompetency label* ICS3, PBL1, SBP3 Method of instruction and feedback Advisor reviews evaluations in portfolio Evaluation method Setting All rotations P1, MK2 P & P Committee reviews Portfolio FP sign-ins P1 PD reviews resident Portfolio FP knowledge PBL2 Advisor reviews portfolio Portfolio Journal Club entries MK1, MK2 PD reviews portfolio entry Portfolio FP and comments PC1, ICS1, PBL4, Preceptor observes, asks and Outpatient P1-3 ICS3 Preceptors model this and ; MSF Outpatient give feedback at end of day 50. Actively manages clinic PC9, PBL1, ICS3, SBP3 Practice Management curriculum; Preclinic ; MSF Outpatient huddle with preceptor 51. Able to function as a Teaching PC2-7 PC9, PBL2, Preceptors observe and ; MSF Attending in clinic and on 4-6, P1-2, SBP3- coach hospital service 5, MK Completes all patient care tasks in a timely, organized and professional manner (charting, flow sheets, phone calls and laboratory data) P1, PC4 Preceptors ask, review, and coach ; MSF Outpatient 53. Resident s pattern of prescribing medications and ordering tests/ancillary services is cost-effective and appropriate for patient s needs and resources PC2-4, MK1, ICS1, SBP2-4 Preceptors ask, review, and coach 54. Resident is seen as an advocate for the FMC and as such encourages patients to choose him/her and the clinic for their ongoing care P1 Faculty and staff are aware of patient comments and MSF Outpatient 55. Complete and assist in billing SBP2, SBP4-5 Advisor reviews portfolio Portfolio HV for 12 home visits entries 56. Works with physician colleagues in a way that garners mutual respect and excellent patient care ICS3, PC9, SBP1 Off-service attending/chief observes and Portfolio All rotations 57. Documents all procedures PC7 Advisor reviews portfolio log Portfolio All rotations performed during 58. Receives an evaluation and feedback for each outside rotation during ICS3, PBL1 Advisor reviews evaluations in portfolio Portfolio All rotations 59. Attends 75% of all noon P1, MK2 P & P Committee reviews Portfolio FP conferences sign-ins 60. Meets expected behaviors and knows content identified in updated/revised Resident Manual P1 PD reviews resident knowledge Portfolio FP (Table continues) 836 Academic Medicine, Vol. 80, No. 9 / September 2005

6 Table 1 (Continued) Resident s year Promotion criteria ACGME subcompetency label* Method of instruction and feedback Evaluation method Setting 61. Critically evaluates literature PBL2 Advisor reviews portfolio Portfolio Journal Club during presentation entries 62. Present a completed COPC PBL4 Advisor reviews portfolio Portfolio FP project entries 63. To meet all six core competency requirements and receive documentation to verify accomplishment All PC, P, MK, ICS, PBL,& SBP PD gives resident certificate of competency completion Portfolio FP 64. Able to practice competently and independently in the field of family medicine All PC, P, MK, ICS, PBL, and SBP Preceptors observe and sign off HV * The authors created these labels as a convenient shorthand to refer to the Accreditation Council for Graduate Medical Education s six competencies and their related subcompetencies. In the table, the labels for the six competencies are PC Patient Care; ICS Interpersonal and Communication Skills; P Professionalism; MK Medical Knowledge; PBL Practice-Based Learning and Improvement; SBP Systems-Based Practice. PD program director, P&P Progress and Promotion Committee. same day preceptor evaluation, MSF multisource feedback, IGE institutional graduate evaluation. COPC community-oriented primary care, HV home visit, FPC family practice center, FP family practice. other. For evaluation of each criterion, three methods were identified: a computer-based competency-based evaluation filled out by the faculty, a resident portfolio, and multisource feedback instruments. Methods used more intermittently, such as an objective structured clinical examination (OSCE) and videotaping, were also listed. To be more specific about our three evaluation methods, the first tool we developed was a computer-based same day preceptor evaluation that faculty complete after each clinic session and/or twice per week on service. Select promotion criteria for each level were listed, and faculty were asked to evaluate residents using a five-point Likert scale where 1 unable to promote, 2 needs improvement, 3 competent, 4 exceeds competent, and 5 able to teach criterion to others. Specific performance, or threshold, anchors were associated with each scale option. For instance, some are all or none and some rely on percentages (e.g., performed 50% of the time). Many of the promotion criteria were evaluated using the same day preceptor evaluation tool, although some criteria required further discussion and reflection, such as learns from experience ( promotion criterion #21) or demonstrates sensitivity to a diverse patient population ( promotion criterion #20). For this reason, we devised a paper-based resident portfolio using the toast rack model: specific curricular tasks with criteria were slotted into portfolio binders and residents were instructed to insert their work into the slots, much like toast is inserted into toaster racks. 10 Finally, three, short multisource feedback instruments were constructed to solicit ratings from (1) resident peers, (2) patients, and (3) ancillary staff (social workers, behavioral science professionals, nurses, front desk staff, and medical record personnel), an approach that may better address some criteria such as works with and motivates all staff in a way that garners mutual respect and efficient patient care (PGY 3 promotion criterion #49). Closing the Loop With faculty development We recognized that faculty buy-in is crucial for the success of any change in our residency educational system, and therefore created an extensive faculty development program that focused on the competency-based promotion criteria. Each teaching faculty member in our program was presented with a competency notebook that included the ACGME competencies and the promotion matrix. An initial faculty development session explained the matrix and its use during subsequent advisor/advisee meetings. Specifically, residents and their faculty advisors would be able to look at performance reviews and determine which promotion criteria had been met. Ultimately, these advisor/ advisee meetings would allow further formative feedback that included positive reinforcement of accomplished tasks and goal setting to reach competence in tasks yet to be completed. Further faculty development sessions were held to prepare the faculty for identifying all of the promotion criteria as well as how to provide feedback about how the resident is progressing. We presented vignettes that demonstrate how residents performances do or do not meet specific criteria, with time for discussion on how to provide feedback in either situation. We also provided faculty with feedback on their performance in utilizing the newly developed tools, specifically the same day preceptor evaluation. We discovered that about half of the faculty used the tool consistently. This information was shared with the faculty as a whole. Our hope is that quarterly feedback of this nature to faculty will improve utilization of these tools. With residents development The same criteria and matrix were presented to each class of residents during their orientation sessions so that each resident could understand what was expected of him or her during the remainder of the year and the years to come. The same vignettes used with Academic Medicine, Vol. 80, No. 9 / September

7 faculty were also shared with the residents as examples of how they could meet each criterion. In addition to the matrix, a promotion criteria checklist was given to each resident to keep in his or her portfolio so that during advisor/advisee meetings, both the advisor and resident could account for each criterion being met in a timely manner. Special emphasis was placed on informing the residents that the criteria specific to their level was expected to be demonstrated by the end of March, the month prior to the time a promotion decision is made for each resident. Currently, each resident s performance is reviewed every quarter by the Progress and Promotion Committee, composed of the residency director, the assistant residency director, the medical educator, and two other residency faculty. Results from preceptor evaluations, multisource feedback, and advisor notes are all considered. Each resident s checklist, beginning in the fall of 2004, has been reviewed quarterly by the advisor and the Progress and Promotion Committee and biannually by the residency director to confirm the resident s progress. If a resident fails to reach the required level of competence for all of the level-specific criteria prior to April, the promotion committee can either decide not to promote the resident or to promote the resident with remediation. Discussion The value of promotion criteria Training residents to become competent health care providers has been and always will be an important and challenging endeavor. With the ACGME s new mandate, many program directors and house office staff are looking for useful guidance to make their residency programs compliant. We believe that developing a set number of promotion criteria that are competency-based should be among the first tasks residency programs should tackle, for several reasons. y First, having defined criteria will make the three to seven years of residency performance assessment more educationally sound. The education literature reminds us that two components are mandatory to a performance assessment: the task and the criteria. 11,12 Since continuous performance assessment is carried out during the majority of a resident s training, the awareness of what is expected of the resident can be raised for both the resident and the faculty by making the task and criteria clearer to them. Residents then are better able to demonstrate that which is expected of them, and faculty are better prepared to evaluate them. y Second, having defined promotion criteria can better target remediation in those residents failing to meet the criteria in a timely manner. Intervention during a time early on in the resident s development is much more beneficial than it is just before making a decision to promote or not promote. y Third, having criteria can improve faculty development, especially in terms of how to assess competence, how to give feedback, and how to account for faculty responsibility in training and evaluating competent physicians. Our experience The experience that we had with the faculty in devising these criteria was very positive. Initially we were concerned about how much the faculty would buyin to recognizing and identifying these competency-based promotion criteria, but the faculty was quick to realize that these criteria were needed and our matrix was an innovative way to objectively assess the residents. Incidentally, both of us were invited to present our work to the Graduate Medical Education Core Curriculum Committee meeting at the University of Kentucky College of Medicine in June The committee members recognized our competencybased promotion criteria as having value to all their residency programs. We felt that this institutional support substantiated the value of our endeavors. Interestingly, after our new-resident orientation in 2004, our residents did not rate the explanation of competencies and promotion criteria during their family medicine orientation as beneficial, despite the fact that this portion is so important to their careers. In their defense, our institution s GME thoroughly covered the core competencies in the GME new-resident orientation, and since our orientation followed that orientation, our interns may have been competencied-out! We believe that our residents will more easily make the transition to competent physicians if they understand the tasks set before them, although it appears that, as new medical school graduates, they have yet to develop the understanding that residency involves active, adult learning processes. We believe these processes should begin during the medical school years in order to ease the transition into residency. Our and residents had previously received instruction and put into practice methods of adult learning evaluation, such as midrotation feedback and evaluations, formative feedback during hospital service, and in-training examinations, all of which often demonstrate areas to emphasize based on the scores obtained. Being more familiar with the adult learning model of feedback and evaluation, our senior residents more readily embraced this model, as indicated by evaluations of their annual orientation sessions. The goal, of course, is to improve GME so that quality medical care is provided by our graduates. The residency division at the University of Kentucky College of Medicine s Department of Family Practice and Community Medicine has taken the first three steps (competency evaluation, determination of competency components and performance levels, and competency evaluation) in the stepwise approach to curricular design. The task before the residency division now is to assess its process. We are currently engaged in analyzing data that more objectively define benchmarks for each of the ACGME competencies using a critical incident survey. This should help describe incidents that reflect good and bad practices and should be helpful in furthering resident and faculty development. In the end, we and our colleagues hope to more surely guide our residents toward their future as competent physicians. References 1 Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: from Flexner to competencies. Acad Med. 2002;77: Altmaier EM, McGuinness G, Wood P, Ross R, Bartley J, Smith W. Defining successful performance among pediatric residents. Pediatrics. 1990;85: Carraccio C, Englander R, Wolfsthal S, Martin C, Ferentz K. Educating the 838 Academic Medicine, Vol. 80, No. 9 / September 2005

8 pediatrician of the 21 st century: defining and implementing a competency-based system. Pediatrics. 2004;113: Thurman GK, Sanders MK. Competencybased education versus traditional education: a comparison of effectiveness. Radiol Technol. 1987;59: Greaves PE, Loquist RS. Impact evaluation: a competency-based approach. Nurs Admin Q. 1983;7: The ACGME Outcome Project Accessed 1 June Tudor J. Performance documentation: how to confirm a resident s progress. J Med Educ. 1978;53: Frey EK, Fogarty J. The new ACGME general competencies: teaching strategies in the FPC. American Academy of Family PhysiciansResidency Assistance Program Workshop; Kansas City, Missouri, Bope ET, Aring A. Fifty competencies for family practice: implementation and evaluation. Workshop for Directors of Family Practice Residencies; Kansas City, Missouri, Webb C, Endacott R, Gray M, et al. Models of portfolios. Med Educ. 2002;36: Arter J. Teaching about performance assessment. Educ Meas. 1999;18: Nitko AJ. Educational Assessment of Students. Upper Saddle River, NJ: Prentice Hall, Did You Know? In 1998, researchers at the University of Hawaii at Manoa John A. Burns School of Medicine became the first to clone mice using adult somatic cells, and the first to clone more than one animal of any species. For other important milestones in medical knowledge and practice credited to academic medical centers, visit the Discoveries and Innovations in Patient Care and Research Database at Academic Medicine, Vol. 80, No. 9 / September