FINAL REPORT. University of Washington School of Medicine Curriculum Renewal. Assessment Committee. Written by: Jan Carline, PhD Sara Kim, PhD, Chair

Transcription

1 FINAL REPORT University of Washington School of Medicine Curriculum Renewal Assessment Committee Written by: Jan Carline, PhD Sara Kim, PhD, Chair Submitted on: April 5, 2013

2 TABLE OF CONTENTS A. Executive Summary... 4 B. Assessment Committee 1. Committee Mandate Members Process... 5 C. Assessment Committee s Consensus on Assessment Vision and Principles... 6 D. Current UWSOM Assessment Approaches... 7 E. Task Forces 1. Charge to Task Forces Synthesized Assessment Approaches across Task Forces... 9 F. Key Recommendations for Future UWSOM Assessment Approaches...11 G. Table 1: Assessment Methods and Purpose by Curriculum Phase and Competency...15 H. Appendix 1. Assessment Committee Literature Review List Task Force Report: Medical Task Force Report: Patient Care Task Force Report: Interpersonal and Skills Task Force Report: Professionalism Task Force Report: -based and Improvement Task Force Report: Systems-based Task Force Report: Technology...58

3 A. EXECUTIVE SUMMARY The curriculum renewal initiative at University of Washington School of Medicine (UWSOM) coincides with a watershed in U.S. healthcare reform and in the formal mandating of the Accreditation Council for Graduate Medical Education (ACGME) milestone competencies for physicians that ensures training programs are producing competent future healthcare providers. Predicting the overall trends in the future healthcare system and medical education is critical to defining the purpose, scope and models of assessing student performance. Since its inception, the 20-member UWSOM Curriculum Renewal Assessment Committee engaged in a series of spirited discussions of assessment in this context, resulting in this final report. Members were assigned to Task Forces that were created based on the ACGME core competencies as a way to align the UME assessment model in the context of the GME framework. Task Forces were asked to (1) identify assessment approaches using the Level 1 and Level 2 ACGME Developmental Milestone Competencies as medical students graduating competencies; (2) recommend approaches to demonstrating progression in student performance over the three Curriculum Renewal phases (Foundation, Patient Care, Career Exploration and Focus); and (3) highlight critical resources to ensure successful and sustainable assessment systems. The overarching themes across Task Force reports for guiding the future assessment approaches at UWSOM are highlighted as follows: Provide students and their mentors with guidance for improvement and continued development, document student outcomes to assure the competency of our graduates longitudinally, and provide a basis on which to measure the quality of our educational work; Provide information to faculty and administrative leadership as well as other stakeholders about the success of the educational program in developing the types of physicians and their skills representing the goals of the school; Broaden and diversify the types of assessments that are used to better capture information about specific competencies and objectives of the curriculum; Sharpen and improve the appropriateness, specificity and reliability of the information provided by assessment activities; Provide flexibility and agility in meeting new or changing assessment needs; and Collect information not only of assessment within educational units, but also for competencies, themes, and specific content that cross educational units. The current UWSOM assessment approaches call for an opportunity to envision a regionally distributed, standardized, robust, and reliable assessment system. Towards this end, we recommend the following major assessment approaches in the new UWSOM curriculum: As the curriculum moves towards instructional units that integrate concepts and skills crossing disciplines and competencies, we recommend the use of Integrated Multiple Choice Questions (MCQ) to mirror the question types of the United States Medical Licensing Examinations. This format can simultaneously target content from combinations of disciplines and content from various threads of the curriculum, such as patient safety and diversity. We strongly recommend an extensive expansion in the new curriculum of direct observation of student performance using Mini-CEX, simulations, and Objective Structured Clinical Examinations (OSCEs) to assess the clinical and communication skills of our students. These assessment techniques can be utilized to document initial and advanced skills in patient

4 communication, physical examination, history taking, and diagnostic reasoning crossing competencies of Medical, Patient Care, Interpersonal and, Professionalism, -based and Systems Based. We recommend more consistent assessment approaches involving patient documentation (patient write-ups, case analyses, or chart notes) throughout training. Specific opportunities for assessment via patient documentation include students ability to analyze patient communication, provide information for other health care professionals, and apply basic science concepts to the analysis of patient findings, or sensitivity to issues of diversity. Assessment techniques that are significantly underutilized in UWSOM include written documentations of student performance such as individual or group student projects, reflections, and written assignments. The Committee recommends these methods be fully integrated to reflect the national trend in combining both quantitative and qualitative modalities to assess students ability to identify a need for quality improvement in health care, to understand the role of a physician in health care teams and the resolution of ethical issues, and to apply basic science concepts to clinical decision making. The resource needs to implement these recommendations are significant. At the same time, an institutional commitment to the following resources is imperative to sustain excellence in the quality of UWSOM medical education in the 21 st century: First, we recommend the establishment of WWAMI regional assessment centers that build upon the existing and additional regional infrastructure and personnel capacity. These regional hubs can (1) coordinate on-line written assessments, particularly in light of the exclusive online NBME subject examinations by 2015 and (2) integrate performance assessments across all phases of the curriculum including simulations, Standardized Patients (SPs), and OSCEs. ISIS (Institute for Simulation and Interprofessional Simulation) may contribute significantly to developing these regional centers. Second, the success of a regional assessment model is critically dependent upon ongoing and consistent faculty development that involves all individuals who are in the role of rating the behaviors of students in courses and clerkships, and particularly those who are tasked with monitoring the progress of students throughout medical school. In addition, the time and effort needed by faculty to accomplish assessment tasks must be supported through financial and time resources that recognize the importance of these tasks. Third, UWSOM is in a critical need of a centralized institutional dashboard and a student portfolio system. The emphasis on competencies and themes cross all phases of the curriculum requires a major reorientation of how multiple sources of assessment results, both quantitative and qualitative, are recorded and reported. The portfolio system must serve multiple goals: rapid and easy access of data from multiple sources, real-time information that charts success and needs for improvement in meeting performance competencies; and continuous quality improvement of the curriculum based on its standards for student outcomes. Lastly, we recommend integrating the currently decentralized assessment activities under the proposed UWSOM Assessment Committee that monitors the quality and consistency of assessment practices, quality, outcomes associated with the UWSOM medical student competencies.

5 B. ASSESSMENT COMMITTEE: MANDATE, MEMBERS & PROCESS 1. Committee Mandate The Assessment Committee was charged with the following 3 mandates: a. Critically assess UW School of Medicine s approaches that currently work well in assessing student performance and those approaches that either need improvement or should be terminated, including justification for any such recommendations. b. Recommend new approaches that will advance and improve student assessment, including justification for any such recommendations, while acknowledging the needs and demands of the additional areas identified as priorities within the School of Medicine curriculum renewal. c. Work collaboratively with other curriculum renewal committees and the Steering Committee to ensure that student assessment is an integral, integrated and vital part of the UW School of Medicine curriculum, which accurately and optimally reflects and helps to advance the performance of our medical students. 2. Committee Membership The committee consisted of the following 20 members listed in an alphabetical order of last names: Michael Campion, Deans' Office Jan Carline, PhD, Medical Education & Evaluation Jamie Cheek, EdD, Dean's Office Freddy Chen, MD, Family Medicine Heidi Combs, MD, Psychiatry Cassie Cusick, PhD, Cell Biology and Neuroscience, Montana State University Rosemary Fernandez, MD, Emergency Medicine Jon Ilgen, MD, Emergency Medicine Susan Johnston, EdD, Graduate Medical Education Sara Kim, PhD, ISIS, Center for Medical Education, Committee Chair Chris Knight, MD, Internal Medicine David Losh, MD, Family Medicine Julie McNalley, PhD, Dean's Office Heather McPhillips, MD, MPH, Pediatrics Phil Mixter, PhD, Immunology, Washington State University Wendy Mouradian, MD, School of Dentistry Annemarie Relyea-Chew, JD, Radiology Lynne Robins, PhD, Medical Education & Evaluation Brian Ross, MD, PhD, Anesthesiology, ISIS Doug Schaad, PhD, Medical Education & Evaluation 3. Committee Process The committee met a total of 10 times, which included 5 meetings devoted to large group brainstorming sessions, 3 meetings for planning task force processes, and 2 meetings for reviewing task force recommendations. An annotated reading list was provided to members that addressed emerging assessment principles and approaches recommended by leading thinkers and researchers in the field of medical education assessment (Appendix 1: Assessment Committee Literature Review List). In addition, five peer institutions were consulted regarding their assessment approaches: Cleveland Clinic Lerner College of Medicine, Columbia School of Medicine, Indiana School of Medicine, University of Pennsylvania Perelman School of Medicine and Weill Cornell Medical College. At the beginning of February, 2013, the committee was organized into 5 Task Forces, which are described more in detail later.

6 C. ASSESSMENT COMMITTEE S CONSENSUS ON ASSESSMENT VISION AND PRINCIPLES The Assessment Committee members spent a significant amount of time brainstorming on the following 2 questions as a way to lay the foundation of the committee s visioning of the new UWSOM assessment approaches: a. What are the major trends in healthcare and medical education that will impact training and assessment of future medical students? b. Which set of assessment principles should guide the future assessment of student performance at UWSOM? Committee members identified the following trends in healthcare and medical education that may impact the future of medical student training and assessment: (1) ACGME competencies will increasingly serve as an organizing framework of medical students competencies that are embedded into teaching and assessment; (2) resident work hour restrictions will significantly impact trainees exposure to and experience with continuity of care and hand-offs; (3) care of complex vs. routine acute cases provided by physicians vs. mid-level providers may possibly impact students exposure to patient panel; (4) growing accountability to the public, society and regulatory agencies may define a new set of physician competencies; (5) IT revolution calls for students EMR competencies; (6) increasing team management skills/team-based care require unique interprofessional communication skills; (7) integration of evidence-driven standard of care; and (8) significant workforce demands for primary care physicians under the new healthcare system. The Committee also set out to define a set of assessment principles that guide the design, implementation and quality improvement of UWSOM assessment model and approaches. Here are foundational assessment principles that anchor the key recommendations the Committee is proposing in this report: Principle 1: UWSOM assessment approaches should be grounded in competencies and objectives that are defined for each curriculum phase and course/block. Principle 2: UWSOM assessment models will be informed by emerging assessment approaches recommended by national thinkers and researchers. These encompass a. Multiple assessment methods that include both quantitative and narrative/reflection oriented assessments; b. Multiple data collection points over time to track student performance longitudinally; c. Innovative application of assessment models such as progressive testing, selfreflections, and peer assessment; d. Consistent implementation of formative assessment with feedback to students for ongoing performance improvement. Principle 3: Student test-taking experiences should align with the NBME exam format and testtaking experiences. Principle 4: The scope of OSCEs should be dramatically expanded to reflect healthcare practices. Principle 5: Assessment data are indispensible tools for continuous quality improvement of curriculum and assessment approaches.

7 Principle 6: A successful implementation of reliable, consistent, high quality and innovative assessment at UWSOM is critically dependent on (a) regional faculty development in assessment; (b) standardized assessment practices across WWAMI; (c) distributed assessments via regional assessment infrastructure; and (d) adequate investment into personnel and IT systems to facilitate a centralized system for quality assurance of assessment materials, data access and reporting, and close monitoring of student progress. D. CURRENT UWSOM ASSESSMENT APPROACHES The primary assessments of student performance are currently completed within the context of successful completion of a series of courses and clerkships. Assessments are designed by course and clerkship faculty based on the specific objectives and content for each educational unit. The primary information from basic science courses reported to the student record is of pass or fail. Most courses utilize Multiple Choice Question (MCQ) tests as the primary assessment, although short answer or essay, problem sets, and short projects are utilized in a number of courses. Assessment across the first year regional campuses includes both common and unique aspects. All courses have a common examination designed to assess core concepts and skills. Additional assessments unique to a site may also be included in a single course. The balance between common and unique assessments may vary significantly within a course as well as between courses. There is little agreement about the balance between the need for consistency in teaching and assessment across sites with support for the unique characteristics of faculty and programs at regional sites. The assessments in Introduction to Clinical Medicine are based on observation of students behaviors with patients and peers, review of patient documentation (write-ups), simulations for physical examination skills, reflections and other similar methods. Assessment in required clerkships include observation of students behavior, written examinations, review of patient documentation, completion of a Mini-Clinical Examination based on observation of a defined clinical skill, recorded experience with patient or disease types (logs), and in some cases including reflections on ethical or professional issues. Three clerkships (OB/GYN, neurology, internal medicine) have adopted National Board of Medical Examiners Subject examinations. Final grades are reported on a four point scale, and include a series of global ratings on a variety of important behavioral characteristics such as communication skills and professional behavior and written comments. Although all courses and clerkships have performance objectives for their curriculum, assessments of performance on individual objectives are not reported (particularly in basic science courses: little is reported centrally other than pass/fail designations). The global ratings of student performance on clerkship assessments represent broad objectives for clinical education as a whole, and not unique to a specific clerkship. A number of clerkships report overall examination scores, and other unique assessments of performance. In addition to the course and clerkship assessments, the SOM does assess students through two OSCEs, at the end of both the second and third years of the curriculum. The end of second year OSCE includes assessment of clinical benchmarks designed by the College faculty. The end of third year includes assessment of representative content from the required clerkships. Although these assessments sample important performance objectives, the limited number of stations cannot be seen as comprising a comprehensive assessment of clinical skills. Students are required to pass Step 1 of the United States Medical Licensure Examination (USMLE) for progress into the clinical years. They are also required to pass Step 2 Clinical and Step 2 Clinical Skills USMLE examinations for graduation.

8 The assessments of student performance are primarily reported in terms of passing courses and clerkships, rather than accomplishment of specific objectives. This does not allow students, faculty or administrators to track the development of specific competencies, such as interpersonal skills or clinical reasoning. Students may pass all courses and clerkships but still not meet performance standards for specific objectives that cross the curriculum. Data reported from the Step examinations gives little information about individual student performance other than a pass or fail designation. Recently, a number of changes in assessment have occurred primarily resulting in improvement of the quality of multiple choice tests. The largest challenge to our curriculum remains the documentation of student growth and success in achieving the competencies needed for practice as a physician. E. TASK FORCES: CHARGE & SYNTHESIZED ASSESSMENT APPROACHES 1. Charge to Task Forces The committee members agreed to organize assessment approaches and recommendations for resources based on the ACGME (Accreditation Council for Graduate Medical Education) competencies: medical knowledge, patient care, interpersonal and communication skills, professionalism, practice-based learning, and systems-based practice. Individual task forces were created under each competency domain. In addition, recognizing the importance that a centralized assessment data infrastructure will play in ensuring a robust tracking and reporting purpose, an additional task force, Technology, was also created. The rationale for basing our approaches on the ACGME competencies is as follows: a. The US graduate medical education is undergoing a major reform to organize residency training and assessment based on ACGME competencies and milestones; b. Level 1 and Level 2 GME milestone competencies across specialties inform medical students graduating competencies as outcomes; c. Therefore, UWSOM s assessment approaches pose an opportunity to align UME and GME competencies both locally and nationally to establish a continuum across training levels and to better prepare UWSOM medical students for the next step in their training and careers. The major intent of the report from each task force was to predict, in general terms, the types of assessment activities and resources that would be needed in each phase of the new curriculum related to the ACGME Competency Domain assigned. The Task Force members engaged in their discussion with the following assumptions: a. The goal for our graduates is to be skilled physicians who are engaged, compassionate, and committed to excellence as stated in the Vision Statement from July b. The curriculum will be structured in three phases: Scientific Foundations, Clinical Foundations, Career Preparation and Exploration. c. The content of the Competency Domain will be taught in a spiral fashion: A concept or skill will be introduced early in the curriculum and revisited and elaborated at later points in the curriculum, there are increasing levels of complexity and difficulty, new learning is related to prior, and the competence of the student increases with repeated learning. d. The expected performance of our graduates within competency domains is captured in the initial milestones (GME Levels 1 & 2 competencies). All Committee members were assigned to one of the Task Forces with a designated chair. They had approximately 3 weeks to meet as a group and submit the final Task Force report. An example of the report template was developed using the Medical competency to guide the Task Force work. The chairs and members of Task Forces are listed in the individual reports appended to this report.

9 2. Synthesized Assessment Approaches across Task Forces The reports of the task force groups list a variety of proposed assessment methods that span Multiple Choice Questions to Standardized Patients, Peer Assessments, and Patient Satisfaction ratings. A significant number of the proposed methods are incorporated into current practices at UWSOM. What distinguishes the proposed assessment activities from current practice is the broadening scope of assessment, increased use of qualitative assessment methods, strong emphasis on self and peer evaluations as well as assessment by faculty, and a focus on assessing student growth in competencies across educational units (courses and clerkships) in a fashion which is currently impossible. These themes are captured in the following aims for the assessment system. The proposed assessment system will: Provide students and their mentors with guidance for improvement and continued development, document student outcomes to assure the competency of our graduates longitudinally, and provide a basis on which to measure the quality of our educational work; Provide information to faculty and administrative leadership as well as other stakeholders about the success of the educational program in developing the types of physicians and their skills representing the goals of the school; Broaden and diversify the types of assessments that are used to better capture information about specific competencies and objectives of the curriculum; Sharpen and improve the appropriateness, specificity and reliability of the information provided by assessment activities; Provide flexibility and agility in meeting new or changing assessment needs; and Collect information not only of assessment within educational units, but also for competencies, themes, and specific content that cross educational units. The frequency of assessment, both formative and summative, will be increased with the implementation of these recommendations, as will be the amount of qualitative information collected about student performance. While scores and grades will continue to be registered for students, the content of the student s written assignments and the comments provided by faculty and peers will provide a meaningful description of a student s progress in mastering the competencies needed to enter the next phase of their education as a physician. The design of our assessment system will need to pay close attention to the developments in Graduate Medical Education, particularly the establishments of milestones within competencies. The Colleges have provided a set of milestones for clinical skills and professionalism. The future work of a curriculum or assessment committee will be to develop additional milestones for all phases of the curriculum and for all the adopted competencies. A summary of the assessment methods discussed by the Task Forces is provided in appendix (Table 1. Assessment Methods and Purpose by Curriculum Phase and Competency). Rather than extensively describing each proposed assessment activity, the following paragraphs will highlight a few of the proposed methods and their uses across competencies. a. Integrated Multiple Choice Questions (MCQ): As the curriculum moves towards instructional units that integrate concepts and skills crossing disciplines and competencies, the MCQs utilized in the assessment of medical knowledge will represent this integration and increasingly mirror the question types included in the United States Medical Licensing Examinations. A single MCQ could represent content from anatomy, pathophysiology, and pharmacology, or any other combination of disciplines. A single item could also represent content from various threads of the curriculum, such as patient safety and diversity, as well as

10 one or more competency. MCQs for integrated courses will need to be reviewed by groups of faculty for appropriateness and accuracy, and scores on items will need to be assigned, tagged, by multiple categories. s for items representing a discipline will need to be summed over courses to gain a full assessment of a student s progress in achieving the appropriate standard in that area. b. Direct Observation of Student Performance Using Mini-CEX, Simulations, Objective Structured Clinical Examinations (OSCEs): While these techniques are currently utilized in our program, there is a strong belief that they need to be expanded dramatically to meet the needs to assess the clinical and communication skills of our students. Well developed and structured, these assessment techniques can be utilized to document initial and advanced skills in patient communication, physical examination, history taking, and diagnostic reasoning crossing the competencies of Medical, Patient Care, Interpersonal and, Professionalism, -based and Systems Based. In order to be effective and valid measures of student competency, these methods need to be carefully developed, and methods to observe and record student behaviors standardized across observers. The technical demands for summative assessments at multiple teaching and testing sites require significant resources to assure that the outcomes are meaningful and truly representative of student competency. Performance across a number of assessments will be needed to map the growth of a student s competency and acquisition of skills required for the practice of medicine. c. Review of Patient Documentation: Currently, review of patient documentation (patient write-ups, case analyses, or chart notes) is undertaken in courses focusing on initial clinical skills and clerkships, with the intention of determining a student s ability to collect and analyze patient information. These assessments appear to be reaching their goals in introductory clinical skills courses, but are reported to be poorly utilized in a number of clerkships. Modifications to the use of this assessment are suggested to allow their use in a more deliberate fashion. Specific aims of assessing student accomplishments should be identified and connected with a particular type of patient documentation to be reviewed by clerkship faculty and or faculty mentor. Documentation could be reviewed specifically for students ability to analyze patient communication, provide information for other health care professionals, apply basic science concepts to the analysis of patient findings, or sensitivity to issues of diversity. A student could be required to identify a patient interaction in which one of these issues was important, provide the documentation for that event, and then, through reflection, indicate how this issue was met or the student s self-perception of additional growth. d. Written Documents: Several of the competency domains contain suggestions that objectives should be assessed through the utilization of individual or group student projects, reflections, and written assignments. Currently, these techniques are used in a limited fashion, primarily to meet III requirements, or presentation of content in a few courses or clerkships. These qualitative measures are proposed as a means to assess the ability of a student or a group of students to identify a need for quality improvement in health care to construct a project that will bring about that improvement, to understand the role of a physician in health care teams and the resolution of ethical issues, and even to display the ability to apply basic science concepts to clinical decision-making. Written documents require faculty time to evaluate, but have the ability to more clearly describe a student s deeper understanding of medical concepts and to use them in the care of patients. e. Additional Thoughts about Methods: There are a number of other methods included in the task force reports that are important to note as either extensions of current practice or new to 10

11 our environment. The methods currently not widely used include 360 Degree Evaluations, Peer- Assessment, Self-Assessment, and Chart-Stimulated Recall. Methods currently used and recommended to be expanded include Reflection, Written Assignments, and Chart Review. The variety of assessments included in the reports emphasizes the need for multiple methods by which to view a student to insure a valid picture of the student s growth and accomplishments. Specific methods may be adapted to include assessment of a variety of curricular themes not discussed in this document. A number of these methods require significant faculty involvement in evaluation and interpretation, resulting in both scores and narrative comments. Ensuring that students are meeting competencies and threads that cross the curriculum requires a centralized map for assessment activities as well as systems necessary to collect information, quantitative and qualitative, from a variety of sources. A systematic approach to assessment will also require the establishment of a body charged with the oversight, implementation, and interpretation of assessment activities. F. KEY RESOURCE RECOMMENDATIONS FOR FUTURE UWSOM ASSESSMENT APPROACHES The resources needed for implementation and maintenance of these assessment methods in a systematic fashion will require investment of resources in faculty, staff, facilities, and support systems. a. Performance Assessments: Regional Assessment Centers The call for increased use of performance assessments across all phases of the curriculum including simulations, Standardized Patients (SPs), and Objective Structured Clinical Examinations (OSCEs) can be supported only if facilities and other resources are available for their use. Collaboration with UWSOM ISIS and other regional simulation centers will be key in mounting these activities on the scale suggested by the task-force reports. WWAMI regional simulation and OSCE centers to perform both formative and summative assessments will be needed to allow faculty and student participation. These centers will need dedicated physical space and resources to support a variety of learning and assessment experiences. These facilities will also be equipped with audio and video recording for use both in assessment and learning activities. Personnel will be needed to manage the center, hire and train standardized patients, recruit observers; work with faculty to author cases; conduct the examinations, manage assessment data systems; and develop performance reports. Cadres of standardized patients competent to conduct each case, assess performance, and provide feedback to students will also be needed. b. On-line Written Assessments: Regional Assessment Centers Currently, on-line assessments are utilized for a small number of courses and clerkships with instructor authored tests as well as National Board of Medical Examiners tests. Locally licensed software supports the former assessments, and subscription to the NBME Custom Assessment System supports the latter. NBME Subject examinations in clerkships are currently administered at regional centers in paper form. By 2015, these examinations will be transferred to on-line only. Increasing use of online assessments for integrated multiple choice questions or other short written assessment tools will require the development of WWAMI regional computer laboratories or other classroom based facilities to allow students to utilize their own computer equipment. These facilities include; adequate WI-Fi bandwidth; desks with electrical outlets for student computers; and adequate space to separate students during testing. Although software is 11

12 currently available to support on-line assessments, resources may be needed to expand the type and frequencies of assessments currently employed. c. Development members assigned to observational assessment of learners need, at least, an orientation to their tasks and practice of the specific skills needed for accurate use of observational tools. Periodic faculty development sessions are needed for all individuals who are in the role of rating the behaviors of students in courses and clerkships. These skills include the observation of individual students as well as students in groups, particularly in the competencies of Interpersonal and, Professionalism and Based. Providing assessment and feedback requires orientation and training, including calibration of observers, to be effective. Providing feedback to students, individuals and groups, related to observation also requires orientation and training to be effective. Attention to characteristics of excellent formative assessment feedback aimed at improvement during instructions and summative assessment at the end of an educational unit is also needed. The observational assessment skills needed for Mini-CEX, OSCEs, Mini-Clinical Examinations, and group-based simulations may be provided for a more limited number of faculty members who are charged with these tasks. members involved with student peer and selfassessments will also need training in methods that will assure the usefulness and success of these learning and assessment activities. members charged with the development of assessment tools will need training in the preparation and refinement of specific tools and the support of professional staff charged with these tasks. d. Support The time and effort needed by faculty to accomplish assessment tasks must be recognized and supported through financial and time resources that recognize the importance of these tasks. Staff and data systems need to be in-place to minimize the routine collection and reporting of assessment information. It may be unreasonable for all faculty to be charged with assessment, and support be offered to a limited number of individuals charged with these tasks. Seed-grant funding should be made available to support faculty scholarship in the development and refinement of assessment instruments as well as data collection and analysis systems. e. Student Assessment Portfolio The emphasis on competencies and themes across all phases of the curriculum requires a major reorientation of how assessment results are recorded and reported. Currently, assessment of learning is primarily focused on the successful completion of courses or clerkships. This is recorded as pass or fail designations for the first two years, a four point scale for clerkship grades, and a series of additional notations for other requirements. In the new curriculum, competencies will be learned and assessed across courses, clerkships, and phases, typically by instruments that mirror the integrated nature of instruction. A single assessment instrument may include results for a number of competencies, curriculum threads, or disciplines. Success in meeting the standards for each of these components will require a system that is able to record data that represents multiple components within a single assessment. For example, the student s success in utilizing pathophysiological concepts to understand illness will be assessed by integrated MCQ tests in phase 1 courses, and by patient assessments in phases 2 and 3. Component scores from a variety of assessment tools will need to be recorded in a central student portfolio to allow students and faculty to view the progress of students towards achieving curriculum competencies and objectives. 12

13 f. Information Technology (IT) Development and Support While many of the assessment methods do not need significant IT support for their development and immediate use, a number of methods have significant IT needs that will need personnel and equipment to support. Perhaps the greatest IT challenge will be in the construction and design of the Student Assessment Portfolio. Careful planning for facilities and systems is warranted both from the importance of the task, and potential costs involved. Some of these needs are outlined below, and informed by the report from the technology task force. Support for Performance Assessments: The performance assessments at regional assessment centers will be recorded through software that captures immediate inputs from all parties, including check lists and ratings provided by patients and observers, student responses to both closed and open ended questions as well as data provided directly from simulation equipment. Results of these assessments will then be transmitted to the student assessment portfolio. Support for On-line Written Assessments: The software implementation of summative assessments, conducting assessments and reporting results are currently supported by central staff in Seattle. Additional regional personnel may be needed as utilization expands to formative assessments. At minimum, personnel needed to proctor examinations as required the SOM policies and the NBME are required. Support for Development: Although much of faculty development may be done inperson, a significant effort will be needed to support on-line and distance learning modalities for faculty development efforts. Software and hardware will be needed, as well as IT support for production of modules and the conducting of synchronous video and audio supported sessions at multiple sites. Support for the Student Assessment Portfolio: The student portfolio system will need data input from a variety of sources including written test scoring software, ratings provided by faculty and peers, results of observational assessments, as well as qualitative statements from faculty and students. Additional systems will be needed to allow rapid and easy accession of data from multiple sources. The portfolio must provide students with information that demonstrates both success and needs for improvement in meeting competencies, faculty mentors and student affairs with information needed to assess student progress and identify strengths and areas needing intervention, and administration and accrediting bodies with information about how well the curriculum is meeting its standards for student outcomes. Currently, the school has a limited number of software components and systems that may support this type of student portfolio. Making the portfolio a functioning reality particularly in a widely dispersed regional environment, will require changes in current reporting of assessment outcomes as well as significant support in the development of systems to collect, manage, and report assessment data. Careful planning, design, and support of systems will be needed for a successful implementation of the portfolio. This will involve significant investment in personnel, facilities, and materials to be realized. g. Structures in Support of Assessment of Student Competency: The assessment methods and resources included in this document are proposed to provide formative and summative information that will be used to foster the development of students and 13

14 to evaluate the success of a student in meeting necessary competencies. All of the assessment results will be collected in a portfolio, both quantitative and qualitative information. As the expected assessment measures will be numerous, and the documentation large for a single student, some formal method and responsibility for the review of this documentation will be needed. members associated with individual educational units will have the role of providing discussion and guidance to students regarding performance in those units. Assessments that are conducted across the curriculum, and that are developmental in nature, will also need attention from faculty to assist students in managing their educational growth. Currently, College Mentors are responsible for review of student performance for their students, particularly in the current second, third and fourth years of the curriculum. This type of effort will need to be expanded as the amount and breadth of assessment information expands. The final structure of mentor or assessment review may include separate mentors for each phase of the curriculum. Whatever the final design of the curriculum, attention and resources will be needed to ensure that a faculty mentor has access to assessment information for his or her students, has the skills needed to interpret assessment information and provide guidance to students, and have the time and resources to accomplish these tasks. h. Structure in Support of Assessment: UWSOM Assessment Committee As the basic unit for assessment moves from departmentalized courses and clerkships to integrated units with assessments that include components for multiple competencies and disciplines, a new structure is needed to oversee the process of assessment. A standing Assessment Committee made up of faculty, students, and professional staff should be established to complete the following charge: 1. Monitor assessment activities at all sites to ensure the quality and consistency of practices and outcomes; 2. Monitor the assessment system to determine its strengths and weaknesses in providing needed assessment data; 3. Advise faculty in the construction, implementation, and interpretation of assessment tools; 4. Advise the academic administration, curriculum committee and faculty regarding the success of our programs and students in meeting competencies and objectives; 5. Develop and suggest changes to assessment practices needed to complete its goals. Sufficient professional staff will be needed to support this work and other efforts of the assessment committee. 14

15 G TABLE 1: ASSESSMENT METHODS AND PURPOSE BY CURRICULUM PHASE AND COMPETENCY Phase Competency Method Purpose Data 1 Medical Brief written Application and analysis, interpretation of patient data, accessing literature 1 Medical Integrated MCQ General 1 Medical 1 Medical 2 Medical 2 Medical 2 Medical 2 Medical 2 Medical 2 Medical 2 Medical 2 Medical 3 Medical 3 Medical 3 Medical 3 Medical 3 Medical 3 Medical Long papers, projects Integration, application of scientific and critical reasoning Reflection Professional, ethical, values Brief written Ability to identify and apply literature findings to patient care Integrated MCQ Assess knowledge and cognitive skills Long papers Application of scientific and critical reasoning to a broad medical topic OSCE / SPs Formative and summative Px, Hx, Dx, communication skills Projects QI, patient safety, team building Reflection Professional growth Review of patient documentation Simulations / checklists Brief written Integrated MCQ Long papers OSCE / SPs Peer-assessment Projects Application of basic science to interpretation of patient information Assess team performance and leadership Statement of career goals, skills and accomplishments, and plans to achieve goals Assess knowledge and cognitive skills Application of scientific and critical reasoning to a broad medical topic Formative and summative Px, Hx, Dx, communication skills Professional and career growth, leadership, team ability QI, patient safety, team building, health care systems, Student 15

16 Phase Competency Method Purpose Data community based medicine 3 Medical Reflection Professional growth, consolidation of and 3 Medical 3 Medical 3 Medical Review of patient documentation Simulations / checklists USMLE Step 1, 2-CK professional identity Application of basic science to interpretation of patient information Assess team performance and leadership Ability to apply basic science and solve problems related to patient health and care 1 Patient Care 360 Degree Assess skills ratings 1 Patient Care Integrated MCQ Assess ability to relate concepts to care 1 Patient Care Mini-CEX/SDOT Document specific skills 1 Patient Care OSCE / SPs Hx, Px, Dx, communicate findings, utilize basic science concepts 1 Patient Care Projects Access and evaluate webbased content 1 Patient Care Script Assess skills of interpretation / Concordance or management Key Feature tests 2 Patient Care 360 Degree General patient care skills 2 Patient Care Brief written Assess skills and knowledge in various curricular threads 2 Patient Care Chart-stimulated recall Clinical reasoning and basic science foundation 2 Patient Care Integrated MCQ Clerkship specific knowledge and conceptual skills 2 Patient Care Mini-CEX/SDOT Specific patient care skills 2 Patient Care OSCE / SPs Core patient care competency 2 Patient Care Presentation Ability to synthesize and present basic science information as applied to patient care 2 Patient Care Reflection Various issues in patient care 2 Patient Care Review of portfolio 2 Patient Care Script Concordance or Key Feature tests 2 Patient Care Simulations / checklists Assess general progress Assess skills of interpretation / management in realistic situations Basic procedural skills / 16

17 Phase Competency Method Purpose Data 2 Patient Care USMLE Step 2- Patient care skills in CS communication, Hx, Px, and Dx 3 Patient Care 360 Degree General patient care skills 3 Patient Care Brief written Self-identification of accomplishments in brief rotations. Student 3 Patient Care Brief written Identification of career and leaning plans needed to achieve personal goals 3 Patient Care Chart-stimulated recall Clinical reasoning and basic science foundation 3 Patient Care Integrated MCQ Focus on medical decision making, e.g. chronic pain management, gross anatomy and procedural skills, utilization of diagnostic testing 3 Patient Care Mini-CEX/SDOT Specific patient care skills 3 Patient Care OSCE / SPs Ability to conduct focused Px and integrate clinical reasoning with Px benchmarks 3 Patient Care Presentation Ability to synthesize and present basic science information as applied to patient care 3 Patient Care Reflection Identification of personal values and interests 3 Patient Care Simulations / Assessment of team behavior, checklists leadership, professional communication, transitions of 3 Patient Care Simulations / checklists 3 Patient Care Simulations / checklists 1 Interpersonal/ 1 Interpersonal/ 1 Interpersonal/ 1 Interpersonal/ 1 Interpersonal/ Student care Basic procedural skills / Assessment of advanced procedural skills and competencies / Checklists Observe / Document skills Global ratings Observe / Document skills OSCE / SPs Document skills Patient Satisfaction Peer-assessment Document skills Re-enforce skills re leadership / Student 17

18 Phase Competency Method Purpose Data 1 Interpersonal/ Reflection Reinforce skills and conceptual frameworks 1 Interpersonal/ Review of Audio/Visual recording Review of patient documentation Self-assessment Reinforce skills and conceptual frameworks 1 Interpersonal/ Ability to collect and analyze patient communication 1 Interpersonal/ Re-enforce skills re Student leadership 2 Interpersonal/ 360 Degree skills with patients and professionals 2 Interpersonal/ Global ratings skills with patients and professionals 2 Interpersonal/ Mini-CEX/SDOT skills with patients and professionals 2 Interpersonal/ Patient Satisfaction and communication Satisfaction skills 2 Interpersonal/ Peer-assessment skills with patients and professionals 2 Interpersonal/ Presentation Presentation communication 2 Interpersonal/ Review of patient Professional communication documentation skills 2 Interpersonal/ Self-assessment skills with Student patients and professionals 3 Interpersonal/ OSCE / SPs Advanced communication skills 1 Professionalism 360 Degree Assess behavior 1 Professionalism Case-based Analysis of professionalism issues 1 Professionalism Global ratings Attendance, general conduct, / peer interaction skills 1 Professionalism OSCE / SPs Analysis of professionalism issues 1 Professionalism Reflection Understanding of professionalism issues 2 Professionalism 360 Degree Interprofessional teamwork 2 Professionalism Global ratings Evaluation of per to peer and student to family interactions, attendance, general demeanor, participation in team 2 Professionalism Reflection Responses to increasing patient responsibilities, Reflection on ethical and unprofessional issues observed 2 Professionalism Review of patient Sensitivity to professional 18

19 Phase Competency Method Purpose Data documentation issues, diversity, cultural sensitivity, etc. 2 Professionalism Simulations / checklists Analysis and application of principles to ethical issues / 3 Professionalism 360 Degree Interprofessional teamwork 3 Professionalism Brief written Formal case review and presentation to assess ability to apply ethical principles to patient care and professional responsibility 3 Professionalism Global ratings Evaluation of per to peer and student to family interactions, attendance, general demeanor, participation in team 3 Professionalism Global ratings Attendance, general conduct, peer interaction skills 3 Professionalism Reflection Responses to increasing patient responsibilities, Reflection on ethical and unprofessional issues observed 1 Based 1 Based 1 Based 1 Based 1 Based 2 Based 2 Based 2 Based 2 Based Self-Assessment Reflection and log Brief written Pre and post assessment of performance and outcomes Document feedback received and reflect Evaluation of scientific advancements that lead to improvement of care / s, Projects Assess computer literacy skills Brief written Article and literature review Self-Assessment Reflection and log Projects Projects Pre and post assessment of performance and outcomes Document feedback received and reflect Ability to develop QI mini projects Assess computer literacy and EMR skills s,, 19

20 Phase Competency Method Purpose Data 2 Based Brief written and log Summary and log of articles in clinical foundations 2 Based Global ratings Evaluations of presentations 3 Based 3 Based 3 Based 3 Based 3 Based 3 Based 1 System Based 1 System Based 1 System Based 2 System Based 2 System Based 2 System Based 2 System Based Reflection selfassessment Document feedback received and reflect on specific educational activities Brief written Develop learning plan for future s, Review of Audio/Visual recording Projects Global ratings Brief written, Presentation Brief written, Presentations Self-assessment, Reflection Simulations / checklists Self-assessment Simulations / checklists Peer-assessment Projects Ability to provide and receive feedback Ability to develop mini QI projects Observation of EMR use and knowledge Ability to analyze and convey practice based learning concepts and outcomes Assessment of understanding of health care systems and implications for patient care Reaction to healthcare system Interprofessional teamwork Preliminary understanding of issues in safety and systems practice Interprofessional teamwork, patient safety Interprofessional team performance Ability to apply systems concepts and principles of patient safety to a specific project (This might be more appropriate in phase 3) Interprofessional teamwork, patient safety, systems analysis / /, s Student / Student 2 System Based OSCE / SPs 2 System Based Peer-assessment Interprofessional teamwork,, 20