2014 CGEA 3/28/2014. Workplace Assessment. Assessment of what trainees (or doctors) actually do in practice. Miller BMJ 2010;341:c5064

Transcription

1 Workplace-based assessment: Capturing written comments with a Cloud-based mobile assessment tool Workplace Assessment Assessment of what trainees (or doctors) actually do in practice. Workplace Assessment One major advantage of workplace based assessment is its ability to evaluate performance in context. Gary Ferenchick, MD David Solomon, PhD Churlson Han, MD Department of Medicine CGEA conference March 2014 Miller BMJ 2010;341:c5064 Miller s Pyramid of Assessment Direct observation Actual patient care: Workplace assessment Does Behavior Demonstration of skills: Standardized Shows How patients Knowledge application: Clinical Knows How problem solving Cognition Knowledge: Multiple choice Knows questions Enabling Step A premise for effective workplace based assessments is that the criteria against which judgments are made need to be explicit and clearly understood by the trainee and the assessor Problems with unprompted observations Standards for judging clinical performance are unclear to evaluators Problems existed in trying to ensure that everyone was working to the same or similar standards (Hill 2009) Faculty unsure of the level of performance expected of the learners (Fernando 2008) Poor agreement among faculty evaluating students due to the fact that Standards for judging clinical competence were not explicit (Hasnain 2004) 1

2 Problem with unprompted observations Standards for judging clinical performance are unclear to evaluators The lack of a clear standard for judging performance was considered problematic (Berendonk 2013) At some point during the interview, almost every (faculty) participant mentioned feelings of insecurity in relation to assessment. (Berendonk 2013) Just In Time A Cloud-based tool.. To enhance the transparency of specific performance standards Can be used in a consistent manner by different observers To provide a structure for assessment & feedback JIT: Sample screen progressions Adv in Health Sci Educ 2013;18:559 JIT: Sample screen progressions JIT: Sample screen progressions Milestone Achievement Dashboard: Individual Learner Score and Grade Captured comments Authoring of content controlled by faculty Authoring of content controlled by faculty 2

3 Internal Medicine: ABIM Milestone Achievement Individual Dashboard Milestone Report: Individual Dashboard Milestone Report: Program Dashboard Hover over gives the specific competency ABIM Milestones Competencies accomplished or not Milestones Milestone Elements Individual assessments green = well done yellow = needs improvement dark = yet to be assessed Example of individual resident dashboard Goals of the JIT app. Facilitate the assessment of a trainees clinical skills during encounters with real patients Increase the transparency of the expected competencies for these observations Create a permanent record of observed skills which track to milestones with minimal administrative effort Capture qualitative comments from evaluators Comment Enhanced CEX Identify one area the student did well Identify one area the student can improve upon Objectives Objective 1. To describe the percentage of Cloudbased assessments which included comments (i.e. comment enhanced evaluations) Objective 2. To describe the percentage captured comments that were General ( Practice more ) Specific ( Excellent neuro exam ) Specific with detail ( Percuss lung fields and compare sides ) Objective 3. To determine if comment type varied by resident vs attending and the purpose of the comment (positive vs corrective), and to estimate the amount of time for feedback 3

4 Methods: Setting College of Human Medicine 8 week clerkship 7 sites in Michigan July 2012 Oct 2013 Comment section added to JIT 256 students Required to do 10 CEX s 309 raters used JIT to guide observations 96 attendings 213 residents Faculty provided no training on providing narrative comments IRB Exempt Methods: Inter-rater reliability All comments downloaded to a spread sheet 3 coders independently coded 100 comments A second round of 209 comments were recoded Inter-rater reliability was established at.67 Each coder rated 33% of the 1287 records Results: Number of comment enhanced CEXs Variable Number Students 256 Students with > 1 comment enhanced CEX 227 (88.7%) Total number of comment enhanced CEXs 1287 Average comment enhanced CEXs per student 4.9 Results: Categories of comments Results: Categories of comments Time for Feedback Well done feedback Residents Attendings Comment Type N Only One area the student did well 288 Only One area the student can improve upon 44 Both 955 Total records 1287 Total comments 2242 % % General Specific Specific with detail Needs improvement feedback Residents Attendings % % Minutes providing feedback Total (%) < > % General Specific Specific with detail

5 Discussion Feedback Essential for development of clinical skills Numerical scores less effective than narrative feedback Specific enough for the student to remediate deficiencies Strategies for improvement is an essential component Discussion Trainees commonly do not view the feedback received from faculty as credible nor influential in learning, especially if the feedback is not immediate and tied to the trainees clinical workplace performance. Discussion Few strategies for getting performance criteria to the student and evaluator at the time of the assessment Watling C,et al Med Educ. 2012;46(2): JIT leverages existing smart mobile technology Smartphone Citations in Pubmed Smartphone Citations in Pubmed Cloud-based assessment Previous work JIT: Leverages smart phones as assessment platforms Delivers customized checklists via self-service interface Criterion-based assessment High user satisfaction Adequate reliability and validity Allows for robust reporting Facilitates capturing of comments tied to specific observed behaviors Ferenchick, et al. PeerJ Sep 17;1:e164. doi: /peerj.164 Ferenchick, et al. Teach Learn Med. 2013;25(4):292-9 Comment specificity If the specificity of the comments is a metric for quality, a high percentage of the comments were either specific (50 61%) or specific with detail (23 38%) 5

6 Comment specificity Via feedback encounter cards Feedback card use prompt students to ask for formative feedback 485 mini-cex assessments 57% of the comments were specific 63% of 5369 comments were specific We found that most evaluators will write something when provided a space to do so. BMC Med Ed 2012;12:97 Med Ed 2008;42:1071 Limitations Single institution Only written comments were assessed. We do not know if written comments paralleled verbal feedback The effect JIT on implementation of feedback and learning of acquired clinical skills not established Conclusions Cloud-based technology can facilitate the completion of written comments on student performance tied to a specific encounter Provide a structure for feedback and help identify area for improvement based upon supportable evidence The specificity of the comments was high Feedback was < 10 minutes in ~ 75% of assessments End 6