Alfredo A. Sadun, MD, PhD F. Thornton Professor of Vision Doheny Eye Inst.-Keck/USC Sch. Med.

Size: px

Start display at page:

Download "Alfredo A. Sadun, MD, PhD F. Thornton Professor of Vision Doheny Eye Inst.-Keck/USC Sch. Med."

Blaze Gilbert
8 years ago
Views:

1 The 360 Degree Evaluation Alfredo A. Sadun, MD, PhD F. Thornton Professor of Vision Doheny Eye Inst.-Keck/USC Sch. Med.

2 Early attempts on creating reliable, quantitatable, testable, measures of resident performance and progress

3 360 Degree Evaluation of From Supervisors Residents Faculty (traditional approach) From Peers* From Nurses* & Techs* From Patients * Probably need anonymity

4 360 degrees = The circle of a resident s s professional contacts More specificity in questions (e.g., ask nurses specifically about resident interpersonal skills with nursing regarding patient orders or discharge care) Less questions per evaluator ( small bites ). Global evaluation with 360 degree view

, ask nurses specifically about resident interpersonal skills with nursing

6 The 360 degree A survey or questionnaire is used to collect data on performance Captures areas not amenable to written testing teamwork communication, management style Patient surveys may assist in the evaluation of general qualitative aspects personal interaction communications skills courtesy, interest, and empathy.

management style Patient surveys may assist in the evaluation of general

7 360 Reliability may be as high as 0.90 for a properly performed institutional patient survey. But patient surveys require multiple responses (20 to 40) to be reliable and are time intensive to collect and report.

8 AAO Task Force medical knowledge Recommend that the assessment of medical knowledge (fund of knowledge) be incorporated into the existing global evaluation forms.

9 AAO Task Force medical knowledge-2 Recommend that the assessment of medical knowledge include documentation of achievement of specific, measurable, explicit, learning objectives Derived from a standardized curriculum and distributed to the learner before the global evaluation is completed.

measurable, explicit, learning objectives Derived from a standardized

10 360 at USC (in preparation) 4 questionnaires with scores for 20 questions each. Test Q year for reliability Test validity by comparing to older system of faculty evaluations

11 For Faculty Questionnaire Medical knowledge Medical judgment Surgical skills Surgical judgment History taking Clinical examination skills Relationships with faculty and staff Attitude, professionalism, ethics, etc.

taking Clinical examination skills Relationships with

12 Ex. from Faculty Questionnaire Does the resident take a comprehensive yet focused history in which he applies a sound factual base in creating and testing a series of hypotheses? 1 = Very cursory history 5 = Competent history but lacks efficiency 9 = Superb, informative, sensitive and subtle history taking

13 For Resident s s Peers Similar to faculty questionnaire but more specific and: More on relationship to co-residents More on relationships with nurses/techs More on conscientiousness More on honesty More on work hours

14 Ex. from co-residents Does your co-resident treat the nurses and technical staff with respect and sensitivity? Circle one

15 For Nurses and Techs Very Specific Questions on the Resident s style A) Approach with Patients such as: Courteous, communicates well, spends enough time, respectful, reassuring, etc. B) Approach with Staff C) Approach with Faculty

Courteous, communicates well, spends enough time,

16 Ex. from Nurses and Techs Q. Does the resident work as a team member coordinating and communicating effectively with faculty, house staff, clerks nurses and technical assistants?

17 For Patients Like a satisfaction survey Specific questions such as: Did the resident see you on time? Did (s)he spend enough time with you? Did he introduce himself, wash his hands? Did he explain things thoroughly? Was he pleasant and congenial? Was he respectful?

18 Ex. from patients Q Did your physician treat you with courtesy and respect; was s(he) a good listener?

19 Evaluation instrument Development Faculty Relevance to competency evaluation (major tools) Stage Burden I. II. III. IV. V. VI. VII. OKAP exams Implemented + 90-degree evaluation form Implemented High + OCEX (Ophthalmic Clinical Evaluation Exercise) In validation Mod + Resident portfolio nil Surgical Skills Assessment Designed High + Journal club evaluation In open trial Mod Surgical logs Implemented Notable incident reports Discussion Mod + Skills checklists Discussion ABMS Patient communication survey Validated degree evaluation (job class specific) Discussion Professionalism tool In development by ABMS Minimal +

nil Surgical Skills Assessment Designed High + Journal club evaluation In open trial Mod Surgical logs Implemented Notable incident reports Discussion Mod +

20 6 Competencies (+1) 1) Pt. Care 2) Medical knowledge 3) Practice based learning 4) Communication skills 5) Professionalism (ethics) 6) Systems (managed care) +1 = Ophthalmology Surgical Skills

21 Examples Evaluation instrument Development (major tools) Stage Faculty Relevance to Comp. Burden OKAP exams Implemented Only II (med. knowl) 90-degree evaluation OCEX (Oph Cl Eval Ex) Implemented High All but III (Pract. Based) In validation Mod I, IV, V Resident portfolio Nil

22 Further examples Burden Surgical Skills Assessment Designed High Journal club evaluation In open trial Mod Surgical logs Implemented Notable incident reports Discussion Mod Skills checklists Discussion Relevance Only VII II and III Only VII III, VI and VII I and VII

23 Examples - 3 Burden Relevance Skills checklists Discussion I and VII ABMS Patient communication survey Validated Only III 360 degree evaluation (job class specific) Discussion All but II and III Particularly high relevancies Professionalism tool In development Minimal Only III

24 6 Competencies (+1) I) Pt. Care II) Medical knowledge III) Practice based learning IV) Communication skills V) Professionalism (ethics) VI) Systems (managed care) +1 = Ophthalmology Surgical Skills

25 6 + 1 Competencies Real instruments that can measure It s s time to start 360 good place to begin Almost fully developed burden especially for faculty Quantitative Can be validated, reliable

27 Proposed instrument Developmental Stage Faculty Burden Patient Care Med know Pract Based Learn Interper Skills & Comm Profes- sionalis m System Based Practice Surg OKAP exams Implemented Strong Standard resident evaluation form Implemented High Med Strong Medium Medium Medium Medi OCEX (Ophthalmic Clinical Evaluation Exercise) Validated Mod Strong Medium Medium Surgical Skills Assessment Form Designed Mod- high Strong Journal club evaluation form Draft in use Mod Medium Med ABMS Patient communication survey Validated Strong Resident portfolio review and follow-up (Standardized table to contents) Discussion Mod Medium Strong 360 degree evaluation (job class specific) Discussion Med Strong Strong Strong Med Chart audits Discussion High Strong Med Medium Notable incident reports Discussion Mod Med Strong Med Skills checklists Discussion Med Med Professionalism tool In development by ABMS Minimal Strong

State of the art - Evaluating teaching in medical schools: Why, What and How

State of the art - Evaluating teaching in medical schools: Why, What and How Linda Snell MD MHPE FRCPC FACP Centre for Medical Education & Department of Medicine, McGill University, Montreal, Canada Visiting