Assessment of communication skills in family medicine

Research Assessment of communication skills in family medicine Web exclusive Douglas Klein MD MSc CCFP Alim Nagji MD CCFP Abstract Objective To assess the communication and interviewing skills of incoming residents and provide formative feedback to residents early in their training. Design New residents completed a 15-minute objective structured clinical examination (OSCE) assessing communication skills and a 12-question, self-administered content quiz at the start of their residency. Each resident was directly observed by a family physician in the OSCE and provided with 15 minutes of structured feedback, with an opportunity for questions and discussion. The entire process remained private and did not affect summative evaluations. Setting Family medicine residency training program at the University of Alberta in Edmonton. Participants First-year family medicine residents. Main outcome measures Residents scores on the OSCE and the content quiz; residents rating of the usefulness of the assessment and the likelihood it would lead to practice change. Results A total of 61 residents (93.8%) completed the skills assessment (50 Canadian graduates, 11 international graduates). The mean score for the content quiz was 20.6 out of a total possible score of 24. Resident scores ranged from 8 to 24. The mean score on the OSCE practice interview was 21.1 out of 30, with a range of 13 to 29. Learner feedback indicated that the skills assessment was useful (4.68 out of 6) and would lead to a change in practice (4.43 out of 6). Conclusion The introductory communication OSCE and quiz offer new residents an opportunity to gauge their baseline skill level, become aware of program expectations early in their training, and garner specific suggestions in a nonthreatening environment. This tailored approach helps orient residents while taking into account their previous experiences. EDITOR S KEY POINTS The diverse backgrounds and previous experience of residents creates the potential for unequal footing during training. This study aimed to assess the communication and interviewing skills of incoming residents and provide early formative feedback. New residents completed an online quiz and an objective structured clinical examination assessing communication skills and received structured feedback from experienced family medicine faculty. Generally, residents performed well on both assessments, with similar performance among Canadian and international medical graduates. The residents found the assessment useful and indicated that it offered the opportunity for introspective learning and had the potential to change their practice. This article has been peer reviewed. Can Fam Physician 2015;61:e412-6 e412 Canadian Family Physician Le Médecin de famille canadien Vol 61: september septembre 2015

Exclusivement sur le web Évaluation de l habileté à communiquer en médecine familiale Recherche Douglas Klein MD MSc CCFP Alim Nagji MD CCFP Résumé Objectif Vérifier la capacité de communiquer et de faire des entrevues chez les nouveaux résidents et leur fournir un feed-back formatif en début de formation. Type d étude Les résidents ont répondu à un examen clinique objectif structuré (ECOS) de 15 minutes vérifiant leur capacité de communiquer et à un quiz de contenu auto-administré de 12 questions au début de leur résidence. Chaque résident était directement observé par un médecin de famille durant l ECOS et avait droit à un feed-back structuré de 15 minutes, avec possibilité de questions et de discussion. Tout ce processus demeurait privé et n affectait en rien les évaluations finales. Contexte Le programme de résidence en médecine familiale de l Université de l Alberta à Edmonton. Participants Des résidents-1 en médecine familiale. Principaux paramètres à l étude Les scores des résidents à l ECOS; leur évaluation de l utilité de cette évaluation; et la probabilité qu elle affecte leur pratique. Résultats Un total de 61 résidents (93,8 %) ont complété cette évaluation de leurs compétences (50 diplômés au Canada et 11 à l étranger). Les scores pour le queiz de contenu variaient entre 8 et 24, avec une moyenne de 20,6 sur un total possible de 24. Les scores à l entrevue simulée de l ECOS allaient de 13 à 29, pour une moyenne de 21,1 sur 30. En rétrospective, les résidents ont indiqué que cette évaluation de leurs compétences était utile (4,68 sur 6) et qu elle Points de repère du rédacteur Parce qu ils ont eu des conditionnements et des expériences différents, les résidents risquent de commencer leur formation sur des bases inégales. Cette étude voulait évaluer la capacité des nouveaux résidents de communiquer et d effectuer des entrevues, tout en leur offrant un feed-back précoce. Les nouveaux résidents ont répondu à un quiz en ligne et à un examen clinique objectif structuré qui évaluaient leur habileté à communiquer pour ensuite profiter d un feed-back structuré de la part de professeurs expérimentés en médecine familiale. Les résidents ont généralement bien performé à ces deux évaluations, les diplômés canadiens et étrangers obtenant des résultats semblables. Les résidents ont trouvé cette évaluation utile et souligné qu elle leur offrait une occasion d apprentissage introspectif susceptible de modifier leur pratique. Cet article fait l objet d une révision par des pairs. Can Fam Physician 2015;61:e412-6 entraînerait des changements dans leur pratique (4,43 sur 6). Conclusion L ECOS initial et le quiz offrent au participant une occasion de mesurer ses compétences de base en communication, de prendre connaissance des attentes du programme tôt dans sa formation et de profiter de suggestions spécifiques dans un contexte non menaçant. Une telle approche personnalisée contribue à orienter le résident tout en tenant compte de ses expériences antérieures. Vol 61: september septembre 2015 Canadian Family Physician Le Médecin de famille canadien e413

Research Assessment of communication skills in family medicine The doctor-patient relationship is the cornerstone of medical care 1 and has been shown to affect treatment outcomes and patient satisfaction. 2-6 The need for strong communication skills has been well explored. 7,8 As such skills are an essential component of medical school and postgraduate curricula, further advances are necessary to ensure the development of competent clinicians and communicators. A variety of assessment tools have been developed to evaluate the doctor-patient relationship, 9 with a particular emphasis on communication. 9,10 With the shift to competency-based curricula, there will be an increased need for evaluation tools that can offer formative and constructive feedback via observed interactions. Formalized communication will therefore be necessary 11,12 and will require trained observers and checklists, with the possible addition of multiple observers or videorecording. 13-15 Objective structured clinical examinations (OSCEs) remain the cornerstone of clinical examination, despite offering poor reliability in evaluating communication skills alongside clinical acumen. 16-18 Many individuals change cities or countries when transitioning from medical school to residency. The resulting diverse backgrounds and previous experience create the potential for unequal footing during residency. In the family medicine program at the University of Alberta in Edmonton, we piloted an initial evaluative OSCE to ascertain the baseline characteristics of residents while offering a structured and formalized avenue for feedback specifically on communication skills. Given the paucity of observed interactions noted in many residency programs, it is essential that residents are offered the opportunity to have direct, specific feedback targeting their style, technique, and execution in patient interviews. As the patient history is key to medical diagnosis and the patient encounter is a critical component of the doctor-patient relationship, investing time and resources to ensure development of baseline expectations might enhance the communication skills of residents. Methods In 2012, the Department of Family Medicine at the University of Alberta introduced a skills assessment at the beginning of residency. The purpose was to assess residents baseline skills in order to better support their training. The skills assessment involved 2 components: an online written quiz and an observed OSCE practice interview. The online quiz evaluated the resident s knowledge of communication microskills, such as paraphrasing and use of open-ended questions. Behaviour relating to personal well-being and resiliency was also assessed. The practice interview was observed live on camera by experienced faculty. Oral and written feedback were provided to the resident on listening skills, verbal and nonverbal communication, and the overall effectiveness of the interview. All the residents completed a formal evaluation of the process following the assessment, rating elements of the assessment on a 6-point Likert scale. The content of the online written quiz and the evaluation criteria for the observed practice interview were determined by a community of family medicine faculty and reflect some basic competencies in interviewing and communication necessary for doctor-patient interaction. This 2-part skills assessment was based on an evaluation framework used by the College of Family Physicians of Canada. 19,20 The skills assessment was then piloted with both faculty and residents in family medicine to ensure acceptability and feasibility. Ethics approval was obtained through the University of Alberta Health Research Ethics Board. RESULTS A total of 61 residents completed the skills assessment process, representing 93.8% of the first-year residents. Of the residents who completed the assessment, 26 were female (42.6%); 50 (82.0%) were Canadian medical graduates, while the remaining 11 were international medical graduates (IMGs). Some residents could not participate owing to scheduling conflicts with mandatory service-based rotations. Residents scored well on the online quiz (Table 1). The mean (SD) score was 20.6 (3.05) out of a total possible score of 24 (range 8 to 24). The mean (SD) score on the observed practice interview was 21.1 (4.32) out of 30 (range 13 to 29). Table 2 shows the results of the practice interviews broken down by types of skills observed. In addition to scoring the resident, the faculty observer looked for specific behaviour during the practice interview. Elements of FIFE (feelings, ideas, function, expectations) were demonstrated by only half of the residents. Table 3 lists the behaviour observed and the frequency with which each was observed among the residents. Overall, the residents found the 2-part skills assessment quite valuable (Table 4), citing introspective learning and potential future integration of feedback, and believed it was an appropriate measure of their skill. Table 1. Results of the online assessment of communication skills Skill (maximum possible score) Mean (SD) score Communication skills (12) 10.0 (1.87) Feedback knowledge (5) 4.4 (1.54) Doctor-patient relationship knowledge (7) 6.3 (0.92) Total (24) 20.6 (3.05) e414 Canadian Family Physician Le Médecin de famille canadien Vol 61: september septembre 2015

Assessment of communication skills in family medicine Research Table 2. Results of the communication OSCE assessment: Each skill is scored out of 5 and the total is scored out of 30. SKILL Mean (SD) score Listening skills 3.7 (0.87) Nonverbal expressive 3.6 (0.85) Nonverbal receptive 3.2 (0.95) Verbal skills 3.8 (0.83) FIFE 3.1 (1.01) Overall 3.6 (0.83) Total 21.1 (4.32) FIFE feelings, ideas, function, expectations; OSCE objective structured clinical examination. DISCUSSION As in previous studies, residents skills assessment scores were highly varied, likely owing to their different experiences before commencing training. 21,22 Therefore, the skills assessment can offer educators insight into the trends and baseline characteristics of the incoming cohort. Further, it creates an organized avenue for residents to receive targeted feedback in an area that will likely go unsupervised for most of their training. Although the online assessment was unsupervised, it is an emerging avenue for assessing knowledge of foundational concepts, including definitions and frameworks. Determining the value of such assessment will require further research to elucidate possible relationships with the skills assessment and with resident performance. Residents who might not have been aware of these key concepts beforehand were able to take the opportunity to supplement their knowledge before commencing their training. It is interesting to note that some residents still performed poorly in the online quiz. We hypothesize 2 possible reasons: either residents thought that they were not able to use resources while completing the quiz, or perhaps they trained at institutions that used alternate methods to impart these skills, as these individuals seemed to perform as well as their peers in the observed OSCE practice interviews. Fortunately, most residents demonstrated solid foundational knowledge in their interviews, possibly owing to the increasing emphasis placed on communication skills in medical schools. Of note, individuals seemed to perform poorest in the FIFE domain, an interesting development given the relative ease of scoring points in this category. Feedback given after the interviews was well received by residents, with by far most indicating that it would lead to practice change and also noting that it offered introspective discovery. Table 3. Behaviour demonstrated by residents during the communication OSCE: N = 61 residents. Behaviour Observed N (%) Allows time for appropriate silences 24 (39.3) Paraphrases 32 (52.5) Responds to patient cues 43 (70.5) Clarifies jargon when used by the patient 5 (8.2) Uses eye contact appropriate for culture and 57 (93.4) patient comfort Is focused on the conversation 49 (80.3) Adjusts demeanour to be appropriate to the 25 (41.0) patient s context Uses physical contact appropriate to the patient s 2 (3.3) comfort Responds appropriately to the patient s discomfort 32 (52.5) Verbally checks the meaning of body language, 5 (8.2) actions, and behaviour Uses language adequate to be understood by the 49 (80.3) patient Is able to converse at an appropriate level for the 46 (75.4) patient s age and education Uses appropriate tone for the situation 43 (70.5) Asks open-ended and closed-ended questions 31 (50.8) appropriately Checks back with patient to ensure understanding 21 (34.4) Facilitates the patient s story 32 (52.5) Provides clear and organized information in a 16 (26.2) way the patient understands Clarifies how the patient would like to be addressed 13 (21.3) Understands the patient s feelings regarding the 30 (49.2) medical illness Learns of the patient s ideas regarding the 25 (41.0) medical illness Learns about the effect of the illness on function 30 (49.2) Understands the patient s expectations for the visit 30 (49.2) Displays empathy 40 (65.6) Is genuine in the interaction with the patient 40 (65.6) Is respectful in dealing with the patient 39 (63.9) Demonstrates appropriate time management 16 (26.2) Demonstrates smooth interview flow 21 (34.4) OSCE objective structured clinical examination. Limitations Comparing IMGs to graduates of the Canadian system poses an interesting challenge, given the relative differences in curriculum designs and the heterogeneity of the former group. Previous literature has documented the need for communication training in IMG populations. 23,24 In our study the groups scored similarly on both assessments (t test, P =.52 for the quiz and P =.81 for the OSCE); Vol 61: september septembre 2015 Canadian Family Physician Le Médecin de famille canadien e415

Research Assessment of communication skills in family medicine Table 4. Results from the resident evaluation of the combined skills assessment Mean (SD) Score Variable (out of 6) The skills assessment was useful 4.68 (1.08) The feedback will change my interviewing 4.43 (1.06) The skills assessment represents my true 4.45 (1.20) performance I learned about myself during the assessment 4.66 (1.10) The online quiz showed my knowledge 3.77 (1.11) Overall, the skills assessment was worthwhile 4.57 (1.12) however, the small sample size and single-site design are notable limitations. In Alberta, IMGs must first complete a longitudinal observership and are prescreened with communication OSCEs before receiving acceptance into our residency program. 25 It is plausible that this leads to a selection bias when considering how this cohort performs on a downstream assessment, and our results might not be generalizable to other programs. Conclusion An introductory assessment of communication skills through observed interviews and an online quiz might be a valuable tool to stratify the baseline characteristics of new trainees. Country of training might not play a pivotal role or might be minimized by the varying entry requirements for IMGs and Canadian graduates. Residents found this to be a useful exercise with the potential to change practices as a result of the specific feedback offered. Further research can examine if these assessments can be used to identify residents who might struggle in the future. Dr Klein is Associate Professor in the Department of Family Medicine at the University of Alberta in Edmonton. Dr Nagji was a third-year resident in family and emergency medicine at the University of Alberta at the time of writing. Contributors Both authors contributed to the concept and design of the study; data gathering, analysis, and interpretation; and preparing the manuscript for submission. Competing interests None declared Correspondence Dr Douglas Klein; e-mail Doug.klein@ualberta.ca References 1. Engel GL. Enduring attributes of medicine relevant for the education of the physician. Ann Intern Med 1973;78(4):587-93. 2. Fuertes JN, Mislowack A, Bennett J, Paul L, Gilbert TC, Fontan G, et al. The physician-patient working alliance. Patient Educ Couns 2007;66(1):29-36. Epub 2006 Dec 22. 3. Martin DJ, Garske JP, Davis MK. Relation of the therapeutic alliance with outcome and other variables: a meta-analytic review. J Consult Clin Psychol 2000;68(3):438-50. 4. Griffith S. A review of the factors associated with patient compliance and the taking of prescribed medicines. Br J Gen Pract 1990;40(332):114-6. 5. Stewart M, Brown JB, Donner A, McWhinney IR, Oates J, Weston WW, et al. The impact of patient-centered care on outcomes. J Fam Pract 2000;49(9):796-804. 6. Kaplan SH, Greenfield S, Ware JE Jr. Assessing the effects of physicianpatient interactions on the outcomes of chronic disease. Med Care 1989;27(3 Suppl):S110-27. 7. Dutta-Bergman MJ. The relation between health-orientation, provider-patient communication, and satisfaction: an individual-difference approach. Health Commun 2005;18(3):291-303. 8. Cegala DJ, Street RL Jr, Clinch CR. The impact of patient participation on physicians information provision during a primary care medical interview. Health Commun 2007;21(2):177-85. 9. Eveleigh RM, Muskens E, van Ravesteijn H, van Dijk I, van Rijswijk E, Lucassen P. An overview of 19 instruments assessing the doctor-patient relationship: different models or concepts are used. J Clin Epidemiol 2012;65(1):10-5. 10. Boon H, Stewart M. Patient-physician communication assessment instruments:1986 to 1996 in review. Patient Educ Couns 1998;35(3):161-76. 11. Herbers JE Jr, Noel GL, Cooper GS, Harvey J, Pangaro LN, Weaver MJ. How accurate are faculty evaluations of clinical competence? J Gen Intern Med 1989;4(3):202-8. 12. Burdick WP, Schoffstall J. Observation of emergency medicine residents at the bedside: how often does it happen? Acad Emerg Med 1995;2(10):909-13. 13. Rosenzweig S, Brigham TP, Snyder RD, Xu G, McDonald AJ. Assessing emergency medicine resident communication skills using videotaped patient encounters: gaps in inter-rater reliability. J Emerg Med 1999;17(2):355-61. 14. Novack DH, Dubé C, Goldstein MG. Teaching medical interviewing: a basic course on interviewing and the physician-patient relationship. Arch Intern Med 1992;152(9):1814-20. 15. Kroboth FJ, Hanusa BH, Parker S, Coulehan JL, Kapoor WN, Brown FH, et al. The inter-rater reliability and internal consistency of a clinical evaluation exercise. J Gen Intern Med 1992;7(2):174-9. 16. Brannick MT, Erol-Korkmaz HT, Prewett M. A systematic review of the reliability of objective structured clinical examination scores. Med Educ 2011;45(12):1181-9. Epub 2011 Oct 11. 17. Nagji A. OSCE: the subjective experience of an objective exam. Univ Alberta Heal Sci J 2012;7(1):2. 18. Thistlethwaite JE. Developing an OSCE station to assess the ability of medical students to share information and decisions with patients: issues relating to interrater reliability and the use of simulated patients. Educ Health (Abingdon) 2002;15(2):170-9. 19. Lawrence KJ, Graves L, MacDonald S, Dalton D, Tatum R, Blais G, et al. Distinguishing a Certificant from a Superior Certificant : exploration of the illness experience. Mississauga, ON: College of Family Physicians of Canada; 2010. 20. Allen T, Bethune C, Brailovsky C, Crichton T, Donoff M, Laughlin T, et al. Defining competence in family medicine for the purposes of Certification by the College of Family Physicians of Canada: the evaluation objectives in family medicine. Mississauga, ON: College of Family Physicians of Canada; 2011. 21. Lypson ML, Frohna JG, Gruppen LD, Woolliscroft JO. Assessing residents competencies at baseline: identifying the gaps. Acad Med 2004;79(6):564-70. 22. Roth CS, Watson KV, Harris IB. A communication assessment and skillbuilding exercise (CASE) for first-year residents. Acad Med 2002;77(7):746-7. 23. Hall P, Keely E, Dojeiji S, Byszewski A, Marks M. Communication skills, cultural challenges and individual support: challenges of international medical graduates in a Canadian healthcare environment. Med Teach 2004;26(2):120-5. 24. Pilotto LS, Duncan GF, Anderson-Wurf J. Issues for clinicians training international medical graduates: a systematic review. Med J Aust 2007;187(4):225-8. 25. Hofmeister M, Lockyer J, Crutcher R. The multiple mini-interview for selection of international medical graduates into family medicine residency education. Med Educ 2009;43(6):573-9. e416 Canadian Family Physician Le Médecin de famille canadien Vol 61: september septembre 2015