A randomized trial of teaching bioethics to surgical residents

The American Journal of Surgery 189 (2005) 453 457 Surgical education A randomized trial of teaching bioethics to surgical residents Anja Robb, M.Ed. a,h, Edward Etchells, M.D., M.Sc. b,e,f, Michael D. Cusimano, M.D., Ph.D., F.R.C.S.C. g,i, Robert Cohen, Ph.D. d, Peter A. Singer, M.D., M.P.H. c,e,f, Martin McKneally, M.D. Ph.D. c,f,g, * a Centre for Research in Education, University of Toronto, Toronto, Canada b Sunnybrook and Women s College Health Sciences Centre, Toronto, Canada c University Health Network, Toronto, Canada d Center for Medical Education, Hebrew University Faculty of Medicine, Jerusalem, Israel e Department of Medicine, University of Toronto, Toronto, Canada f Joint Centre for Bioethics, University of Toronto, Toronto, Canada g Department of Surgery, University of Toronto, Toronto, Canada h Department of Family and Community Medicine, University of Toronto, Toronto, Canada i St. Michael s Hospital, Toronto, Canada Manuscript received February 4, 2004; revised manuscript August 2, 2004 Abstract Background: Bioethics education has been recommended as a formal component of surgical residency training. The best method for teaching bioethics to residents is unclear. We compared the effectiveness of a standardized patient (SP)-based seminar to a traditional seminar format for teaching bioethics to surgical residents. Methods: We randomized 31 first- and second-year surgical residents to either a SP-based seminar or a traditional seminar on informed consent. Immediately after the seminars, we evaluated resident performance in patient encounters on informed consent by using an objective structured clinical examination. Their knowledge of informed consent was also evaluated by using a 20-question short-answer written examination immediately after the seminars and then 3 weeks later. Results: Twenty-nine residents completed the study; two withdrew because of an emergency. The SP seminar group had lower SP interview scores on the 22 item checklist compared with the traditional seminar group (57% versus 66%; difference 9%; 95% confidence interval [CI], 17% to 1%, P.03). The SP seminar group also had lower knowledge scores on the questionnaire immediately after the seminar (60% versus 73%; difference 13%; 95% CI, 21% to 4%, P.003). The difference in knowledge scores persisted at 3 weeks (41% for the SP group, 59% for the traditional seminar group; difference 18%; 95% CI, 29% to 7%; P.002). Conclusions: A traditional seminar was superior to an SP-based seminar for teaching informed consent to surgical residents. 2005 Excerpta Medica Inc. All rights reserved. Keywords: Informed consent; Standardized patients; Surgical education; Ethics Peter A. Singer is a Canadian Institutes of Health Research Investigator. Anja Robb is Director of the Standardized Patient Program, University of Toronto. * Corresponding author. Tel. 1-416-946-8084; fax: 1-416-978-1911. E-mail address: martin.mckneally@utoronto.ca Bioethics is an important component of undergraduate and postgraduate medical curricula [1,2]. The American College of Surgeons is currently developing a curriculum for surgical residents. The best techniques for teaching and evaluating ethics teaching are not well defined. Traditional ethics seminars and lectures for undergraduates are associated with improved scores on reflectiveness [3] and moral reasoning [4,5]. A text-based ethics course with lectures and small group seminars was associated with improved knowledge but no change in attitudes or analysis of case vignettes [6]. In a randomized trial, a combination of lectures and small group conferences improved the care of patients with do not resuscitate orders; fewer medical residents offered futile therapy to patients with do not resuscitate orders after the intervention [7,8]. Ethics education can also improve house officers knowledge and self-confidence regarding medical ethics [9]. Standardized patients (SPs) have been used increasingly 0002-9610/05/$ see front matter 2005 Excerpta Medica Inc. All rights reserved. doi:10.1016/j.amjsurg.2004.08.066

454 A. Robb et al. / The American Journal of Surgery 189 (2005) 453 457 for case-based teaching and testing of medical skills. Ninety-seven percent of medical schools in the United States and Canada use SPs for teaching [10]. SPs have many potential advantages, including eliminating the variability and risks associated with using actual patients, providing practice with difficult patients or sensitive situations, providing an opportunity for direct feedback about interpersonal skills, and reducing the pressures of time generally experienced in actual clinical situations [11]. They may be particularly well suited for the sensitive and difficult issues in surgical ethics. We have used SPs for teaching and evaluating clinical ethics for the past 12 years [12,13]. Our objective was to evaluate the effectiveness of a SP-based seminar compared with a traditional seminar for teaching informed consent to surgical residents. We chose this topic because it is an essential skill for surgeons, and we targeted first- and second-year surgical residents because their clinical responsibilities include ensuring that informed consent is obtained for surgical procedures. Our primary outcome was resident performance in an objective structured clinical examination. Our secondary outcome was resident knowledge using a 20-question short-answer written examination. Methods Study design The study was described to potential participants 1 week before implementation, at the 90-minute principles of surgery lecture given weekly at 0730. Willing participants gave informed consent at that time. On the day of the study, participants were randomized to either a SP-based seminar group or a traditional seminar group. Randomization was done in blocks of 10, and residents were given sealed opaque consecutively numbered envelopes on the day of the study containing their allocations. A traditional surgical seminar on pulmonary embolism was given simultaneously in an adjacent classroom. The participants switched classrooms during the second hour. Participants The participants were first- and second-year residents in the Department of Surgery at the University of Toronto. Intervention: traditional seminar group The traditional seminar group was led and facilitated by a senior surgeon with bioethics expertise (M.M.). He conducted a 1-hour interactive discussion that covered relevant legal and ethical issues involved in the process of informed decision making using a one-page case vignette on consent to carotid endarterectomy after an episode of transient monocular blindness. A structured discussion was built around a lesson plan by using a standard series of questions such as Do you give the patient statistics? Are you required to give a minicourse in medicine? What risks must be disclosed? and so on as described in the Royal College of Physicians and Surgeons of Canada Surgical Ethics Curriculum [14]. The residents discussed interactively how they would apply their knowledge of informed decision making to the carotid endarterectomy case. Intervention: SP seminar group The SP group participated in an interactive discussion about the relevant components of informed consent for 1 hour, led by the same surgical ethics facilitator using the same questions and lesson plan. In lieu of the case vignette, students were presented with a SP who portrayed the patient in the carotid endarterectomy vignette. Students were able to interact with the SP and use a time-in, time-out technique [15]. This technique enables the interview to be stopped in progress, allowing the instructor to ensure that the relevant legal and ethical issues were discussed. After a time-out, the SP interaction is restarted with a time-in, either with the same or a different resident. The residents watched, critiqued, and discussed their colleagues technique. The standardized patients were trained actors well versed in their roles. Their script included questions like why operate on my neck if the problem is in my eye? Outcome measures The primary outcome measure was an evaluation conducted by observing the residents obtain informed consent for elective laparoscopic cholecystectomy, using their own words, during a 10-minute SP interview. Three independent observers assessed each resident performance during the gallbladder objective structured clinical evaluation by checking it against specified criteria outlined in Table 1. Observers were blind to the intervention. At least one observer in each room was a physician. Observers included eight physicians, five bioethics graduate students, three nurses, and one pharmacist. Observers were trained before the study in a 30-minute session, during which they observed the gall bladder consent scenario, used the checklist to assess the performer, and asked questions. The checklist items included clinical information for the examiners where appropriate, such as risk estimates and expected length of hospital stay for cholecystectomy. Before the start of the study, the scenario and checklists were field tested by three general surgeons who routinely perform the operation. An expert panel, including three ethicists, a physician, a nurse, and a lawyer, further validated the checklist. (The checklist items are listed in Table 1.) The secondary outcome measure was knowledge about the key ethical considerations and legal requirements for informed decision making. We developed a knowledge questionnaire based on the content of the lectures and handouts. We administered the questionnaire immediately after

A. Robb et al. / The American Journal of Surgery 189 (2005) 453 457 455 Table 1 Individual item analysis for SP checklist Checklist item the intervention and 3 weeks later. Two independent ethics graduate students, blinded to the group allocation, scored the questionnaires. Analysis Judged present by all three observers (n 29) Overall score Identifies his/her position as resident on 18.79 surgical team Reviews patient s understanding that specific 17.80 and characteristic symptoms of biliary colic are due to gallstones Reviews patient s understanding that vague, 1.08 nonspecific abdominal symptoms are not necessarily due to gallstones Explains the procedure What is going to be done 26.95 How long the operation takes 8.47 How long hospital stay will be 28.99 How long to return to normal activities 11.62 Who will do the surgery 27.95 Evaluates patient s understanding; asks 3.41 patient to explain in their own words Explains that surgery will almost certainly 10.57 relieve specific symptoms of biliary colic Explains that surgery will have uncertain 1.10 effect on nonspecific abdominal complaints Discusses alternatives to surgery Delaying the surgery 13.55 Refusing the surgery 6.28 Discloses risks of procedure Death (0.2%; range 0 1% low/very low) 6.30 Need for big incision (1%; range 0.3 24.91 3%, low/very low) Injury to the bile duct (range 0.3 3% low/ 16.62 very low) Explores potential impact of surgery on patient s particular situation Personal life (eg, caring for 3.16 children/family) Work life 18.72 Creates an atmosphere that is conducive to 18.79 asking questions Invites patient to ask questions at end of 25.95 interview or offers opportunity for further discussion Makes a recommendation if requested by 18.76 patient Uses appropriate language for explanations 17.81 The mean scores on the SP interview and the knowledge test were compared by using analysis of variance. The significance level was set at P.05 (two-tailed). We estimated internal consistency reliability of the SP interview checklist by using Cronbach s alpha. We calculated three Cronbach alpha estimates, one for each of the three sets of SP checklist scores. We estimated interobserver reliability for the SP checklist scores and knowledge questionnaire scores by using intraclass correlation coefficients. We also conducted an exploratory (post hoc) item analysis for the SP checklist. We calculated an overall score for each item as follows: the 29 interviews had 3 observers, so the maximum possible score for each checklist item was 87 (29 interviews 3 observers). We counted the total number of checks for each item and divided by 87 to get an overall score. We noted that the distribution of scoring was not symmetric, so we did a second analysis where the student would get credit for the item if checked by all three observers. Finally, we explored the possibility that content specificity or familiarity with cholelithiasis and laparoscopic cholecystectomy may have affected resident performance on the SP interview. We categorized items as content specific if the resident would require specific knowledge about cholelithiasis or laparoscopic cholecystectomy to complete the item satisfactorily. Ethics The study protocol was approved by the Human Subjects Review Committee at the University of Toronto. All subjects provided written consent to participate in this study. Results Thirty-one residents began the study, but two residents in the SP group left the seminar because of an emergency. Twenty-nine residents completed the study. Despite randomization, there were differences in gender distribution, prior experience with standardized patients, level of training, and self-assessment of skills (Table 2). The SP seminar group had lower performance scores on the 22-item checklist (57% for the SP seminar group, 66% for traditional seminar group; difference 9%; 95% CI, 17% to 1%; P.03) (Table 3). The intraclass corre- Table 2 Characteristics of the study groups SP-based seminar (%) (N 13) Second-year resident 31 44 Female 0 31 Medical school graduation 38 6 more than 2 years ago Experience with SPs 62 100* Self-rated ability high for obtaining consent for laparoscopic cholecystectomy 62 25 *n 15. One resident did not respond to this question. Traditional seminar (%) (N 16)

456 A. Robb et al. / The American Journal of Surgery 189 (2005) 453 457 Table 3 Results SP group (%) (n 13) Seminar group (%) (n 16) Difference (%) P value 95% CI (%) Performance in SP interview 53 66 9.03 17 to 1 Knowledge: immediately after seminar 60 73 13.003 21 to 4 Knowledge: 3 weeks after seminar 41 59 18.002 29 to 7 lation coefficient for the checklist scores was 0.81 (95% CI, 0.66-0.91), indicating excellent agreement between the multiple observers. The SP seminar group had lower knowledge scores on the knowledge questionnaire immediately after the seminar (60% for SP seminar group, 73% for traditional seminar group; difference 13%; 95% CI 21% to 4%; P.003). The difference in knowledge scores persisted at 1 month (41% for the SP group, 59% for the traditional seminar group; difference 18%; 95% CI 29% to 7%; P.002). The Cronbach s alpha was 0.55 to 0.70, indicating good internal consistency reliability. The intraclass correlation coefficient for the knowledge questionnaire was 0.73 (95% CI, 0.50-0.92), indicating good agreement between the two markers. The exploratory individual item analysis showed that performance was skewed (Table 1). Ten items had an overall score of 75% or greater, whereas 7 items had an overall score of less than 50%. We judged that 12 of the 22 checklist items required specific surgical knowledge about cholelithiasis and laparoscopic cholecystectomy. The overall score was greater than 75% for 4 of 12 of these items. Ten items did not require surgical knowledge but were uniquely related to informed consent, such as the item explores the impact of surgery on the patient s particular situation. The overall score was greater than 75% for 6 of these 10 items. Comments We found that the traditional seminar group did significantly better on both the performance and knowledge measures. Why did the traditional seminar group perform better? The facilitator reported that more active and interactive discussion took place in the traditional seminar group. Residents in the SP group may have felt uncomfortable criticizing the performance of their peers or they may have been less engaged by the process. When one resident was on stage with the SP, others may have felt more passive and unengaged than they might in a traditional seminar, where peer-to-peer interaction and competitive participation contributes to active learning [16]. A major advantage of an SP-based seminar is the opportunity for residents to actually perform and to receive feedback on their performance; however, not all of the residents had an opportunity to perform and receive feedback during the 1-hour session. We found that the overall performance of the surgical residents on the laparoscopic scenario was uneven. We specifically chose laparoscopic cholecystectomy because it is familiar to all surgical residents at a junior level. Our exploratory item analysis found performance problems with both surgery specific and generic consent checklist items. We doubt that the residents performance would have been better if a different surgical procedure had been chosen for the outcome scenario. The performance was very good in many areas, including explaining what will be done and the length of hospital stay, stating who will perform the surgery, and inviting questions. Some generic skills requiring improvement were evaluation of patient understanding of the procedure, discussion of alternatives of delay and refusal, and exploring the potential impact of surgery on the patient s personal life. Our results suggest ways to improve the approach to teaching informed consent to surgical residents. Our seminars and outcome measures focussed on the content of the informed consent interview, such as discussion of risks and benefits. There was less emphasis on teaching the communication and interpersonal aspects. SP-based seminars may be more useful for teaching communication and interpersonal performance, whereas traditional seminars may be better for content. Because effective communication and interpersonal skills are essential in the informed consent process, an observed expert SP interview by a local role model might have more impact on improving resident performance in this domain. In our setting, staff surgeons generally conduct the informed consent discussion of the planned surgical procedure in their office before admission. This discussion often occurs in the absence of the residents because of their responsibilities elsewhere. An idealized recreation of the consent interview using one or more SPs in a class setting could help remedy this problem. To have a lasting and informative impact, postgraduate bioethics teaching should ideally involve respected clinicians [17]. An SP-based seminar with faculty as interviewers and learners might underline the importance attached to bioethics by the faculty and promote better teaching and role modeling of this important skill in the clinical setting. Our study had several strengths, including randomized allocation, blinded outcome assessment, and high reliability of the outcome measures. We also carefully developed our SP checklist based on expert input from multiple disciplines.

A. Robb et al. / The American Journal of Surgery 189 (2005) 453 457 457 Our study has several limitations. First, our control and intervention groups had baseline differences, despite randomization, and our sample size was too small to perform adjusted analyses to account for these differences. The traditional seminar group had more recent graduates who were more experienced with SPs, so they may have been more comfortable having their performance evaluated using an SP. We did not assess the extent of prior ethics education among the residents; ethics lectures had been added to the curriculum in most Canadian medical schools during the period of the residents undergraduate education. Second, our generalizability is limited because of the small sample size, an inherent limitation of most surgical training programs, and the narrow focus of the seminars on consent for surgery. In summary, a traditional seminar was superior to a SP-based seminar for teaching informed consent to surgical residents. Acknowledgments We are grateful to the surgeons, ethicists, and colleagues who contributed to the development of this research study and to several anonymous reviewers for helpful suggestions that improved the clarity of this article. References [1] Caelleigh AS, Lane LW, Miles SH (eds): Special issue: Teaching medical ethics. Acad Med 1989;64:699 764. [2] Scott CS, Barrows HS, Brock DM, et al. Clinical behaviours and skills that faculty from 12 institutions judged were essential for medical students to acquire. Acad Med 1991;66:106 111. [3] Siegler M, Rezler A, Connell K. Using simulated case studies to evaluate a clinical ethics course for junior students. J Med Educ 1982;57:380 385. [4] Self DJ, Wolinsky F, Baldwin D. The effect of teaching medical ethics on medical students moral reasoning. Acad Med 1989;64:755 759. [5] Self DJ, Baldwin D, Wolinsky F. Evaluation of teaching medical ethics by assessment of moral reasoning. Med Educ 1992;26:178 184. [6] Shorr A, Hayes R, Finnerty J. The effect of a class in medical ethics on first-year medical students. Acad Med 1994;69:998 1000. [7] Sulmasy D, Geller G, Faden R, et al. The quality of mercy: Caring for patients with do not resuscitate orders. JAMA 1992;267:682 686. [8] Sulmasy D, Terry P, Faden R, et al. Long-term effects of ethics education on the quality of care for patients who have do-not-resuscitate orders. J Gen Intern Med 1994;9:622 626. [9] Sulmasy D, Geller G, Levine K, et al. Medical house officers knowledge, attitudes, and confidence regarding medical ethics. Arch Intern Med 1990;150:2509 2513. [10] Emerging Trends in the Use of Standardized Patients. Contemporary Issues in Medical Education Data Brief. AAMC, May 1998, Vol 1, Number 7. [11] Barrows HS. Simulated (Standardized) Patients and Other Human Simulations. Chapel Hill: Health Sciences Consortium, 1987. [12] Singer PA, Cohen R, Robb A, et al. The ethics objective structured clinical examination. J Gen Intern Med 1993;8:23 27. [13] Singer PA, Robb A, Cohen R, et al. Evaluation of a multicenter ethics objective structured clinical examination. J Gen Int Med 1994;9:690 692. [14] Available at: http://rcpsc.medical.org/english/ethics/surgery/index. php3. Accessed September 19, 2003. [15] Barrows HS, Tamblyn R. Problem-Based Learning: An Approach to Medical Education. New York: Springer, 1980. [16] McKeachie WJ. Teaching Tips. New York: Houghton Mifflin Co.; 2002:188. [17] McKneally MF, Singer PA. Teaching bioethics in the clinical setting. CMAJ 2001;164:1163 1167.