1 Journal of Substance Abuse Treatment 26 (2004) Regular article Beliefs about evidence-based practices in addiction treatment: A survey of Veterans Administration program leaders Mark L. Willenbring, M.D. a, *, Daniel Kivlahan, Ph.D. b, Marie Kenny, B.A. a, Michael Grillo, B.A. a, Hildi Hagedorn, Ph.D. a, Andrea Postier, M.P.H. a a Quality Enhancement Research Initiative-SUD Module, Minneapolis VA Medical Center (116A), One Veterans Drive, Minneapolis, MN 55417, USA b Center of Excellence in Substance Abuse Treatment and Education, Veterans Administration Puget Sound Healthcare System, 1660 South Columbian Way 116 ATC, Seattle, WA , USA Received 18 May 2003; received in revised form 29 September 2003; accepted 1 October 2003 Abstract The purpose of this study was to determine Veterans Administration addiction treatment program leaders familiarity with evidence-based practices, understanding of and attitudes toward existing VA treatment guidelines, the extent to which specific practices should be implemented, and perceived barriers to implementation. Two hundred and twenty surveys were mailed to program leaders between November 1999 and May One hundred and seventy-four (79%) were completed, representing 135 out of 162 (83%) facilities. Program leaders saw guidelines as educational tools that improved quality of care and could be implemented into existing programs. However, they also perceived staff resistance to implementation. The most strongly cited barriers to implementation were lack of administrative support, insufficient staff time, and lack of skills or knowledge. Several treatments were seen as strongly evidentiary, but were not widely implemented, suggesting possible foci for future translation studies. D 2004 Elsevier Inc. All rights reserved. Keywords: Evidence-based practice; Practice guidelines; Guideline barriers; Translation; Substance use disorders 1. Introduction * Corresponding author. Tel.: ; fax: address: (M.L. Willenbring). Practice guidelines have been defined as statements systematically developed from efficacy and effectiveness research and clinical consensus for practitioners and patients to use in making decisions about appropriate care under different clinical circumstances (Lohr, Eleazer, & Mauskopf, 1998). The purposes of practice guidelines are to make more efficient use of resources, reduce inappropriate variation in clinical practice, and act as a means of getting synthesized research evidence to clinicians and patients (Hutchinson, 1998). Unfortunately, despite wide dissemination, practice guidelines have had a limited impact on physician behavior and patients health outcomes (Davis & Taylor-Vaisey, 1997; Kosecoff et al., 1987; Lohr et al., 1998; Lomas et al., 1989; Woolf, Grol, Hutchinson, Eccles, & Grimshaw, 1999). Studies of the impact of guideline implementation efforts in primary and specialty medical care identify the importance of providers knowledge, attitudes and beliefs about practice guidelines (Hayward, Guyatt, Moore, McKibbon, & Carter, 1997; Tunis et al., 1994) and their perceptions of implementation barriers or facilitating strategies (Cabana et al., 1999). However, no prior report has investigated these factors with respect to perception of practice guidelines among addiction treatment providers. The Health Services Research and Development Service of the Veterans Administration (VA) has initiated a large-scale effort to improve the quality and outcomes of treatment for nine prevalent chronic diseases (Demakis, McQueen, Kizer, & Feussner, 2000), including substance use disorders (Finney, Willenbring, & Moos, 2000). The Quality Enhancement Research Initiative (QUERI) seeks to determine whether implementing evidence-based clinical practices and guidelines will result in reduced variation in practices, increased use of evidence-based treatments, and better outcomes and quality of life for patients. The VA and the United States Department of Defense (DoD) have collaborated on a new, comprehensive, evidence-based practice guideline for the management of substance use disorders (SUD; VA/DoD Evidence-Based /04/$ see front matter D 2004 Elsevier Inc. All rights reserved. doi: /s (03)
2 80 M.L. Willenbring et al. / Journal of Substance Abuse Treatment 26 (2004) Table 1 Respondent characteristics N Mean age (SD) 48.9 (7.7) Male gender % Mean years in current position (SD) (6.0) Highest degree earned Doctoral degree % Masters degree % Physician (MD or DO) % All others % Program affiliated with medical school % Research investigator % Principal investigator for VA or NIH grant % Total 174 Clinical Practice Guideline Working Group revised, 2001). This new guideline employs an algorithmic format and addresses the full continuum of care, including identification and treatment of SUDs in primary care settings and specialty SUD treatment programs. In order to guide implementation of the VA/DoD practice guideline, the Substance Use Disorder Module of QUERI conducted a nationwide survey of leaders of VA SUD treatment programs. The purposes of this survey were to assess program leaders knowledge, attitudes, and current behaviors related to evidence-based practices and practice guidelines for SUDs, and to identify perceived barriers to implementation of practice guidelines in SUD treatment programs. The survey also sought to determine beliefs about potential strategies to overcome these perceived barriers to implementation. To permit comparison with practice-guideline implementation efforts in other settings, the survey included some items used in prior studies of health care providers, as well as many new items specific to treatment of SUD. 2. Materials and methods Each VA facility was contacted via telephone and asked to identify SUD treatment programs and their leaders. A total of 220 distinct programs were determined as being providers of autonomous services to a defined group of veterans. A pre-notice letter was mailed during October 1999, and the 195-item surveys were mailed one week later in November Two weeks later a reminder letter was sent, followed by a replacement survey a week after that. In addition, all non-responders were contacted by telephone after the first month and new questionnaires were sent if requested. The overall response rate was 79% (174/220) at the program level and 83% at the medical center level (135/162). This survey was completed at 108 medical centers, while 27 centers had two or more program leaders responding. The survey gathered information on the program leaders background (gender, age, education, tenure, affiliation with a medical school, and research experience), their familiarity with specific SUD treatment recommendations in the guideline, and their attitudes and opinions regarding guidelines and evidence-based practices. Included also were questions about perceived barriers to implementation of the guidelines and perceived usefulness of possible ways to overcome barriers. Respondents were asked to rate the strength of evidence for 13 specific treatment recommendations, whether or not these practices should be routinely recommended, and the current level of implementation in their program. The treatment recommendations were selected by the two senior authors (MW and DK) to represent interventions having different levels of evidence supporting their efficacy. Interventions with substantial evidence from randomized controlled trials supporting efficacy were all included, but others had little or no evidence supporting efficacy, although they may be widely practiced. The Executive Committee of the VA Quality Enhancement Research Initiative-Substance Use Disorder Module (QUERI-SUD), which is composed of senior researchers and clinicians in substance use disorders from across the Veterans Health Administration, reviewed the initial selections. Comments and suggestions by this group were incorporated into the final survey. Respondents were also asked to rate the degree to which seven specific barriers prevented them from implementing these practices Table 2 General beliefs about clinical practice guidelines Strongly disagree Disagree Neutral Agree Strongly agree Improve quality of care Promote cookbook care Control costs Too general to apply Educational tool Reduce autonomy Convenient source of advice Too rigid to apply Can be implemented in existing programs Do not consider a clinician s experience and judgment Will improve outcomes Will be used to discipline staff Are implemented without adequate training
3 M.L. Willenbring et al. / Journal of Substance Abuse Treatment 26 (2004) Table 3 Beliefs about implementing clinical practice guidelines Strongly disagree Disagree Neutral Agree Strongly agree Most staff members understand clinical practice guidelines Most staff members support clinical practice guidelines Clinical experience is more valid than randomized clinical trials Professional responsibility to use modalities proven by randomized clinical trials Program philosophy is more important Randomized clinical trials are not useful in program Randomized clinical trial patients are too different from those in program in their program. Program leaders were asked to estimate the approximate percentage of unique patients who received specific treatment services, and to describe the number of full-time equivalent employees (FTEE) in their programs. 3. Results Respondent characteristics are shown in Table 1. Respondents to this survey were typically mid-career men (70.5%); more than half had earned a degree at the M.D. or Ph.D. level, a majority had participated in research, and one fifth had been a principal investigator on a competitive research grant. Similarly, most programs were affiliated with medical schools. Mean total-program staffing was 13.4 (SD = 10.6) FTEE, the median was 10, and 50% of programs had staff levels between 5.5 and 18. Over 90% of programs had addiction counselors. Although most programs had dedicated psychiatric, psychological, social work, and nursing staff, 17% had no dedicated access to a psychiatrist, 20% had no social workers, and 26% did not have any psychology services. A minority of programs reported having other professional staff, such as recreational therapists and chaplains. Based on program leader estimates, the approximate percentages of patients estimated as receiving various treatment services demonstrated considerable variability in the range of services available across programs. Services available at more than 85% of programs were outpatient and continuing care, pharmacotherapy for co-occurring psychiatric disorders, psychotherapy for co-occurring psychiatric disorders, addiction-related self-help groups, and routine urine toxicology screening. However, the percentage of programs in which more than half the patients received these services ranged from 72.9% for routine urine toxicology and 64.1% for addiction-related self-help groups to less than 30% for psychotherapy for co-occurring psychiatric disorders. Other treatment modalities were used much less frequently. Nearly 75% of programs did not offer opioid agonist therapy. Other services unavailable at over half the programs were inpatient detoxification in a specialty SUD bed section, inpatient rehabilitation other than detoxification, and residential rehabilitation treatment. A total of 32% of programs did not offer naltrexone for alcohol dependence, and 24% did not offer smoking cessation services. At facilities where more than one program existed, however, other programs in the facility might offer those services. Program leaders were at least somewhat familiar with most existing practice guidelines, such as those published by the American Psychiatric Association (1995) or the Veterans Administration and Department of Defense (VA/ DoD Evidence-Based Clinical Practice Guideline Working Group revised, 2001). However, 48% were very familiar with the Patient Placement Criteria of the American Society of Addiction Medicine revised (Mee-Lee, Shulman, Fishman, Gastfriend, & Griffith, 2001). As shown in Table 2, a large majority of program leaders agreed or strongly agreed that practice guidelines are useful in improving quality of care, are a helpful educational tool and a convenient source of advice, and can be implemented into existing programs. A majority of respondents also agreed that practice guidelines: (1) improve outcomes if properly implemented, (2) consider a clinician s experience and judgment, (3) are not too general or too rigid to apply to individual patients, and (4) do not promote oversimplified cookbook care. However, approximately 46% of program leaders felt practice guidelines were implemented without adequate training. Although there was considerable support overall for practice guidelines, opinion was divided in many Table 4 Perceived barriers to practice guideline implementation Barriers Not at all important Somewhat important Insufficient staff time Lack of administrative support Pharmacy or formulary restrictions Lack of staff skills or knowledge Would require expert consultation to implement Guidelines are too complicated or confusing Lack of belief in usefulness Inadequate information management systems Very important
4 82 M.L. Willenbring et al. / Journal of Substance Abuse Treatment 26 (2004) Table 5 Perceived usefulness of implementation strategies Strategy Not at all useful Somewhat useful Very useful Staff training Easy access in clinical setting Pocket-sized cards summarizing guidelines Short pamphlet summaries Computerized reminders of practices Official manual containing guidelines Support of opinion leaders Flow chart or clinical algorithm Expert consultation Review in peer review journal Note: Values in percentages, listed in order of usefulness. instances, with other respondents endorsing a wait and see neutrality. On many items, 20 45% rated themselves as neutral, and on some items, a significant proportion of respondents agreed with a position critical of practice guidelines. For example, 29% agreed that practice guidelines will reduce autonomy and another 25% were neutral. Similarly, 24% agreed that practice guidelines do not consider a clinician s experience and judgment and 26% were neutral. Although only 16% agreed that practice guidelines would be used to discipline staff, 45% were neutral. When asked about implementing guidelines into their own program (Table 3), respondents were also positive in their endorsements. Most providers held positive views about randomized controlled trials (RCT), although almost half (45%) felt that their clinical experience is more valid than RCTs. Nearly 65% disagreed that program philosophy was more important than results from RCTs when determining treatment techniques. Staff training was raised again as a concern, with almost half of respondents disagreeing that staff members understood practice guidelines. Leaders were predominately neutral when reporting on their staff members support of guidelines, and nearly one third disagreed that staff supported them. Lack of administrative support (50%) and insufficient staff time (49.4%) were the most frequently perceived barriers to implementation (Table 4). Lack of staff skills or knowledge were also important barriers. Many implementation strategies were perceived as potentially valuable (Table 5). Program leaders most strongly endorsed staff training (84.5% = Very Useful), followed by providing easy access to the guidelines in the clinical setting (79.9%) and pocket cards (64.4%). Rated least useful were a systematic review published in a peer-reviewed journal (17.8% = Not at all useful) and the support of locally respected clinicians (13.2%). Participants were given a list of 13 different treatment recommendations, including a brief description of each (Table 6). They were asked to indicate the strength of scientific evidence, whether or not they felt the practice should be routinely recommended, and how often the recommendation was currently implemented in their program. Of the 13 recommendations, extended continuing care (maintaining contact with outpatients for 6 months after a shorter period of more intensive treatment), and integrated treatment of psychiatric disorders were rated as having the strongest evidence. They were followed closely by cognitive behavioral relapse prevention and smoking-cessation treatment. The lowest rated practice was verbal confrontation, with more than half rating the strength of evidence as low or none. The recommendation that respondents seemed least familiar with was methadone dosing level (33.5% Don t Know), with contingency management and manualized addiction therapy also not well known (slightly over 23% = Don t Know). When asked which of these treatments should be routinely recommended, 93% agreed that both integrated treatment of psychiatric disorders and cognitive behavioral Table 6 Awareness, endorsement, and implementation of treatment methods Strength of evidence Should be routinely recommended Level of current implementation Low Medium High DK Disagree Neutral Agree DK Low Medium High DK Residential treatment Patient education Cognitive behavioral relapse prevention Extended continuing care Integrated treatment of psychiatric disorders Verbal confrontation Behavioral marital therapy Naltrexone for alcohol dependence Disulfiram for alcohol dependence Contingency management Manualized addiction therapy Smoking cessation treatment Methadone dosing requirements ( > 60 mg / day) D.K. = Don t Know.
5 M.L. Willenbring et al. / Journal of Substance Abuse Treatment 26 (2004) Table 7 Barriers to implementation of specific treatment modalities Lack of administrative support Pharmacy or formulary restrictions Lack of skills or knowledge Lack of staff time Lack of confidence in effectiveness Low demand or low priority Naltrexone for alcohol dependence Smoking cessation treatment Methadone dosing requirements Disulfiram for alcohol dependence Contingency management 39.9 N/A Verbal confrontation 17.5 N/A Behavioral marital therapy 15.8 N/A Conflict with program philosophy relapse prevention should be routinely recommended. Extended continuing care and smoking cessation treatment were also highly endorsed. Over half (55%) disagreed that verbal confrontation should be recommended, and slightly over a third disagreed that disulfiram treatment for alcohol dependence should be recommended. Nearly a third (27.6%) did not know whether or not a methadone dose of z 60 mg/day should be recommended. Program leaders were asked next about barriers to the implementation of each of these practices (Table 7). The practices with the highest number of perceived barriers were contingency management, behavioral marital therapy, and methadone dosing level, with half or more of the respondents endorsing three barriers. Barriers to naltrexone treatment for alcohol dependence were rated as relatively rare, with the exception of the barrier low demand/low priority (58.5% responding Yes ). 4. Discussion This report describes results of a national survey of attitudes, beliefs, and practices related to evidence-based clinical practice guidelines among substance use disorder treatment program leaders in U.S. Department of Veterans Affairs medical centers. This survey is the first to extensively examine these issues in the field of SUD treatment. Program leaders in VA SUD endorsed general agreement with stated guideline benefits and disagreement with guideline criticisms. This finding is not surprising, given the advanced education of program leaders in the VA, the medical school affiliation of many programs, and the relatively large proportion of respondents who had participated in or led research projects. Compared to surveys of physicians in general medical practice (Hayward et al., 1997; Tunis et al., 1994), this sample of program leaders indicated stronger support for benefits of practice guidelines. Respondents in the current study were also more likely than Canadian physicians (Hayward et al., 1997) to disagree that practice guidelines are too rigid and restrictive, promote cookbook care, or would be used in staff disciplinary actions. On the other hand, sizable minorities agreed that practice guidelines might reduce clinician autonomy, fail to consider clinical experience, or would be used for disciplinary purposes. Furthermore, a significant proportion of respondents indicated neutrality towards many items, suggesting they are withholding judgment until presented with concrete implementation. Respondents were not very familiar with existing SUD practice guidelines, suggesting that existing guidelines may not have been disseminated effectively, or are not perceived as useful enough to justify the investment of time required to learn them more completely. This knowledge gap also demonstrates a critical challenge for guideline implementation: clinicians favorably disposed toward guidelines often fail to implement them, or even to closely examine them. The Patient Placement Criteria of the American Society of Addiction Medicine (Mee-Lee et al., 2001) were the only guidelines with which a substantial proportion of leaders felt very familiar. These utilization criteria for determining level of care do not constitute a full practice guideline, but they have pragmatic utility in guiding important clinical decisions, and are also useful for defending the need for services in the face of budgetary pressures. That these criteria were very familiar to nearly one half of respondents suggests the importance of such pragmatic factors to program leaders interest. Lack of time, knowledge and skills were perceived as major barriers to implementation of practice guidelines, and overall, non-supervisory staff members were perceived as neutral or opposed to guidelines (Box 1). Staff reductions in VA SUD treatment programs in recent years and a significant difference in education level between program leaders and addiction therapists possibly account for the perceived gap between program leaders and line staff. Contrary to expectations, conflict with program philosophy was not Box 1. Most Commonly Perceived Implemention Barriers Lack of administrative support Insufficient staff time Lack of staff skills or knowledge Inadequate information management systems Lack of belief in usefulness
6 84 M.L. Willenbring et al. / Journal of Substance Abuse Treatment 26 (2004) rated as a significant barrier to implementation of evidencebased practices, although program clinicians may see this differently. Further study is needed to directly assess the beliefs and attitudes of SUD treatment staff members in addition to program leaders. Opinions regarding specific treatment modalities conformed only partially to the strength of the available evidence supporting them. For example, integrated treatment of psychiatric and substance use disorders has a modest evidence base at best, although it has considerable intuitive and practical appeal (Booth, Cook, Blow, & Bunn, 1992; RachBeisel, Scott, & Dixon, 1999). The strength of evidence for extended continuing care was rated as high, and although there is considerable descriptive evidence, there are few controlled trials (Booth et al., 1992). Although evidence for efficacy of smoking cessation programs in general medical settings is high, there is much less evidence for the efficacy of smoking cessation in SUD treatment programs (Hays et al., 1999). Patient education was rated as having relatively strong support, although studies on its efficacy are lacking. On the other hand, although the evidence for efficacy of naltrexone during the first 12 twelve weeks of treatment is stronger than that for disulfiram (Garbutt, West, Carey, Lohr, & Crews, 1999), they were rated similarly. This survey was carried out before the results of a large VA cooperative study showed lack of efficacy for naltrexone in this veteran population (Krystal, Cramer, Krol, Kirk, & Rosenheck, 2001). That naltrexone was not perceived as efficacious by VA program leaders suggests that their clinical experience with VA patients had not validated results from earlier efficacy trials in non-va populations. Other clearly efficacious modalities such as methadone maintenance (Farre, Mas, Torrens, Moreno, & Cami, 2002), contingency management (Griffith, Rowan- Szal, Roark, & Simpson, 2000), manualized counseling (Carroll, Nich, Ball, McCance, & Rounsaville, 1998; Woody et al., 1983), and behavioral marital therapy (O Farrell & Fals-Stewart, 2000) were not rated as strongly supported. Many program leaders expressed lack of knowledge regarding their efficacy, suggesting that education would be necessary. As for behavioral marital therapy, it probably has less relevance to many VA patients because so many are unattached when they present for treatment (Moos, Finney, Federman, & Suchinsky, 2000). This study has several limitations. Results from this study may not generalize well to other settings outside of the VA. It is likely that community-based programs have fewer program leaders with advanced degrees and research experience, and fewer non-va programs are likely to have an academic affiliation and teaching mission. It is difficult to know how these differences might affect their perceptions. In this study, only program leaders were surveyed, and only regarding their perceptions and opinions. Perceptions and opinions of line staff may be different from how program leaders see them. Directly surveying line staff would be a necessary and important step to help guide implementation of guidelines. Finally, program characteristics, such as the type of services offered, and the numbers of patients receiving them, are estimates made by program leaders and were not directly measured. These limitations notwithstanding, the results reported here provide some guidance about possible interventions that could take place to implement evidence-based treatments. Treatment modalities perceived by program leaders as efficacious and appropriately recommended, but which are less widely implemented, are fertile areas for quality improvement activities and implementation trials. Particular examples identified in this study are integrated psychiatric and SUD treatment, smoking cessation treatment, and contingency management. Lack of skills and knowledge was perceived as a key barrier to implementation. However, education, while a necessary component of implementation, is unlikely to be sufficient in itself. Facilitation of implementation with computer reminders, clinical pathways, and pocket cards, are likely to be necessary as well. Another important barrier is the perception of low priority or demand for a specific treatment modality. This finding suggests that programs may be comfortable with current routines. Lack of familiarity with a new modality, and a belief that its implementation will not change outcomes, are factors that will need to be addressed. Program staff may also be very busy with current activities, so that learning about and implementing a new modality may seem overwhelming. Changing this situation will require informing clinicians about the evidence base, and providing information and support in a format that is flexible and can be integrated into staff schedules. Direct study of SUD program staff members would complement and extend information derived from this survey of program leaders. Acknowledgments This work was supported by U.S. Department of Veterans Affairs Health Services and Development QUERI awards #SUBQ and SUT Heather Ormont contributed to the writing of the report. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs. References American Psychiatric Association. (1995). Practice guidelines for the treatment of patients with substance use disorders: Alcohol, cocaine, opioids. The American Journal of Psychiatry, 152 (Suppl 11), Booth, B. M., Cook, C. A., Blow, F. C., & Bunn, J. Y. (1992). Utilization of outpatient mental health services after inpatient alcohol treatment. Journal of Mental Health Administration, 19 (1), Cabana, M. D., Rand, C. S., Powe, N. R., Wu, A. W., Wilson, M. H.,