Tailored Design Method, Response Rates and Systematic Limitation: A Comparative Study

Transcription

1 Tailored Design Method, Response Rates and Systematic Limitation: A Comparative Study Kevin P. Mulvey, PhD Center for Substance Abuse Treatment Substance Abuse and Mental Health Services Administration Judy Huang, PhD Johnson, Bassin and Shaw, Inc Susan Hubbard, PhD Johnson, Bassin and Shaw, Inc Susan Hayashi, PhD Johnson, Bassin and Shaw, Inc Corresponding Author: Kevin P. Mulvey, PhD Social Science Analyst, Center for Substance abuse Treatment, Substance Abuse and Mental Health Services Administration 5600 Fishers Lane, Rockwall II Suite 840 Rockville Md

2 I. Introduction This paper provides a comparison of two studies sponsored by the Substance Abuse and Mental Health Services focusing on the dissemination of best practices and the need for adequate response rates. Along with providing the context for understanding this research, this paper will examine the following questions: (1) What is the Substance Abuse and Mental Health Services Administration and The Center for Substance Abuse Treatment?; (2) What are Treatment Improvement Protocols?; (3) What is the Treatment Improvement Protocols Evaluation Project?; (4) What is diffusion theory?; (5) How is diffusion theory applied to the TIPs Evaluation Project?; (6) What is the Tailored Design Method?; (7) What is an appropriate Response rate?; (8) How do the Retrospective and TIP #24 studies compare with regards to response rates?; and (9) What factors affect response rates even using the Tailored Design Method? II. Understanding this Study in its Broader Context? A. What is the Substance Abuse and Mental Health Services Administration and The Center for Substance Abuse Treatment? To emphasize the need for separate organizations to provide services and conduct services research, the U. S. Congress reorganized the substance abuse treatment research and services administration in 1992, dissolving Alcohol, Drug Abuse and Mental Health Administration (ADAMHA) and forming the Substance Abuse and Mental Health Services Administration (SAMHSA) to focus on services. The research components of ADAMHA moved to the National Institutes of Health (NIH) and consisted of the National Institute on Alcohol Abuse and Alcoholism (NIAAA), the National Institute of Drug Addiction (NIDA), and the National Institute of Mental Health (NIMH). In its current structure, SAMHSA consists of three service centers (i.e., the Center for Substance Abuse Treatment (CSAT), the Center for 2

3 Substance Abuse Prevention (CSAP), and the Center for Mental Health Services (CMHS)), and three offices (i.e., the Office of the Administrator (OA), the Office of Applied Studies (OAS), and the Office of Program Services (OPS)). The three service centers work to carry out SAMHSA s mission of improving the quality and availability of prevention, treatment, and rehabilitation services in order to reduce illness, death, disability, and cost to society resulting from substance abuse and mental illnesses. As one of the services centers of SAMHSA, CSAT s mission is to improve the lives of individuals and families affected by alcohol and drug abuse by ensuring access to clinically sound, cost-effective addiction treatment that reduces the health and social costs to our communities and the nation. As the sponsoring agency of the TIPs Evaluation Project, CSAT is working towards fulfilling one of its primary charges, which is to support and improve substance related treatment throughout the United States. In addition, CSAT collaborates with other Federal agencies, such as the Departments of Justice and Veteran Affairs, to achieve the goals set forth by the Administration s Office of National Drug Control Policy. CSAT also works with multiple stakeholders to develop The National Treatment Plan, which represents an effort to design and implement a coordinated approach to deal with our nation s substance abuse problems. As part of the National Treatment plan, CSAT currently focuses on providing services through two types of programs (i.e., the Substance Abuse Prevention and Treatment Block Grant and the Targeted Capacity Expansion), to ensure that thousands of Americans with substance abuse problems have access to the best treatment services possible, when and where they need them. CSAT also ensures the effectiveness of substance abuse treatment services through its Knowledge and Application Program, which exists to bridge the gap between research and service providers in local communities. 3

4 B. What are Treatment Improvement Protocols? CSAT s Treatment Improvement Protocols (TIPs), developed since 1992, are best practice guidelines for the treatment of substance abuse. CSAT's Office of Evaluation, Scientific Analysis, and Synthesis draws on the experience and knowledge of clinical, research, and administrative experts to produce the TIPs, which are distributed to a growing number of facilities and individuals across the country. The audience for the TIPs is expanding beyond public and private substance abuse treatment facilities as alcohol and other drug disorders are increasingly recognized as a major problem. The TIPs Editorial Advisory Board, a distinguished group of substance abuse experts and professionals in such related fields as primary care, mental health, and social services, and the State Alcohol and Other Drug Abuse Directors, generate topics for the TIPs based on the field's current needs for information and guidance. After selecting a topic, CSAT invites staff from pertinent Federal agencies and national organizations to a Resource Panel that recommends specific areas of focus as well as resources that should be considered in developing the content for the TIP. Soon after that a consensus panel is held: non-federal experts who are familiar with the topic and are nominated by their peers participate in panel discussions over five days. The information and recommendations on which they reach consensus form the foundation of the TIP. The members of each Consensus Panel represent substance abuse treatment programs, hospitals, community health centers, counseling programs, criminal justice and child welfare agencies, and private practitioners. A panel chair ensures that the guidelines mirror the results of the group's collaboration. 4

5 A large and diverse group of experts reviews the draft document. Once the changes recommended by the field reviewers have been incorporated, the TIP is prepared for publication. While each TIP strives to include an evidence base for the practices it recommends, CSAT recognizes that the field of substance abuse treatment is evolving, and research frequently lags behind the innovations pioneered in the field. A major goal of each TIP is to convey "front-line" information quickly but responsibly. For this reason, recommendations proffered in the TIP are attributed to either Panelists' clinical experience or the literature. If there is research to suggest a particular approach, citations are provided. Today's technology permits access to the TIPs documents not only in print but online as well. The TIPs can be accessed via the Internet from the National Library of Medicine. C. What is the Treatment Improvement Protocols Evaluation Project? Development of practice guidelines in the substance abuse treatment field is a component of the Knowledge and Application Program of CSAT that brings to light the Science to the Service paradigm. However, while SAMHSA has been progressive in the dissemination of basic information and in the real life applications, there has been a paucity of evaluation data regarding the process by which guidelines can most effectively be developed, disseminated, and evaluated. Toward this aim, CSAT supported a multi-year evaluation study to examine the effectiveness of the Federal Government s efforts to develop and disseminate best practice guidelines in substance abuse treatment. CSAT was specifically interested in assessing the impact of their Treatment Improvement Protocol series (TIPs) on practices within the substance abuse treatment field. The TIPs Evaluation Project, which began in 1997, was a 4-year initiative consisting of a Retrospective Study, an Addiction Technology Transfer Center (ATTC) Study, a 5

6 Prospective Study, and a study of primary care physicians based on TIP #24: A Guide to Substance Abuse Services for Primary Care Clinicians. The project also conducted a study on the implementation of TIP #35: Enhancing the Motivation for Change in Substance Abuse Treatment. Together, these studies were designed to meet the following objectives: To assess the extent to which members of the TIPs target audience are aware of, read, and implement TIPs. To evaluate the effects of TIPs on the delivery of addiction treatment services To learn ways to strengthen the development, formatting, marketing, dissemination, use, and evaluation of TIPs to ensure effective use of future Federal resources devoted to TIPs The TIPs Evaluation Project represents the effort of the Federal government to assess the effectiveness of practice guidelines on the process and outcomes of addiction treatment. The results of this evaluation project offer valuable information to other Federal agencies involved in the development and dissemination of practice guidelines. D. What is diffusion theory? Diffusion of innovations theory (Rogers, 1995) provided the theoretical framework for the TIPs Evaluation Project. Diffusion is defined by Rogers (1995) as the process by which an innovation is communicated through certain channels over time among members of a social system (p. 5). The theory attempts to explain how new ideas and practices spread throughout a community over time. Three important components characterize diffusion: (1) it occurs over time; (2) people do not adopt innovations instantly, but rather pass through stages in the adoption process; and (3) characteristics of the innovation, the organization, and the individuals affect the rate of adoption. Fundamental to the theory is that the diffusion of an innovations takes place over an extended period of time. In general, a new idea or practice enters a community from an external source; however, the adoption of the new idea or practice then flows through interpersonal 6

7 contact networks. As might be expected, not everyone adopts an innovation immediately. Some people adopt an idea or practice right away, while others wait to see how successful it is before deciding to adopt. These individuals are classified as early adopters and late adopters, respectively. Over time the proportion of individuals who engage in this new practice increases until the potential audience for the new idea or product is saturated. This accumulation of people who accept the new behavior often results in an S-shaped adoption curve. The S-shaped pattern of diffusion has been demonstrated in prior research in the diffusion of medical innovations (Coleman, Katz, and Menzel, 1957), educational innovations (Carlson, 1965), new medical technology (Anderson & Jay, 1985; Greer, 1977), policy innovations among States (Walker, 1966), and many more topics (Griliches, 1957; Ryan & Gross, 1943; Valente and Rogers, 1995; see Hamblin, Jacobson, & Miller, 1973 and Rogers 1995 for reviews). Although some people may adopt an innovation earlier than others, all people pass through five stages on their way to adoption. These stages include knowledge, persuasion, decision, implementation, and, finally, adoption. Before individuals adopt a new practice, they must be aware that the innovation exists. Awareness of innovations thus precedes adoption, and many dissemination programs are created to raise awareness of some issue or behavior. After people are exposed to the new idea or practice, they intentionally or unintentionally learn more about it. Then, they make a conscious decision to either try or not try the innovation. If a decision is made to try the innovation, the person must obtain it. Implementing, or actually trying, the innovation is the next step. Finally, if implementation leads to positive outcomes, the individual adopts the behavior into his or her normal way of doing things. In many diffusion studies (Rogers, 1995) these stages have been collapsed into the three most important ones: 7

8 knowledge, attitudes and practices. Studies using this framework are often referred to as KAP studies (i.e., knowledge, attitudes, and practices studies). The third major element of diffusion theory asserts that the rate of diffusion is influenced by the perceived characteristics of the innovation, as well as by the structural characteristics of the organization and the personal characteristics of the target audience. Characteristics of the innovation include the innovation s relative advantage compared to current practices and behaviors, its compatibility with current practices, the complexity of the innovation (which relates to the ease with which individuals can try the innovation), how radical the innovation is compared to existing practices, the ability to observe others using it effectively, and the cost of implementing the innovation into practice. Structural characteristics of the organization that affect the rate of adoption include the organization s centralization (the degree to which power and control in an organization are concentrated in the hands of a few individuals), its complexity (the degree to which members of an organization possess a high degree of knowledge and expertise), its formalization (the degree to which an organization stresses following rules and procedures), its interconnectedness (the degree to which the units in a social system are linked by interpersonal networks), and its organizational slack (the degree to which uncommitted resources are available to an organization). Generally, the diffusion process is facilitated in organizations with low centralization, high complexity, low formalization, high interconnectedness, and a lot of organizational slack (Rogers 1995, p. 379). Finally, personal characteristics such as age and socioeconomic status, in addition to characteristics of the targeted community, have been found to affect the rate of adoption (Rogers 1995). 8

9 Diffusion theory was used as the theoretical framework for the TIPs Evaluation Project to understand the extent to which TIPs, and the guidance they offer, have spread throughout the substance abuse treatment system. The TIPs Diffusion Model presents a model which applied the key elements of diffusion theory to the dissemination and adoption of TIPs. This model was used to structure the TIPs Evaluation Project by providing a conceptual framework which was used to: (1) formulate the research questions for the studies, and (2) identify key dependent measures used to design the measurement instruments for the project s three major studies. E. How is diffusion theory is applied to the TIPs Evaluation Project? TIPs represent best practices in substance abuse treatment. The ultimate goal for TIPs is adoption of the practice guidelines by the substance abuse treatment community. With this goal in mind, an evaluation of TIPs would need to examine both the extent to which and how TIPs have been adopted. By using the process of diffusion documented in the literature to create the TIPs Diffusion Model, CSAT was able to identify and refine three research questions which guided the entire TIPs Evaluation Project. These questions were: (1) Are substance abuse treatment professionals aware of TIPs and knowledgeable about them? (2) What are substance abuse treatment professionals attitudes towards TIPs? (3) How do substance abuse treatment professionals use TIPs in practice, and how has using TIPs impacted the substance abuse treatment field? From these questions, three key dependent measures were identified and used in each of the project studies. These dependent measures were substance abuse treatment professionals (1) knowledge, (2) attitudes, and (3) practices regarding the TIP series. With this background information it is now important to focus on the purpose of this paper. 9

10 III. Focus of the Study: Tailored Design Method, Response Rates & Systemic Limitations: A Comparative Study A. What is the Tailored Design Method? Response rates, particularly in survey research, are important in the generalizability and overall validity of the study s findings. To achieve high response rates, researchers may employ established procedures such as TDM (Dillman, 2002). The Tailored Design Method provides guidelines for instrument development, as well as specifics on the type and timing for initial contact, follow-up, and incentives. The TDM approach makes it possible to achieve a response rate of percent or higher. However, TDM does not guarantee a high response rate. In some cases, despite the best efforts of researchers to follow the TDM procedures, the response rate may turn out to be less than adequate. B. What is an appropriate response rate? According to Baker (1994), there is a considerable debate regarding what is an appropriate response rate. It is generally suggested by the research community that with a well drawn sample a researcher should strive for a 70% response rate. The response rate would enable the researcher to have confidence that the sample would be largely representative of the population from which the sample was drawn and thus the results would be generalizable to the population. C. How do the Retrospective and TIP #24 studies compare with regards to response rates? The two studies are a part of a 4-year evaluation of consensus-based best practice guidelines (i.e., Treatment Improvement Protocols [TIPs]) intended to provide salient and effective approaches to substance abuse (SA) treatment. Using Rogers (1995) Diffusion of 10

11 innovations theory as a framework, the evaluation was designed to determine the impact of TIPs on the SA treatment field. The Retrospective Study was the first major study under the TIPs Evaluation Project. Data for the Retrospective Study were collected in 2 waves. The Wave 1 survey focused on the TIP series in general and the 28 specific TIPs published at the time of the study. The Wave 2 survey collected data on the most useful and the least useful TIP as identified by the respondents. The goal of the Retrospective Study was to determine: (1) if TIPs are reaching their intended audiences; (2) if TIPs contain appropriate information for use by the target audiences; (3) how readers use information in TIPs, and (4) the impact of TIPs on changing substance abuse treatment practices. Using TDM data collection procedures, Wave 1 of the Retrospective Study collected data from 3,267 individuals affiliated with State recognized SA-treatment programs (i.e., Single State Agency [SSA] directors, treatment facility program directors, clinical supervisors, and program counselors) and yielded an 80.1% response rate. The Wave 2 survey focused on objectives 2 and 3 and also employed TDM data collection procedures. A total of 1,028 SA treatment professionals participated and a 74.1% response rate was obtained. The TIP #24 Study was a special study under the TIPs Evaluation Project. The purpose of this study was to examine (1) primary healthcare professionals awareness of TIPs, including TIP #24, A Guide to Substance Abuse Services for Primary Care Physicians; (2) primary healthcare professionals attitudes toward the alternative versions of the TIP #24 materials; (3) if the TIP #24 materials contain appropriate information for use by members of primary healthcare organizations; and (4) how primary healthcare professionals use the TIP #24 materials and the impact of the TIP #24 materials on clinical practices. The TIP #24 Study surveyed 600 primary care professionals selected from a population of 2508 leaders from 24 primary healthcare 11

12 organizations but obtained responses from only 137 individuals. Like the Retrospective Study, the data collection procedure used for the TIP #24 Study was structured around the TDM. Furthermore, experts were consulted in the study design and guidance was obtained regarding the use of primary healthcare organizations. Nevertheless, only a 22.8% response rate was obtained. The large difference in response rates between these two studies, despite the use of the same TDM procedures, called for an examination of the factors that may have contributed to this discrepancy. One obvious difference is that these two studies sampled from different populations. The target population in the Retrospective Study was SA treatment professionals while the TIP #24 Study targeted primary care physicians and related healthcare professionals who do not work in SA treatment programs. Although the use of similar procedures to design the studies and collect the data was expected to yield similar response rates, there may have been more subtle difference between these populations that may have influenced the response rates. This paper explores what other, possibly more subtle factors, may have influenced the varied response rates and could have an impact on future research efforts. D. What factors affect response rates? There are several factors that can affect the response rates. These factors are: (1)the extent of participants interest in the study; (2) the importance of obtaining a more comprehensive understanding of participants; (3) considerations for using mailing lists provided through professional organizations; (4) factors impacting the delivery of mailed survey to individuals in leadership positions; (5) determining appropriate incentives; and (6) name recognition of the study sponsor. 12

13 References Anderson J.G., & Jay S.J. (1985). Computers and clinical judgment: The role of physician networks. Social Science Medicine, 20 (10), Baker, T. L. (1994). Doing Social Research (2 nd edition). New York: McGraw-Hill. Coleman, J.S., Katz, E., Menzel, H. (1957). The diffusion of an innovations among physicians. Sociometry, 20, Dillman, D.A. (2000). Mail and Internet surveys: The tailored design method (2nd edition). New York: John Wiley & Sons, Inc. Greer A.L. (1977). Advances in the study of diffusion of innovations in health care organizations. Health and Society, 55 (4), Griliches, Z. (1957). Hybrid corn: An exploration in the economics of technological change. Econometrica, 25, Hamblin, R.L., Jacobson, R.B., & Miller, J.L. (1973). A mathematical theory of social change. New York: Wiley. Rogers, E.M. (1995). Diffusion of innovations (4th ed.). New York: Free Press. Ryan, B., & Gross, N. (1943). The diffusion of hybrid seed corn in two Iowa communities. Rural Sociology, 8, Valente, T.W., Rogers, E.M. (1995). The origins and development of the diffusion of innovations paradigm as an example of scientific growth. Science Communication, 16 (3), Walker, J.L. (1966). The diffusion of innovations among the American States. American Political Science Review, 63,

14 14