A Naturalistic Test of the Predictive Validity of the American Society of Addiction Medicine (ASAM) Patient Placement Criteria

Transcription

1 A Naturalistic Test of the Predictive Validity of the American Society of Addiction Medicine (ASAM) Patient Placement Criteria Debra Gresham 1 PhD, Bruce A. Thyer 2 PhD, LCSW, BCBA-D, & Raymond J. Waller 3 PhD. Abstract Background: The American Society of Addiction Medicine (ASAM) placement criteria are widely used to make treatment recommendations for substance abuse clients. Objective: To examine the treatment completion rate of clients who received ASAM-consistent treatment compared to those who received ASAM-inconsistent treatment. Methods: Data from the Florida Substance Abuse Program Office for the fiscal years were used to obtain a sample of clients with whom the ASAM was used. Results: Out of an initial sample of 458 clients who met this study s inclusionary criteria, treatment completion data were obtainable for only 49 clients. Among these 49 participants, 80% were assigned to outpatient treatment. Clients who received treatment consistent with ASAM recommendations completed treatment at a higher rate (41%) than those who received treatment ASAM-inconsistent treatment (29%). Conclusions: Our exploratory results derived from actual state-level collected data support the use of the ASAM as a method of allocating clients to types of treatment. In acknowledging our study s limitations, we provide some recommendations for future research in this area. 1. Dr. Debra Gresham is Assistant Professor of Social Work at Thomas University, Thomasville, GA. 2. Dr. Bruce A. Thyer is Professor of Social Work at Florida State University, Tallahassee, FL. 3. Dr. Raymond J. Waller is Associate Professor of Social Work at Troy University, Troy, AL.

2 Addiction, Recovery and Aftercare 56 Vol 1(1) Substance abuse disorders approach epidemic proportions in the United States. Almost 9% of adults aged 18 years and older are estimated to meet the DSM-5 criteria for Alcohol Use Disorder in any given 12 month period (American Psychiatric Association, 2013, p. 493). Estimates such as this vary demographically, with ages, race, gender and geographical location all impacting risk. The costs of alcoholism are immense, to the afflicted individuals, their families, and to society at large. In Florida, for example, the State Estimates of Substance Use from the National Surveys on Drug Use and Health (NSDUH) was used to obtain information on substance use (Wright, Sathe & Spagnola, 2005). This report provided estimates for 23 measures of substance abuse or mental health problems based on the NSDUHs from 2004 and During state fiscal year , 47,114 adults and 13,694 children in Florida received substance abuse treatment, with substance abuse expenditures (excluding prevention) of $103.2 million and $48.0 million, respectively. A recent statewide survey found that publicly funded treatment for substance abuse cost the state of Florida over $184 million. This is a serious underestimate, given that only 71% of eligible agencies provided cost information (Alexandre, Beulaygue, French, McCollister, Popovici & Sayed, 2012). The effective diagnosis and treatment of substance abusers is promoted by the use of assessment instruments with strong psychometric properties. One assessment method that is widely used in the determination of level of care for substance abusers is promoted by the American Society for Addiction Medicine, and is known as the ASAM (ASAM, 1996, 2005). The current version of the ASAM PPC-2R used in Florida was revised in 2001 and includes recommendations for five basic levels of care, as follows: Level 0.5: Early Intervention. This level provides professional services but is not to be confused with prevention. A program within this level can include a DUI or DWI with a variable length. Level I: Outpatient Treatment. Level I encompasses organized services that may be delivered in a wide variety of settings. Addiction or mental health treatment personnel provide professionally directed evaluation, treatment and recovery service. Such services are provided in regularly scheduled sessions and follow a defined set of policies and procedures or medical protocols usually fewer than nine contact hours a week. Gresham, Thyer & Waller

3 Addiction, Recovery and Aftercare Vol 1(1) 57 Level II: Intensive Outpatient/Partial Hospitalization Treatment, an organized outpatient service that delivers treatment programming during the day, before or after work or school, in the evening or on weekends. Level III: Residential/Inpatient Treatment. Level III encompasses organized services staffed by designated addiction treatment and mental health personnel who provide a planned regimen of care in a 24-hour live-in setting. They are housed in, or affiliated with, permanent facilities where patients can reside safely and are staffed 24 hours a day. The defining characteristic of all Level III programs is that they serve individuals who need safe and stable living environments in order to develop their recovery skills. Such living environments may be housed in the same facility where treatment services are provided or they may be in a separate facility affiliated with the treatment provider. Level IV: Medically Managed Intensive Inpatient Treatment. Level IV programs provide a planned regimen of 24-hour medically directed evaluation, care and treatment of mental and substance-related disorders in an acute care inpatient setting. They are staffed by designated addiction-credentialed physicians, including psychiatrists, as well as other mental health- and addiction-credentialed clinicians. See Mee-Lee, Shulman, Fishman, Gastfriend & Griffith, 2001, pp. 2-4) for a complete description of the ASAM-designated levels of care. Without adequate and credible evaluation research on the effectiveness or predictive validity of ASAM, it is difficult to determine the clinical utility of ASAM in making treatment decisions, particularly in large systems of care such as the state of Florida. Relatively few studies have examined one crucial element of the ASAM s validity, namely determining if clients who receive ASAM-consistent treatment fare better than those who receive substance abuse treatment that is inconsistent with ASAM-recommendations. It has been specifically claimed for example, that Mismatched placement, according to the American Society of Addiction Medicine s (ASAM) Patient Placement Criteria (PPC), promotes no-shows to treatment. (Angarita, Reif, Pirard, Sharon & Gastfriend, 2007, p. 79). The literature contains limited research on the predictive validity of the ASAM Patient Placement Criteria despite its widespread use (e.g., Deck, Gabrield, Knudsen & Grams, 2003; Heatherton, 2000; Kosanke, Magura, Staines, Foote & jcharltonpublishing.com

4 58 Addiction, Recovery and Aftercare Vol 1(1) DeLuca, 2002; Magura, Staines, Kosanke, Rosenblum, Foote, DeLuca & Bali, 2003; Sharon, Krebs, Turner, Desai, Binus, Penk & Gastfriend, 2003; Sharon, Rubin, Turner & Gastfriend, 2000) since its inception in 1991 (Hoffman, Halikas, Mee-Lee & Weedman, 1991). A good summary of this research up to about 2003 appears in Gastfriend (2004) who concluded that the available evidence generally supports the use of the ASAM placement criteria. One related study conducted since Gastfriend s (2004) book is Angarita, Reif, Pirard, Sharon and Gastfriend (2007) in an analysis of 700 patients who were randomly assigned to treatment matched, or mismatched, with the ASAM treatment recommendations. Patients whose treatment matched those of the ASAM recommendations had a no-show rate of 43.5%, while those who received mismatched treatment had a noshow rate of 53.4%, a statistically significant but modest difference supporting the recommended practice of matching patients to ASAM-recommended treatment. The current researchers had an opportunity to conduct a secondary analysis of substance abuse treatment provided by the state of Florida that further examined the value of placement matching according to ASAM criteria. Methods Following administrative policies of the Florida Department of Children and Families (DCF), we obtained existing clinical data audited by DCF and used to determine the extent to which ASAM PPC forms were being completed correctly. By linking this information, which included an analysis of whether the patient received ASAM--consistent treatment, or not, with the limited available outcome measures in the DCF database, we could examine the predictive validity of the ASAM. Knowing this information has the potential to increase the state s ability to assess its overall success in substance abuse treatment, along with its other performance measures. The protocol for this study was approved by the human subjects institutional review boards of the Florida Department of Health and of Florida State University. The Florida DCF Substance Abuse Program Office (SAPO) provided the first author (at the time a SAPO employee) with a database containing selected information from 458 ASAM forms that had been randomly audited by state DCF staff during It is important to note that the ASAM forms of all clients receiving substance services in Florida (an estimated 60,000 per year) were not examined in this study. Rather, the limited Gresham, Thyer & Waller

5 Vol 1(1) Addiction, Recovery and Aftercare 59 subset of those randomly selected by DCF staff for auditing was used. Prior to being provided to the authors, the database was purged by other SAPO staff of all personally identifying information (e.g., names, social security numbers, gender, ethnicity, age, etc.) of clients, thus the authors could not trace any information in the database back to a given individual. Nor could such demographic factors be subjected to statistical analysis. This limited database contained the following information: 1. Whether or not the client received services consistent with or inconsistent with ASAM recommendations, 2. Whether the client completed treatment, 3. Whether the client was readmitted within two years of the completion of treatment, and 4. The length of time between the initial completion of treatment, and the first readmission (for those readmitted to treatment). The database contained data from 15 DCF districts within the state of Florida. The present study was predicated on being able to match individual client ASAM records obtained pretreatment with those completed after therapy was concluded. With an initial pool of 458 cases available with pretreatment data, it was anticipated that a final sample size would be obtained which would be sufficiently large to permit the inferential analysis of this study s hypotheses. Design The results from the two samples (those placed consistently with ASAM versus those not placed consistently with ASAM) were compared with respect to treatment completion rates. These frequency data were analyzed using a 2 by 2 X 2 test for data obtained from two independent samples. The independent variable (intervention) was construed as whether or not the client received ASAM-consistent treatment, with the dependent variable (outcome measure) being whether or not the client completed treatment. Completion of treatment can be viewed as a dichotomous (categorical) variable (yes, treatment was completed, vs. no, treatment was not completed); as can the independent variable. It is important to note that these determinations were made independently by the state DCF auditors of the ASAM files, not by the authors of the present study. This provided a real-world evaluation to test the hypothesis that treatment provided which jcharltonpublishing.com

6 60 Addiction, Recovery and Aftercare Vol 1(1) is consistent with ASAM recommendations results in a higher rate of completion, relative to clients who received treatment that was inconsistent with the ASAM recommendations. The readmission rates for the clients placed consistently with ASAM versus those not placed consistently with ASAM were also examined. The comparison was limited to two years following the client s readmission into treatment using data through June Readmission was not limited to clients who only completed substance abuse treatment, but included all clients who were admitted to treatment regardless of their completion of treatment. The readmission data were analyzed in a manner similar to that used in answering the first outcome measure, a 2 by 2 contingency table. It was hypothesized that treatment which was consistent with ASAM guidelines would result in lower client readmission, relative to clients who received treatment which was inconsistent with ASAM recommendations. Clients The initial sample totaled 458 audited records of clients who had received substance abuse services in Florida during the period of July 1, 1999 to June 30, The Department of Health Institutional Review Board did not allow the researchers access to demographic data since they considered any demographic information as personally identifying information. Demographic data could have been used with clients signed consent, but since this study used retrospective archival data, past clients could not be contacted to give consent to participate in this study. Results Upon receipt of the data involving 458 former clients from DCF, an attempt was made by the DCF data-analyst (not the authors) to match initial ASAM data with follow-up ASAM data, at the level of individual clients, using a supposedly unique identification number used by DCF with each client. The DCF data system allows providers to input a client s SSN, or a pseudo-ssn (following an algorithm) or an assigned State Integrated Substance Abuse Report number for identification. With the exception of the social security number, the other identification numbers could change each time a client sought treatment from a different provider agency. The DCF data analyst was only able to reliably match clients through the use of social security numbers (SSN), but not all clients had their SSN used as their unique identifier. After reviewing the data for the original dataset Gresham, Thyer & Waller

7 Vol 1(1) Addiction, Recovery and Aftercare 61 of 458 clients and discarding adolescent records, disappointingly, 83 records of adult clients remained. Of the 83 records, 31 were repeats and 3 records were missing data, leaving only 49 unduplicated clients remaining to comprise this study s sample; see Table 1. This unexpected limitation, only emerging after the authors receipt of the database, dramatically reduced the number of clients available for analysis. Table 1. Overview of ASAM Matched vs. Unmatched Treatment ASAM Levels of Care* ASAM Compliant Treatment? Level 0.5 Level I Level II Level III Lelvel IV YES NO Total (%) 22(45) 27(55) *Only those ASAM levels of care actually used are represented. Hypothesis 1: Clients receiving ASAM consistent-treatment will have higher completion rates than clients who received treatment inconsistent with ASAM guidelines. With respect to the first outcome measure, a cross tabulation was completed using an X 2 test. The results depicted in Table 2 corroborated Hypothesis 1. Table 2. Hypothesis 1 Results (N=49) Was treatment consistent with ASAM guidelines? Was Treatment Completed? YES NO YES 20 4 NO Clients who received treatment consistent with ASAM recommendations completed treatment at a higher rate (41%) than those who received jcharltonpublishing.com

8 62 Addiction, Recovery and Aftercare Vol 1(1) treatment inconsistent with the ASAM recommendations (29%). Only 4 clients of 24 who received treatment consistent with ASAM guidelines failed to complete treatment. However, among those clients who received treatment inconsistent with ASAM guidelines, 11 of 25 (44%) failed to complete treatment. In other words, treatment failure occurred in 8% of those clients receiving ASAM-consistent treatment, compared to 22% of those receiving ASAM-inconsistent treatment [X 2 (1) = 4.31; p = 0.037]. Due to the small N in one of the cells, Yate s Correction for Continuity was applied, as recommended by Pett (1997) and obtained X 2 (1) = 3.11; p = 0.038, which remained statistically significant. Hypothesis 2: Clients who received ASAM-consistent treatment will have lower readmission rates than clients who received treatment inconsistent with the ASAM guidelines. Of 49 clients, a total of six (12%) had been readmitted to treatment, and all six had received treatment inconsistent with ASAM guidelines. These individuals had all been recommended for Level III (Residential Treatment) but actually received Level II (Intensive Outpatient Treatment). No patients who received ASAM-consistent treatment were readmitted (see Table 3). A 2 by 2 X 2 test [X 2 (1) = 6.56; p =. 010] corroborated Hypothesis 2. A further X 2 test using Yate s correction [X 2 (1) = 4.52; p = 0.012] continued to support the prediction that matched patients would have lower readmission rates than patients who received mismatched treatment. Table 3. Hypothesis 2 Results (N=49) Was treatment consistent with ASAM guidelines? YES NO Was Client Readmitted? YES NO Hypothesis 3: Clients who received ASAM-consistent treatment will have longer periods of readmission than those clients who received treatment inconsistent with the ASAM guidelines. Because the database contained NO clients readmitted who received treatment consistent with ASAM guidelines, the planned comparison of the length of time prior to readmission for the ASAM-compliant versus non-compliant patients could not be investigated. Gresham, Thyer & Waller

9 Vol 1(1) Addiction, Recovery and Aftercare 63 Limitations Several serious limitations in using archival data were encountered. The sample size initially anticipated was dramatically reduced due to DCF s use of varying approaches to input individual identification numbers, making analysis impossible in over 90% of the cases. The Department of Health (DOH) Institutional Review Board process also contributed to the dramatic reduction of the original sample size by requiring the data analyst to purge the file of demographic information such as ethnicity, drugs used, gender, age and race. The lack of this information, combined with ambiguous client identification numbers, made it much more difficult to determine an individual s follow-up status. Without these additional variables, more detailed analyses could not be performed. By analyzing the existing but deleted demographic data, more could have been learned about the nature of the relationships between ASAM matching and outcomes. We could not test our third hypothesis since no clients who were readmitted had received treatment consistent with ASAM guidelines. The present data were obtained from only one state, Florida, and the generalizability of the limited findings are open to question, and call for replication across other states and regions of the United States. The substantial amount of missing data is a grave concern, since lacking an adequate sample of initial ASAM recommendations and corresponding treatment dispositions it is impossible to legitimately evaluate the usefulness of the ASAM method of assessment and allocation to treatments. Given the costs to Florida to adopt and apply the ASAM statewide, it seems remiss to not have a more effective system in place to assure greater consistency of data-entry. The data were also old, being obtained from clients who received treatment during , thus we have no way of ascertaining whether or not our tentative positive findings are being maintained by more current cohorts of substance abuse treatment clients. In addition, the state collects data on treatment completion, which usually means participating in so many treatment sessions over a given period. However, treatment completion does not necessarily mean treatment success. Obtaining credible follow-up information on employment, earnings, abstinence from illicit substances and alcohol remain major challenges for all drug abuse treatment facilities. jcharltonpublishing.com

10 64 Addiction, Recovery and Aftercare Vol 1(1) Implications for Practice, Research and Policy These data provide provisional support to use the ASAM. Higher treatment completion rates were attained by clients who received ASAM-consistent treatment compared to those who received ASAM-inconsistent treatment. Achieving this is one of the purposes of using the ASAM. The present study should be replicated with a more current cohort of clients served by Florida substance abuse treatment agencies. It is hoped that advances over the past ten years in data entry and retrieval technology would make available larger and more representative samples of clients. Some forms of cost-benefit analysis of using the ASAM would also be informative. Given estimates of the numbers of clients who completed treatment due to use of the ASAM, it should be possible to factor in the costs of using the licensed ASAM assessment method and estimate if this method results in any monetary savings. Conclusions Even with the dramatically attenuated sample of patients (49 out of an original random size of 458) it was found that the larger majority of patients were receiving treatment that was consistent with ASAM recommendations. Moreover, clients who received ASAM- consistent treatment were statistically significantly more likely to actually complete treatment, compared to clients who received treatment inconsistent with ASAM recommendations. Of the six clients who were readmitted to treatment, all six had received ASAM-inconsistent treatment. The data did not permit an examination of whether or not ASAM consistent-treatment had a bearing on length of time until recidivism, as assessed by a readmission to treatment. While small in scope, these data corroborate the predictive validity of complying with ASAM treatment recommendations. The inability of the State to make effective use of its own data, to match clients with treatment outcomes, is a major area of concern and an area of improvement to be addressed by the Florida DCF Substance Abuse Program Office (SAPO). Consistently and validly determining an ASAM compliance rate is a potentially viable outcome measure that can be used to evaluate districts and provider agencies. The data are currently being collected in the data system and their analysis could be included in the SAPO standard district reports produced monthly, quarterly and annually. An analysis of compliance rates could also indicate training and technical assistance needs of districts and/or provider agencies to be provided by the Gresham, Thyer & Waller

11 Vol 1(1) Addiction, Recovery and Aftercare 65 SAPO. Difficult economic conditions place constant pressure on helping organizations to maximize the effectiveness and efficiency of services provided. It is here, perhaps, rather than the modest support this study provides to the psychometric properties of the ASAM, that this work makes a contribution. Despite mandates and incentives to collect and use outcome data, large service areas and organizations can easily amass data, often with cumbersome collection requirements and with great expense, that serve little or no use whatsoever. We would like for this paper to serve as a call to action for service organizations, ensuring that outcome data are collected with integrity, such that the data are sufficient to inform practice in the field at large. Perhaps no single action can ensure the integrity of data collection procedures as effectively as actually using the data. Nothing will identify deficiencies in data collection as quickly as exposing them to the rigors of empirical quantitative analysis. Such was the case in this project, which saw a presumed sample of more than 450 shrink to a fraction of that whole by exposing significant systemic flaws in the collection procedures. Substance abuse treatment facilities, the supervisory state departments, and service organizations should develop collaborative relationships with university resources and personnel in both the development of data collection procedures and in the ethical practice of actually using the data collected for improvement of service and informing the field. Strong collaborative relationships such as described would give new meaning and significance to the concept of research to practice. jcharltonpublishing.com

12 66 Addiction, Recovery and Aftercare Vol 1(1) References Alexandre, P.K., Beulaygue, I.C., French, M.T., McCollister, K.E., Popovici, I., & Sayed, B.A. (2012). The economic cost of substance abuse treatment in Florida. Evaluation Review, 36, American Psychiatric Association (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: American Psychiatric Press. American Society of Addiction Medicine. (1996). Patient placement criteria for the treatment of substance-related disorders (2nd ed.) (ASAM PPC-2). Chevy Chase, MD: American Society of Addiction Medicine, Inc. American Society of Addiction Medicine (ASAM) What we do. (2005). Retrieved December 12, 2005, from Whatwedo.htm Angarita, G.A., Reif, S., Pirard, S., Sharon, E., & Gastfriend, D.R. (2007). No-show for treatment in substance abuse patients with co-morbid symptomatology: Validity results from a controlled trial of the ASAM patient placement criteria. Journal of Addiction Medicine, 1(2), Deck, D., Gabriel, R., Knudsen, J., & Grams, G. (2003). Impact of patient placement criteria on substance abuse treatment under the Oregon Health Plan. Journal of Addictive Diseases, 22, Gastfriend, D.R. (Ed.) (2004). Addiction treatment matching: Research foundations of the American society of Addiction Medicine (ASAM) Criteria. Binghamton, NY: The Haworth Medical Press, Inc. Heatherton, B. (2000). Implementing the ASAM criteria in community treatment centers in Illinois: Opportunities and challenges. Journal of Addictive Diseases, 19, Gresham, Thyer & Waller

13 Vol 1(1) Addiction, Recovery and Aftercare 67 Hoffman, N.G., Halikas, J.A., Mee-Lee, D., & Weedman, R.D. (1991). Patient placement criteria for the treatment of psychoactive substance use disorders. Washington, DC: American Society of Addiction Medicine. Kosanke, N., Magura, S., Staines, G., Foote, J., & DeLuca, A. (2002). Feasibility of matching alcohol patients to ASAM levels of care. American Journal on Addictions, 11, Magura, S., Staines, G., Kosanke, N., Rosenblum, A., Foote, J., DeLuca, A., & Bali, P. (2003). Predictive validity of the ASAM patient placement criteria for naturalistically matched vs. mismatched alcoholism patients. The American Journal on Addictions, 12, Mee-Lee, D. Shulman, G.D., Fishman, M., Gastfriend, D.R., & Griffith, J.H. (Eds.) (2001). ASAM patient placement criteria for the treatment of substance-related disorders, second edition-revised (ASAM PPC-2R). Chevy Chase, MD: American Society of Addiction Medicine, Inc. Pett, M.A. (1997). Nonparametric statistics for health care research: Statistics for small samples and unusual distributions. Thousand Oaks, CA: Sage. Sharon, E., Krebs, C., Turner, W., Desai, N., Binus, G., Penk, W., & Gastfriend, D.R. (2003). Predictive validity of the ASAM patient placement criteria for hospital utilization. Journal of Addictive Diseases, 22, Sharon, E., Rubin, A., Turner, W. & Gastfriend, D.R. (2000). Challenges of engaging patients in ASAM criteria placements. Journal of Addictive Diseases, 19, 150. (abstract). Wright, D., Sathe, N., & Spagnola, K. (2005). State estimates of substance use from the National Surveys on Drug Use and Health. Washington, DC: Office of Applied Studies (SAMHSA). jcharltonpublishing.com