Cost-effectiveness analysis of addiction treatment: paradoxes of multiple outcomes

Transcription

1 Drug and Alcohol Dependence 73 (2004) Cost-effectiveness analysis of addiction treatment: paradoxes of multiple outcomes Jody L. Sindelar a,b,, Mireia Jofre-Bonet c, Michael T. French d, A. Thomas McLellan e a School of Public Health and Medical School, Yale University, LEPH 304, P.O. Box , 60 College Street, New Haven, CT , USA b National Bureau of Economic Research, Cambridge, MA 02138, USA c London School of Hygiene and Tropical Medicine and Department of Economics, London School of Economics, Houghton Street S682, London WC2A 2AE, UK d Department of Sociology, University of Miami, Coral Gables, FL 33124, USA e Department of Psychiatry, Treatment Research Institute and the Department of Psychiatry, University of Pennsylvania, Philadelphia, PA 19106, USA Received 15 July 2002; received in revised form 2 September 2003; accepted 2 September 2003 Abstract This paper identifies and illustrates the challenges of conducting cost-effectiveness analysis (CEA) of addiction treatments given the multiple important outcomes of substance abuse treatment (SAT). Potential problems arise because CEA is intended primarily for single outcome programs, yet addiction treatment results in a variety of outcomes such as reduced drug use and crime and increased employment. Methodological principles, empirical examples, and practical advice are offered on how to conduct an economic evaluation given multiple outcomes. An empirical example is provided to illustrate some of the conflicts in cost-effectiveness (CE) ratios that may arise across the range of outcomes. The data are from the Philadelphia Target Cities quasi-experimental field study of standard versus enhanced (e.g. case management and added social services) drug treatment. Outcomes are derived from of the Addiction Severity Index (ASI), while cost data were collected and analyzed using the Drug Abuse Treatment Cost Analysis Program (DATCAP). While the results are illustrative only, they indicate that cost-effectiveness ratios for each of several different outcomes can produce conflicting implications. These findings suggest that multiple outcomes should be considered in any economic analysis of addiction treatments because focusing on a single outcome may lead to inadequate and possibly incorrect policy inferences. However, incorporating multiple outcomes into a CEA of addiction treatment is difficult. Cost-benefit analysis (CBA) may be a preferable and more appropriate approach in some cases Elsevier Ireland Ltd. All rights reserved. Keywords: Cost-effectiveness analysis; Treatment; Services; Economics 1. Introduction There is increasing interest in the cost-effectiveness (CE) of treatments for dependence on illicit drugs. Policy makers and payers are asking not only whether treatments are effective, but also whether they are cost-effective. We focus on a salient, but often ignored, issue in cost-effectiveness analysis (CEA) of substance abuse treatment (SAT). The issue is that CEA is predicated on there being only one key outcome, but addiction treatment can lead to multiple, important outcomes. Only if the single outcome is highly predictive of Corresponding author. Tel.: ; fax: address: jody.sindelar@yale.edu (J.L. Sindelar). the other outcomes can a single indicator be sufficient for a CEA. The substance abuse treatment literature, however, suggests that the outcomes are not always highly correlated (McLellan et al., 1981). Thus, an issue is how to handle the multiple, important outcomes of substance abuse treatment in an economic analysis of substance abuse treatment. We illustrate the problems that the multiplicity of outcomes might pose to CEA using data from the Philadelphia Target Cities Project (McLellan et al., 1998, 1999). We use these data only to illustrate potential problems in a real world setting. The importance of this article is methodological, not the particular data set or the empirical application. The study that we use for our illustration is a large evaluation of publicly funded, outpatient addiction treatment programs contrasting standard care to enhanced care. Thus, we /$ see front matter 2003 Elsevier Ireland Ltd. All rights reserved. doi: /j.drugalcdep

2 42 J.L. Sindelar et al. / Drug and Alcohol Dependence 73 (2004) have two treatments to compare in terms of both effectiveness and costs. Based on our empirical example, we suggest that relying on the outcome of reduced drug use alone may provide misleading evidence as to which SAT is most cost-effective overall. For example, one approach may be more cost-effective in reducing drug use while another approach may be more cost-effective in improving other domains. We provide some guidance on how to perform economic analyses of substance abuse treatments in the presence of multiple outcomes. 2. Background CEA by definition is based on a single program outcome. When there are several program outcomes, one can use a key outcome if it is positively and highly correlated with the other outcomes. In drug treatment, there are multiple outcomes of concern. The use of reduced drug use as a key indicator of success in addiction treatment is an obvious choice as the single outcome. However, changes in drug use may not be strong predictors of changes in other outcomes. Further, reduced crime and enhanced employment may be of equal or greater concern to society (McLellan and Weisner, 1996), but changes in these social outcomes may not always be highly, positively correlated with reduced drug use. Only if changes in drug use were very highly predictive of changes in the other areas would it be a sufficient statistic to use alone in CEA. Thus, if some outcomes are uncorrelated, or even negatively correlated, then the use of only one outcome (e.g. drug use) in CEA may provide inadequate, inaccurate or misleading evidence as to which treatment is most cost-effective in the given setting. There is relatively little extant literature that performs cost-effectiveness analyses of substance abuse treatment. See Cartwright (1998, 2000) and French et al. (2002) for reviews of CEA and cost-benefit analysis (CBA) as applied to treatment for drug abuse. The few articles that perform CEAs of treatment for drug abuse have used only a single outcome, as is standard in CEAs. Most have used a measure of drug use such as days of use or abstinence as an outcome. Cartwright (1998) suggests the use of an abstinence-based measure for CEA. Kraft et al. (1997) and Zarkin et al. (2002) use reduced drug use while Barnett and Swindle (1997) use an indirect measure of drug use. Shephard et al. (1999) use the percentage of the year that an individual is abstinent from illicit drugs as the measure of success. There are only a few studies that use an alternative to drug use measures in CEA. For example, Barnett et al. (2001) conduct CEAs of substance abuse treatment and the transmission of HIV/AIDS using quality of life year (QALYs) measures. Daley et al. (2000) use crime measures, and McCollister et al. (2003a,b) use the number of days reincarcerated over the follow-up period as the effectiveness measure in studies of the impact of prison-based SAT. 3. Methods 3.1. Data and setting The Philadelphia Target Cities Project is a quasi-experimental field study that investigated the effectiveness of standard outpatient counseling versus the same treatment enhanced with a case manager and access to prepaid social services. Nine abstinence-based outpatient treatment programs from particularly poorly served areas located in inner city North and West Philadelphia were selected for participation from 1993 to The programs were publicly supported (through Medicaid and Block Grant Funds) and served primarily inner city clients (McLellan et al., 1998, 1999). These programs used primarily 12-step group counseling aimed at abstinence through the provision of 2 3 hours of group sessions per week. A 4 6 weeks length of stay comprised a complete program, however, most clients did not complete the program. The average length of stay was a bit more than 22 days. Six of the programs received enhancements and three programs continued to provide treatment as usual, or standard care. The enhancements included a clinical case manager (social worker) assigned to each of the selected programs and funds made available to provide pre-contracted social support services outside of the treatment facility. The clinical case manager was available to assess the social service needs of the clients and help to make connections to the needed services offered in the community. Services offered in the community were identified and prepaid, thus allowing greater assurance that the patients would actually receive the intended services. The pre-contracted services included health care, employment, education, parenting, nutrition, and housing services. Patients were recruited at admission to treatment at the central intake unit and were selected on the basis of consecutive entry into the treatment programs. Approximately 80% of those eligible were enrolled in the study. Individuals were surveyed at baseline before treatment and at follow-up 7 months later regardless of whether they finished the entire planned program of treatment. The Addiction Severity Index (ASI), the most commonly used instrument in substance abuse treatment, was used at baseline and follow-up. The ASI is a 40 min structured interview designed to measure lifetime and recent (past 30 days) patient problems in seven areas relevant to substance abusers (McLellan et al., 1980a,b). Our sample size is 431 patients. Sixty percent are male and 20% white. The average age is almost 35 years, and 12 years is the average level of education completed Economic costs of treatment services Treatment costs were determined using the Drug Abuse Treatment Cost Analysis Program (DATCAP). The DAT- CAP was administered separately in each of 2 years to three

3 J.L. Sindelar et al. / Drug and Alcohol Dependence 73 (2004) of the programs (French et al., 1997, 2002). The DATCAP is an on-site data collection instrument that assembles the information necessary to estimate the economic costs of treatment services. The instrument is structured along standard categories including personnel, supplies and materials, contracted services, buildings and facilities, equipment, and miscellaneous items. A detailed explanation of the DATCAP methodology, as well as a summary of some of its empirical findings, can be found in French and McGeary (1997), Salomé and French (2001), Roebuck et al. (2003) and at DATCAP data on costs were collected on only three of the programs, all three of which were enhanced programs. Due to this limitation, we had to use the average weekly cost estimates at these three programs as a proxy for the costs of all the enhanced programs. We view these estimates as acceptable proxies as it is likely that the cost structures for all participating programs would be similar given that they are in the same geographical area and thus face similar prices for rent and counselor salaries. As all three programs that were interviewed with the DATCAP were enhanced programs, we do not have a direct measure of standard care. Instead, we use estimates of the costs of the enhanced care and subtract the costs of the enhancements from the estimated costs of the enhanced program. We can estimate the per patient cost of the enhancements through case manager records, expenditures on ancillary services, and estimates of the number of patients treated in a year. To obtain the total cost of the episode, the per day cost estimate of the enhanced programs was multiplied by the average numbers of days (22) in treatment. The greatest source of uncertainty in the data is the costs of the enhancements. Thus, we conduct sensitivity analysis using alternative estimates of these Outcomes Outcomes for this study are selected from each of the seven domains of the ASI: drug and alcohol use, family, social relationships, medical problems, psychiatric symptoms, employment, and illegal activities (Table 1). Within a domain, we selected variables that provide relatively comprehensive coverage of the domain, have face validity, and are reported without many missing or illogical responses. All relevant ASI measures use a reference period of the last 30 days. In order to compare outcomes in common units, whenever feasible, we select outcomes that are measured as days with problems (e.g. How many days out of the last 30 did you have psychiatric problems? ). We do not use the ASI composite scores for several reasons, including that they are not reported in units that can be interpreted directly either in economic or clinical terms. Further, individual measures, not composite scores, are the standard in CEA studies of SAT. When several good measures are available within a particular domain, we analyze alternatives and examine how sensitive our findings are to specific measures. Table 1 lists the outcome measures by domain, and provides the variable name and definition. Table 2 displays some descriptive statistics for our sample Measures of effectiveness As our measure of effectiveness, for each outcome, we use the change from follow-up to baseline. The average length of stay in treatment is a little more than 3 weeks and follow-up occurs about 6 months later. For outcomes that reflect days of problems, a reduction from baseline to follow-up from treatment is an improvement. Thus, a negative value or a decrease in the problem frequency represents an improvement. In contrast, for employment, a positive value (i.e. an Table 1 Domains of outcomes, definitions and variable names for outcomes from the seven addiction severity index domains Domains Definitions Variable name Drugs Sum of number of different drugs used in the last 30 days (i.e. if heroin, cocaine, numdrugs cannabis and amphetamines were used in the last month, the variable is equal to four) Sum of number of days reported using a drug across all drugs (e.g. 1 day using cocaine drugdays and 1 day using heroin is equal to two, even if the drugs are used on the same day) Dollars spent on illicit drugs in the last 30 days drgspnt Days out of the last 30 days suffering from drug related problems drgprob Alcohol Dollars spent on alcohol in the last 30 days alcspnt Days out of the last 30 days suffering from alcohol related problems alcprob Family Days out of the last 30 days suffering from serious family conflict famprob Social relationships Days out of the last 30 days suffering from serious conflict with others socprob Medical Days out of the last 30 days with medical problems medprob Psychiatric symptoms Days out of the last 30 days with psychiatric problems psyprob Employment Dollars earned from employment during last 30 days monyjob Days out of the last 30 days with paid employment daypaid Illegal activities Days out of the last 30 days engaged in illegal activities ilact Dollars earned from illegal activities during last 30 days monyill

4 44 J.L. Sindelar et al. / Drug and Alcohol Dependence 73 (2004) Table 2 Summary statistics of variables at baseline: from the seven domains of the Addiction Severity Index (observations: 431) Domains Variable Mean S.D. Minimum Maximum Drugs numdrugs drugdays drgspnt drgprob Alcohol alcspnt alcprob Family famprob Social relationships socprob Medical medprob Psychiatric symptoms psyprob Employment monyjob daypaid Illegal activities ilact monyill increase in paid days) represents an improvement. When reviewing the tables and interpreting the results, one must be aware of the signs of the outcome variables Correlations A key question of this investigation is whether reduced drug use is a good predictor of changes in other outcome measures. If so, then drug use can be used as the key indicator in CEAs of addiction treatments without much concern for other outcomes, as they would all move together. We explore the extent to which reduced drug use is correlated with improvements in other domains in our Target Cities sample. As a first step, we examine the correlations across the changes in different outcomes. If measures in different domains are affected quite similarly (e.g. significantly and highly correlated) by treatment, then a CEA on a single outcome could provide sufficient information to deter- mine which treatment regime would be most cost-effective. If the outcomes are not similarly affected, then CEAs of alternative domains might provide conflicting evidence on cost-effectiveness. Table 3 shows the pair-wise correlation of the change from baseline to follow-up in the selected set of outcome measures. We use the Pearson product moment correlation coefficients between the selected variables using case-wise deletion, i.e. using the sample for which all variables are non-missing. The 10% significance level for each correlation is obtained using the unadjusted significance level statistic. This statistic follows a two-tailed cumulative Student s t distribution with n 2 degrees of freedom. An asterisk is displayed in the table if the correlation coefficient is significant at least at the 10% level. In other words, if the probability that T >r[ n 2/ 1 r 2 ] is at most 10%, where r is the sample Pearson product moment correlation coefficient. See Table 3 Pair-wise correlations of changes in ASI outcomes from baseline to follow-up numdrugs drugdays drgspnt drgprob alcspnt alcprob famprob socprob medprob psyprob monyjob daypaid ilact monyill numdrugs 1 drugdays drgspnt drgprob alcspnt alcprob famprob socprob medprob psyprob monyjob daypaid ilact monyill Indicates the 10% level of confidence.

5 J.L. Sindelar et al. / Drug and Alcohol Dependence 73 (2004) Greene (2002), Snedecor and Cochran (1989), and Edwards (1984) for a more detailed explanation of these methods. We find that the outcome measures within the drug domain are almost all positively and significantly correlated with each other at the 0.10 level. This can be seen in the first section of the correlation matrix. In contrast, improvements in drug measures are insignificantly correlated with employment outcomes. They are positively and significantly correlated with changes in crime, but the correlation is relatively low for most measures of drug use. Family, social and psychiatric problems are generally significantly related to drug measures and are of the expected sign, but the magnitude of the correlation is relatively low (typically below 0.18). This simple descriptive analysis of correlations suggests that drug use may not be a sufficient predictor of changes in the other outcomes in this sample to be used as a single outcome measure in CEA of SAT. However, the use of the correlations is only to be indicative of potential problems, the further evidence of CE ratios is needed to draw any firm conclusions about the implications for CEA. Further, this is only one example; correlation rates will vary over different time frames, data sets, and situations. The point is that the outcomes cannot be counted on to always be highly, positively correlated. In the second and most important phase of the analysis we explore the practical implications of alternative outcome measures for a CEA of the Philadelphia Target Cities Project. We calculate cost-effectiveness ratios for each domain to see if they provide conflicting evidence as to whether standard or enhanced treatment is more cost-effective Calculation of CE ratios In order to derive the necessary cost-effectiveness ratios, we must first calculate the incremental effects and the incremental costs of standard care relative to no care, and of enhanced care relative to standard care (see Drummond et al., 1997; Gold et al., 1996). With information on the incremental effectiveness and incremental costs, we can then form the incremental cost-effectiveness ratios. The no care component is represented by baseline characteristics and has no program costs Treatment costs Using cost data from DATCAP, we estimate the cost of standard drug-free care to be US$ 200 for the 22-day length of stay in treatment (French et al., 2002). The marginal costs of enhancements are estimated to be US$ 18 per person in the enhanced programs and are equal to the total costs of enhancements divided by the number of patients in the enhanced programs Treatment effectiveness To calculate the incremental gains due to standard care as opposed to no care, we compare the baseline characteristics of the standard care group to their characteristics at follow-up. Similarly, the baseline characteristics of the enhanced care group are compared to their follow-up characteristics to determine the incremental effectiveness of standard plus enhanced care relative to no treatment. We do this for each outcome separately to obtain measures of effectiveness by domain. These calculations can be seen in Table 4. The baseline characteristics of those in standard care (enhanced care) are shown in the column indicated as B st (B e ) and the follow-up means for those in standard care (enhanced care) are shown in the column marked as F st (F e ). The differences between follow-up and baseline are indicated by the columns marked as st ( e ). The significance levels of the increments st and e, and [B s B e ]intable 4 are obtained by performing a test of equality of means, assuming unequal variances. The test for equality of means, u st and u e, when their respective standard errors σ st and σ e are unknown and assumed unequal is given by t = (u st u e )/ [(s st ) 2 /n s ] + [(s e ) 2 /n e ], where s j stands for the sample standard error for group j and n j the number of observations in group j (for j equal to e and st). The resulting statistic is distributed as a Student s t with v degrees of freedom, where v is given by [(s s ) 2 /n s + (s e ) 2 /n e ] 2 /[((s s ) 2 /n s )/(n s 1) + ((s e ) 2 /n e )/(n e 1)] (Satterthwaite, 1946) Incremental gain to enhancements In order to estimate the incremental change in outcomes associated with enhanced care, we subtract the gains due to standard care from those that accrue to enhanced care. We do this because the baseline characteristics of the standard and enhanced groups are significantly different (indicated in the last column of Table 4). Thus, a simple comparison of the follow-up outcomes for the control and the enhanced groups would not be a proper estimate of the effectiveness of enhanced care. Instead, we calculate the relative improvement of enhanced over standard care. The results are found in the column marked e st intable 5. The significance levels of the differences ( e st )intable 5 are obtained by performing tests on the equality of the mean change for the standard group with respect to the mean change for the enhanced group, taking into account that st and e are increments themselves. Note that we calculate confidence levels for the effectiveness indicators, but not for the CE ratios. This is because we only have average treatment costs by modality, so there is no variation across individuals in costs. Thus, the levels of significance of the measures of effectiveness are the relevant levels for the CE ratios. 4. Results Below we present the findings on effectiveness and incremental effectiveness and then use these results to calculate incremental cost-effectiveness ratios.

6 46 J.L. Sindelar et al. / Drug and Alcohol Dependence 73 (2004) Table 4 Comparison of mean outcomes in standard vs. enhanced addiction treatment Domains Standard care (88 observations) Enhanced care (343 observations) B st B e baselines Base, B st Follow, F st Increment, st Base, B e Follow, F e Increment, e Drugs numdrugs drugdays drgspnt drgprob Alcohol alcspnt alcprob Family relations famprob Social relations socprob Medical and psychiatric medprob psyprob Employment monyjob daypaid Illegal activities ilact monyill Baseline, follow-up and change data are shown for the seven domains of the Addiction Severity Index. B st and B e indicate characteristics at baseline for standard and enhanced care, respectively; the stands for changes in each of these. Similarly, F st and F e indicate values at follow-up for standard and enhanced care respectively. The B st B e baseline column indicates whether the means for control and enhanced groups were significantly different at baseline or not. Pre- and post-treatment outcome means are significantly different at the 90% level of confidence. Pre- and post-treatment outcome means are significantly different at the 95% level of confidence. Pre- and post-treatment outcome means are significantly different at the 99% level of confidence Effectiveness As can be seen in Table 4, drug use decreases significantly due to treatment for several drug outcome measures for both standard and enhanced treatment. However, the gains are more significant and of greater magnitude for enhanced care. Standard care reduces alcohol use, but not significantly. Both the amount of money spent on alcohol and days using alcohol are significantly reduced for those in enhanced care. Problems with family, social relationships, physical health and mental health are reduced in both standard and enhanced treatment. These improvements are significant for those in enhanced care, with the exception of family problems. In contrast, for those in standard care, the improvements are significant only for medical problems. Both measures of employment are significantly improved from baseline for each standard and enhanced care. Surprisingly, the improvement is greater for the standard as compared to the enhanced group for both measures. Illegal activities, as measured by days of crime or illegal income, worsen for both those in standard and enhanced care in most cases; the exception is money from crime in enhanced care. None of the changes, however, are significant Incremental effectiveness In Table 5, the column marked incremental benefit shows the magnitude and significance of improvements of enhanced care as compared to standard care for each outcome. We report the absolute value of the numbers in order to report improvements as positive numbers. Thus, a positive number in this column implies that enhanced care offers greater improvement as compared to standard care. By taking the absolute value, we changed the signs on all entries but the employment measures. We find that those in the enhanced program had significantly greater improvements in every measure of drug and alcohol use as compared to those in standard care. Measures of social relations, psychological problems and money from crime showed greater, but not significantly greater, gains from enhanced care. In other domains and measures, standard care was often associated with greater improvement. This is the case for family problems, medical

7 J.L. Sindelar et al. / Drug and Alcohol Dependence 73 (2004) Table 5 Cost-effectiveness ratios by domains of the Addiction Severity Index Domains Change outcome Incremental Standard, benefit, e st st CEA standard ratio, US$ 200 CEA enhanced ratio: alternative costs of enhanced care Enhanced, US$ 18 US$ 36 US$ 72 e US$ US$ 18/ 200/ st ( e st ) CE US$ 36/ ( e st ) CE US$ 72/ ( e st ) Drugs numdrugs e 70.9 e e drugdays e 12.0 e 24.0 e drgspnt e 0.4 e 0.8 e drgprob e 8.7 e 17.5 e Alcohol alcspnt e 2.4 e 4.8 e alcprob e 15.0 e 30.0 e Family relations famprob NA st NA st NA st Social relations socprob e 88.0 e e Medical and psychiatric medprob NA st NA st NA st psyprob e 27.7 e 55.3 e Employment monyjob NA st NA st NA st daypaid NA st NA st NA st Illegal activities ilact NA NA st NA st NA st monyill NA 1.8 e 3.7 e 7.4 e The incremental benefit represents an improvement from standard treatment to enhanced and is shown as a positive number. Therefore, values in the e st column take on a positive value when the improvement is greater for the enhanced group. The CE column indicates which is the most cost-effective treatment for the particular outcome: e if it is the enhanced treatment; st if it is standard. When NA appears in the CEA ratio column, this means that enhanced treatment is dominated by the standard since it is both more expensive and less effective and a formal CE ratio is not appropriate, thus the NA designation. The average change in outcome is significantly different between the standard treatment group and the enhanced group at the 90% levels of confidence. The average change in outcome is significantly different between the standard treatment group and the enhanced group at the 99% levels of confidence. CE problems, employment income, days employed, and crime as measured by days of illegal activities Cost-effectiveness ratios To calculate the various CE ratios, we divide the incremental cost by the incremental effect for each outcome. Thus, for standard care we divide the cost of care per client (US$ 200) by the change in each effectiveness measure for the move from no care (baseline) to standard care. To calculate the CE ratio for enhanced care, we divide the estimated per client cost of enhanced care (US$ 18) by the comparative improvement in outcomes. The improvements reported in the first three columns of Table 5 are estimates of improvements over a 30-day period. However, the improvement may have accumulated over the entire period of follow-up after an average of 22.5 days in treatment. Therefore, in order to calculate the CE ratios, we multiply the 30-day improvement by 6.25, which is representative of the number of months between the end of treatment and the 7-month follow-up. We assume that the incremental effects are constant over the more than 6 months following treatment. We use this arbitrary assumption given that we do not have any data on the time pattern of benefits. This transformation merely puts the figures in context of a longer period. Because we are multiplying all effects by a constant, the ordinal ranking of the cost-effectiveness of treatment modalities is not affected. The fourth and fifth columns of Table 5 show the cost-effectiveness ratios of going from no care to standard care and from standard care to enhanced care for each of the outcomes. As can be seen, the CE ratios from different outcome variables provide conflicting evidence on cost-effectiveness. For all indicators of drug and alcohol dependence, enhanced care has incremental CE ratios that are smaller than those of standard care. This implies that each unit of effect is achieved at a lower cost for enhanced care as compared to standard. In these cases, enhanced care is said to weakly dominate standard care and enhanced care is more cost-effective. The logic of weakly dominated is that, since standard care has been accepted at the policy level and actually implemented, the cost per output is acceptable if standard care is considered to be acceptable. Therefore,

8 48 J.L. Sindelar et al. / Drug and Alcohol Dependence 73 (2004) the lower cost per unit of outcome achieved with enhanced care will also be acceptable. That enhanced care weakly dominates standard care for the drug and alcohol measures of effectiveness is fortuitous in terms of interpreting our CE ratios. Without this situation, one would need a threshold or benchmark value for the cost-effectiveness of drug treatment, but the substance abuse treatment field does not have such information. If enhanced care both costs more and provides greater outcome, then one needs a threshold level of society s willingness to pay for these outcomes in order to make a policy decision. In some cases, the cheaper standard care is more effective or equally as effective as enhanced care. In these cases, it is obviously not necessary, and is not meaningful, to calculate CE ratios and NA (not applicable) appears in the corresponding column in Table 5. This situation occurs for problems with family relationships, physical health problems, both measures of employment, and for days of illegal activity. In these cases, the cheaper standard care produces greater or similar benefits as compared to enhanced care. Thus, standard care is obviously the most cost-effective choice for these outcomes. Given the uncertainty associated with the per capita costs of enhancements, we recalculate the CE ratios for enhanced care using alternative, higher estimates of the cost of enhanced care (US$ 36 and 72). We focus on higher costs due to our concern that the estimated cost of enhanced care might be too low. We use reasonable alternative, higher estimates in order to see if the conclusions would change with different estimates. We find that the qualitative results do not change even with the higher estimated costs of enhanced care (Table 5). Our overall conclusion is that alternative measures and domains of outcomes can produce conflicting results as to which treatment should be selected on the grounds of cost-effectiveness. This, combined with the evidence on the correlations across outcomes, suggests that the analyst cannot merely assume that CE ratios for drug outcomes (or any other outcomes) can be used to infer CE ratios for outcomes in other domains. The conflict in the CE ratios suggests that use of only the drug use outcome in CEA may be misleading as to the overall cost-effectiveness of one type of treatment versus another. In addition, policy-makers may be more concerned with other outcomes such as reduced crime and reduced reliance on welfare and the focus on only drugs would be misguided. 5. Discussion 5.1. Limitations There are several potential limitations relating to the field study and corresponding data. However, the data set and study are only an empirical illustration of the kinds of problems that can arise with the multiple outcomes of substance abuse treatment. Thus, potential problems with the data and the study are not as important as is the methodological point. None the less, we indicate potential issues relating to the study and the data. These include: use of a quasi-experimental field study rather than a clinical trial with random assignment; the use of patient self-reports of problems; the demonstrated difference in group means at baseline; the lack of precise estimates of the costs of enhancements; the relatively small sample size; data available at two time points; and the lack of a no treatment arm. Reduction in AIDS risk and housing stability are other outcomes that would be relevant, but lacking data, we did not incorporate these empirically. All of these limitations might reduce confidence in the specific numbers, but we believe that the qualitative conclusion remains. The data are used only to illustrate the methodological problems of applying CEA to substance abuse treatment. We believe that the data illustrate that in some cases it may be that comparisons across different outcomes may yield conflicting conclusions about the relative cost-effectiveness of competing alternative treatments Advice and alternatives Despite potential problems with CEAs in general and with specific studies, CEAs will be desired to make policy decisions. Various countries are using CEAs as part of a policy decision-making process (Baker and Mounteney, 1999). CEAs are now used internationally and guidelines for CEAs have been established in multiple countries (Langley, 1996; Garattini et al., 1995; Department of Human Services and Health, 1995; Canadian Coordinating Office for Health Technology Assessment, 1997). CEAs are also desired in part to facilitate comparisons across other medical interventions. They may also be used because of the belief that they are fairly easy to conduct as compared to cost-benefit analysis. This latter hope seems to be misplaced as it is difficult to conduct a CEA well. However, given the potential demand for CEAs, the following suggestions may be helpful in conducting a CEA of substance abuse treatment and in focusing further research on new methods. We suggest that in conducting a CEA of drug treatment, as a first step, outcomes from multiple domains should be analyzed separately; a CEA should not rely solely on inferences based on the single outcome of drug use or abstinence. (Note that CE ratios could well vary by drug selected for study as well. See Rounsaville et al. (2003) for the suggestion to take multiple drugs of abuse into account in effectiveness analysis.) One should first check to see if there is conflict across CE ratios of key outcomes. Without conflicts, one could be fairly confident of the policy prescription that would come from a CEA comparing one method of drug treatment to another. If, however, there is conflict, the resolution is more complicated. One way to resolve the ambiguity due to conflicting results is to aggregate the domains into a single indicator. The domains can be aggregated in several ways,

9 J.L. Sindelar et al. / Drug and Alcohol Dependence 73 (2004) such as incorporating evidence on preferences or valuing all outcomes in pecuniary terms. Preferences for different outcomes could vary by region, country, or by a decision-maker. Thus, preferences would have to match to or be appropriate to the location and decision-maker. Use of a preference measure converts a CEA to a cost-utility analysis (CUA). A quality adjusted life year ( QALY ) is a preference-based measure. However, presently there are no such indices designed to capture the most essential outcomes for substance abuse. Further, QALYs do not capture the important external effects such as the impact of reduced crime on non-drug users and the impact of welfare expenditures on taxpayers. These externalities may be what most motivate governments to fund treatment programs. Direct evidence on preferences might be used to aggregate outcomes, but such data on preferences are rare and not yet well developed. Further, as preferences may be specific to a certain region or country, a single set of preferences may not be suitable for all studies. A cost-benefit analysis aggregates all of the costs and benefits into a single quantity using dollars as the common unit. The CBA is the preferred alternative on conceptual grounds (Kenkel, 1997; Reuter, 1999). CBA aggregates across the multiple outcomes and includes the external effects on society. In theory, CBA offers a comprehensive answer to the question of value (see French et al., 1997, 2002; Kenkel, 1997; Sindelar and Manning, 1997). However, it is difficult to measure such disparate outcomes in pecuniary terms. Future CBAs will be easier to conduct because several cost-benefit analyses of drug addiction treatment programs have been conducted and have made advances in valuing key outcomes in dollars (see Godfrey et al., 2000; Gutzwiller and Steffen, 2000; French et al., 1991, 1996, 2002; Cartwright, 1998, 2000, for recent reviews). A final note is that if CEAs of substance abuse treatments are compared to CEAs of other interventions (either medical or non, e.g. welfare or criminal justice), using only reduced drug use as an outcome could unfairly handicap the case for drug treatment when vying for scarce resources. One would want to measure the full set of benefits of substance abuse treatment in order to make meaningful comparisons. Omitting critical benefits of drug treatment would severely handicap substance abuse treatment in comparisons to other interventions. 6. Conclusion Economic evaluations of treatment of drug dependence are in demand in order to assure that government and/or society at large are getting value for resources spent. CEAs are popular because they can appear to be relatively easy to conduct and also because comparisons can be made across extant studies. Part of the comparative ease of CEA as opposed to CBA is that only one measure of outcome is used at a time. However, this paper illustrates the potential problems in conducting a CEA of treatment for drug dependence in the face of multiple, important outcomes. Our findings illustrate that indicators of drug use are not consistently, highly, and significantly correlated with other important outcomes. Further, one cannot count on consistency in CE ratios of treatment of drug dependence across domains. We find that CEAs using only one outcome, i.e. reduced drug use, for evaluation of treatment for drug dependence can be misleading as to which type of service is most cost-effective overall. Reduced drug use is the most direct outcome, but it is not the only outcome, nor is it necessarily the most important one to payers and to society. It could be argued that reduced crime, better family functioning, and higher earnings, are of greater concern to society than are reductions in drug use per se. Conducting CEAs separately for multiple outcomes can produce ambiguity. Alternatives to CEA include cost-benefit analyses or development of appropriate preference weights to aggregate the disparate outcomes into a single index. Both alternatives are difficult because it is hard to aggregate outcomes in pecuniary terms or to develop preference weights. CBA is preferred for several reasons including that methods are available to place costs and outcomes in pecuniary units. In sum, researchers should incorporate the many, important outcomes of drug treatment when conducting economic evaluations. Further research should be aimed at advancing methods to overcome obstacles to obtaining policy-relevant information on the value of treatment for substance abuse. Acknowledgements The authors acknowledge the support of the Veterans Administration Research Service, the Center for Substance Abuse Treatment, the National Institute on Drug Abuse (P50 DA09241, U10DA1308, P50 DA07705, R01 DA11506), and the National Institute on Drug Abuse (grant to Yale University, R01 DA1471). References Baker, O., Mounteney, J., Evaluating the Treatment of Drug Abuse in the European Union. Office for Official Publications of the European Communities, Luxembourg. Barnett, P., Swindle, R., Cost-effectiveness of inpatient substance abuse treatment. Health Serv. Res. 32 (5), Barnett, P.G., Zaric, G.S., Brandeau, M.L., The cost-effectiveness of buprenorphine maintenance therapy for opiate addiction in the United States. Addiction 96 (9), Canadian Coordinating Office for Health Technology Assessment, Guidelines for Economic Evaluation Pharmaceuticals: Canada, second ed. CCOHTA, Ottawa. Cartwright, W.S., Cost benefit and cost-effectiveness analysis of drug abuse treatment services. Eval. Rev. 22 (5), Cartwright, W.S., Cost-benefit analysis of drug treatment services: review of the literature. J. Ment. Health Policy Econ. 3,

10 50 J.L. Sindelar et al. / Drug and Alcohol Dependence 73 (2004) Daley, M., Argeriou, M., McCarty, D., Callahan Jr., J.J., Shepard, D.S., Williams, C.N., The costs of crime and the benefits of substance abuse treatment for pregnant women. J. Subst. Abuse Treat. 19 (4), Department of Human Services and Health, Guidelines for the Pharmaceutical Industry on Preparation of Submissions to the Pharmaceutical Benefits Advisory Committee. Australian Government Publishing Service, Canberra. Drummond, M.F., O Brien, B., Stoddart, G.L., Torrance, G.W., Methods for the Economic Evaluation of Health Care Programs, second ed. Oxford University Press, New York. Edwards, A.L., An Introduction to Linear Regression and Correlation, second ed. Freeman, New York. French, M.T., McGeary, K.A., Estimating the economic cost of substance abuse treatment. Health Econ. 6, French, M.T., Rachal, J.V., Hubbard, R.L., Conceptual framework for estimating the social cost of drug abuse. J. Health Soc. Policy 2 (3), French, M.T., Mausopf, J.A., Teague, J.L., Roland, J., Estimating the dollar value of health outcomes from drug abuse interventions. Med. Care 34 (9), French, M.T., Dunlap, L.J., Zarkin, G.A., McGeary, K.A., McLellan, A.T., A structured instrument for estimating the economic cost of drug abuse treatment: the Drug Abuse Treatment Cost Analysis Program (DATCAP). J. Subst. Abuse Treat. 14 (4), French, M.T., Salome, H.I.J., Sindelar, J.L., McLellan, A.T., Benefit cost analysis of addiction treatment: methodological guidelines and applications using the DATCAP and ASI. Health Serv. Res. 37 (2), Garattini, A., Grilli, R., Scopellitti, D., Mantovani, L., A proposal for Italian guidelines in pharmacoeconomics. PharmacoEconomics 7, 1 6. Godfrey, C., Eaton, G., McDougall, C., Culyer, A., The Economic and Social Costs of Class A Drug Use in England and Wales. Home Office Research Development and Statistics Directorate, London. Gold, M.R., Siegel, J.E., Russell, L.B., Weinstein, M.C., Costeffectiveness in Health and Medicine. Oxford University Press, New York. Greene, W.H., Econometric Analysis. Prentice-Hall, Englewood Cliffs, NJ. Gutzwiller, F., Steffen, T., Cost Benefit Analysis of Heroin Maintenance Treatment. Karger, Basel, Switzerland. Kenkel, D., On valuing morbidity, cost-effectiveness analysis, and being rude. J. Health Econ. 16, Kraft, M.K., Rothbard, A.B., Hadley, T.R., McLellan, A.T., Asch, D.A., Are supplementary services provided during methadone maintenance really cost effective? Am. J. Psychiatr. 154 (9), Langley, P.C., The November 1995 revised Australian guidelines for the economic evaluation of pharmaceuticals. PharmacoEconomics 9, McCollister, K.E., French, M.T., Inciardi, J.A., Butzin, C.A., Martin, S.S., Hooper, R.M., 2003a. Post-release substance abuse treatment for criminal offenders: a cost-effectiveness analysis. J. Quant. Criminol., in press. McCollister, K.E., French, M.T., Prendergast, M., Wexler, H., Sacks, S., Hall, E., 2003b. Is in-prison treatment enough? A cost-effectiveness analysis of prison-based treatment and aftercare services for substance-abusing offenders. Law and Policy 25 (1), McLellan, A.T., Weisner, C., Achieving the public health and safety potential of substance abuse treatment. In: Bikle, W., De Grandpre, R.J. (Eds.), Drug Policy and Human Nature: Psychological Perspectives on the Prevention, Management and Treatment of Illicit Drug. Pleneus Press, New York, pp McLellan, A.T., Luborsky, L., Erdlen, F., LaPorte, D., 1980a. The addiction severity index: a diagnostic/evaluative instrument for substance abuse clients. In: Gottheil, E. (Ed.), Substance Abuse and Psychiatric Illness. Pergamon Press, New York. McLellan, A.T., Luborsky, L., O Brien, C.P., Woody, G.E., 1980b. An improved evaluation instrument for substance abuse patients: the Addiction Severity Index. J. Nerv. Ment. Dis. 168, 26 33, 38. McLellan, A.T., O Brien, C.P., Luborsky, L., Woody, G.E., Kron, R., Are the addiction-related problems of substance abusers really related? J. Nerv. Ment. Dis. 169 (4), McLellan, A.T., Hagan, T.A., Meyers, K., Levine, M., Gould, F., Bencivengo, M., Durell, J., Jaffe, J., Supplemental social services improve outcomes in public addiction treatment. Addiction 93 (10), McLellan, A.T., Levine, M., Hagan, T.A., Meyers, K., Randall, M., Gould, F., Bencivengo, M., Durell, J., Does clinical case management improve outpatient addiction treatment? Drug Alcohol Depend. 55 (1 2), Reuter, P., Are calculations of the economic costs of drug abuse either possible or useful? Addiction 94 (5), Roebuck, M.C., French, M.T., McLellan, A.T., DATStats: summary results from 85 completed Drug Abuse Treatment Cost Analysis Programs (DATCAPs). J. Subst. Abuse Treat. 25 (1), Rounsaville, B.J., Petry, N., Carroll, K., Single vs. multiple drug focus in substance abuse clinical trails. Drug Alcohol Depend. 70 (2), 21, Salomé, H.J., French, M.T., Using cost and financing instruments for economic evaluation of substance abuse treatment services. In: Galanter, M. (Ed.), Services Research in the Era of Managed Care, Section III. Recent Developments in Alcoholism, vol. 15. Kluwer Academic Publishers/Plenum Publishers, New York, Chapter 11, pp Satterthwaite, F.E., An approximate distribution of estimates of variance components. Biometrics Bull. 2, Shephard, D., Larson, M.J., Hoffman, N., Cost-effectiveness of substance abuse services. Addict. Disord. 22 (2), Sindelar, J.L., Manning, W.G., Cost-benefit and cost-benefit analysis: issues in the evaluation of the treatment of illicit drug abuse. In: Egertson, J., Fox, D., Leshner, A. (Eds.), Treating Drug Abusers Effectively. Milbank/Blackwell, pp Snedecor, G., Cochran, W.G., Statistical Methods, eighth ed. Iowa State University Press, Ames, IA. Zarkin, G., Lindrooth, R., Demiralp, B., Wechsberg, W., The cost and cost-effectiveness of an enhanced intervention for people with substance abuse problems at risk for HIV. Health Serv. Res. 36 (2),