Impact and Cost- Benefit Analysis of the Anchorage Wellness Court

Transcription

1 Impact and Cost- Benefit Analysis of the Anchorage Wellness Court RESEARCH REPORT JULY 2008 John K. Roman Aaron Chalfin Jay Reid Shannon Reid research for safer communities URBAN INSTITUTE Justice Policy Center

2 This document was prepared under a grant from the National Institute of Justice and the University of Alaska-Anchorage. The views expressed are those of the authors and should not be attributed to the Urban Institute, its trustees, or its funders. i

3 ACKNOWLEDGMENTS The authors would like to thank those who assisted our efforts in collecting, preparing and analyzing the data used in this study. Colleen Owens, Michael Kane, Kevin Roland, Courtney Schafer, Aaron Sundquist, and Bogdan Tereshchenko provided exceptional research assistance. We would like to thank Janice Munsterman, Sandra Woehrle, and Marilyn Moses at the National Institute of Justice who monitored this project for their patience and guidance. Numerous people in Anchorage found time to discuss the project with us and their help in understanding the Anchorage Wellness Court was invaluable: Conrad Brown, Steve Christopher, Judith Conti, Richard Gustafson, Robyn Johnson, Jeeni Jurvig, Michael Krukar, Julie Linnell, Janet McCabe, Steve Priddle, Judge Stephanie Rhoades, Ron Taylor, Christine Thoreson, Ann Vanbockel, and Judge James Wannamaker. We wish to thank our project partners at the University of Alaska-Anchorage; Bob Langworthy who served as principal investigator; Ron Everett who guided the project to completion; Alan McKelvie who procured (and explained) the data for our portion of the project; and our partners at Alaska Judicial Council, Larry Cohn and Susie Dosik. It was a delight to work with such a capable and congenial team. Special thanks to Jenny Miller who spent countless hours answering questions about who did what with whom, and served as our tour guide to all things Anchorage. Despite their guidance, all remaining errors are our own. ii

4 Contents ACKNOWLEDGMENTS...II EXECUTIVE SUMMARY...V Results... vi Results of the Evaluation of the and Comparison Groups... vii Bivariate Results... vii Multivariate Results... vii Results of the Evaluation of the Opt-In, Opt-Out and Comparison Groups...viii Bivariate Results Opt-In Group vs. Comparison Group...viii Bivariate Results Opt-Out Group vs. Comparison Group...viii Multivariate Results Opt-In Group vs. Comparison Group...viii Multivariate Results Opt-Out Group vs. Comparison Group... ix Summary... ix BACKGROUND...1 Prior Research...2 THE RESEARCH DESIGN...5 Data...5 Hypotheses...7 Sample Characteristics...8 Sub-Groups...8 Sub-group 1 The Opt-In Group...9 Sub-group 2 The Opt-Out Group...9 Sub-group 3 successes (graduates)...9 Sub-group 4 failures...9 Creating Anchor Dates...11 Methods...12 Selection Models...12 Multivariate Models...13 Dependent Variables...15 Cost and Benefit Models...16 Survival Analysis...16 Outcome Models...17 RESULTS...18 Bivariate Comparisons of Recidivism...18 Group vs. Comparison Group...21 Opt-In Group vs. Comparison Group...21 Opt-In Successes vs. Comparison Group...22 Opt-In Failures vs. Comparison Group...22 Opt-Out Group vs. the Comparison Group...22 Multivariate Comparisons of Recidivism...22 Arrest...23 Conviction...27 Multivariate Results...30 Cost-Benefit Analysis...33 Costs...35 Court Costs...35 Costs...36 Benefits...38 Net Benefits...44 Outcome Analysis...45 DISCUSSION...46 REFERENCES...48 iii

5 APPENDIX I UNWEIGHTED RESULTS...55 APPENDIX II COST-BENEFIT RESULTS...56 APPENDIX III OUTCOME ANALYSIS...6 iv

6 EXECUTIVE SUMMARY The primary goal of this research is to estimate the costs and benefits of serving misdemeanor DUI offenders in the Anchorage Wellness Court (AWC), a specialized court employing principles of therapeutic jurisprudence. The Urban Institute, as the subcontractor to the University of Alaska- Anchorage, conducted an impact and a cost-benefit analysis (CBA) to estimate the effectiveness of the AWC. The study focused on the impact of the program on reducing the prevalence and incidence of new criminal justice system contact. Costs were collected to estimate the opportunity cost of the AWC. Recidivism variables were monetized to estimate the benefits from crime reductions. Outcomes were observed at 24, 30, 36, and 48 months. The Anchorage Wellness Court began serving misdemeanor DUI offenders in Anchorage, AK in August, 1999, with the goal of reducing alcohol-related offending through treatment and increased individual accountability. The Anchorage Wellness Court began as a bail and sentencing option. Arrestees with an identified alcohol problem were released into the community where they received substance abuse treatment and regular judicial supervision. Over time, the AWC expanded operations to include more components of therapeutic jurisprudence, eventually evolving into a mature therapeutic court. Program components included substance abuse treatment, moral reconation therapy (MRT), recovery meetings (such as Alcoholics Anonymous), employment and financial responsibility, case management and substance abuse monitoring, judicial supervision, and complex criminal justice collaborations. 1 Participant eligibility was determined by clinical staff and prosecutors. Defendants voluntarily enrolled into the program and received reductions in jail terms and fines if they successfully completed the program, which usually required about 18 months. In this study we make two sets of comparisons to estimate the effect of AWC on participant behavior. First, we compare the outcomes for 277 individuals who were eligible for the Anchorage Wellness Court (AWC) 141 individuals who had no contact with the program (the Comparison Group), and 136 who were referred to the program (the Group). Although not everyone who was referred to the program formally enrolled, all who were referred received at least some exposure to AWC 2. We refer to those who formally opt-in to the program as the Opt-In Group, and those who were referred but did not formally enroll as the Opt-Out Group. To account for the 1 For a complete description of the program and operations of the Anchorage Wellness Court and a discussion of the transferability of the model, please see Susie Mason Dosik. (2008). Transferability of the Anchorage Wellness Court Model. Anchorage, AK: Alaska Judicial Council Dosik notes that [a]significant lag time sometimes months might elapse between referral and formal enrollment, and the defendant was receiving substance abuse treatment and program services during that time (2008:4). v

7 presence of two distinct groups within the Group, we then compare outcomes for the Opt-In Group (91, including those who ultimately graduate (44) and those who fail (47)), the Opt- Out Group (45), and the Comparison Group (141). 3 Figure 1. Flow of Cases into the Anchorage Wellness Court Eligible DUI Misdemeanors N=277 Referred To AWC? Yes Group N=136 Enrolled? Yes Opt-In Group N=91 Graduated? Yes Opt-In Success N=44 No No No Comparison Group Opt-Out Group Opt-In Failure N=141 N=45 N=47 Source: urban Institute Analysis of program data. Given the complicated enrollment process, both tests are necessary to understand the effectiveness of AWC. Those who eventually opt-out of AWC may receive considerable services from AWC before exiting the program. In effect, the decision to enroll is an intermediate outcome where those who are doing well (or are expected to do well) formally enroll and those who are not, exit. If this initial success or failure is used to determine group composition, final outcomes are confounded. Those who are on a positive path and opt-in include individuals who would be expected to do better than the average person referred to AWC, since the Comparison Group includes both those who would have opted-in and also those who would have opted-out. If such a decision rule is used to determine who is in the Group, the results are likely to be biased. As a result, the impact of AWC on all who are referred must be tested. 4 However, it is also important to determine whether those who received the full program had better outcomes, and thus we include a second set of tests where the outcomes of the Opt-In and Opt-Out Groups are estimated separately. RESULTS Overall we find that AWC reduced recidivism and reconviction for the Group. Despite the decrease in the prevalence of recidivism, the Group returned negative benefits in the form of significantly higher costs to the criminal justice system and victims that result from their 3 A complete explanation of this graphic can be found on page 7 of this report. 4 There is an additional reason to evaluate the effectiveness of AWC on all who are referred to the program. That is, one critical measure of program effectiveness is how successful the program was in getting referred individuals to enroll and receive services. If those who are treated do well, but few who are eligible ultimately enroll, it is prudent to ask whether that program should be deemed successful. Including all those who were assigned to AWC within the Group allows for this type of comparison. vi

8 new offending. However, when the Group is divided into the Opt-in and Opt-Out Groups, a much different pattern emerges. We find that the Opt-In Group had significantly lower likelihood of any rearrest and reconviction and significantly fewer Opt-In Group members were rearrested and reconvicted in all four follow-up periods. Those in the Opt-In Group had large and significant benefits to the criminal justice system and crime victims, returning over three dollars in benefits for each dollar in program costs. By contrast the Opt-Out Group has worse outcomes than the Comparison Group on almost all measures. RESULTS OF THE EVALUATION OF THE TREATMENT AND COMPARISON GROUPS Bivariate Results The general pattern of results is that the Group had better outcomes on most indicators of success, including the likelihood of a new arrest and the number of new arrests, but that the program was more costly to administer than the comparison, and the harms from new offending were greater. In the bivariate analysis, we find that at 24 months, 37 percent of the Group had been rearrested, compared to 53 percent of the Comparison Group. These significant differences (p<0.01) persist through 48 months where 47 percent of the Group had been rearrested compared to 66 percent of the Comparison Group. Those in the Group were less likely to be re-convicted as well, although the difference is only significant at 24 months. There were no significant differences in the number of rearrests or reconvictions. While the arrest and conviction prevalence were lower for the Group, we find negative benefits to the criminal justice system and the public from new offending that is, the harms from new offending were higher for the Group than the Comparison Group. Overall, we estimate that the cost of the program was about $3,300 per participant. When the cost of AWC and the costs of new offending are combined, we find that AWC was not cost-beneficial. Multivariate Results There are differences between the and Comparison Groups in terms of the attributes of each group s membership. To control for any bias this may introduce, we ran multivariate analyses to confirm the bivariate findings. In general, the same results are returned. The odds of a Group member being rearrested are lower than the Comparison Group in all four periods, as are the odds of a reconviction (but again the differences are only statistically significant at 24 months). We again find no statistically significant differences in the number of re-arrests and the number of reconvictions. We also find no significant differences in the time to rearrest, though there is a significantly longer time to reconviction for the Group. In the multivariate models, we again find large negative benefits (additional costs) associated with the new offending of the Group. These differences are significant in the first three follow-up periods and average about $7,800 per participant (and these costs are in addition to the $3,300 in new costs associated with AWC programming. vii

9 RESULTS OF THE EVALUATION OF THE OPT-IN, OPT-OUT AND COMPARISON GROUPS Bivariate Results Opt-In Group vs. Comparison Group The Opt-In Group had better outcomes than the Comparison Group on virtually all indicators of success. In the bivariate analysis, we find that at 24 months, 26 percent of the Opt-In Group had been rearrested, compared to 53 percent of the Comparison Group. These significant differences (p<0.01) persist through 48 months where 42 percent of the Opt-In Group had been rearrested compared to 66 percent of the Comparison Group. Those in the Opt-In Group were less likely to be re-convicted as well in all four follow-up periods. The Opt-In Group had fewer reconvictions in all four periods and fewer rearrests at 36 and 48 months. We find no difference in the bivariate comparisons of benefits to the criminal justice system and the public from new offending although costs to police from new offending by the Opt-In Group were significantly lower, costs to supervision agencies were significantly higher. Overall, we estimate that the cost of the program was higher for the Opt-In Group then the Comparison Group, averaging about $3,900 per participant. Bivariate Results Opt-Out Group vs. Comparison Group For the Opt-Out Group, we largely find no effect or negative effects. In the bivariate analysis, we find that at 24 months, 55 percent of the Opt-Out Group had been rearrested, compared to 53 percent of the Comparison Group, and there were no differences at any of the follow-up periods. At 48 months 55 percent of the Opt-Out Group had been rearrested compared to 66 percent of the Comparison Group, but the difference is not statistically significant 5. There are no differences between the Opt-Out Group and the Comparison Group on measures of the likelihood of a reconviction or the number of rearrests and reconviction. However, in the bivariate comparisons of benefits to the criminal justice system and the public from new offending, the Opt-Out Group had large and significant negative benefits. That is, at 48 months, the costs of new offending by the Comparison Group were about $25,300, while the costs associated with new offending by the Opt- Out Group were about $37,500. We estimate that the program expenditures were much lower for the Opt-Out Group then the Group, averaging about $700 per participant. Multivariate Results Opt-In Group vs. Comparison Group Because there are differences between the Opt-In Group and Comparison Groups in terms of the attributes of their membership, we ran multivariate analyses to confirm the bivariate findings. In general, the same results are returned. The odds of an Opt-In Group member being rearrested are significantly lower for the Opt-In Group in the first three periods, as are the odds of a reconviction (but the differences are only significant at 24 and 30 months). We find significant reductions in the number of re-arrests and re-convictions (at 24 and 36 months). In the multivariate models, we again find large positive benefits (a reduction in costs) associated with the new offending of the Opt-In Group at 24 months (a savings of about $13,400) and 30 months (a savings of about $11,900). These 5 Unless otherwise noted in the text, throughout this paper, results are considered to be statistically significant if p<0.05. viii

10 differences are significant and more than offset the additional cost of about $3,900 of treating this group. In addition, there is a significantly longer time to re-arrest for the Opt-In Group. Multivariate Results Opt-Out Group vs. Comparison Group Multivariate models were run to control for baseline differences in attributes between the Opt-Out Group and Comparison Groups. Again, similar results are returned. The odds of an Opt-Out Group member being rearrested are higher than the Comparison Group in the first two follow-up periods and lower in the last two follow-up periods, although none of the differences are significant. Interestingly, by 48 months, the odds of re-arrest for the Opt-In Group and the Opt-Out Group are almost identical. Odds of a reconviction are higher for the Opt-Out Group than the Comparison Group in all four periods, but none of the differences are significant. An identical pattern for the odds of any reconviction and the number of reconvictions is observed. In the multivariate models, we again find large and significant negative benefits (an increase in harms associated with new offending) for the Opt-In Group at all four periods, and the Opt-Out Group had average negative benefits of $15,900-$17,400. These differences are significant and add to the additional cost of about $700 of treating each member of the Opt-Out Group. SUMMARY In general, we find that the AWC was effective in reducing recidivism and associated harms for the Opt-In Group. Among those who were referred to the program, but who did not enter the program (the Opt-Out Group), there was no effect on some outcomes and negative effects on other outcomes including a finding that this group contributed substantial additional harms to society. Thus, if the AWC is evaluated only on the effectiveness of serving those who were sufficiently motivated to formally enroll in the program, the results are an unqualified success. If a more expansive lens is used, and the effectiveness of the program considers whether the program was effective in serving all who were referred, which is surely a goal of the program, then the effectiveness of the program is modest. ix

11 BACKGROUND Over the last two decades, agencies within the criminal justice system have developed and implemented several innovative programs to link intensive criminal justice system surveillance with substance abuse (including alcohol) treatment. These programs are designed to address the underlying causes of criminal behavior among a substance using population with the goal of reducing the burden of chronic drug use on private citizens and the criminal justice system. The most acclaimed of these interventions has been specialized court dockets, including drug courts and DWI/DUI courts. DWI/DUI courts identify and treat offenders whose alcohol use and abuse is determined to be a significant cause of their criminality. Alcohol treatment is generally provided in the community to a population that otherwise would have been incarcerated. DWI/DUI programs can be implemented using a variety of models, but most are short-term diversion programs that provide pretrial services. These programs generally use direct judicial monitoring in combination with treatment and ensure compliance with a program of graduated sanctions. The Anchorage Wellness Court began serving misdemeanor DUI offenders in Anchorage, AK in August, 1999 with the goal of reducing alcohol-related offending through treatment and increased individual accountability. The Anchorage Wellness Court began as a bail and sentencing option. Arrestees with an identified alcohol problem were released into the community where they received substance abuse treatment and regular judicial supervision. Over time, the AWC expanded operations to include more components of therapeutic jurisprudence, eventually evolving into a mature therapeutic court. Program components included substance abuse treatment, moral reconation therapy (MRT), recovery meetings (such as Alcoholics Anonymous), employment and financial responsibility, case management and substance abuse monitoring, judicial supervision, and complex criminal justice collaborations. 6 Participant eligibility was determined by clinical staff and prosecutors. Defendants voluntarily enrolled into the program and received reductions in jail terms and fines if they successfully completed the program, which usually required about 18 months. The goal of this research is to determine how cost-effective the AWC is in delivering services to arrestees with alcohol problems. How much does it cost to treat a client in AWC? How many crimes are prevented when arrestees receive treatment? Do the benefits vary according to how much treatment is delivered, and in particular, do those who are exposed to the program but not treated still have positive outcomes? Is there a longer time to rearrest for treated clients? Can some ex ante predictors of treatment success be identified to inform other similar programs? 6 For a complete description of the program and operations of the Anchorage Wellness Court and a discussion of the transferability of the model, please see Susie Marie Dosik. (2008). Transferability of the Anchorage Wellness Court Model. Anchorage, AK: Alaska Judicial Council

12 PRIOR RESEARCH The Relationship between Alcohol and Crime Substance use (both drug and alcohol) has been routinely linked to criminality (Anglin and Perrochet 1998; Boyum and Kleiman 2002; DeLeon 1988a; DeLeon 1988b; Harrison and Gfroerer 1992; Inciardi et al. 1996; Inciardi 1992; Inciardi and Pottieger 1994; MacCoun and Reuter 2001; Miller and Gold 1994). Much of the cause of alcohol-related crime is psychopharmacologic, where the effects of alcohol consumption lead users to commit crimes while intoxicated (Greenfeld 1998; Stuart et al. 2006). Alcohol abusers are also more likely to be the victims of violence than those who are not (Greenfeld 1998; Stuart, et al. 2006). Alcohol-related violence is associated with more severe injuries and more chronic cases of violence (Reider et al. 1988; Brecklin 2002; Graham, Plant, and Plant 2004; Leonard and Senchak, 1996; Martin and Bachman, 1997). The total economic costs associated with alcohol use are estimated to be more than $235 billion in 2007 dollars (Harwood 2000). The cost of alcohol-related automobile crashes in the United States put the figure at more than $141 billion, including more than $50 billion in monetary costs and almost $90 billion in quality of life losses (Miller, Spicer, and Levy 1999). When taking into account all crimes, those attributed to alcohol or other drugs (AOD) cost society over $263 billion (when taking into account tangible expenses and the value of pain and suffering, and lost quality of life) (Miller et al. 2006). Expenditures by the criminal justice system make up a significant percentage of these costs. Alcohol related motor vehicle accidents and the harms caused by long-term alcohol use (medical costs and lost productivity due to alcohol-related mortality and morbidity) make up about two-thirds of alcohol-related costs (Harwood 2000). The economic costs of alcohol abuse and dependence are not completely internalized by the user, and negative externalities include alcohol-related crime, lost productivity and taxpayer subsidized health care (Simon et al. 2005; Harwood 2000; NIDA 1992; NIAAA 1991). Alcohol abuse and dependence is related to an increased risk of traumatic injury (Woodruff and Baron 1994; Holt et al. 1980; Honkanen and Visuri 1976; Stephens 1985; Rund, Summers and Levin 1981). Long-term health effects of alcohol abuse and alcoholism include alcohol psychosis, cirrhosis of the liver, alcohol poisoning, heart disease or pancreatitis (Solfrizzi et al. 2007; NIAA 2000; Pearson 1996; Monforte et. al 1995). With respect to criminality, the Bureau of Justice Statistics (1988) reported that alcohol use contributed to about 40 percent of violent victimizations (by the offender or the victim). A similar percentage of offenders self-report using alcohol at the time of the offense. The National Highway Traffic Safety Administration (NHTSA) estimates that alcohol-related crashes were associated with more than $50 billion in economic costs in 2000 (Blincoe et al. 2002). The costs of DWI accidents estimated by Miller, Cohen and Wiersma (1996) to be about $6,000 per drunk driving accident, and $18,000 if the pain and suffering is included. Costs associated with alcohol-related accidents vary widely. Miller and colleagues (1996) estimate that the DWI cost totals $71,000 if there is an injury and $2,700 if there is no injury. Costs from alcohol-related accidents in Alaska are estimated to exceed $500 million (Blincoe et al. 2002). While costs associated with drunk 2

13 driving in Alaska are slightly below national averages, Alaskans are estimated to drive while impaired from alcohol consumption more often than residents of any other state, and drive while impaired at almost three times the national average (Liu et al. 1997). It is estimated that Alaska residents spend 2.5 percent of per capita personal income on costs associated with drunk driving accidents (Blincoe et al. 2002). Effectiveness of Alcohol A substantial literature has linked reductions in alcohol-related offending to treatment. Harms from alcohol-related automobile accidents can be minimized by combining treatment with licensing actions (DeYoung 1997; Donovan et al 1988; Donovan et al 1985). Existing research suggests treatment has a small but positive effect on reducing the rate of repeated DUI offenses and involvement in alcohol-related crashes (Wells-Parker 1994). Results from a comprehensive metaanalysis find DUI remediation reduces recidivism by at least 7-9 percent (Wells-Parker et al. 1995). Moreover, treatment has a modest effect on reducing driving under the influence and alcohol-related crashes among offenders who are recipients of mandated intervention (DeYoung 1997; Wells-Parker and Williams 2002). Moreover, diversion to treatment substantially reduces long-term mortality rates (Mann et al. 1994). Economic studies have found there is no relationship between cost and effectiveness across 36 different treatment modalities though outpatient treatment is generally more cost effective than inpatient treatment for most individuals (Finney and Monahan 1996; Dill and Wells-Parker 2006; NIAAA 2000). Criminal Justice System Interventions with Alcohol-Dependent Offenders Given the preponderance of evidence that treatment can be effective, various criminal justice system innovations have emerged to treat drug and alcohol abusing offenders in the community with criminal justice system oversight over the last three decades. The first widespread intervention, Accountability for Safe Communities (TASC), redirected drug offenders from the court system into treatment facilities. TASC provided the link between the judicial system and treatment services and offered participation incentives in the form of case dismissal for successful completion (Nolan 2001). In the late 1980s, a more rigorous program, Intensive Supervision Probation (ISP), was developed to monitor drug offenders in the community as an alternative to incarceration with the goal of reducing prison crowding and providing more thorough supervision than regular probation (Tonry 1990). Similar programs such as those mandated under Proposition 36 passed in California in 2001 and the Drug Offender Sentencing Alternative (DOSA) program in Washington serve large numbers of offenders in several states, and these have generally been found to yield better treatment and criminal justice outcomes (Aos et al. 2005; Longshore et al. 2004). While each of these programs has linkages to the criminal justice system, they did not fully exploit the coercive powers of the criminal justice system to incentivize compliance with treatment protocols. Under the rubric of therapeutic jurisprudence, a more formal model of intensive courtbased supervision, referred to as drug treatment courts, emerged in the 1990s (Hora, Schma and Rosenthal 1999; Senjo and Leip 2001; Slobogin 1995; Wexler and Winick 1991). The therapeutic 3

14 jurisprudence model posits that legal rules and procedures can be used to improve psychosocial outcomes, an idea supported by a growing research consensus that coerced treatment is as effective as voluntary treatment (Anglin et al. 1990; Belenko 1999; Collins and Allison 1983; DeLeon 1988a; DeLeon 1988b; Hubbard et al. 1989; Lawental et al. 1996; Siddall and Conway 1988; Trone and Young 1996). A number of studies have found that drug treatment court participation reduces recidivism rates (Finigan, 1998; Goldkamp and Weiland 1993; Gottfredson and Exum 2002; Harrell and Roman 2002; Jameson and Peterson 1995; Peters and Murrin 2000; Wilson et al. 2006). In response, the model has proliferated, and by 2005 there were about 600 adult drug courts in operation in the United States, including most medium and large counties (Rossman, Zweig, and Roman 2008). The model was originally developed to treat those at risk of drug abuse and dependence, but co-occurring alcohol problems led to the extension of this model to alcohol courts (Cook and Reuter 2007, 1184). Results from a recent study of the Rio Hondo DUI court in Los Angeles County, California found no evidence that the program reduced DUI recidivism and drinking and driving (MacDonald, et al. 2007). However, the absence of statistically significant results may be due to the lack of a credible threat of jail time thereby reducing incentives to comply with treatment (18). A study of the same program found the DUI court was cost-effective only for third-time offenders due to the high costs of jail time (Eibner, et al. 2006). Alternatively, a study of an Oregon program emphasizing intensive supervision and treatment, similar to a DUI court, found statistically significant results suggesting the program reduced recidivism (Lapham, et al. 2006). Based on available evidence on the general crime reducing benefits that accrue by treating drug involved offenders, our research was motivated by the need to provide policy makers some guidance on the prospects of going to scale. That is, what crime reductions can we reasonably expect if more drug involved offenders received treatment? We explain our analytical strategy for exploring this issue next. 4

15 THE RESEARCH DESIGN In this section of the paper, we first describe our data sources and then specify the hypotheses we test in the evaluation. This is followed by a discussion of the baseline characteristics of the full sample, the and Comparison Groups, and of the Opt-In and Opt-Out Groups that comprise the Group. We then turn to the methods to be used in the analysis, first discussing the propensity scores used to balance the samples, and then briefly describe the multivariate models used in the impact evaluation, the survival analysis, the cost-benefit analysis, and the outcome evaluation. DATA Data used to construct our sample and to measure criminal history and recidivism, program costs, and demographic information were provided to us from a number of sources including the Alaska Department of Public Safety, the Alaska Alcohol Safety Action Program and the University of Alaska Anchorage. Criminal History and Recidivism The Alaska Department of Public Safety provided arrest and conviction data for all participants in the evaluation. These data were at the level of arrest event and included the disposition for each event. For the Group, prior arrests were counted if they occurred prior to the date of opt-in to the AWC (see discussion of anchor dates below). For the Comparison Group, prior arrests were counted if they occurred prior to the arrest date of the judicial case that landed them in the Comparison Group. Recidivism arrests and convictions were counted if they occurred after these respective dates depending on the follow-up period. If the follow-up period was two years, then recidivism events were only counted if they occurred within two years of the aforementioned start dates. Recidivism in this study therefore does not include probation or parole violations. Data The Alaska Alcohol Safety Action Program (ASAP) provides substance abuse screening, case management and supervision for individuals with DWI or other alcohol-related misdemeanor court cases. Individuals are referred to ASAP from District Court for an assessment. Once assessed, ASAP links the criminal justice system and health care providers by monitoring cases as individuals enter education and/or treatment programs. 5

16 An extract of the ASAP case management database was provided to the Urban Institute for each individual referred to ASAP by the courts. The Anchorage Wellness Court or the district court referred 190 offenders, of the 298 enrolled in the evaluation, to ASAP for screening and monitoring. The information collected by ASAP and provided to Urban includes demographic data, court case data (including incident information), case management notes and information on treatment and outreach assignments. Data were provided in hierarchical files that were flattened to the personlevel. Cost estimates Cost estimates for treatment and judicial oversight were obtained from semi-structured interviews with individuals that were associated with the court system and treatment providers. Three site visits were conducted in Anchorage, and more than two dozen interviews were conducted. The initial interviews were to inform development of data collection instruments. These interviews solicited information about the types of services provided and whether those services were exclusive to AWC participants. Subsequent interviews gather detailed data about the prices and quantities of services delivered (a detailed discussion of the cost-benefit method can be found later in the report). For the most part, price data were obtained from budgets or expense reports, and data on quantities were obtained from administrative data. Estimates of the amount of court time were developed from direct observations. Cost data on jail/prison stays were provided by the University of Alaska Anchorage. This data included length of stay and type of incarceration for each individual in our study, if applicable. Demographic and sample definition Information on the characteristics of the study s participants was obtained through a database that was maintained by the University of Alaska in Anchorage. This data included all those referred to AWC between 2000 and In addition, a matched sample of individuals who were arrested in the same period, and were eligible for AWC but not referred was used to construct the Comparison Group. Subjects were retained if their data met the following criteria: the date of the arrest or enrollment into AWC occurred early enough for a minimum of a two-year follow-up period to be observed, they had an arrest date, initial hearing date, or date of enrollment that was not missing they had a nonmissing date of enrollment into the AWC if their date of arrest was prior to the year 2000 when the AWC began. These criteria yield a final sample of 277 individuals included in the study. Figure 2 describes the flow of cases into the Anchorage Wellness Court. In total, we identified 277 offenders who were eligible for AWC based on their initial charge, and were matched to AWC participants on several characteristics, including current charge, age, gender and number of prior 6

17 arrests. 7 Of these 277 arrestees, 136 were referred to AWC, and 141 were not. Those who were not referred became the studies comparison group. Those who were referred became the studies treatment group. Of the 136 who were referred to AWC, 91 formally signed a rule 11 agreement the formal plea agreement -- and entered the program. These 91 individual s comprise the Opt-In group. Of those 81 individuals, 44 ultimately graduated from the program and are labeled Opt-In Successes and 47 did not graduate and are labeled as Opt-In Failure. The other 45 of the 136 who were referred to AWC did not formally sign a plea agreement. This group is labeled as the Opt-Out group. Dosik and Cohn (2008) note that those who opted out of AWC may have received substantial services before opting out. They state, A significant lag time sometimes months occurred between the defendant being identified as appropriate for Anchorage Wellness Court and the entry of the defendant s plea agreement the defendant was receiving substance abuse treatment and program services during that time (2008:4). Therefore, it is appropriate to include this group in the Group. Figure 2. Flow of Cases into the Anchorage Wellness Court Eligible DUI Misdemeanors N=277 Referred To AWC? Yes Group N=136 Enrolled? Yes Opt-In Group N=91 Graduated? Yes Opt-In Success N=44 No No No Comparison Group Opt-Out Group Opt-In Failure N=141 N=45 N=47 Source: urban Institute Analysis of program data. HYPOTHESES In order to evaluate the Alaska Wellness Court s effectiveness in improving public safety we tested the following hypotheses: (1) Those who were referred to AWC (the Group) will have lower odds of any rearrest and reconviction than the Comparison Group; 7 Due to limited data availability at the time the match was performed, the match was based on categorical variables. Thus, for example, an offender entering AWC may be matched to an individual who was not referred based on matches across each of these variables. However, a match would occur if both individuals had two or more prior arrests. Since the mean number of prior arrests was much greater than 2, significant differences remained between the groups, even after the matching was used to select the cohorts. Thus, additional controls for selection were employed in this analysis. 7

18 (2) Those who were referred to AWC and enrolled in the program (the Opt-In Group) will have lower odds of any rearrest and reconviction than the Comparison Group and the Opt-Out Group. (3) Those who were referred to AWC (the Group) will have fewer rearrests and reconvictions than the Comparison Group. (4) Those who were referred to AWC and enrolled in the program (the Opt-In Group) will fewer rearrests and reconvictions than the Comparison Group and the Opt-Out Group. (5) Those who were referred to AWC (the Group) will have a larger benefit to cost ratio that the Comparison Group. (6) Those who were referred to AWC and enrolled in the program (the Opt-In Group) will have a larger benefit to cost ratio than the Comparison Group and the Opt-Out Group. (7) More days will pass before those who were referred to AWC (the Group) will be rearrested and reconvicted than the Comparison Group. (8) More days will pass before the Opt-In Group will be rearrested and reconvicted than the Comparison Group and the Opt-Out Group. Each of these outcomes captures a different dimension of recidivism, where each hypothesis tests a different dimension of the general hypothesis that the AWC will reduce recidivism. Sample Characteristics The sample is comprised of arrestees who enter the Anchorage Wellness Court (n=136), and a matched sample of comparisons (n=141) who resemble those who enrolled in AWC, but who did not participate in the program. The last column on the right of Table 1 describes the characteristics of the full sample (n=277). Overall, the sample is older than the population of arrestee s entering the criminal justice system with an average age of almost 39. The sample also includes a higher percentage of women (29 percent) then the criminal justice system as a whole. Whites are about half the sample (51 percent) and Alaska natives (29 percent) and American Indians (12 percent) are the largest minority groups. The sample has substantial prior criminal justice system contact, averaging about 10 prior arrests. While public order (society) offenses, which include prior DWI/DUI offenses accounts for about half of these prior arrests, sample members also average about two prior violent arrests and almost three prior arrests for a property crime. Sub-Groups The Group includes several heterogeneous groups. The Group includes (1) those who actually participated in the AWC (i.e. the Opt-In Group) and (2) individuals who initially intended to participate in the AWC but opted out shortly thereafter (i.e. the Opt-Out Group). For most of this report, we concentrate on the differences between the full Group and the comparison, and the differences between these two sub-groups of treatment clients and the comparison. It is worth noting that within the Opt-In Group there are two additional sub-groups: 8

19 (3) those who graduated from AWC (i.e. treatment successes) and (4) those who did not graduate (i.e. treatment failures). We describe the characteristics of each of these groups below, again referring to Table 1. In general, each of these sub-groups was similar to the pooled Group in that there had significantly more prior arrests, had a significantly greater percentage of Alaska Natives in the group then were observed in the comparison, and had a significantly smaller percentage of American Indians in the group then were observed in the comparison. The ethnic composition of all four treatment sub-groups was almost identical to the full group. Sub-group 1 The Opt-In Group The Opt-In Group (n=91) were slightly, though not significantly, older than the average individual in the Group. While the number of prior arrests was very similar, it was distributed slightly differently across crime types. The biggest differences were that there were fewer prior property arrests and more public order (society) arrests. Sub-group 2 The Opt-Out Group Among the four sub-groups of treatment participants, those who initially attended AWC but chose to drop out (n=45) before formerly enrolling in the program looked least like the full treatment sample. Those who opted out of AWC were about three and a half years younger than the full treatment sample, and significantly younger than the Comparison Group. With respect to prior criminal history, the treatment opt-outs had fewer prior violent arrests, more prior property arrests, and fewer public order (society) arrests. Sub-group 3 successes (graduates) Among the four sub-groups of treatment participants, those who successfully completed AWC (n=44) had the highest average age. This sub-group also had the fewest prior arrests. Within arrest categories, treatment successes had the fewest past arrests for violence, property crimes, and drugs. Despite having the fewest total prior arrests, this sub-group had the largest number of prior arrests for a public order (society) crime, and almost half of prior arrests were in this category. Sub-group 4 failures The sub-group of those who entered but ultimately failed AWC is slightly younger than the average of the full Group. This sub-group had the most prior arrests of any of the four subgroups, and the most prior arrests for drug and violent crimes. 9

20 Table 1. Unweighted sample characteristics Group Opt-In Successes Failures Opt-Out Comparison Group Entire Sample Age (years) * * (8.94) (7.90) (6.98) (8.54) (10.31) (8.94) (8.67) Male (%) (0.46) (0.46) (0.47) (0.46) (0.46) (0.45) (0.45) Alaskan native (%) 0.38*** 0.38*** * 0.36** (0.49) (0.49) (0.50) (0.48) (0.48) (0.41) (0.45) American indian (%) 0.06*** 0.05*** 0.05*** 0.06** 0.07** (0.24) (0.23) (0.21) (0.25) (0.25) (0.38) (0.32) White (%) (0.50) (0.50) (0.50) (0.51) (0.51) (0.50) (0.50) Other race (%) (0.30) (0.31) (0.32) (0.31) (0.25) (0.27) (0.28) Number of prior arrests 13.96*** 14.02*** 13.09*** 14.89*** 13.82* (19.60) (14.14) (14.51) (13.89) (27.75) (7.74) (15.32) Number of prior violent arrests 2.65*** 2.84*** *** 2.29*** (3.96) (4.43) (3.65) (4.95) (2.81) (2.08) (3.17) Number of prior property arrests 3.86** 3.12** * (11.48) (7.06) (8.13) (5.99) (17.30) (3.04) (8.40) Number of prior drug arrests 0.56** 0.68*** *** (1.01) (1.14) (0.90) (1.32) (0.60) (0.72) (0.88) Number of prior society arrests 6.88*** 7.38*** 7.64*** 7.15*** 5.87* (7.43) (6.01) (6.29) (5.79) (9.69) (4.28) (6.33) N Significance testing is based on independent sample t-tests comparing each treatment group to the comparison group. Significance: * p < 0.10, ** p < 0.05, *** p < 0.01 In Table 1, the Group (the first column) can be compared to the Comparison Group (in the second column from the right) along each of eleven demographic variables to generate an initial assessment of how well the two groups are balanced. If the means of the Group are significantly different than the means of the Comparison Group, there is the potential for selection effects which confound interpretation of the impact of AWC. If the two groups differ on characteristics that may have an independent effect on outcomes. For example, as shown in Table 1, the Group averages more than twice as many prior arrests. Because the main outcome is recidivism, these differences in baseline characteristics suggest that the Group is at much higher risk of a new arrest (since past criminality is the best predictor of future criminality). In addition, the Group averages more prior arrests in each of the four arrest categories. The Group also has a higher percentage of Alaska Natives and a lower percentage of American Indians than the Comparison Group. In order to minimize selection bias, it is important to have treatment and Comparison Groups that are as similar as possible along all characteristics, except for the variable which is being manipulated (which in this case is whether an individual received AWC). We return to the subject of selection below. Some possible associations between characteristics at baseline and an individual s ultimate disposition can be inferred from these data, although there is no way to independently verify this hypothesis. For instance, those with the high percentage of public order (society) crimes were most 10

21 likely to be in a group that succeeded. For instance, 58 percent of prior crimes for the graduate subgroup were public order, compared to only 42 percent for treatment opt-outs and 48 percent for treatment failures. This may be due to a high correlation between the severity of the individuals alcohol problem and the number of prior public order arrests (which include DUI/DWI arrests). These differences suggest that this group may have had less severe alcohol problems and chose not to opt-in as a result, and those with the most severe alcohol problems opted-in. Creating Anchor Dates In order to capture a consistent date that sample members were enrolled into the evaluation cohort, an anchor date was assigned for each observation in the sample.. The date of enrollment into the sample was measured slightly differently for different sub-groups (Table 2). For the Comparison Group, the date of the arrest that made them eligible for inclusion in the study was used as the starting date for the purpose of observing and counting new arrests. For the treatment sub-group of opt-outs, the date they first appeared in Wellness Court was used as the starting date if such a date was available, and arrest date was used if no starting date was observed. For the Opt-In Group treatment sub-group, the date of first appearance at AWC was used as the starting date. The different dates were used so that only new arrests after any AWC treatment was received were counted. For many in the Group, there was often an additional arrest after the arrest that was associated with AWC participation. That is, a client was arrested, was in the court system awaiting disposition for that charge, but had not participated in AWC in any manner, and was subsequently rearrested before enrolling in AWC. Since no AWC treatment had been received at the time of the second arrest, we did not count that arrest as a post-awc recidivism event. The enrollment period for Comparison Group observations extends beyond the time of AWC operation, since a sufficient number of appropriate comparisons could not be found within the period of AWC operation. Table 2. Number of AWC Group members enrolled from Opt-In Successes Failures Year Opt-outs % 14% 29% 27% 22% 8% 0% 20% 27% 25% 25% 2% 0% 9% 30% 30% 19% 13% 2% 7% 40% 11% 13% 27% 1% 12% 32% 22% 19% 14% Sample Size

22 METHODS Selection Models A common problem in nonrandom experiments is that individuals in the Group may differ systematically from individuals in the Comparison Group. If those systematic differences are related to outcomes (rearrest, reconviction) and are not observable in the covariates of a multivariate model, then the observed program impact is biased. For instance, if one group systematically differs on a characteristic that makes members of that group less likely to recidivate (such as motivation to get a job or desist from crime), that group would be expected to have better outcomes whether or not the group received the intervention. Since the Group has significantly more prior criminal justice system contact, it is reasonable to presume that unless those differences are controlled for in the modeling of outcomes, the Group will do worse than the comparison on measured of recidivism, all else being equal. One approach to control for these selection effects is to run propensity score models to balance the samples, to the extent possible, ex ante. This approach parses out (1) the independent effect of control variables (such as age and prior criminal history) on assignment to the treatment or control group from (2) the independent effect of those controls on the outcomes. In doing so, they allow for weights to be constructed that balance the two groups using information from (1). If the model in (1) has a lot of predictive power, then the weighted observations will resemble an experimental design. In order to account for the effect of self-selection into AWC in the multivariate outcome models, inverse weights were generated using the fitted predicted values from a logistic regression model on treatment. observations were weighted according to their inverse probability of being in the Group and comparison observations were weighted according to their inverse probability of being in the Comparison Group. That is, the treatment observations were assigned a weight of one divided by the predicted probability of being in the Group. Thus, if a treatment observation had a low predicted probability of being in the Group and thus had attributes similar to the Comparison Group then that observation was given a higher weight. Conversely, if the available information predicted that a treatment observation would have been in the Group then that observation was given a low weight. The same approach was used to weight the Comparison Group. Each observation was weighted as one divided by the predicted probability of being in the Comparison Group. Thus, comparison observations that were predicted by the model to be similar to treatment observations would be given the largest weights. For those Comparison Group observations that had high predicted probabilities of being in the Comparison Group, a very small weight is assigned, so those observations would count less. A complete description of this processed can be found in Appendix IV. 12