Evaluation of Substance Abuse Treatment Outcomes

Transcription

1 Evaluation of Substance Abuse Outcomes Office of Standards and Evaluation March 1998 Lawton Chiles Governor Edward A. Feaver Secretary

2 The Department of Children and Families and the Office of Standards and Evaluation would like to acknowledge the clients and providers who participated in this evaluation and made it possible. For reprints of this report, please contact: FL Department of Children and Families Office of Standards and Evaluation (OSEV) 1317 Winewood Blvd. Bldg. 1, Room 306 Tallahassee, FL For additional information contact: Nina B. Barrios, Ph.D., Director Glenn Mitchell, Ph.D., Sr. Mgt. Analyst (850) File reference: OSEV98_2

3 Executive Summary of Department of Children and Families Evaluation of Substance Abuse Outcomes The Office of Standards and Evaluation of the Department of Children and Families evaluated substance abuse treatment outcomes to address questions of program effectiveness and assess the validity and reliability of measures used for performance budgeting. The office assessed client outcomes six months after treatment using both a telephone survey of clients and state wage data. The evaluation compared outcomes of two groups of clients, those who completed treatment free from drug and alcohol use and clients who entered treatment with the same problems, but did not complete it. The study also analyzed information on all clients and services in to identify factors related to outcomes and assess data quality, using the program s State Integrated Substance Abuse Reports (SISAR) database. Findings of the Evaluation Florida s alcohol and drug treatment programs produce positive outcomes for clients after they leave treatment. Six months after discharge, clients who complete treatment have fewer substance abuse problems and better employment than clients who leave before completing treatment. Comparison of outcomes for completers versus non-completers validates the importance of the current performance measure of drug free completion of treatment. $114.7 million was appropriated for FY , with which the department contracted with 135 agencies. Services such as residential and outpatient treatment, detoxification, intervention assessment and referral, and prevention were provided to 136,300 adults and 62,000 children in FY Primary referral sources for clients are criminal justice and juvenile justice. Clients have different patterns of substance abuse and outcomes. Alcohol is the most common substance abuse problem for adults and marijuana is most common for adolescents. Clients abusing crack/cocaine have the lowest treatment success. Most substance abusers are male. Compared to other substances, a higher proportion of clients abusing crack/cocaine are female. The data system used for performance reporting provides useful information on client characteristics and services. Due to problems with reporting discharges, the system lacks outcome information on many clients. To correct data problems, the program is working with providers and districts to implement data validation and simplify reporting. Performance measures have not been developed for services such as detoxification and intervention. Subgroups used to report populations served have substantial overlap, so information about characteristics of clients is lost. Summary of Recommendations: The substance abuse program should: Implement systematic follow-up of clients after treatment to determine instances of drug-free behavior and employment. Maintain use of completion of treatment as a performance measure. Use quality improvement processes to increase client completion rate. Develop performance measures for management and accountability of all primary services and evaluate the current subgroup reporting structure to better communicate client characteristics and service needs. Continue to improve data quality and establish measurable objectives for improvements in timeliness, completeness and accuracy. Substance Abuse Evaluation Page i

4

5 The Office of Standards and Evaluation of the Department of Children and Families evaluated substance abuse treatment outcomes to address questions of program effectiveness and assess the validity and reliability of measures used for performance based budgeting. The Legislature and departmental senior management questioned whether measures of completion of treatment reflected program effectiveness. They also questioned the reliability of data, given the low number of clients with reported outcomes. PURPOSE OF THIS EVALUATION The department s Office of Standards and Evaluation prepared an evaluation study to assess client outcomes six months after treatment. The study compared the outcomes of two groups of clients using both a telephone survey of clients and independent employment data. One group was clients who completed treatment free from drug and alcohol use. The comparison group was clients who entered treatment with the same problems, but did not complete it. The study also analyzed information on all clients served by substance abuse programs in , to provide descriptive information and identify factors related to outcomes. The primary source of these data is the program State Integrated Substance Abuse Reports (SISAR) database of client admission and discharge reports from each component of service. Results from analysis of this information assess the quality of the data and the utility for performance management. Statutory authority for the substance abuse services program is Chapters 20, 394, and 397, Florida Statutes. The legislative intent of the program is to provide comprehensive prevention, intervention and treatment services to meet the needs of individuals and reduce consequences to Florida of substance abuse. The program is to provide services primarily through community-based, private not-for-profit providers (Chapter 397, F.S.). The program serves two groups: adults and children under age 18. The department s goals for the program are for adults with substance abuse problems to be drug free and economically self-sufficient, and for children with, or at risk of, substance abuse problems to be drug free. The program reports over 136,300 adults and 62,000 children under age 18 were served in Fiscal Year The major types of services provided by the program are: BACKGROUND: THE SUBSTANCE ABUSE PROGRAM AND ITS RESOURCES The Program Provides Comprehensive Services Through Private Not-For-Profit Providers Goals are for Clients to be Drug Free and Economically Self-sufficient 1 In this evaluation, analysis of client characteristics and services are based on admissions of 71,000 adults and 41,000 children during FY The number served includes clients who are not yet discharged. Because of the low reporting of discharges, the reported number served may include clients never discharged and not receiving services. Annual admission data was determined to better reflect client characteristics and services. Substance Abuse Evaluation Page 1

6 Non-residential treatment: Includes assessment, counseling, family therapy and substance abuse education in outpatient and day treatment settings. Residential treatment: Assessment, counseling, rehabilitation, family therapy and education services in an intensive, 24 hour, therapeutic environment. Residential detoxification and stabilization services:provide for stabilization and withdrawal from substance abuse dependence. Intervention services: Assessment, short-term counseling, referral to treatment when needed, and case management. Prevention services: Targeted prevention: For children at risk of substance abuse, including children of drug abusers and poor school achievers. This includes classroom-based activities in schools to increase achievement and reduce substance abuse risk in grades 4 through 8. Primary prevention: Information and education for the general population to increase public awareness and reduce the incidence of substance abuse in the community. Licensing: In addition to funding direct services, the department licenses all providers of substance abuse treatment services, both public and private. The Department of Children and Families contracts with 135 community substance abuse agencies to provide services. The department s 15 service districts manage contracts with providers. Statewide, there are 78 district and central office positions to manage the system of services in FY Funds are not appropriated for staff to provide direct services. In addition to managing contracted services, administrative staff are responsible for routine monitoring and licensure activities. $114.7 Million was Appropriated for FY The Legislature appropriated $114.7 million to the Department of Children and Families for substance abuse services, up $3.2 million from Sources and dispositions of funds include: Program Appropriation for Services in Millions Adults Children Total General Revenue $26.5 $27.3 $53.8 Trust Funds* $40.5 $20.4 $60.9 Total $67.0 $47.7 $ Adults Children Total General Revenue $28.3 $22.0 $50.3 Trust Funds* ** $39.8 $21.4 $61.2 Total $68.1 $43.4 $111.5 * Includes Federal funding ** Excludes $3.5 unfunded budget Page 2 Substance Abuse Evaluation

7 Funding for substance abuse services gradually increased during the 1980s and then decreased through the 1990s. There were large increases in and with increased Federal Block Grant funds and the state trust fund for children and adolescents established by the Wheels Bill. Budget reductions in the 1990s have resulted in cuts in four of the last seven years, so that current funding is lower than Federal funds provide almost 50% of the Legislature s appropriation. According to the US Government Accounting Office, combined Federal, state and local public resources represent the majority of funding available for substance abuse treatment and prevention. Nationwide estimates based on limited private sector data indicate private insurance and client fees may have accounted for as low as 15% of resources for treatment and prevention in 1993 (GAO, 1996) i. To implement requirements of Florida s Government Performance and Accountability Act of 1994, the department developed performance measures of outcomes for clients of the substance abuse treatment programs. The Legislature approved the measures in the General Appropriations Act. Both the Legislature and the department continue to refine the measures for budgeting and contract management. Measurable Outcomes are Required for Performance Based Budgeting Performance Measures Approved by the Legislature for : For Adults With Substance Abuse Problems: Percent of discharges completing treatment with no alcohol or other drug use during the month prior to discharge. Percent reduction in frequency of substance use for unsuccessful discharges during the month prior to discharge compared to the month prior to admission. Percent of clients successfully completing treatment who are readmitted for substance abuse services during the 12 months following discharge. Percent of clients employed at discharge from substance abuse treatment services. Percent of women pregnant during substance abuse treatment who give birth to substance free newborns. Percent of pregnant women receiving substance abuse treatment who deliver infants with normal birth weights. Change in percent of clients with arrests within 90 days following discharge compared to arrests 90 days prior to admission. Average level of satisfaction on the Behavioral Healthcare Rating of Satisfaction scale. Percent of community partners satisfied based on survey. Substance Abuse Evaluation Page 3

8 For Children Abusing Substances: Percent of discharges completing treatment with no alcohol or drug use during the month prior to discharge. Percent reduction in frequency of substance use by those who do not successfully complete treatment. Percent of clients successfully completing treatment who are readmitted for substance abuse treatment services during the 12 months following discharge. Percent of children committed or recommitted to Juvenile Justice during the 12 months following treatment completion. Percent of families reporting average or above level of satisfaction on the Family Centered Behavior Scale for parents of children receiving treatment. Percent of clients satisfied. Percent of community partners satisfied based upon survey. For Children At Risk of Substance Abuse: Percentage of children in targeted prevention programs who achieve expected level of improvement in math. Percentage of children in targeted prevention programs who achieve expected level of improvement in reading. Percentage of children in targeted prevention programs who are admitted to substance abuse services during the 12 months following completion of prevention services. Percentage of children in targeted prevention programs who perceive substance use to be harmful at the time of discharge when compared to admission. Percentage of children in the general population who perceive substance use as harmful. Percent reduction in monthly or more use of alcohol and other drugs by middle and high school students as measured on a statewide survey of students. Proposed Changes In Measures Include Six Month Follow-up The department has recommended a new measure for both adults and children for FY based on experience with the first year of budgeting and management with these measures. For increased validity, the percentage drug free for six months following completion of treatment will replace the current measures of completion of treatment, which will be reported as output measures. Because of questions of validity and reliability, the department will defer until the two measures of children s perceptions and use of substances based on a survey of the general population. Page 4 Substance Abuse Evaluation

9 With increased government accountability and the shift to managed care systems for behavioral health care, substance abuse treatment nationwide is beginning to use measurable indicators to demonstrate effectiveness and improve client outcomes. National studies demonstrate the cost benefit of treatment to society. reduces criminal activity, alcohol and drug use, and health costs, and increases employment (CALDATA 1994; NTIES, 1996; Oregon, 1966; DATOS, 1997) ii. These studies support Florida s use of measures of substance use, employment, arrests and other desired outcomes for performance management. Florida s Measures Reflect National Findings of Effects The consensus of existing research and clinical experience is that alcohol and drug abuse and dependence are complex disorders involving biological, psychological and social factors ( Protocol Effectiveness Study by Office of National Drug Control Policy, 1996). Substance abuse is generally a chronic, long term condition, not a brief episode. Abusers tend to relapse, especially during high risk events in their lives. People referred by the criminal justice system or employers may be in denial about their problems. Their attempts to become drug free may fail several times. Hard-core drug users, such as cocaine users involved in criminal activity, are difficult to reach. At the current level of development of substance abuse treatment, it continues to be difficult to identify the most effective type of treatment and for whom it might be most effective. What intensive studies show is that regardless of the substance abused or the treatment provided, treatment works when: those who abuse drugs are engaged in their treatment; they remain in treatment to completion; and other needed services such as job training and placement, housing, health care and education, are integrated with drug treatment to help clients resolve problems that accompany their drug use. I. How effective is substance abuse treatment in Florida? FINDINGS Substance abuse treatment results in long term benefits of reduced substance abuse and other problems, and better employment. Substance Abuse Evaluation Page 5

10 I. A. Adult Substance Use Six Months After Clients Were Surveyed Six Months After The study surveyed a random sample of adult clients who completed treatment, and clients who did not, to assess the effectiveness of treatment from their perspectives and to learn whether treatment resulted in reduced substance abuse problems. The survey asked former clients general questions about their conditions since leaving treatment. The questions included whether clients lives and personal relationships were better or worse six months after treatment, and whether they were employed. Clients were asked if they had a problem with substance abuse in the last 30 days. The question was worded to prevent self incrimination. When asked directly about frequency and amount of substance use, fewer persons indicated a problem. Answers about substance abuse problems were consistent with those to questions about life, relationships, and employment. The survey also asked former clients if they were satisfied with services, and whether they had suggestions to improve services. Any methodology involving self report has the potential for bias. However, studies have found a close relationship between self reported drug use and drug tests (NTIES, 1996). In addition, use of a comparison group controls for potential bias in self reported drug use since it is the difference between the groups, not the absolute level of response, that is examined. Clients Who Completed Reported Fewer Substance Abuse Problems and Better Employment Outcomes Findings of the survey: Both adults who completed treatment and those who did not report they were treated with courtesy (93%) and satisfied with services (81%). Six months after leaving treatment, less than half as many clients who completed treatment reported having a substance abuse problem as clients who did not complete treatment. Six months later, 11% of completers reported they had a substance abuse problem in the last 30 days compared to 26% of non-completers. Only 3% of completers said they had a major problem with substance abuse within the last 30 days compared to 9% of noncompleters. More than three-fourths (77%) of clients who completed treatment reported being employed compared to 58% of non-completers. About two-thirds (66%) of completers said they were employed fulltime (35 or more hours per week), compared to half (51%) of noncompleters. Only one percent of clients who completed treatment said their lives were more difficult compared to seven percent of clients who did not complete treatment. Page 6 Substance Abuse Evaluation

11 Half as many clients who successfully completed treatment (6%) said they had re-entered a treatment program, compared to clients who did not complete (12%). 100% Self Reported Substance Abuse Problems Six Months Following, by Completion Status 80% 60% 40% 20% 0% 11% Completed Drug Free 26% Did Not Complete (Noncompliant or left) Some or Major Substance Abuse Problem in Last 30 Days 100% 80% 60% 40% 20% 0% Self Reported Employment Six Months Following, by Completion Status 77% Completed Drug Free 58% Did Not Complete (Noncompliant or left) * Source: Telephone survey of adult clients. n=148 Employed Full or Part Time These differences in substance abuse problems, employment, and other outcomes six months following treatment provide evidence that substance abuse treatment is effective for clients who complete treatment. These survey findings are supported by state wage data and other information examined in this evaluation. In addition to providing information about client outcomes, the survey served as a pilot test of a proposed follow-up measure for performance budgeting. The lessons of the pilot suggest that such follow-up of substance abuse clients is difficult. All existing records and analysis use only identification numbers to ensure confidentiality. The department had to obtain names and telephone numbers of clients in the sample from providers to be able to contact respondents after treatment. Some clients move frequently and do not want to have contact with the treatment system. Substance Abuse Evaluation Page 7

12 Survey Results of Completers and Noncompleters were Compared The findings reported above are from a comparison group survey. The department contracted with the Florida International University Institute for Public Opinion Research (IPOR) to administer a short telephone survey to a random sample of adult completers and non-completers of treatment. The department chose a telephone survey over other options to provide objectivity. Findings from a carefully constructed sample are representative because the sample reflects characteristics of the population already known from department data. The sample included adults discharged from residential treatment and outpatient treatment between April and June was for alcohol, cocaine and marijuana use. The sample included an equal number of completers and non-completers from each provider to control for variations in treatment. The telephone survey resulted in information from 148 people (91 completers and 57 non-completers) out of a random sample of 780 adults. The primary reasons surveys were not completed: 22% were wrong numbers. 16% had no answer after 6 attempts. 12% phones were not in service. 8% refused to participate. Other reasons surveys were not completed include phone numbers that were for businesses, beepers and voice mail. To protect client confidentiality, the surveyors did not use these numbers. We did not survey children for two reasons. Children cannot legally give consent to answer confidential substance use questions over the phone. Also, it would be difficult to get sufficient participation because of the small number of children in residential or outpatient treatment. I. B. Adult Employment Six Months After State Wage Data Shows Higher Employment for Clients Who Complete To assess client outcomes after treatment, employment was also examined using state wage data. Employment outcomes were compared for the 3,263 completers and 2,763 non-completers discharged from treatment in the first quarter of Wage data support findings of the telephone survey. The relative difference between the completers and non-completers is in the same direction and of a similar amount. Both sources of information indicate the program is achieving the desired outcome of increased employment. Because the employment information from the Florida Education and Training Placement Information Program (FETPIP) includes all clients, not just those able to be reached by telephone, it indicates a lower rate of employment overall. The employment rate is 29% less for non-completers than for completers. State wage data indicate that six months later 49% of the completers successfully discharged from treatment in the first quarter Page 8 Substance Abuse Evaluation

13 were employed in the third quarter of 96-97, compared to 35% of noncompleters. Earnings of those employed in both groups differed by 35%. Completers who were employed earned on average $3,663 during the quarter compared to $2,391 earning during the quarter by noncompleters. Full-time minimum wage earnings per quarter at the time were $2, % Percent of Adults Employed Six Months Following, by Completion Status 80% 60% 40% 20% 0% 49% Completed Drug Free 35% Did Not Complete (Non-compliant or left) Average Quarterly Earnings of Adults Six Months Following, by Completion Status $5,000 $4,000 $3,000 $2,000 $1,000 $0 $3,663 Completed Drug Free $2,391 Did Not Complete (Non-compliant or left) * Full-time quarterly minimum wage earnings at the time were $2,375. ** Source: FETPIP Employment Data 1-3/97. n=6,025 The findings support Department use of drug free completion of treatment as a program performance measure. Use of these data demonstrates the feasibility of obtaining follow-up employment outcomes from FETPIP on a routine basis. The findings reported above resulted from comparison of two groups who entered treatment with similar problems. One group had the full benefit of treatment and the other did not. We compared 3,263 clients who completed treatment and were discharged drug free to 2,763 clients who left before completing treatment or were discharged for noncompliance. Employment after treatment cannot be compared to employment before treatment because of differences in criteria and data between the databases used. Employment Data for Completers and Non- Completers were Compared Substance Abuse Evaluation Page 9

14 To assess employment outcomes, we matched clients discharged from treatment July-September 1996, to Florida Education and Training Placement Information Program (FETPIP) employment data for January-March We compared employment rates of all 6,025 completers and non-completers reported. FETPIP obtains the employment data from the Department of Labor and Employment Security s Unemployment Insurance wage record data to assess the outcomes of vocational education and training programs. The data are unduplicated, cleaned and validated by FETPIP and are available for analysis. II. Who are our clients and how well do we serve them? Clients abuse primarily alcohol, crack/cocaine or marijuana depending on their age and sex. Most adults are males, age 25 to 44, who abuse alcohol or crack/cocaine. Most children are high school males who abuse marijuana. Clients with alcohol problems have higher rates of successful treatment. Clients with crack or cocaine problems tend to receive more intensive and expensive residential treatment and have lower success rates. Patterns of substance abuse are as varied as the people of Florida. The primary drugs abused by those served are alcohol, crack or cocaine, and marijuana. While the predominant substance abuse problem continues to be alcohol, substance abuse patterns continue to change as the availability of substances change. In , heroin was the major new problem in the country. In , the problem was heroin and psychoactive drugs. In the 1990s, cocaine became the most significant new substance abuse problem. Drug abuse patterns vary by age and sex. Adults and children have different substance abuse problems and legal status, and require different treatment approaches. Page 10 Substance Abuse Evaluation

15 II. A. Adults Served The program reports over 136,300 adults were served in The analysis of characteristics of clients and the services they receive is based on 71,095 admitted to services during the year. 2 Non-residential and residential treatment Comprise the largest expense of substance abuse services provided to clients. In providers receiving state funds admitted from one to 8,452 clients. Most clients (63%) received non-residential or residential treatment that represented 69% of expenditures. Forty-seven percent received non-residential treatment. Twenty-nine percent of admissions were for intervention services, primarily Alternatives to Street Crime (TASC) for males and community intervention for females. Intervention includes assessment, brief counseling and referral to treatment when appropriate. 71,000 Adults Entered Services in with $68.1 Million in Expenditures Expenditures and Adults Admitted by Service, Services Expenditures (Millions) Admitted Amount Percent Number Percent Non-residential $ % 32,955 47% Residential $ % 11,485 16% Detoxification $ % 4,875 7% Intervention (assessment & $5.6 8% 20,692 29% referral, & case management) Prevention $3.0 4% 134 0% Special Overlay $0.1 0% 954 1% Other $0.1 0% Total $ % 71, % * Source: Department cost center allocations and SISAR client information. The main referral source into substance abuse services is the criminal justice system (57%), primarily from probation and parole and driving under the influence (DUI). Twenty-six percent of admissions are self referrals. Generally adult clients do not have a pattern of multiple arrests prior to treatment. About one-fourth had two or more arrests prior to treatment while three-fourths had one arrest (39%) or none (35%). Adults Primarily Enter Services From Criminal Justice or Self Referral 2 The program reports 136,325 adults were served in , with 79,248 receiving non-residential treatment and 15,174 receiving residential treatment. This report uses admission counts rather than number served for analysis of client characteristics and services. The number served is calculated to include those admitted prior to the year and not yet discharged. Substance Abuse Evaluation Page 11

16 Referrals of Adults to Substance Abuse Services Criminal Justice (Inc. Probation, DUI and Other Court and Law Enforcement) 57% 40,726 Self Referral 26% 18,314 Cases Admitted in FY (n=71,095) Community Social Service Agencies 4% 3,009 Dept. Children & Families Family Safety Preserv. 3% 2,098 Other Referrals (Inc. SubAbuse MHealth, Employers & Unk) 10% 6,948 Intervention (Assessment, Counseling, Referral, & Case Management) 29% 20,692 Detox 7% 4,875 Other 1% 1,088 Non-Residential Outpatient & Day 47% 32,955 Residential 16% 11,485 * Source: Dept. of Children and Families SISAR Admissions ** Numbers are admissions, not total number served. The program reports 136,300 served, including 79,248 in non-residential and 15,174 in residential treatment. Adults Primarily Abuse Alcohol, Crack/Cocaine and Marijuana The three primary substances abused by clients receiving services are alcohol, crack/cocaine and marijuana. Predominant Substances Abused Adults Admitted to All Services, Alcohol and Drug Use Number Admitted Percent Alcohol 26,966 38% Crack/Cocaine 19,199 27% Marijuana/Hashish 9,954 14% Heroin 1,548 2% Other Opiates 908 1% Benzodiazepine & Other 361 1% Tranquilizers & Sedatives Methamphetamine & other 334 0% Amphetamines Hallucinogens 89 0% Other Drugs 139 0% Unknown/Denies 2,934 4% None (inc. spouse receiving support) 3,379 5% No Data 5,284 7% Total 71, % * Source: Dept. of Children and Families SISAR Admissions Page 12 Substance Abuse Evaluation

17 The majority of both alcohol and marijuana users admitted to treatment are male (76% and 80% respectively). A higher percentage of crack/cocaine users are female (37%) than for other drugs. Primary Substances Abused by Sex Adults Admitted to All Services, Most Clients are Males Crack/Cocaine is the Drug with the Largest Proportion of Female Clients Major Substances Total Number Males Percent Females Percent Alcohol 26,966 76% 24% Crack/cocaine 19,199 63% 37% Marijuana 9,954 80% 19% Heroin 1,548 68% 32% Total 57,667 72% 28% * Source: Dept. of Children and Families SISAR Admissions Marijuana use is most common among young adults. Alcohol is most common among older clients. The age group with the highest percentage of crack/cocaine use is Primary Substances Abused by Age Group, Adults Admitted to All Services, Age Total Number Alcohol Crack/ Cocaine Marijuana Heroin ,138 32% 23% 43% 2% ,968 42% 41% 15% 2% ,406 52% 36% 9% 3% 45+ 7,155 73% 19% 5% 3% Total 57,667 47% 33% 17% 3% * Source: Dept. of Children and Families SISAR Admissions Substance Abuse Varies by Age Group While the majority of clients are in outpatient, non-residential treatment, a high percentage of crack/cocaine and heroin users are in more intensive and more costly residential treatment. Crack/cocaine users are the second largest group of clients who enter treatment. They have the worst characteristics for successful completion and long term outcomes. The majority, 78%, use crack. Clients using crack/cocaine are admitted with higher rates of multiple arrests (35% two or more) and prior admissions (67%), and lower employment rates, than other clients. Most Clients Receive Outpatient : A Higher Percentage of Crack/Cocaine Abusers Receive More Intensive and Costly Residential Substance Abuse Evaluation Page 13

18 Type of by Primary Substances Abused Adults Admitted to, Major Substances Admitted to Residential Non-residential Number Percent Percent Alcohol 18,490 19% 81% Crack/cocaine 14,371 39% 61% Marijuana 6,777 18% 82% Heroin % 51% Total 40,322 27% 73% * Source: Dept. of Children and Families SISAR Admissions Abusers of Alcohol Have Higher Success Rates than Abusers of Crack/Cocaine Differences in a client s drug use and source of referral to services have the strongest relation to differences in successful completion of treatment. is most successful for clients with alcohol problems. Comparing the ratio of successful completers to unsuccessful discharges, as used for the follow-up studies, 64% of those in alcohol treatment are successful, compared to 45% of crack/cocaine users. Ratio of Successful Completion by Primary Substances Abused Adults Discharged from, Total Successful & Unsuccessful Successful: Completed Drug Free Unsuccessful: Discharged Non-Compliant or Left Major Substances Alcohol 7,023 64% 36% Crack/cocaine 5,469 45% 55% Marijuana 2,587 52% 48% Heroin % 48% Total ** 15, % 45% * Source: Dept. of Children and Families SISAR Discharges ** Not including 548 completing treatment not drug free and 5851 discharges to other services, deaths, etc. Does not include adults in treatment for other drugs. Clients referred from criminal justice sources such as probation or DUI have a higher ratio of successful completion of treatment (65%) compared to self referrals (43%). For the largest number of clients, those in treatment for alcohol problems, residential and non-residential treatment have similar success rates. This is not true for crack/cocaine treatment, where the ratio of success for non-residential treatment is 35% successful to 65% unsuccessful, compared to a ratio for residential treatment of 57% successful to 43% unsuccessful. Page 14 Substance Abuse Evaluation

19 Ratio of Successful Completion by and Primary Substances Abused Adults Discharged from, Residential Non-residential Major Substances Successful & Unsuccessful Percent Successful Successful & Unsuccessful Percent Successful Alcohol 1,383 66% 5,640 64% Crack/cocaine 2,402 57% 3,067 35% Marijuana % 2,133 51% Heroin % % Total 4,347 60% 10,966 53% * Source: Dept. of Children and Families SISAR Discharges The analysis of the characteristics of adults served and their services identified several areas where improvements can be made in performance reporting. Department performance measures assess outcomes for clients in non-residential and residential treatment services that receive the majority (69%) of resources. Detoxification services with 18% of resources does not have performance measures. Intervention services receiving 8% of resources, but serving 29% of clients, do not have measures other than client satisfaction. The program loses information about the characteristics of the population it serves by the procedure it uses to disaggregate clients into four subgroups related to service needs. It uses a mixture of clinical, demographic and legal criteria that do not distinguish unique groups. The four groups are intravenous drug users, clients in need of mental health services, parents putting children at risk by their substance use, and clients involved in the criminal justice system. One third of the population, 34%, are both parents and criminal justice involved. Because criminal justice involvement, which is a characteristic of 84% of the population, is disaggregated last, it appears to represent less than half of the population. For planning and management purposes it would be more useful to report all these treatment related characteristics. Reporting of Performance Omits Some Programs Serving Adults Subgroups Have Substantial Overlap II. B. Children Served The program reports over 62,000 children under age 18 served in The analysis of characteristics of clients and the services they receive is based on 41,079 children admitted to services in Services provided to children under age 18 include prevention, intervention and treatment. The major prevention program called Alpha- 41,000 Children Entered Services in with $43.4 Million in Expenditures 3 The program reports 62,005 children under age 18 were served in , with 19,414 receiving non-residential treatment and 2,551 receiving residential treatment. As indicated for adults, for the purpose of this report, current year admissions was determined to be a better source of information for analysis of the characteristics of clients and the services they receive. Substance Abuse Evaluation Page 15

20 Beta targets younger children in elementary and middle schools who are at risk of substance abuse. It is provided in 100 classrooms across the state by community agencies in partnership with local school boards, to improve educational achievement, parent involvement and behavior. Intervention, which includes assessment, brief counseling, referral and case management, is provided to all children involved in the juvenile justice system. includes both residential and non-residential outpatient and day treatment. Non-residential and residential treatment are the largest expenditures (50%) for services provided although they involve less than one-fifth (16%) of children reported admitted to services. Intervention is provided to 69% of children admitted. Expenditures and Children Admitted by Service, Expenditures (Millions) Admitted Services Amount Percent Number Percent Non-Residential $8.2 19% 5,101 12% Residential $ % 1,467 4% Detoxification $3.0 7% 2,050 5% Intervention (assessment & $6.5 15% 28,294 69% referral, case management) Targeted Prevention $8.1 19% 2,400 6% Special Overlay & Aftercare $2.6 6% 1,767 4% Other $0.9 2% Total $ % 41, % * Source: Department cost center allocations and SISAR client information. Children Enter Services Primarily From the Department of Juvenile Justice The primary referral source to substance abuse services is the Department of Juvenile Justice, 67%. The majority receive intervention services for assessment, brief counseling and referral to treatment if needed. Page 16 Substance Abuse Evaluation

21 Referrals of Children to Substance Abuse Services Cases Admitted in FY (n=41,079) Dept of Juvenile Justice 67% 25,738 Other Criminal Justice (Inc. Courts, Probation & TASC) 19% 7,764 Schools 10% 4,010 Self and Family Referral 4% 1,767 Other Referrals (Inc. Substance Abuse, Other Agencies, Dept. Children Families & Unkn) 4% 1,800 Special Services (Inc Follow-up) 4% 1,767 Intervention (Assessment, Counseling, Referral, & Case Management) 69% 28,294 Detox 5% 2,050 Targeted Prevention 6% 2,400 Residential 4% 1,467 Non-Residential Outpatient and Day 12% 5,101 * Source: Dept. of Children and Families SISAR Admissions. ** Numbers are admissions, not total number served. The program reports 62,005 served, including 19,414 in non-residential and 2,551 in residential treatment Marijuana is the most common drug problem among children admitted to services abusing substances, followed by alcohol. Predominant Substances Abused Children Admitted to All Services, Children Served Primarily Abuse Marijuana Alcohol and Drug Use Number Admitted Percent Marijuana/Hashish 13,722 33% Alcohol 4,190 10% Crack/Cocaine 697 2% Hallucinogens 172 0% Methamphetamine & Other 76 0% Amphetamines Benzodiazepine & Other 61 0% Tranquilizers & Sedatives Heroin 48 0% Other Opiates 14 0% Other Drugs 642 2% Unknown/Denies 3,971 10% 8,016 20% None (Inc. clients assessed with no drug use, or in prevention programs) No Data 9,470 23% Total 41,079 58% * Source: Dept. of Children and Families SISAR Admissions Substance Abuse Evaluation Page 17

22 Most Children Served are Adolescent Males The average age of children served is 16. Of those served, 78% are age Seventy-four percent are males. Among clients abusing marijuana, alcohol and crack/cocaine (95% of the total), the majority (58%) are males using marijuana. The next common clients are males using alcohol, and females using marijuana and alcohol. Primary Substances Used by Sex Children Admitted to All Services, Sex and Major Total Substances Abused Number Percent Male Marijuana 10,610 58% Male Alcohol 2,990 16% Female Marijuana 2,973 16% Female Alcohol 1,156 6% Male Crack/cocaine 429 2% Female Crack/cocaine 259 2% Total ** 18, % * Source: Dept. of Children and Families SISAR Admissions ** Does not include children admitted for other substance use. Most Do Not Have a Pattern of Multiple Arrests Similar to adults, the majority of children do not have a pattern of multiple arrests. Thirty percent are reported as having no arrests in the two years prior to admission, and 27% reported only one. Twenty-eight percent have two or more arrests before admission to services. Crack/cocaine users have the highest rates of multiple arrests. Arrests in the 24 Months Prior to Admission Children Admitted to All Services, Arrests in 24 Months Number Percent Prior to Admission No Arrests 12,459 30% One Arrest 10,946 27% Two Arrests 5,114 13% Three or More Arrests 6,219 15% No Data 6,341 15% Total 41, % * Source: Dept. of Children and Families SISAR Admissions A Higher Percentage of Crack/Cocaine Abusers Receive Residential Of the small proportion of children reported admitted to treatment services, the majority are in outpatient, non-residential treatment. A higher percentage of crack/cocaine users are admitted to more intensive and costly residential treatment. Page 18 Substance Abuse Evaluation

23 Type of by Primary Substances Abused Children Admitted to, Major Substances Admitted to Residential Non-Residential Number Percent Percent Marijuana 3,799 18% 82% Alcohol % 81% Crack/cocaine % 61% Total ** 5,099 25% 75% * Source: Dept. of Children and Families SISAR Admissions ** Does not include children admitted for other substance use. Differences in the primary substance abused have a strong relation to successful drug free completion of treatment. As with adults, the treatment of alcohol use has the highest success ratio (60%) and crack/cocaine the lowest (39%). Crack/cocaine also has the highest rate of prior admissions. Abusers of Alcohol Have Higher Success Rates than Abusers of Crack/Cocaine Ratio of Successful Completion by Substances Abused Children Discharged from, Successful: Completed Drug Free Unsuccessful: Discharged Non-Compliant or Left Major Substances Successful & Unsuccessful Number Percent Percent Marijuana 2,144 53% 47% Alcohol % 40% Crack/cocaine % 61% Total ** 2,762 54% 46% * Source: Dept. of Children and Families SISAR Admissions. ** Not including 547 completing treatment and not drug free, and 1,112 discharges to other services, deaths, etc. Does not include children receiving treatment for other drugs. The analysis of children s characteristics and services identified areas where performance reporting can be improved similar to adults. A comparatively low proportion of children were reported admitted to treatment programs in relative to the total served. Two-thirds (69%) were admitted to intervention services that are provided to all children entering the juvenile justice system. Other than client satisfaction, there are no performance measures to provide accountability for services such as detoxification and intervention. As with the adult population, some of the subgroups used to identify service needs do not clarify performance management strategies. The subgroup of children under the supervision of the state is the priority subgroup of the Department. Because this subgroup includes all children referred through the juvenile justice system, irrespective of their substance abuse Reporting of Performance Can Be Improved Substance Abuse Evaluation Page 19

24 problem, it represents 86% of children served. Only a small percentage of children served are not included in the subgroup. Analysis of Clients and Services Are Based on Extensive Program Data The above findings of both adult and child client characteristics and services are based on the SISAR database of all 181,074 admission and discharge records of clients of providers receiving state funding during This study did not include clients served by private providers licensed by the state that do not receive funds from state appropriations. The client group studied included all clients admitted and all clients discharged during the year. Client characteristics, referrals and service information were analyzed using data from admission forms. Completion of treatment information is based on client discharge forms. III. How good are the data used for performance management? The data system provides useful information on the characteristics of clients served and their substance abuse at admission to services. Reporting problems and data errors reduce outcome information. The program is aggressively addressing data issues. The Program s Data System Provides Useful Client and Service Information There are Problems Reporting Discharge Information The program office uses the State Integrated Substance Abuse Reports (SISAR) in the Alcohol, Drug Abuse and Mental Health data warehouse to report and manage performance. Performance information is available for each service district and provider. The SISAR data system was built with federal funding to track characteristics of those served and services provided. It includes information from admission and discharge forms for each component of service. The large number of clients included in the database provides good information on the population served and lends itself to analysis. Security measures ensure client confidentiality. No names, addresses or phone numbers of individuals are included in the data system. A significant problem with data used for performance measurement is the low ratio of reported discharges to admissions in the current database. This indicates a large loss of information on client outcomes. When the department began to use the data system to report outcome performance it found the number of clients reported as completing treatment to be very small compared to the number admitted. For example, compared to the 1,393 children admitted to residential treatment in , only 664 were reported discharged. Performance outcomes can only be reported by the department for these 664 children. Page 20 Substance Abuse Evaluation

25 The following table illustrates the magnitude of this problem and the extent to which clients are lost to follow-up. Compared to the number of adults reported admitted in , only 47% were reported discharged. This indicates as many as 53% of adults who are treated do not have discharge information in the department s database and their client outcomes cannot be reported. For children, only 41% of the number admitted to treatment are reported to be discharged, so that 59% have no outcome information. Because a low number of children are reported to be admitted to treatment and a low percentage of those admitted are reported discharged, accountability for treatment is based on only 2,448 discharges. Follow-up Status of Persons Discharged by, Admissions Discharges No Discharges Percent Percent Compared to Compared to Number Number Admissions Admissions Adults Non-residential 29,643 13,528 46% 54% Residential 10,991 5,403 49% 51% Total 40,634 18,931 47% 53% Children Non-residential 4,520 1,784 39% 61% Residential 1, % 52% Total 5,913 2,448 41% 59% * Source: SISAR Admission and Discharge Data The program has identified the primary reason for the low ratio of discharges to admissions as discharge forms not accepted by the database. Discharge forms are not accepted unless they match an admission form for the same service. The program has found reported service dates, and coding of provider and client identification numbers, often do not match. Reasons for these problems include formatting of data, errors in completing and entering scan forms, and some providers non-compliance with reporting requirements. The program is aggressively improving data quality. Performance information is regularly reported back to districts and providers so reporting problems can be identified. District staff now have on-line access to the database for analysis and contract monitoring. The program is checking database records against client records with providers to identify and correct data errors and reporting problems. The Program is Aggressively Addressing Data Problems The program is carrying out a data validation process with each contracted provider to determine the accuracy of key data elements. The process will determine if the data in the SISAR database reflects the actual population served, is consistent with information in client files, and whether key data elements required for performance reported are Substance Abuse Evaluation Page 21

26 completed. The process was developed with the Florida Mental Health Institute. The program began implementing the process January 1, Each district is responsible for completing the validation process with each one of their contracted providers and will report all information to the central office by April 15, The process will then be incorporated into the contract monitoring process. A program process improvement team with central office, district and provider members is addressing data problems. The team identified underlying problems that include lack of timely reporting by providers, provider misunderstanding about procedures and requirements, and a reporting process that is complex and increases the probability of problems and errors. Many of these problems reflect the large quantity of information reported. All providers, both those state funded and those not funded but licensed, are currently required to send data forms for each component of service for each client. This generates a very large number of client forms that are difficult to manage to ensure completeness and quality. Some information on the forms, such as client demographics, is redundant and some is only relevant for a small portion of clients and services. Much of the information is not needed or used for performance management and accountability. Required state reporting is in addition to internal provider record keeping. To address these underlying problems, the process improvement team is reengineering the data reporting system to reduce the amount of data that needs to be reported and improve quality. Information will only be reported for the entire episode during which a client receives services from a provider. Data are being submitted electronically through a system developed by the department that includes built-in data checks. Electronic submission from the majority of providers already demonstrates that it is timely and error free. Non-essential data elements are being eliminated. These changes to correct existing data problems will be phased in beginning July 1, 1998 and completed January 1, Page 22 Substance Abuse Evaluation

27 In conclusion, the findings of this evaluation are: Completion of substance abuse treatment produces positive outcomes of reduced substance abuse problems and increased employment, based on a six month follow-up. Comparison of outcomes for completers versus non-completers validates the importance of the current performance measure of drug free completion of treatment. The substance abuse program provides an array of services to clients with very different characteristics, depending on age, sex and primary substance use. Alcohol, crack/cocaine and marijuana are the most common substances abused by clients. Alcohol is the most common problem for adults and marijuana is most common for adolescents. Clients abusing crack/cocaine, the second most common drug used by adults, have the lowest treatment success. Most substance abusers are male. A higher proportion of clients abusing crack/cocaine are female compared to other substances. Alcohol and drug treatment services are operated primarily by private, not-for-profit providers under contract to the Department of Children and Families. The department receives state General Revenue and substantial federal money to fund contracts. Due to data problems with reporting discharges, the program lacks outcome information on a large percentage of clients. Performance measures have not been developed to hold all services accountable, especially detoxification and intervention. Subgroups used to report populations served have substantial overlap, so information about characteristics of clients is lost. The program is reengineering its data reporting system to correct data problems. It is working with providers and districts to simplify reporting, and expand use of data for contract management and better accountability. SUMMARY AND CONCLUSIONS Based on this evaluation, the following recommendations are offered: 1. The substance abuse program should work with providers to develop and implement systematic follow-up of clients after treatment to demonstrate that desired outcomes of drug-free behavior and employment are met. 2. The program should maintain the use of completion of treatment as a performance measure. The program should apply quality improvement processes to increase the rate of successful drug free completion of treatment. This should include identifying RECOMMENDATIONS Substance Abuse Evaluation Page 23

28 characteristics of clients with low success rates and working with providers to improve treatment completion. 3. The program should further evaluate the current subgroup reporting structure to better communicate client characteristics and service needs. 4. The program should develop performance measures for management and accountability of all primary services, including detoxification and intervention. The program should identify desired outcomes and measurable indicators of how services drive program goals of drug free behavior and employment. 5. The substance abuse program should establish measurable objectives for improvements to the data system in terms of timeliness of reporting, completeness and accuracy. 6. Data validation efforts should be implemented system-wide as an integral function of contract monitoring and data reporting during the next fiscal year. i General Accounting Office, 1996, Drug and Alcohol Abuse: Billions Spent Annually for and Prevention Activities (GAO/HEHS-97-12). ii California Department of Alcohol and Drug Programs, 1994, Evaluating Recovery Services: The California Drug and Alcohol Assessment (CALDATA). US Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, 1996, National Improvement Evaluation Study (NTIES). Oregon Department of Human Resources, Office of Alcohol and Drug Abuse Programs, 1996, Societal Outcomes and Cost Savings of Drug and Alcohol in the State of Oregon. National Institute on Drug Abuse, 1997, Drug Abuse Outcome Survey (DATOS). Page 24 Substance Abuse Evaluation