PB 2 Performance Report

PB 2 Performance Report No. 98-49 February 1999 Performance Data Problems Limit Conclusions About Substance Abuse Subprogram Success This report assesses the performance of the Department of Children and Families' Substance Abuse Subprogram based on measures and comments on measures proposed for 1999-2000 under performance-based program budgeting (PB²). Summary The Legislature cannot use the PB² measures to assess the performance of the Department of Children and Families' Substance Abuse Subprogram because of questionable performance data for Fiscal Year. To enable an assessment of the subprogram's performance, the department must improve the reliability of its performance data. The department has made efforts to improve its data collection and verification processes in order to enhance the reliability of performance data for the 1998-99 fiscal year. Suggested modifications to the proposed Fiscal Year 1999-2000 measures and standards should allow the Legislature to evaluate program outcomes. Background The Legislature intended the Substance Abuse Subprogram to provide prevention, intervention and treatment services in order to meet the needs of substance abusers and reduce the social and financial consequences of substance abuse. The subprogram provides services to enable adults with substance abuse problems to be stable, economically self-sufficient, and drug free and to enable children with or at-risk of substance abuse problems to be drug free. The department contracts with 169 community-based, private not-for-profit contractors to provide a variety of services, such as residential and outpatient treatment. Office of Program Policy Analysis and Government Accountability an office of the Florida Legislature

See Appendix C for a description of program services. Clients Served The subprogram is divided into two basic categories of clients, adults with substance abuse problems and children with or at-risk of substance abuse problems. The department estimates that 118,755 adults and 60,337 children were served during Fiscal Year. Adults served by the subprogram parents who put their children at risk because of their drug abuse (e.g., pregnant women and parents referred by the department's Family Safety and Preservation Program) adults involved in the criminal justice system who abuse substances dually diagnosed individuals (mental illness and substance abuse) intravenous drug users Children served by the subprogram children under the supervision of the state who are abusing substances school-age children not under the supervision of the state who are abusing substances children at risk of substance abuse See Appendix D for the distribution of the adult and children substance abuse target populations. Program Resources The Department of Children and Families allocated approximately $134 million for subprogram services. Appendix E details the state and federal funds allocated to the subprogram for adult and children substance abuse treatment for Fiscal Year. In Fiscal Year, the department assigned 59 full-time equivalent (FTE) positions to the Substance Abuse Subprogram. These employees are housed in the department's central office and throughout the department s 15 service districts. Of the 59 FTEs, 36 staff monitor services provided to adults while the remaining 23 staff monitor children s services. 2

Performance The Legislature should not use the subprogram's PB² information to make conclusions about performance due to the questionable reliability of its performance data for the fiscal year. The department did not sufficiently validate its data, and available evidence suggests that the data are not accurate or complete. As a result, the Legislature cannot confidently use the data to evaluate program outcomes. See page 5 for more information about limitations with the department s performance data. See Appendix A for a more detailed discussion of program performance for each of its measures. Proposed Performance Measures The department's Fiscal Year 1999-2000 proposed performance measures are an improvement from its initial set of measures but need some modifications to provide more meaningful information to the Legislature. For example, the department is proposing a measure of post-treatment success that will provide the Legislature with useful information about program results. We recommend that the Legislature add additional measures, such as unit cost measures, and that the Legislature modify the wording and methodologies of selected measures. We further recommend that the Legislature eliminate measures from the General Appropriations Act that do not provide meaningful information for policy decisions. For example, we recommend that the Legislature eliminate the four targeted prevention measures because they measure a small sub-population of total clients served. See Appendix B for a more detailed discussion of our recommendations for the program measures. 3

Rating of Program Accountability A key factor in PB² is that agencies need to develop strong accountability systems that enable the Legislature and the public to assess program performance. An accountability system consists of these key elements: program purpose or goals, performance measures, a process for valid and reliable data, and credible reports of performance that can be used to manage the program. Our rating tells decision-makers whether they can rely on the program's performance information. We compare the components of an accountability system against our established criteria to determine the rating. Accountability System Component Meets Expectations Needs Some Modifications Needs Major Modifications Program Purpose and Goals X Performance Measures X Data Reliability X Reporting Information and Use by Management X Source: OPPAGA analysis Program Purpose and Goals. The Substance Abuse Subprogram has developed clearly stated and comprehensive goals and objectives. The subprogram s objectives are measurable and consistent with subprogram goals. Performance Measures. Although the subprogram's performance-based program budgeting measures are related to its statutory purpose and are generally appropriate indicators of performance, they have some limitations. Some measures are not based on sound methodologies, which limits their meaningfulness. For example, one measure relies on clients' self-reporting of their substance use following treatment. The substance abuse literature questions the reliability of self-reported information. None of the performance measures address the subprogram s cost efficiency, such as cost per client or per treatment modality, or its cost effectiveness, such as the cost per treatment success, thus limiting the usefulness of measures for making budgetary decisions. Some measures may create disincentives for providers to serve clients with the greatest need. For example, providers may be reluctant to readmit a client into treatment after discharge because it would adversely affect the provider's performance goals. 4

Data Reliability. Despite improvements to the department's data system, this aspect of accountability needs major modifications. During Fiscal Year, the department initiated a number of improvements to its data collection process to help ensure data reliability. For example, the department developed software to reduce data entry errors and implemented a work group to continuously re-evaluate the data collection process. However, despite these improvements, we are not confident that the Fiscal Year performance data are reliable because the data have not been validated, and available evidence suggests that the data are inaccurate and incomplete. Though required to do so by statute, the department s inspector general has not sufficiently validated the program's performance-based budgeting data for Fiscal Year. In an effort to assess data reliability, subprogram staff independently conducted a limited validation of the subprogram's data. Although the study's sample size was small and not representative of the client population, the results indicated that the data were neither accurate nor complete. For example, one-third of data elements checked against the client records were not accurate. In addition, when staff compared client admission to discharge data, they found that only half of the key data elements matched for adult records and only a third matched for children's records. The study's findings are consistent with the concerns expressed by subprogram, district, and provider staff that the Fiscal Year data is inaccurate and incomplete. Beginning in Fiscal Year 1998-99, the subprogram implemented a number of new procedures for data collection and verification to improve data. For example, new data entry software will link admission and discharge data, thereby improving data completeness. The subprogram may earn an improved data reliability rating pending a review by the inspector general of the Fiscal Year 1998-99 performance data. Reporting and Use by Management. Performance information is reported to the Legislature and the general public through various publications, including the Legislative Budget Request, the Agency Performance Report, and the department's website. Department managers have used performance data on a limited basis to assess district and provider performance. For example, performance measures are included in all provider contracts. In addition, the department is beginning to monitor each district s performance. However, performance data have not yet been used in a systematic manner to improve services, redirect resources, identify "best practices," or make changes to types or mix of services provided by the state. Improving data reliability in Fiscal Year 1998-99 may enable district staff to receive more accurate and timely feedback on performance, which should help to improve services. 5

For More Information See FGAR profile at http://www.oppaga.state.fl.us/profiles/5046/ or call Marti W. Harkness (850) 487-9233 or Gene Bowers at (850) 487-1760. Information from the department is available on its website at http://www.state.fl.us/cf_web/adm/ or by calling (850) 487-2920. 6

Appendix A Analysis of DCF's Substance Abuse Subprogram for Each of Its Performance Measures Combined Adult and Children Client Group Outcome Measures Performance Standard Met Standard? Comments Increase successful completion of treatment. Percentage of discharges successfully completing treatment with no alcohol or other drug use during the month prior to discharge Children = 53.6% Adults = 55.0% Children = 52.0% Adults = 51.0% Unable to assess Unable to assess The Legislature should not use this performance data to assess the subprogram's performance in increasing successful treatment completions due to questionable Fiscal Year data. Reduced substance use. Percentage reduction in frequency of substance use for unsuccessful discharges during the month prior to discharge compared to the month prior to admission Children = 38.0% Adults = 44.6% Children = 61.0% Adults = 64.0% Unable to assess Unable to assess The Legislature should not use this performance data to assess the subprogram s performance due to questionable Fiscal Year data. Because it is difficult to track and collect information on these clients, the 1998 Legislature deleted this measure. Reduce readmissions for substance abuse. Percentage of clients successfully completing treatment who are readmitted for substance abuse services during the 12 months following discharge Children Baseline = 8.0% Adult Baseline = 4.7% Identified in FY GAA as a FY 1998-99 measure; no standard N/A N/A This measure was intended to indicate how many clients were relapsing after discharge, however (1) due to treatment waiting lists, it may be measuring access to treatment rather than measuring relapse, and (2) does not capture other negative posttreatment outcomes, such as admission into jail, DJJ, or state prison. 7

Combined Adult and Children Client Group Outcome Measures Performance Standard Met Standard? Comments Increase community satisfaction. Percentage achieving expected level of performance on key informant community satisfaction scale Children Baseline = 77.1% Adults Baseline = 78.0% Identified in FY GAA as a FY 1998-99 measure; no standard N/A N/A The baseline was established using agency-wide survey results from 1998 and is reasonable. Overall results suggest that the department cooperates well with community partners but may lack the flexibility and sensitivity to address their needs. Adult Client Group Outcome Measures Performance Standard Met Standard? Comments Increase employment at discharge. Percentage of adults employed upon discharge from treatment services 55.6% 53.0% Unable to assess The performance data cannot be used to assess the subprogram's ability to link clients with employment opportunities due to questionable Fiscal Year data. (See discussion of data reliability on page 5.) Reduce substance-exposed newborns. Percentage of adult women pregnant during treatment who give birth to substance-free newborns 85.0% 70.0% Unable to assess The performance data cannot be used to assess the subprogram's ability to reduce the percentage of substance-exposed newborns due to questionable Fiscal Year data. (See discussion of data reliability on page 5.) 8

Adult Client Group Outcome Measures Performance Standard Met Standard? Comments Reduce low birth weight newborns. Percentage of pregnant women receiving substance abuse treatment who deliver infants with normal birth weight No baseline Identified in FY GAA as a FY 1998-99 measure; no standard N/A The department has been unable to establish a baseline for this measure due to difficulties it has experienced in matching subprogram data to Department of Health data. Decrease arrest rates. Percentage change in adults arrested during the 90 days following discharge as compared to the 90 days prior to admission Baseline = 51.0% Identified in FY GAA as a FY 1998-99 measure; no standard N/A The department established the 51% baseline using a sample of 5,000 clients from 1995. Increase client satisfaction (adults). Average level of satisfaction on the Behavioral Healthcare Rating of Satisfaction 132 138 Unable to assess The subprogram's performance on this measure is based on a survey with a return rate of approximately 43.7%. The department has not determined whether survey results are representative of the total client population. 9

Adult Client Group Output Measures Performance Standard Met Standard? Comments Number of adults served with substance abuse problems 93,527 100,379 Unable to assess Due to data reporting improvements, the department noted that the standard was not accurate. The performance reported in the department's LBR reflects clients served from June 1, 1997, to May 31, 1998. The department actually served 118,755 clients during Fiscal Year. Child Client Group Measurements Performance Standard Met Standard? Comments Decrease juvenile justice commitments/recommitments. Percentage of children under the supervision of the state receiving substance abuse treatment who are not committed or recommitted to DJJ during the 12 months following treatment completion Baseline = 79.0% Identified in FY GAA as a FY 1998-99 measure; no standard N/A The measure's baseline was established using a sample of 16,000 juvenile clients and is reasonable. The department has experienced problems matching subprogram data with Department of Juvenile Justice (DJJ) data because client identifier data is collected differently in the agencies' databases. Reduce Admissions to Substance Abuse Services. Percentage of children in targeted prevention services that are admitted to substance abuse services during the 12 months following completion of prevention services Baseline = 9.0% Identified in FY GAA as a FY 1998-99 measure; no standard N/A The baseline for this measure was based on questionable Fiscal Year 1996-97 performance data. (See discussion of data reliability on page 5.) 10

Child Client Group Measurements Performance Standard Met Standard? Comments Increase perception of substance use as harmful (targeted prevention). Percentage of children in targeted prevention programs who perceive substance use to be harmful at the time of discharge when compared to admission Baseline = 68.0% Identified in FY GAA as a FY 1998-99 measure; no standard N/A This measure identifies the percentage change in targeted prevention clients who perceive substance abuse to be harmful at the time of discharge when compared to admission. The baseline for this measure was based on questionable Fiscal Year 1996-97 performance data. (See discussion of data reliability on page 5.) Increase perception of substance use as harmful (general population). Percentage of children in the general population who perceive substance use to be harmful. No baseline Identified in FY GAA as a FY 1998-99 measure; no standard N/A The 1998 Legislature deleted this measure. As a result, the department has not collected baseline data on this measure. Increase family satisfaction. Average level of satisfaction on the Family Centered Behavior Scale for parents of children receiving substance abuse services. Baseline = 89.9% Identified in FY GAA as a FY 1998-99 measure; no standard N/A The baseline was established using survey results with a return rate of approximately 23.4%. The department has not determined whether survey results are representative of the total client population. Increase client satisfaction (children). Average level of satisfaction for children in treatment as scored on a reliable scale of client satisfaction. Baseline =85.0% Identified in FY GAA as a FY 1998-99 measure; no standard N/A The baseline was established using Fiscal Year 1996-97 results from the department-wide satisfaction survey which was statistically valid at the state level. 11

Child Client Group Measurements Performance Standard Met Standard? Comments Increase education achievement (math). Percentage of children in targeted prevention programs who achieve expected level of improvement in math 71.0% 71.0% Yes The department reports that schools provide accurate and timely data for this measure. Increase education achievement (reading). Percentage of children in targeted prevention programs who achieve expected level of improvement in reading 70.0% 67.0% Yes The department reports that schools provide accurate and timely data for this measure. Decrease substance use by middle and high school students. Percentage reduction in monthly or more use of alcohol and other drugs by middle and high school students as measured on a statewide survey of students. No baseline Identified in FY GAA as a FY 1998-99 measure; no standard N/A The 1998 Legislature deleted this measure. Output Measures Performance Standard Met Standard? Comments Number of children served with or at risk of substance abuse problems 46,446 30,574 Unable to assess Due to data reporting improvements, the department noted that the standard was not accurate. The performance reported in the department's LBR reflects clients served from June 1, 1997, to May 31, 1998. The department actually served 60,337 clients during Fiscal Year. Source: ADM Legislative Budget Request and OPPAGA analysis 12

Appendix B OPPAGA Recommendations for the DCF Substance Abuse Subprogram's Fiscal Year 1999-2000 Measures Combined Adult and Children Client Group Outcome Measures, Fiscal Year 1999-2000 Measures Proposed by Agency Proposed Standards OPPAGA Recommendations/Comments Percentage of clients discharged for completing treatment with no alcohol or other drug use during the month prior to discharge Percentage of community partners satisfied based on survey Children = 72.0% Adults = 68.0% Children = 90.0% Adults = 90.0% This measure assesses successful completion, which the department defines as having no reported drug use during the month prior to discharge. Performance attainment for this measure is based primarily on selfreported data regarding drug use. The literature questions the reliability of self-reported information. Nevertheless, the measure can be used as an indicator of treatment success. We recommend modification of this measure. The measure should be reworded to read "Percentage of clients who complete treatment." The department has developed specific criteria that define "treatment completion." We recommend adoption of this measure. The community satisfaction survey results reflect the satisfaction of community partners (e.g., law enforcement, judiciary) using specific concepts (e.g., timeliness, cooperation), as well as overall satisfaction with Department of Children and Families' services. The survey was designed to produce reliable results only at the state level. Satisfaction survey results are most useful for improving program services when the results apply to specific districts and client groups. The department should make the community partner satisfaction survey more useful to districts by identifying program-specific services that need to be improved and by disseminating these results to the districts. 13

Combined Adult and Children Client Group Outcome Measures, Fiscal Year 1999-2000 Measures Proposed by Agency Proposed Standards OPPAGA Recommendations/Comments Percentage of clients completing treatment who are not readmitted for substance abuse services during the 12 months following discharge Children = 95.0% Adults = 96.0% We recommend deletion of this measure. According to the department, this measure is an indirect indicator that clients are drug-free. It assumes that no readmission means the individual is not abusing substances. This measure does not provide meaningful information to the Legislature because (1) due to treatment waiting lists, it may be measuring access to treatment rather than measuring relapse, (2) the measure does not capture other negative posttreatment outcomes, such as admission into jail, DJJ, or state prison, and (3) it may create an unwanted incentive for providers not to admit a client they have previously discharged or to move them farther down the waiting list. Relapse and readmission are part of the recovery process. Allowing a client to be readmitted should not be discouraged because it contributes to the long-term success of the client. The department has proposed a better measure of posttreatment success: "Percentage of adults drug-free six months following completion of treatment." Adult Client Group Outcome Measures, Fiscal Year 1999-2000 Measures Proposed by Agency Proposed Standards OPPAGA Recommendations/Comments Percentage of adults employed upon discharge from treatment services 61.0% We recommend modification of this measure. The measure itself does not assess treatment effectiveness, however employment is an important factor for client success. The department should identify how well the treatment provider was able to link the client with employment opportunities. The measure should read "Percentage of adults employed upon discharge who were unemployed at admission." The department should also adjust the proposed standard to reflect the new methodology. Percentage drug free at six months following completion of treatment No baseline available We recommend adoption of this measure. 14

Adult Client Group Outcome Measures, Fiscal Year 1999-2000 Measures Proposed by Agency Proposed Standards OPPAGA Recommendations/Comments Percentage of adult women pregnant during treatment who give birth to substance-free newborns 89.0% We recommend adoption of this measure. Percentage of pregnant women receiving substance abuse treatment who deliver infants with normal birth weight No baseline available We recommend deletion of this measure. It is unnecessary to have two measures for the same outcome (healthy babies). The measure associated with substance-free newborns is more directly related to substance use, whereas low birth weight could be attributed to other prenatal factors. Percentage of adults who reduce the frequency of arrest during the 90 days following discharge as compared to the 90 days prior to treatment admission Average level of satisfaction on the Behavioral Healthcare Rating of Satisfaction 57.0% We recommend modification of this measure. The department is attempting to assess change in the arrest rates of clients before versus after treatment. The department should modify its methodology and reword the measure to read "Percentage of adults with criminal justice involvement who are not re-arrested within one year following discharge from treatment." The department should adjust the proposed standard to reflect the new methodology. 138 We recommend modification of this measure. The department should reword the measure to read "Average score on the Behavioral Healthcare Rating of Satisfaction." Percentage of individuals in protective supervision who have case plans requiring substance abuse treatment who are receiving treatment No baseline available We recommend adoption of this measure. Because this measure requires significant coordination between the ADM and Family Safety and Preservation programs, staff need to develop formal processes to ensure that protective supervision clients are identified and placed in treatment and to resolve data problems inherent in matching clients across these two programs. 15

Adult Client Group Output Measures, Fiscal Year 1999-2000 Measures Proposed by Agency Proposed Standards OPPAGA Recommendations/Comments Number of adults served 95,059 We recommend adoption of this measure. Number of individuals in protective supervision who have case plans requiring substance abuse treatment who are receiving treatment No baseline available We recommend adoption of this measure. Child Client Group Outcome Measures, Fiscal Year 1999-2000 Measures Proposed by Agency Proposed Standards OPPAGA Recommendations/Comments Percentage of children under the supervision of the state receiving substance abuse treatment who are not committed or recommitted to DJJ during the 12 months following treatment completion Percentage of children in targeted prevention programs who achieve expected level of improvement in math 85.0% We recommend modification of this measure. The department reported difficulty matching department data with Department of Juvenile Justice (DJJ) data. Because client identifier data is collected differently in these agencies' databases, the department reported matching records for less than 3% of all clients. Unless the department can identify a better methodology to match department and DJJ data, this measure should be removed from the GAA because it does not provide the Legislature with meaningful information. 75.0% We recommend deletion of this measure. PB² measures should cover major activities of the subprogram or functions that account for major subprogram expenditures. Targeted prevention clients represented less than 5% of the children receiving services in Fiscal Year. Although it may be inappropriate to retain this as a PB² measure, the department should continue to collect and maintain data internally on targeted prevention services to monitor and improve services. 16

Child Client Group Outcome Measures, Fiscal Year 1999-2000 Measures Proposed by Agency Proposed Standards OPPAGA Recommendations/Comments Percentage of children in targeted prevention programs who achieve expected level of improvement in reading Percentage of children in targeted prevention services that are not admitted to substance abuse services during the 12 months after completion of prevention services Percentage of children in targeted prevention programs who perceive substance use to be harmful at the time of discharge when compared to admission Percentage of parents of children receiving services reporting average or above average level of satisfaction on Family Centered Behavior Scale Percentage of children receiving services who are satisfied based on survey 75.0% We recommend deletion of this measure. PB² measures should cover major activities of the subprogram or functions that account for major subprogram expenditures. Targeted prevention clients represented less than 5% of the children receiving services in Fiscal Year. Although it may be inappropriate to retain this as a PB² measure, the department should continue to collect and maintain data internally on targeted prevention services to monitor and improve services. 95.0% We recommend deletion of this measure. PB² measures should cover major activities of the subprogram or functions that account for major subprogram expenditures. Targeted prevention clients represented less than 5% of the children receiving services in Fiscal Year. Although it may be inappropriate to retain this as a PB² measure, the department should continue to collect and maintain data internally on targeted prevention services to monitor and improve services. 96.0% We recommend deletion of this measure. PB² measures should cover major activities of the subprogram or functions that account for major subprogram expenditures. Targeted prevention clients represented less than 5% of the children receiving services in Fiscal Year. Although it may be inappropriate to retain this as a PB² measure, the department should continue to collect and maintain data internally on targeted prevention services to monitor and improve services. 95.0% We recommend adoption of this measure. 90.0% We recommend adoption of this measure. 17

Child Client Group Output Measures, Fiscal Year 1999-2000 Measures Proposed by Agency Proposed Standards OPPAGA Recommendations/Comments Number of children served 44,361 We recommend adoption of this measure. Number of targeted prevention children served Number of children completing substance abuse treatment 6,233 We recommend deletion of this measure. PB² measures should cover major activities of the subprogram or functions that account for major subprogram expenditures. Targeted prevention clients represented less than 5% of the children receiving services in Fiscal Year. Although it may be inappropriate to retain this as a PB² measure, the department should continue to collect and maintain data internally on targeted prevention services to monitor and improve services. 4,500 We recommend adoption of this measure. This proposed measure provides the number of successful completions, which is used to generate the current measure "Percentage of children discharged for completing treatment with no alcohol or other drug use during the month prior to discharge." OPPAGA Recommendations for Additional Measures, Fiscal Year 1999-2000 Measures Comments Number of adults completing substance abuse treatment This measure complements the "Percentage of adults discharged for completing treatment with no alcohol or other drug use during the month prior to discharge" measure reported for adult clients. A similar measure was proposed for child clients, but was not included for the adult population. 18

OPPAGA Recommendations for Additional Measures, Fiscal Year 1999-2000 Measures Comments Unit cost measures, such as "Cost per completion" or "Cost per client group." Performance measures for other service components. Unit cost measures are important because they are needed to link changes in funding to predictable changes in performance. The department can identify the state costs (general revenue and trust funds) and Medicaid costs per client to provide treatment services. However, unit cost figures that simply divide the total expenditures by the total number of clients are not meaningful for budgeting purposes because they do not take into account the differences in client severity and mix of treatment services. In December 1998, the department submitted a budget amendment to the Executive Office of the Governor for funding to conduct a study on client severity and service mix. This information should help the department develop meaningful unit cost measures that would allow the Legislature to link cost to performance. A number of subprogram services, such as assessment, intervention, and detoxification, are not addressed by performance measures. These services account for almost a third of total subprogram expenditures. The department should consider measures that assess the impact of these services. For example, the department could develop a measure that would assess how well detoxification services stabilize clients. The department should determine which of these services would be most appropriate to measure for performance-based budgeting purposes. Source: ADM Legislative Budget Request/comments by OPPAGA 19

Appendix C Program Services Intervention services identify clients at-risk of substance abuse problems and provide short-term counseling, referral, and outreach services. Prevention services consist of school-based programs designed to reduce substance abuse risk factors, such as the Alpha and Beta programs. These programs aim to improve the educational achievement of students in grades 4 through 8 and are provided through community substance abuse agencies in partnership with county school boards. Assessment services involve gathering and evaluating information to determine the nature and severity of clients' substance abuse problems and their need and motivation for services. Non-residential treatment provides a range of assessment and counseling services, including a structured schedule of treatment, education, and rehabilitative services for substance abuse clients living in the community. Detoxification services assist clients in their efforts to withdraw from the effects of substance abuse. The services use medical and psychological procedures and a supportive counseling program. Residential treatment provides a range of assessment, treatment, rehabilitation, and education programs in an intensive therapeutic environment. 20

Appendix D Clients Served by the Substance Abuse Subprogram, Fiscal Year Three-quarters of the adults served are involved with the criminal justice system and twothirds of the children served are under state supervision as delinquent or dependent children. Adults 66% (n=118,755) Children 34% (n=60,337) Adult Sub-populations 1 Children Sub-populations 2 74% Criminal Justice Involved Children At-risk 67% Parents putting children at risk Children Not Under 41% (includes pregnant women, State Supervision parents with children, and (includes referrals from 26% parents referred by Family Safety and Preservation) schools, other community agencies, and self or family 12% referrals) Dually Diagnosed 7% Children Under State Supervision (includes Department of Juvenile Justice or Family Safety and Preservation involvement) 12% Other 10% IV Drug Users Notes: 1 Percentages are based on duplicated counts, i.e., an adult may be an IV drug user and involved in the criminal justice system. 2 Percentages are based on unduplicated counts, i.e., a child can only be categorized in one of the three mutually exclusive categories. Source: Data provided by Substance Abuse Subprogram staff, September 1998 21

Appendix E Program Resources: Program Allocations, Medicaid, and Local Match Expenditure, Fiscal Year Funding Source Adult Substance Abuse Child Substance Abuse Administration (Districts and Central Office) Totals General Revenue $26,545,982 $27,298,408 $6,622,886 $ 60,467,276 Trust Funds¹ 40,525,896 20,440,909 60,966,805 Medicaid² 8,060,000 4,940,000 13,000,000 Total³ $75,131,878 $52,679,317 $6,622,886 $134,434,081 ¹Trust funds include federal grants ²The Agency for Health Care Administration administers Medicaid funding ³This does not include local match dollars. The department estimates that an additional $20.8 million is provided by local match. Source: Department of Children and Families General Revenue 45% Trust Funds 45% Medicaid 10% Note: N=$134.4 million. This does not include local match dollars, which provides an estimated additional $20.8 million in cash or in-kind contributions, such as free office space. Source: OPPAGA 22

Appendix F Response From the Florida Department of Children and Families FLORIDA DEPARTMENT OF CHILDREN & FAMILIES Jeb Bush Governor Kathleen A. Kearney Secretary January 15,1999 Mr. John W. Turcotte, Director Office of Program Policy Analysis and Government Accountability 111 West Madison Street Room 312, Claude Pepper Building Tallahassee, FL 32301 Dear Mr. Turcotte: Thank you for your December 22 letter enclosing the preliminary performance-based budgeting standards report for The Department of Children and Families' Substance Abuse Subprogram. Overall, the report captures the fundamental issues and challenges associated with the implementation of performance-based budgeting. We acknowledge the data issues are a primary concern, and they have been the fundamental focus of our improvement activities both within the department and with our providers. We have made substantial progress over the two years since the Substance Abuse Program initiated performance measurement activities. Last year, the Legislature added several new measures to increase our accountability. We made significant data system modifications, which were implemented in FY 1998-1999 and will significantly improve data validity and reliability. Additionally, the Substance Abuse 1317 Winewood Boulevard Tallahassee, Florida 32399-0700 Working in partnership with local communities to help people be self-suffcient and live in stable families and communities. 23

Mr. John Turcotte Page 2 Program, with the assistance of the Florida Mental Health Institute (FMHI), initiated a data validation methodology as part of an internal quality improvement process. A draft copy of a report on the validation process is enclosed. One of the major conclusions of the OPPAGA report is that the Legislature cannot use the FY PB 2 data to assess the performance of the Substance Abuse Program because of questionable data reliability. We are concerned that the across-the-board conclusions of being unable to assess all of our measures, including outputs, at the state level, do not take into consideration the differences in reliability at the state, district and provider level and the different methodologies used to calculate our measures. For example, we believe our report of the numbers served for both adult and children in the substance abuse program are an accurate reflection of that activity. For the reasons outlined below, we believe our data at the state level can be used with qualifications. 1. Data Validation and Accuracy: The inspector general's reliability and validity assessments began with FY 1998-99 performance data in November 1998 and are ongoing. The Substance Abuse Program Office conducted a validation study of FY performance data through a contract with the Florida Mental Health Institute (FMHI). The OPPAGA performance report, which was written prior to the FMHI report, relied only on an internal program office analysis of very limited and incomplete validation data. The major conclusions of the FMHI report regarding data validity and accuracy are as follows: State Interim Substance Abuse Report (SISAR) admission and discharge forms seem to be completed and maintained in agency client records at a reasonable rate for both children and adults in substance treatment. Other sources of information in the agency client records support the validity and data integrity of the information submitted to the Department of Children and Families on SISAR forms. Although there may have been problems with consistency in using and relying on the paper forms, there is little evidence to indicate that provider agencies are reporting false information to improve their outcomes. 2. Data Completeness: The OPPAGA report indicates that a significant number of discharge records have no matching admission records; we concur with that finding. In FY, 66 percent of discharge records had matching admission records. Our goal is that 95 percent of these records will match. During FY, 118,755 adults and 65,000 children were served. This number includes all adults and children who received services (not just those children who received outpatient and residential treatment services) such as detox, assessment and intervention. To evaluate either group, a random sample of 1,200 cases would be needed 24

Mr. John Turcotte Page 3 PB²Performance Reports to achieve a confidence level of 99 percent. Since the program office does not use sampling, the number of records used for performance measures had to exceed these sample sizes in order to achieve the same confidence level. All adults (n=26,796) and children (n=4,730) who completed treatment were included in the analysis to determine the level of performance for that performance measure. DATA IMPROVEMENTS The following data improvement initiatives were implemented in FY 1998-99, as part of the recommendations made in FY by the Alcohol, Drug Abuse, and Mental Health Data (ADM) Improvement Workgroups: Re-engineering of the substance abuse data system and simplification of the mental health performance outcome data (PB 2 ) to reduce the reporting requirements in FY 1998-99, compared to previous years. Implementation of the Mental Health and Substance Abuse software for electronic submission of the data to reduce the processing time and to improve the quality of the data by eliminating errors associated with scannable forms. Development of a checker program, including data validation edits for providers who use their own database software, to further reduce errors in data reported. Designation of at least one staff person per district to serve as a data liaison to coordinate between the ADM central program offices and the providers. Direct access to both the ADM Home Page and the ADM Data Warehouse was provided for district liaisons to allow timely feedback of critical information to their respective providers, including the monthly submission of erroneous records that need corrective actions from providers. Data validation by the offices of the Inspector General and Standards and Evaluation staff to check the accuracy and validity of the data submitted by providers to ADM central program offices. We have recently reinstated the statewide data workgroup to address a number of data and performance measure issues, including the concerns outlined in OPPAGA's report. We have also created the Mental Health and Substance Abuse Steering Committee to serve as a forum for discussion of critical ADM performance monitoring issues, a vehicle for dialogue about these issues among diverse stakeholders, and a policy recommendation and oversight body. 25

Mr. John Turcotte Page 4 PERFORMANCE MEASURES Overall, we agree several of the measures need modifications to better evaluate the program's performance. We concur with the recommendation to delete the readmission for services measure for the reasons outlined in the report. We will begin modifying several of the existing measures and developing new measures as directed by the Legislature. Regarding the self-report nature of the successful completion of treatment measure, it is important to note that the source document is completed by provider staff, usually a clinician, not the client. Information is derived not only from the client, but also from random urinalysis over the course of treatment, collateral contacts, and the clinician's own knowledge of the client's progress in treatment. Suspicion of substance use usually requires the client to submit to urinalysis, though not required at discharge. REPORTING AND USE BY MANAGEMENT The Legislature has supported the use of Competitive Technologies, Inc. as a consultant in developing and implementing the department's performance improvement process, which has been underway for almost two years. Central to that process is data-driven decision-making. The use of performance data has been an integral part of this process. Within the Substance Abuse Program, the strategic focus has been on the performance measure of successful completion of treatment. In FY 97-98, this measure was included in the district performance agreements with the secretary, as well as in provider contracts. Improvement activities during FY 97-98 focused primarily on data improvement issues. In June, 1998, the substance abuse program staff and several providers, along with Competitive Technologies, mapped the processes contributing to the performance of this outcome. Subsequent data analysis substantiated the correlation between family participation in treatment and completion of treatment. That analysis has been provided to every district in the state; each district has developed countermeasures to improve the performance of this measure. Data from the first quarter of FY 98-99 indicate improvement in services and performance is taking place. We appreciate the opportunity to comment on this draft report and will continue to improve the accountability and effectiveness of these services to the citizens of Florida. We also appreciate the effort put into this review by Mr. Harkness and Mr. Bowers of your staff. They participated in many meetings and discussions with central office staff, district staff, and community providers to better understand the program. Sincerely, Melissa C. Jacoby Executive Staff Director 26

THIS PAGE INTENTIONALLY LEFT BLANK 27

The Florida Legislature Office of Program Policy Analysis and Government Accountability Visit The Florida Monitor, OPPAGA s online service. This site monitors the performance and accountability of Florida government by making OPPAGA's four primary products available online. OPPAGA Publications and Contracted Reviews, such as policy analyses and performance reviews, assess the efficiency and effectiveness of state policies and programs and recommend improvements for Florida government. Performance-Based Program Budgeting (PB²) Reports and Information offer a variety of tools. Program Evaluation and Justification Reviews assess state programs operating under performance-based program budgeting. Also offered is performance measures information and our assessments of measures. Florida Government Accountability Report (FGAR) is an Internet encyclopedia of Florida state government. FGAR offers concise information about state programs, policy issues, and performance. Check out the ratings of the accountability systems of 13 state programs. Best Financial Management Practice Reviews for Florida School Districts. OPPAGA and the Auditor General jointly conduct reviews to determine if a school district is using best financial management practices to help school districts meet the challenge of educating their students in a cost-efficient manner. OPPAGA provides objective, independent, professional analyses of state policies and services to assist the Florida Legislature in decision making, to ensure government accountability, and to recommend the best use of public resources. This project was conducted in accordance with applicable evaluation standards. Copies of this report in print or alternate accessible format may be obtained by telephone (850/488-0021 or 800/531-2477), by FAX (850/487-3804), in person (Claude Pepper Building, Room 312, 111 W. Madison St.), or by mail (OPPAGA Report Production, P.O. Box 1735, Tallahassee, FL 32302). The Florida Monitor: http://www.oppaga.state.fl.us Project supervised by: Frank Alvarez (850/487-9274) Project conducted by: Marti Harkness (850/487-9233) Gene Bowers (850/487-1760)

29