How To Measure Program Performance In Addiction Treatment

Transcription

1 Measuring Program Performance in Addiction Treatment: Why and How? CALDAR Summer Institute, August 2008 Richard Rawson, Ph.D. UCLA Integrated Substance Abuse Programs 1640 S. Sepulveda Blvd. Suite 200 Los Angeles CA

2 Now for something completely different.

3 33 year old man, high on methamphetamine admitted to emergency room r complaining of severe headache in Portland Oregon. X ray revealed 12, 2 inch nails (6 on each side) in his head, administered with aq nail gun. The man at first claimed it was an accident, but he later admitted that it was a suicide attempt. The nails were removed, and the man survived without w any serious permanent damage. He was eventually transferred to psychiatric care; he stayed for almost one month under court order but then left against doctors orders MSNBC TV

4 MORIARTY, N.M. Principal Wayne Marshall said a teacher at Moriarty High School made a misjudgment on an algebra test for his students A question on the test incorporated a fictional methamphetamine dealer. The question read, "Smoky J. sells meth. Smoky's source says he has to sell a G's worth of meth by the end of the month. If Smoky sold 240 dollars the first week and 532 dollars the second week, how much money must Smoky make if he wants to avoid the beat down from his connection?"

5 Acknowledgements Mady Chalk Tom McLellan Connie Weisner Victor Capoccia Dennis McCarty Deb Garnick Jack Kemp Peter Luongo Jonathan Fielding

6 Improving Addiction Treatment: The Holy Grail Quality Improvement Increased Effectiveness Outcomes Management Evidence based Practices Program Licensing Staff Licensing JCAHO Accreditation Etc.

7 Outcomes Measuring Outcomes National Outcomes Monitoring System (NOMS). Reimbursement for Improved Outcomes But..

8 Healthcare Performance The National Quality Forum (NQF) has been established as a private, not for forprofit, open membership, public benefit corporation for the purposes of developing consensus about standardized health care performance measures, reporting mechanisms, and a national strategy for health care quality improvement.

9 Performance Measures Urgency of need for measuring quality using performance measures has been identified by the IOM in its recent report and by the work of the National Quality Forum (NQF)

10 Performance Leaders in Addiction Washington Circle Group Treatment Convened in 1998 by SAMHSA s s Center for Substance Abuse Treatment Goals: Develop and pilot test performance measures for substance abuse treatment Promote adoption of these measures by public and private stakeholders (McCorry et al., 2000) 2003 NCQA adopts measures, Oklahoma adapts two measures for regional reports

11 Performance Webster s s definition: Performance the manner in which or the efficiency with which something reacts or fulfills its intended purpose

12 Performance Performance in Addiction Treatment Performance Performance in addiction treatment: The degree to which an addiction treatment organization meets specific benchmarks with regard to specified measures of patient/client behavior; staff behaviors; and/or organizational activity.

13 Performance vs Outcomes Performance Performance refers to the functioning of treatment organizations/programs/sites/clinics, etc. Outcomes Outcomes refers to the functioning of patients/clients. Outcome and performance measurement may use the same measures when appropriate (eg( eg.. retention). There is an implicit assumption (not yet proven) that treatment programs that demonstrate good performance will promote better outcomes for their patients/clients.

14 Why Measure Performance? It is presumed that there is a link between effective performance measurement and successful client outcomes. (eg( eg. Garnick et al. 2007) Performance measurement allows payers to reward programs with better performance, Performance measurement allows payers to identify programs with poorer performance and target technical assistance in response. It can provide a foundation for implementation of performance based management and contracting

15 Performance Measurement There is no one fits all approach to using performance measures for different services. Measuring the performance of a narcotic treatment program (methadone treatment) is likely to use different measures than measuring the performance of a residential program. As chronic care or continuing care or recovery recoveryoriented systems of care are developed, performance measures may include transfers between different levels of care and/or different organizations.

16 Developing Performance Measures Performance measures need to be developed with clear reference to practical service considerations AND in consultation with operational experts, i.e. providers Operational staff (treatment program staff) need to be routinely consulted about the design of a performance measurement, monitoring, and (ultimately) contracting program.

17 Developing Performance Measures What do you want to change (improve) and how do you want to change (improve) it? Criteria include: relevance and importance to improving treatment services, feasibility of collecting data on the measure, reliability and validity of data measures.

18 Performance Measures 1 What do you want to improve? Access to treatment; reducing rates of early drop outs, etc; are there specific groups or services you want to specifically address (eg( eg.. women, urinalysis results)? 2 What will be your measures: : depends on objectives of the organization or system; can be focused on specific program benchmarks or on benchmarks across a system of care 3 How will feedback be provided to programs? 4 How will benchmarks be established? 5 What rewards can be used for good performance w/o moving to performance contracting?

19 Some Performance Measure Candidates Candidates Program Access: : How many days does it take to get an initial assessment interview? Engagement: What % of individuals who are admitted, successfully complete 30 days in treatment (operationally defined by a specific ic number of days, doses, visits). Achievement of Specific Clinical Benchmarks: : What % of individuals admitted to treatment successfully complete 90 days/one year, etc. in treatment. Retention in Treatment: Mean number of days in treatment (number of days from admission to discharge). Participation in Treatment: : Mean number of treatment service units (clinic visits, doses of medication, urine tests, discharge interviews) delivered to a client during treatment (from admission to discharge).

20 Some Performance Measure Candidates Candidates Specific Service Delivery: eg.. Mean number of individual counseling sessions, family sessions, EBPs,, % tested for HIV/Hepatitis). Program Completion Completion : : % of clients who complete prescribed treatment service set. Widely used presently. Very poorly defined. May be b e of limited use in some settings (eg( methadone treatment) and the concept of completion completion may not be congruent with a chronic care model. Successful Performance in Treatment. % of clients whose treatment participation is rated as successful by clinician. Currently widely w used, however, validity and reliability of clinical ratings are poor and a as treatment goals of programs may vary (eg( abstinence oriented vs harm reduction), problems with operational definitions. Successful Transfers Between Levels of Care: What % of clients are successfully moved between levels of care. Eg.. a performance measure for detox programs may be: What % of individuals transfer from detox to rehab? In addition If a chronic care model is successful there should be a substantial transfer rate between levels of care. (eg( eg.. Currently in California, 85% of all treatment episodes episodes are single service sets.

21 What can be done with performance data? Performance Monitoring; ; modifying data needs; Providing feedback Performance Management; ; using feedback to promote change; setting benchmarks benchmarks Performance Contracting Rewarding good performance Indentifying programs that need improvement Providing technical assistance Technical assistance; who needs what help?

22 Methods to promote effective performance improvement Promote evidence based practices as part of TA and training activity Consult with experts who have experience in this work. Involve operational experts (providers) early in the process Flexibility: allow room for innovation Don t t forget to recognize and reward good performance

23 Performance Contracting The Delaware Experiment

24 Delaware s s Situation Outpatient Providers Limited Budget No success with outcome evaluation Providers won t/can t/can t t use EBPs

25 Delaware s s Performance based Contracting 2002 Budget 90% of 2001 Budget Opportunity to Make 106% Two Criteria: 80% Utilization/Occupancy Active Participation Audit for accuracy and access

26 Delaware s s Results Years 1 and 2 One program lost contract Two new providers entered, did well Mental Health and Employment Programs Programs worked together First, common sense business practices Second, incentives for teams or counselors 5 programs learned MI and MET

27 Utilization Average Daily Census

28 % Attending >30 days >60 days

29 California Treatment System Publicly Funded Treatment System Privately Funded Treatment System Individual Practice Treatment Service Delivery Entity State Certified Residential Treatment Programs Narcotic Treatment Programs Outpatient Treatment Programs Detox Programs Approximately 1200 programs Licensed Private Treatment Programs State Certified Treatment Programs HMO Providers Sober Living Facilities Addiction M.D. Addiction Psychiatrist Addiction Psychologist Social Worker MFT Number of Patients per Year 168,670*?? Annual Funding Approximately $750,000,000??

30 Performance Measures and CQI Essential Public Health Tools Using Performance and Outcome to Improve Treatment Summit Los Angeles, CA March 20, 2008 Jonathan E. Fielding, M.D., M.P.H., M.B.A Director of Public Health and Health Officer L.A. County Department of Public Health

31 Los Angeles County Background 4,300 square miles 89 incorporated cities and 2 islands 9.9 million residents (more than 42 States) 46% Latino, 32% White, 13% Asian/Pacific Islander, 10% African American, 0.3% American Indian Over 100 different languages spoken by significant size populations 15% living in poverty (14% of families & 24% <18) 22% of adults & 8% of children have no health insurance

32 Why We Try to Continuously Improve Quality of Services Offered There are demands for accountability for usage of County funds From the Board of Supervisors and CEO From County residents/taxpayers From consumers Want the best results for our limited funding Need to look at return on investment (ROI), both in terms of $ and in terms of health Need to ensure that we are using evidence based best practices

33 Continuous Quality Improvement Process The PDCA (or PDSA) Cycle was originally conceived by Walter Shewhart in 1930's, and later adopted by W. Edwards Deming. The model provides a framework for the improvement of a process or system.

34 Design Principles for Measurable Objectives INPUT QUANTITY How Much Service Did We Deliver? QUALITY How Well Did We Deliver Service? Focus on Quality of Outcomes More Than Quantity if Possible OUTPUT How Much Effect or Change Did We Produce? What Quality of Effect or Change Did We Produce? What difference did the program make?

35 Four Key Performance/Outcome Measures 1. Engagement the individual has attended at least one session during the first 30 days and is an ongoing active participant. This measure is considered an indicator of treatment and participant fit, but also program efforts and participant commitments. 2. Retention identifies 90 days of continuous program participation; the point in time where treatment begin to influence the participants attitude and behavior.

36 Four Key Performance/Outcome Measures 3. Exit Interview is a scheduled interaction, preferably face to to face, between program staff and the participant at the completion of his/her individualized program treatment plan. A key part of this exit interview is the completion of the Los Angeles County Participant Reporting discharge questionnaire. 4. Positive Compliance refers to participants who completed treatment or left before completing treatment with satisfactory progress. While completed treatment is the most successful conclusion, left before completion with satisfactory progress is also perceived as a positive outcome.

37 Los Angeles County Residential Programs Program Number and Size 75 residential programs located throughout Los Angeles County. Programs range from 12 participant admissions per year to 500 1,500 admissions per year. Programs by Gender 29 (39%) residential programs provide treatment services for men and women 27 (36%) residential programs provide treatment services for women only 19 (36%) residential programs provide treatment services for men only Programs by Race/Ethnicity 2 (3%) residential programs were predominantly White (> 70%) 9 (12%) residential programs were predominantly Black/African American (> 70%) 6 (8%) residential programs were predominantly Latino/Hispanic (> 70%)

38 Residential Programs by Performance Measures: Engagement N = 75 Mean = 70.6 Median = 71.0 Std. Dev. = 15.4 Skewness =.823 Kurtosis = Minimum = 13.4 Maximum = 100.0

39 Residential Programs by Performance Measures: Retention N = 75 Mean = 43.7 Median = 44.4 Std. Dev. = 16.1 Skewness =.473 Kurtosis =.250 Minimum = 1.7 Maximum = 77.8

40 Residential Programs by Performance Measures: Positive Compliance N = 75 Mean = 57.7 Median = 60.0 Std. Dev. = 21.1 Skewness =.380 Kurtosis =.213 Minimum = 11.1 Maximum = 99.1

41 Residential Programs by Performance Measures: Exit Interview N = 75 Mean = 81.0 Median = 84.2 Std. Dev. = 17.0 Skewness = Kurtosis = Minimum = 17.5 Maximum = 100.0

42 Residential Programs Performance Measures Fiscal Year Performance Measures Mean Median Std. Dev. Range Engagement 71 71% 15% % 100% Retention 44 44% 16% 2 78% Exit Interviews 81 84% 17% % 100% Positive Compliance 58 60% 21% 11 99%

43 Individual Residential Programs and Performance Measures Percent Program A Program B Engagement Retention Exit Interviews Positive Compliance

44 Current and Future Efforts to Improve Performance Measures & Implement CQI Design and Implement Approved Performance Measures and Standards Continued Implementation of Evidence Based Practices Design and Implementation of Performance Based Contracting

45 Research questions concerning performance measurement Are improvement in performance measures associated with improved client outcomes? Are performance measures valid and reliable? Does performance contracting improve treatment quantity and quality? Are there unintended consequences of performance contracting? Is case mix adjustment needed for performance contracting? If so, how?

46 Challenges in establishing a performance based system Limited experience of treatment staff in use of data to manage services. High turnover of staff Limited technology Definitions of many terms are not standardized Extensive portion of addiction service delivery is outside traditional programs programs. (eg eg. Suboxone, high end programs, sober living facilities).

47 Thank you Rick Rawson