Transcription

1 ATCA Symposium 2010 Building Tomorrow s Services on Today s Information. Barry Evans Executive Director, The Buttery 1

2 In a QMS Review of The Buttery November 2001 the report commented; The ongoing contribution to State and National information collections is noted. However the service does not appear to use this information to systematically evaluate outcomes of service. Issues relating to statistical cohorts are acknowledged, but this has inhibited analysis of interim outcomes or the impact of strategies and treatment. 2

3 Today NGOs are increasingly expected to be accountable for the service they provide, the resources they expend and the outcomes they achieve. 3

4 Service accreditation and evidence based practice are now routine requirements for funding. These disciplines of management are required to sit alongside the traditional disciplines of social justice, access and equity in the welfare field where resourcing is perennially limited. 4

5 While evaluation research is seen as a necessity in our climate of increasing accountability, few individual agencies have the resources to carry it out. NADA Health Outcomes Project Final Report. Nov

6 History of Program Evaluation in The Buttery Therapeutic Community - 6

7 C.A.R.A. The 1990 s Client At Residential Agencies data did not monitor patterns of substance use, social and medical history, family background. CARA information was only available months after collection. CARA did not include a follow up of residents from which medium to long term effectiveness of the program may be inferred. 7

8 1995 The Buttery participates in an ATCA Peer Review which addresses:addresses:organization and Management Physical Environment Records Management Assessment and Treatment Rights of Clients Research, Planning and Evaluation Staff Development and Education Community Liaison and Participation 8

9 Treatment outcomes were not evaluated by this quality assurance process. 9

10 1995 ADCA Forum Treatment and ResearchResearch Where to from Here? Commonwealth Department of Health and Family Services supported the Minimum Data Set Project for AOD Treatment. NDARC and the Australian Institute of Health and Welfare designed and implemented the collection of the NMDS for all Commonwealth funded AOD services. 10

11 1996 NSW NADA Health Outcomes project. Objective:To Objective: To develop a client record data set that was; 1.Locally maintained by participating agencies. 2.Recorded consistent demographic data. 3.Facilitated a process of client outcome monitoring. 4.Contributed to the development of benchmarks (Health Outcome Indicators) for treatment services. 11

12 The Buttery implemented the NADA designed data set and contributed a cohort of fifty five residents to an initial composite data set of some eight hundred and eighty clients from the initial thirteen participating agencies in

13 The Buttery followed this cohort in 1998 successfully contacting 69% of the original residents, establishing positive outcomes in all of the original indicators except for nicotine consumption. This result reinforced the value of the Health Outcomes project as a feedback mechanism to The Buttery and it s potential to provide evidence based outcomes to funding providers. 13

14 1998 NSW Department of Health funds NDARC to develop and trial NSW MDS. This data set was not designed to measure treatment outcomes but was promoted as a necessary precursor to the comparable measurement of outcomes across different settings in NSW. 14

15 July 2000 the statewide collection of data from all NSW Government funded AOD treatment agencies in NSW begins 15

16 National and State Minimum data set requirements and an appreciation of the value of the Health Outcomes data set led to the development of a Buttery multipurpose database for internal evaluation purposes. Three data sets were combined in one program together with clinical information required by The Buttery program. 16

17 Health Outcome followfollow-up studies by The Buttery have been conducted in

18 The inclusion of the health outcomes fields in The Buttery data base provided the basis for future research projects which potentially tied treatment outcomes to the treatment process. The thesis:thesis:- that the program design within the treatment context of the therapeutic community at The Buttery resulted in improved health outcomes for program participants. 18

19 . Considerations for Evaluation in The Buttery and other therapeutic communities Randomized trials are problematic in therapeutic communities for ethical reasons. Evidence of the efficacy of a therapeutic community s treatment approach could come from follow up studies. Internal evaluation of program elements identifies which parts of the program are well received and relevant to treatment from a residents perspective. 19

20 The Buttery research objectives; 1.To determine whether program objectives have been met. 2. To determine the success or otherwise of the program. 3. To determine if the program is targeted effectively? 4. To determine how the program can be improved. 20

21 Study design:design:two levels. 1. Internal evaluation of program elements based on Exit Surveys of residents. 2. External evaluation of program based on pre and post treatment surveys to determine health outcomes for Buttery residents. 21

22 Level 1: Internal Evaluation Exit Interviews, Computer based.

23

24

25

26 RESPECT FOR CONFIDENTIALITY (VOL) good/very good ok/not very good don't know % st QTR nd Half st Half nd Half st Half

27 HOW WELL THEY RUN GROUPS (CASEWORKERS) good/very good ok/not very good don't know % st QTR nd Half st Half nd Half st Half

28 NEW RESIDENTS GROUP very/fairly useful a bit/not useful not sure % st QTR nd Half st Half nd Half st Half

29 Level 2: External evaluation. The design of the external evaluation follows that of the NADA Health Outcomes Pilot Project. Questionnaire administered:administered: Before and after treatment, 12 months followfollow-up by telephone, Generally using an interviewer not previously known to the respondent. Permission to contact the respondent for followfollow-up is obtained in writing during first interview. Attempts made to contact all eligible clients who had signed consent forms.

30 Successful follow up rate is partly attributable to the fact that the half way houses are in Byron Bay and many exexresidents live in the area for a time after leaving The Buttery. 30

31 Information collected for the Buttery Outcome Studies includes; 1. Principal drug of concern. 2. Profiles of substances used for each drug used:used:frequency and length of use. Pattern of use. Whether problem use. 3. Risk behaviour. 31

32 4.Quality of relationships. 5.Main source of income. 6.Criminal activity. 7.Employment history. 8.SF36. 32

33 Additional information distinct from the MDS included:-included: length of stay, completion of program phases, referral source, postcode of origin etc. average age by sex. average delay before admission. 33

34 The information is routinely collected for each client on admission and entered on MS Access database along with MDS. The follow up Health Outcome interview is then added to the database at 12 months to enable comparisons to be made between the two sets of data. 34

35 The Outcomes: Is the size of the effect clinically important? 35

36 The 2002 Study sample: sample:1. Data was collected on admission for the period 01/06/1997 to 11/02/2001 resulting in a raw database of 178 residents. 2. Eligibility: (a) a minimum of 3 months stay, must have left more than 12 months ago and less than 18 months. (b) participants had signed an agreement to take part in the outcome study. 36

37 (c) participants had remained in The Buttery for at least 2 months or had completed Program One (now 3months but it had previously been 10 weeks). 3. This resulted in an eligible sample for follow up of 98 exex-residents. Non--eligible = 80 (45% of the population) Non 37

38 Of the 98 clients included in the survey, 67 (68%) were successfully followed upup- a high success rate for a survey of this kind. Of the 67 residents followed up only 37 had both a pre and a post test result on the SF36. 38

39 Percent using DrugUse At ProgramEntry &12MonthsPost ProgramExit Pre-Treatment Post-Treatment Alc oho l Her oin Can nab is Am phe tam ine s Pren=176, Post n=67 39

40 ProblemDrugUse Percent Pre-Treatment Post-Treatment Alco hol Her oin Can Ben Am Coc phe nab zod tam aine iaza is ines pine s 40

41 Crime 100 Percentage Involved Pre-Treatment Charged Post-Treatment 41

42 Risk Behaviour 100 Percent ShareNeedles OperateMachinery UnsafeSex Pre-Treatment Post-Treatment 42

43 SatisfactionwithRelationships Percentage No ts ati sfi ed Al ittle So me wh at Mo stl y Ve ry Pretreatment Post treatment 43

44 SF-36MeanScores Genera M B l Health ental Health odilypain Physica R R S V l Functio olephysical oleemotional ocial Function itality ning Pre-Treatment Post-Treatment 44

45 With the exception of physical functioning all SF36 health indicators in this sample remained below the NSW norms two years after entering treatment. 45

46 2008 Follow up study

47 Criteria for participation: Successfully completed at least three months and exited the Buttery between Jan 04 and Dec 08 (i.e. at least 12 months had elapsed since departure); Consented to Follow Up; and Completed a Short Form (SF) 36 - Personal Well Being self assessment on arrival to the Buttery and on departure from the Buttery.

48 This left 147 (58% of 255) exex-residents that could potentially be followed up. Of these 147, 53 (36%) were successfully followed up. Those who couldn t be followed up had refused to be interviewed or failed to respond to contacts (8) or were untraceable in the time available to complete the Follow Up (85).

49

50

51

52

53

54

55 These outcomes reflect the need for a clear understanding of realistic outcome expectations for the treatment group and agencies offering treatment to a group with severely compromised health exacerbated by drug/alcohol dependence. Abstinence is only one limited measure of a problem which manifests in a range of life matters. 55

56 Limitations of the results. Is there a real effect (or could the findings be due to bias, or the play of chance)? It was possible that the follow up samples were not representative of the entry population since the followfollow-up sample included only persons staying at The Buttery for a minimum of 3 months or to the end of Program 1. 56

57 Of the initial cohorts approximately 50% stayed less than the required time or did not give permission to be followed up. This cohort may have been a higher risk group in terms of continued drug use, criminal behaviour etc. but they did not stay in treatment long enough for it to have an impact. 57

58 Methodological and practical considerations of conducting outcome evaluation without the benefit of separate research funding. Lack of dedicated research personnel means existing clinical and administration staff must be used for initial data collection, follow up and data entryentry- potentially compromising objectivity and producing respondent bias. 58

59 Specialised help is needed to ensure data is collected systematically and competently in order to:to:1. minimise errors 2. identify missing information. 3. analyse the data and interpret the results. Computer and data program skills are necessary in order to maintain an agency database. database. 59

60 Clinical staff are often focused on service delivery, counseling and therapeutic issues. The discipline required to ensure the recording of accurate and reliable data is not a priority which easily takes priority over the day to day demands of working with a taxing client group. 60

61 Analysis and review of data collected requires specialised knowledge and skills not always readily available to NGOs. Outcome studies are time consuming and they require dedicated resources. 61

62 Some of the limitations were addressed by The Buttery with the development of a specialised client data base which incorporated client information including personal details, details of previous and current treatment, psychological test scores and the outcome measures described previously. In 2009 the database was modified to collate survey results into reports and represent them in tables or graphs. 62

63 Although costly to set up, once in place the database has the facility to generate reports for statistical and monitoring purposes and the monthly MDS as well as enabling The Buttery to report on resident treatment outcomes in an easily read format. 63

64 The Buttery datasets provide an opportunity for control of information collected by it with the cooperation of participating residents. 64

65 There needs to be a link between treatment elements, treatment experiences and treatment outcomes to substantiate the contribution of the TC to longlong-term recoveries. de Leon G (2000). The therapeutic community: Theory, model, and method. Springer Publishing Co, Inc: New York, NY, USA. 65

66 The program experience of residents at The Buttery completing at least three months of treatment and positive health outcomes identified in follow up surveys suggests a causal relationship between the program design, model and elements and the treatment results. This link is demonstrated in the Level 1 Evaluation ie: Exit Surveys of program elements completed by exiting residents and; Level 2 Evaluation ie: Health Outcome assessment pre and post treatment.