Taking research to practice: Program design and evaluation LESSONS FROM THE DOORWAY PILOT

Transcription

1 Taking research to practice: Program design and evaluation LESSONS FROM THE DOORWAY PILOT

2 Contents 1. Overview of the model and summary of outcomes (20 mins) 2. Personal reflections on the Doorway program (10 mins) 3. Lessons from the Doorway evaluation (10 mins) 4. Open discussion (20 mins)

3 1. Overview of the model and summary of outcomes (20 mins) 2. Personal reflections on the Doorway program (10 mins) 3. Lessons from the Doorway evaluation (10 mins) 4. Open discussion (20 mins)

4 Overview of Doorway Designed to enhance the capacity of individuals with a serious mental illness who were homeless or at risk of homelessness to lead independent, healthy and meaningful lives in stable housing within communities of their choice Pilot was implemented by MI Fellowship with funding from the Victorian Department of Health (now DHHS) Pilot housed and supported 59 individuals over a three year period between 2011 and 2014 in two metro and one regional AMHS catchment

5 Demand for the program ~ ? Queries Referrals received Intake Housed Housed (11/13) Chose not to continue Left Doorway 18 9

6 Core elements of the model Eligibility criteria Serious mental illness and requiring service from an AMHS Homeless or at risk of homelessness Eligible for segment 1 of the public housing waiting list (but may not be currently on the list) Currently case-managed by AMHS Want to live in the designated area Willing to accept support Willing to contribute 30% of household income to rent Integrated team Core elements Family members, friends and community members Case manager from AMHS Housing & Recovery Worker Participant Flexible elements Employment Consultant AOD worker Physical health professionals Cultural and spiritual supports Values Choice Social Inclusion Sustainability

7 Key enhancements to Housing First Adaptations that reflect Doorway s local operating context Integrated teams Emphasis on psychosocial support Adaptations intended to improve non housing outcomes Focus on natural support networks Specialist employment assistance Adaptations intended to enhance sustainability Open rental market Participants hold the lease Combined housing and recovery role

8 Health outcomes The average time in bed-based clinical mental health services per participant per year decreased by half in the 12 months preand post-housing The total Emergency Department presentations across all participants in the 12 months pre- and post-housing decreased by one third The mental health of one-third of participants improved to the point of their being able to be discharged from their Area Mental Health Service Participants largely attributed their improved mental health outcomes to having stable accommodation and an integrated support team two firsts for many participants.

9 Economic outcomes The percentage of participants engaged in paid and unpaid work increased from 16% to 27% at the time of the evaluation The proportions of Doorway participants accessing education and vocational training opportunities and receiving qualifications whilst in the program increased

10 Housing outcomes Most participants reported feeling more independent, having greater levels of self-respect and pride and finding greater meaning in their lives as a direct result of having more stable and secure accommodation The number of tenancy related incidents was relatively low, with six lease breaks by participants, ten breach of duty notices and no evictions Participants developed tenancy related skills, including money management

11 Comparison of housing costs Figure 1: Comparison of housing costs ( data where available) $26,802 $28,105 $21,900 $16,060 $10,136 $9,417 Doorway housing (average p/a) Public housing^ Community housing Crisis accomm - Hostel style^ Crisis accomm/transitional housing^ Other supported accomm^ ^ All social housing options include the cost of capital to Government with the exception of community housing given that data is not available.

12 1. Overview of the model and summary of outcomes (20 mins) 2. Personal reflections on the Doorway program (10 mins) 3. Lessons from the Doorway evaluation (10 mins) 4. Open discussion (20 mins)

13

14 1. Overview of the model and summary of outcomes (20 mins) 2. Personal reflections on the Doorway program (10 mins) 3. Lessons from the Doorway evaluation (10 mins) 4. Open discussion (20 mins)

15 Our work on the Doorway pilot program taught us four important lessons as evaluators 1. The importance of commencing evaluation work before a program is implemented 2. The benefits of genuine partnership between the evaluator and implementing agency 3. The value of being able to draw on multiple data sources to evaluate progress 4. Sometimes you have more than one client nousgroup.com 15

16 Lesson 1: The importance of commencing evaluation work before a program is implemented Program logic framework completed before first participant joined Evaluation tools purpose built to align with indicators Data collection practices embedded into service delivery Able to see the evolution of the story from day one nousgroup.com 16

17 Lesson 2: The benefits of genuine partnership between the evaluator and implementing agency Composition of teams remained unchanged Presented evaluation findings on monthly basis Interaction across all levels of MI Fellowship Genuine value in reflective practice Able to evolve and enhance model as new evidence emerged nousgroup.com 17

18 Lesson 3: The value of being able to draw on multiple data sources to evaluate progress Diverse array of data sources Reduced dependence on single data sources Multiple avenues for exploring and validating outcomes Figure 2: Primary data sources used in Doorway evaluation Quantitative data Six monthly data collection by staff Outcomes measurement tools Department of Health (Vic) datasets (CMI-ODS, VAED, VEMD) Qualitative data Six monthly data collection by staff Participant and carer focus groups Key stakeholder interviews nousgroup.com 18

19 Lesson 4: Sometimes you have more than one client MI Fellowship as the primary client DHHS s role as independent but very interested funders Strong levels of buy-in by Doorway participants nousgroup.com 19

20 1. Overview of the model and summary of outcomes (20 mins) 2. Personal reflections on the Doorway program (10 mins) 3. Lessons from the Doorway evaluation (10 mins) 4. Open discussion (20 mins)

21 Question for your consideration 1. What roles should evaluators play in multi-year programs? 2. How can we design program evaluations that enable program participants to effectively participate? 3. What role should funders play in evaluations?