Alberta Health Services Addiction and Mental Health Primary and Community Care

Transcription

1

2 Alberta Health Services Addiction and Mental Health Primary and Community Care May 30, 2013

3 Acknowledgements Without our partnered organizations good will and support, this evaluation would not have been possible. Additionally, the significant contributions of many Alberta Health Services (AHS) In Roads staff whose willingness to participate was invaluable to the completion of the evaluation. Finally, a heartfelt thank you two individuals who sorted through volumes of information to provide such a thorough and comprehensive evaluative report: Kate Woodman, PhD AHS Evaluator Nicole Van Kuppeveld, MBA Manager, DTFP In Roads We are grateful for the opportunity to deliver the In Roads program in Alberta and we look forward to continuing the program in 2013/14.

4 Final Evaluation - ii TABLE OF CONTENTS LIST OF TABLES... iv LIST OF FIGURES... v EXECUTIVE SUMMARY... vii SECTION 1: INTRODUCTION... 1 Background... 1 Purpose of the Evaluation... 2 Evaluation Implementation... 2 Evaluation Participation... 3 SECTION 2: PROJECT DESCRIPTION... 3 Project Goal... 3 Project Objectives... 4 Target Population... 7 Activities... 7 Outputs Expected Outcomes Across the Three Phases Operating Context Project Management/Governance Structure and Administration Project Stakeholder Relationships SECTION 3: EVALUATION SCOPE AND METHODS Evaluation Scope Evaluation Issues, Questions and Indicators Analytical Procedures for Both Qualitative and Quantitative Information Process Used to Arrive at Conclusions and Recommendations Methodological Limitations SECTION 4: INTERIM EVALUATION FINDINGS Was the project implemented as Intended Were the expected activities undertaken and outputs delivered What Course Corrections were made, why and what is the impact on the project s ability to reach its expected outcomes What challenges were encountered and how were they addressed... 67

5 Final Evaluation - iii What worked particularly well What could have been improved What are the key lessons from this project To what extent was progress made toward the project s expected outcomes What changes are recommended to help ensure the project reaches its expected outcomes SECTION 5: FINAL EVALUATION FINDINGS The DTFP and AHS Immediate (Short Term Outcomes Immediate Outcomes Summary The DTFP and AHS Intermediate Outcomes Intermediate Outcomes Summary What are the key lessons from the project What are the implications for future work on systems and/or service improvement What are the project s key contributions to the DTFP outcomes To what extent has the project provided good value for DTFP s money invested What recommendations (stakeholders) flow from the project What are the findings by other project outcomes Success Stories SECTION 6: PROJECT LESSONS Summary of lessons learned Lessons to Improve Evidence Informed Practice Lessons to Advance Achievement of DTFP Outcomes SECTION 7: CONCLUSION AND RECOMMENDATIONS Conclusion Statement of Project Findings Recommendations SECTION 8: LIST OF REFERENCES SECTION 9: APPENDICES Appendix A: Evaluation Tools Appendix B: Supporting Documentation ACRONYMS

6 Final Evaluation - iv LIST OF TABLES Table 3.1 In Roads Evaluation Issues & Questions Table 3.2 In Roads Indicators & Data Sources for the First Phase of In Roads Table 3.3 Indicators & Data Sources for Capacity Building with Community Agencies.. 21 Table 3.4 Indicators & Data Sources for Service Delivery to Clients Table 3.5 Outcomes & Indicators of the In Roads Project Table 3.6 Summary of Data Collection Methods Table 3.7 Topics Covered by the Agency & Service Provider Needs Assessments Table 3.8 Semi-structured Interview Key Areas Table 4.1 Outputs, Indicators & Data Sources for Phase 1 In Roads Program Table 4.2 In Roads Partner Agency Program Categories Table 4.3 Outputs, Indicators & Data Sources for Phase 2 In Roads Program Table 4.4 Knowledge & Skills Include in the In Roads Project Table 4.5 Number of Activities, Participants & Hours Spent on Capacity Building Table 4.6 Calgary Number of activities, participants & hours spent Table 4.7 Edmonton Number of activities, participants & hours spent Table 4.8 Central Number of activities, participants & hours spent Table 4.9 Knowledge exchange activities with non-partnered agencies Table 4.10 Outputs, Indicators & Data Sources for Phase 3 In Roads Table 5.1 Pre-post paired sample t-test of participants - substance abuse Table 5.2 Pre-post paired sample t-test of participants - counseling practices Table 5.3 Pre-post paired sample t-test of participants in addiction screen learning Table 5.4 Pre-post paired sample t-test of participants knowledge in intervention Table 5.5 Pre-post paired sample t-test of participants confident in mental health Table 5.6 Self report on increased capacity - summary of data by highest responses.. 99 Table 5.7 Pre-post mentee survey summary of data by highest responses

7 Final Evaluation - v LIST OF FIGURES Figure 1 In Roads Logic Model... 5 Figure 2. In Roads Governance Structure Figure 3. Job classifications of In Road s Mentees Figure 4. Mentee Educational Background Figure 5. Would the Following Learning Areas be of Interest to your Staff? Figure 6. Mentee Self-Report on Previous Training Figure 7. Mentee Interest in the In Roads Learning Areas Figure 8. Pre-mentorship Rating of Knowledge, Skill & Knowledge Figure 9. Service Provider Responses on Mentorship & Resources Figure 10 Attendee Feedback on All Presentation Provided by In Roads Figure 11 Percent of Sample Response: Participation in Presentation/workshop Figure 12 Participant Feedback on Putting Unity in Community Conference Figure 13 Percent of Sample Response: Action Towards Collaboration in Community Figure 14 Client Gender Figure 15 Client Age Figure 16 Employment or Education status of Clients Figure 17 Housing Status of Clients Figure 18 Percent of Sample Response: Was client administered a screener? Figure 19 Why screening was not administered? Figure 20 Percent of Sample Responses: Were any brief interventions used? Figure 21 In Roads interventions used during client interactions Figure 22 Percent of Sample Response: Was client referred for additional services Figure 23 Number & type of referrals given to clients at partnered agencies Figure 24 Agency Representative Survey Responses Figure 25 Participant perspective on collaboration & networking Figure 26 Pre-post comparison of participants knowledge, skills & confidence Figure 27 Number of participants reporting decreased Figure 28 Do you plan on incorporating some of the knowledge & skills Figure 29 Pre-post comparison of participants knowledge, skills & confidence Figure 30 Number of participants reporting in counseling practices Figure 31 Do you plan on incorporating some of the knowledge & skills... 83

8 Final Evaluation - vi Figure 32 Pre-post comparison in addiction screening Figure 33 Number of participants reporting in addiction screening learning area Figure 34 Incorporating knowledge & skills from learning area Figure 35 Pre-post comparison of participants in intervention techniques Figure 36 Number of participants reporting decreased in intervention techniques Figure 37 Incorporating knowledge & skills from learning area Figure 38 Pre-post comparison of participants knowledge in mental health learning Figure 39 Participants reporting decreased or improved knowledge Figure 40 Incorporating knowledge & skills from learning area to practice Figure 41 How likely are you to work with clients at your organization Figure 42 Participant perspectives on impact of In Roads program on client Figure 43 Client feedback responses Figure 44 Percent of Sample Response: Would you like more help Figure 45 What kind of services are you considering... 97

9 Final Evaluation - vii Executive summary Overview of the In Roads Project In 2009, Alberta Health Services (AHS) received an $8.05 million grant from health Canada s Drug Treatment Funding Program (DTFP) to strengthen the overall addiction service system in Alberta. In partnership with Health Canada s DTFP program, AHS Addiction and Mental Health created the In Roads project. The project was intended to improve Alberta s addiction service system by coordinating and collaborating with community agencies that serve high-risk youth and young adults. The aim of In Roads was to assist community agencies in developing capacity to incorporate evidence-based addiction practices into their service provision to high-risk youth and young adults (ages years) with substance abuse issues. Through knowledge exchange activities, education and a mentorship model that paired addiction counselors with community providers, In Roads increased the capacity of community agencies. The addiction counsellor (mentor) and the community provider (mentee) worked collaboratively to determine how to implement evidence-informed practices such as addiction screening and early intervention into day-to-day interactions with clients. Over a period of 32 months, In Roads partnered with 48 agencies in three centres in Alberta: Calgary, Edmonton, Red Deer. Final Evaluation The purpose of the final project evaluation was to assess whether In Roads achieved the stated Health Canada and Alberta Health Services projected outcomes. Evaluation results confirm the project was implemented as planned; it delivered the activities and outputs identified at project inception; it was successful in terms of the identified target population. Not only does it meet immediate outcomes, it also shows trends towards contributing to longer-term objectives. The Final Evaluation elaborates on the four key conclusions garnered from In Roads evaluation findings: a) collaboration is essential and improves service; b) the mentorship model is effective; c) clients indicated they received key services from the mentees; and, d) developing an evidence-informed curriculum creates demand for the resources.

10 Final Evaluation - viii The evaluation process used multiple lines of evidence and multiple methods (including pre/post mentee assessment). Change in sector capacity was assessed. The evaluation process was divided into three key phases that occurred in tandem depending upon the readiness and current competency levels of the stakeholder organization, its staff, and the time at which the agency joined the In Roads program. Phase One: Partnership with Community Agencies Phase Two: Capacity Building with Service Providers Phase Three: Service Delivery to Clients The evaluation data includes aggregates on the activities and outputs delivered, from metrics gathered across three cohorts of agency intake: Cohort 1: August 2010 Cohort 2: May 2011 Cohort 3: May 2012 Some Key Findings service providers indicate increased confidence in assisting the target population (youth and young adults with substance abuse and addictions concerns) community service providers have increased understanding of substance use and addiction utilization of evidence-informed practice includes improvement at time of intake; new capacities to accept clients and commit to working with them; new abilities to implement brief intervention; and increased empathy for clients adoption of a Harm Reduction Model; clients are reportedly staying in programs longer and referrals to AHS Addiction and Mental Health Services are more accurate and appropriate all agency leads indicated observing implementation of skills and learning from In Roads; service providers affirmed their ongoing commitment to the utilization of evidence-informed approaches and techniques service providers report improved capacity to engage parents and care providers as well as increasingly calling AHS to find resources for their clients

11 Final Evaluation - ix Conclusion In Roads has achieved its objective of linkages and knowledge exchange between agencies, service sectors and jurisdictions. Stakeholders report nearly unanimous agreement of the value of In Roads and subsequent improvement in service delivery. Conclusions of the Evaluation demonstrate improved competency in delivering services and affirmation that the learning achieved through In Roads is and will continue to be used to inform practice change.

12 Final Evaluation - 1 SECTION 1: INTRODUCTION Background In partnership with the Health Canada Drug Treatment Funding Program (DTFP), Alberta Health Services (AHS) created the In Roads project. The initiative was intended to strengthen the overall addiction service system through innovative coordination and collaboration with community agencies that serve at-risk youth and young adults (12 24 years old). 1 The aim of the In Roads project was to: Partner with community agencies; Assist them in developing capacity to incorporate evidence-based addiction practices into their service provision; Increase collaboration across the addiction treatment continuum and Increase awareness of available resources. Throughout the project cycle In Roads partnered with 48 community agencies located in Calgary, Edmonton, and Red Deer. The initiative was implemented from August 2010 March The In Roads project increased community agency capacity by providing knowledge exchange and skill development through direct mentorship relationships between an experienced addictions counselor (mentor) and the community service providers (mentee). The two worked collaboratively to determine how to implement basic evidence-based practices, such as addiction screening, into the mentee s day-to-day interactions with youth. The In Roads Knowledge Exchange training model involved the development of customized learning programs, designed to address the identified learning needs, previous training, and clients served by each mentee. The In Roads project integrated and supported the DTFP s key investment areas of enhancement of linkages; the exchange of knowledge between people, organization service sectors and jurisdictions; and, increasing the implementation of evidence-informed practice information. 1 Note that throughout this report, youth and young adults stands for this target population of at-risk youth and young adults who are vulnerable to substance abuse and addictions.

13 Final Evaluation - 2 Purpose of the Evaluation This final (and summative) evaluation is designed to measure outcomes achieved in providing addiction-related services by the community agencies and their staff to the youth and young adults they serve. Utilizing multiple lines of evidence, and multiple methods (including pre/post mentee assessment), change in sector capacity is assessed. The approach and direction taken for the In Road s project evaluation was developed in conjunction with Health Canada (HC) to ensure alignment with the national DTFP outcomes. The purpose of this final project evaluation is to assess whether the In Roads initiative has achieved the stated HC and AHS outcomes. Evaluation data in support of this final report includes aggregates on the activities and outputs delivered, from metrics gathered across three cohorts of agency intake (n=48): 2 Cohort 1: August 2010 (n=19 agencies) Cohort 2: May 2011 (n=10 agencies) Cohort 3: May 2012 (n=19 agencies) Evaluation Implementation To support project development and evidence-based decision making, the evaluation was implemented by an internal AHS evaluator, supported by a team of data input and analysis staff. This team was overseen by the In Roads Provincial Program Manager. Over the course of the project, two evaluators worked with the team; one managed the formative report (delivered January 2012) and the second one completed the final (summative) report (May 15, 2013). The In Roads evaluation plan was developed collaboratively with Health Canada to ensure alignment with national DTFP program evaluation goals. In addition to direction from HC, and the provincial team noted above, provincial and zone program coordinators and addiction counselor supervisors also participated in the evaluation process. Finally, In Roads partnered community agencies and their clients provided data for the evaluation. 2 Note that the first wave of formative data (from Cohort 1, only) was reported on in the In Roads January 2012 Interim Report to Health Canada.

14 Final Evaluation - 3 Evaluation Participation The In Roads project evaluation utilized multiple lines of evidence from key stakeholders: AHS In Roads team staff (Community Facilitators, Mentor Supervisors, and Addiction Counselor Mentors) Partner agency representatives/ leads Partner agency service providers (mentees) Agency clients These stakeholders participated in multi-method data collection, responding to surveys, semistructured group interviews, and providing administrative data (e.g., quarterly reports, meeting minutes). SECTION 2: PROJECT DESCRIPTION Project Goal As agreed to at project inception, the In Roads initiative is unique in that it contains aspects of both the treatment systems and treatment services targets of the DTFP investment areas. The overall goal of the In Roads project is to increase access to evidence-based, early intervention treatment services for youth and young adults who are at-risk or experiencing substance abuse problems. The initiative achieves this goal through wider system-level changes and practice changes at the individual, service provider level. Best practices for assisting youth at-risk of, or with, an addiction problem were created by Health Canada (2001, 2008) and suggest that to be effective, services for youth at-risk need to be accessible "where they are both physically and developmentally. The Alberta In Roads project was designed understanding that the first point of contact for many high risk youth is community organizations (e.g., emergency shelters, youth drop-in centers) rather than the formalized substance use treatment system. In addition, many high risk clients may not enter (or remain in) the formalized addiction treatment system due to their readiness to change, the extent of their addiction or various other barriers. The In Roads project is based on the premise that, for these hard to reach clients, a community agency may be a less intimidating initial engagement and

15 Final Evaluation - 4 service delivery setting than a more formalized treatment system. The initiative theorizes that atrisk youth are already attending community agencies to meet a variety of needs (e.g. food, clothing, shelter, employment assistance) and have established relationships with the agencies service providers. However, while these community service providers have established relationships with the youth, they may lack evidence-based knowledge, and the skills required to detect problematic substance use, discuss and intervene appropriately and match clients to additional treatment services. The In Roads project was designed to address this gap by facilitating knowledge exchange and skill development within and among these community agencies. The goal was not to enable community services providers to become experts in substance abuse, but to build capacity in community agencies by helping them develop basic evidence-based knowledge and skills regarding substance use and addiction. They in turn are then better able to understand and engage in conversations about substance use, recognize and screen for substance use and mental health issues, provide some early interventions and refer to additional services where necessary (In Roads Project Manual, 2013). Project Objectives The In Roads project was undertaken in three phases (outlined below); each phase aligned with and contributed to achieving the following three objectives: Engage stakeholder organizations and service providers who work with high risk youth and young adults in Edmonton, Red Deer and Calgary; Enhance the substance abuse-related competencies of service providers who are in contact with at-risk youth and young adults; and, Community service providers provide screening, referrals and brief early interventions to the youth and young adults that frequent their agencies. The three phases were delivered through a two streamed approach to assisting community agencies develop their capacity to address substance abuse and addiction challenges with their clients (see Figure 1 Logic Model).

16 Final Evaluation - 5 Figure 1 IN ROADS LOGIC MODEL Increase access to evidence-informed, early intervention treatment services for youth and young adults at risk of or experiencing substance abuse problems GOAL Phase 1 C apacity to D eliver Service Phase 2 Phase 3 Service Delivery OBJECTITVES Engage stakeholder organizations and service providers who work with high risk youth and young adults in Edmonton, Red Deer and Calgary Enhance the substance abuse-related competencies of service providers who are in contact with at-risk youth and young adults. At-risk youth and young adults are screened, receive early intervention and referred as appropriate for substance abuse problems. A C TIVITIES Identify stakeholder organizations that provide services to at-risk youth and young adults 12 to 24 Identify service providers within the organization for training/mentorship Collaborate with organizations to determine organization/staff training needs Undertake knowledge sharing and exchange activities Mentoring/training of organization s service providers Develop and conduct presentations and workshops Develop resource materials Organization service providers provide early intervention addiction services to youth including screening and brief interventions Organization service providers provide referrals to additional treatment or services TARGET POPULATIONS Stakeholder organizations and their service providers that are in contact with at-risk youth and young adults Service providers and front line workers who are in contact with at-risk youth and young adults At-risk youth and young adults OUTPUTS Partnerships between AHS and stakeholder organizations established Service providers for mentorship/training identified Organization/staff training needs identified Mentorship relationships established Training provided Presentations and workshops conducted Resource materials developed Screening, brief intervention and additional treatment or service referrals are conducted SHORT-TERM OUTCOMES Enhanced Collaboration between AHS and stakeholder organizations Increased service providers knowledge and skills on substance abuse screening, early intervention and referral. Increased awareness of treatment options for youth and young adults at-risk of or experiencing substance abuse problems. Youth and young adults have increased knowledge about substance abuse issues/effects and treatment options available IN TER M ED IA TE OUTCOMES Enhanced capacity to deliver evidence-informed early intervention services for at-risk youth and young adults. Increased access and use of evidence-informed practice information. Increased engagement in early intervention activities within partner organizations Increased access to sustainable, evidence-informed early intervention treatment services by at-risk youth and young adults Note: Shaded outcomes show alignment with DTFP outcomes

17 Final Evaluation - 6 The first stream was the primary modality in which the initiative helps community service providers gain substance use capacities and it was where the majority of resources and activities were devoted. The first stream imparted substance use knowledge and skills through a direct mentorship relationship. In this approach, and based on a needs assessment of relevant knowledge and skills, an AHS addiction counselor mentored a community service provider on how to implement evidence-based practices into their everyday work with clients. In other words, the two service providers worked collaboratively to address the substance use issues typically found in the environment where the agency service provider worked with youth. To avoid disruption of the regular services provided at the agency, the In Roads project provided partnered agencies with funding to cover-off for the staff that was participating in the phase I portion of the initiative. The In Roads project took a needs-based learning approach, where the transfer of knowledge and skills required to work with the target population was tailored to the agency s service need, client demographics, as well as the individual service provider s previous learning and training experiences. In addition to adapting the individual learning, the AHS mentor worked with the community service providers to facilitate the incorporation of knowledge and skills into their everyday work with clients, largely through one-to-one engagement. The information was delivered to the agency and service providers through direct mentorship training, information sharing, job shadowing, presentations or workshops, and tours and observations of AHS formalized treatment settings such as detoxification or residential treatment services. According to project developers the strength of this learning approach was that it allowed for a community service provider to participate fully in learning evidence-based knowledge and skills. Further, it provided them time to gain practice in using and incorporating these skills in their regular work environment (In Roads Project Manual, 2013). 3 The second stream of the In Roads project was developed in recognition that to address system level change, it was important to collaborate with a wide variety of community agencies and programs (In Roads Project Manual, 2013). However, it was recognized that some community 3 Note that this training design is congruent with adult education principles, such as problem-based and collaborative, with equality between the teacher and learner (Zmeyov 1998; Fidishun 2000).

18 Final Evaluation - 7 agencies would be not able to participate in the in-depth mentorship and capacity building involved in the first stream. The In Roads project developers wanted to ensure that a wide variety of community agencies would have the opportunity to benefit from the evidence-based information provided through the project initiative. As a result, the second stream of the In Roads initiative provided non-partnered agencies with knowledge exchange opportunities such as presentations or workshops on topics of interest to their organization and staff (In Roads Project Manual, 2013). In sum, the objectives of the project were to collaborate with community agencies to determine how to build evidence-based capacity, using a needs-based learning approach, to promote a more coordinated and accessible system of addiction services. The initiative was based on the premise that: Strengthened and collaborative partnerships and increased capacity among community agencies to provide preliminary substance use information and services contributes to, Youth and young adults at-risk having greater access to addiction information and basic services through the community agencies that they frequent; resulting in, A strengthened addiction system that provides evidence-based services to at-risk youth and young adults (In Roads Project Manual, 2013). Target Population At times, clients encounter difficulties navigating complicated and diverse systems to receive addiction treatment. This decreases the likelihood of follow-through and engagement in a treatment process. To that end, the In Roads project targeted youth and young adults at-risk of substance abuse (12 24 years of age), who were living in what has been coined the drug corridor of Edmonton, Red Deer, and Calgary. 4 Activities The In Roads project was undertaken in three phases: 1. The first phase was the implementation of the In Roads project within partnered community agencies; 4 The target population is referred to throughout this report as youth and young adults.

19 Final Evaluation The second phase involved enhancing capacity to deliver services using a needs-based learning approach; and, 3. The third phase described the activities and outcomes involved in the delivery of services to the client. These phases occurred concurrently depending upon the readiness and current competency levels of the stakeholder organizations and their staff members and the time/ cohort at which the agency joined the In Roads project. Phase One: Partnership with Community Agencies Engaging stakeholders was the first objective of the In Roads project. Based on the results outlined in the initial situational analysis 5, the target populations for this objective were community organizations and their service providers within the cities of Edmonton, Red Deer and Calgary, who were already in contact with at-risk youth and young adults aged 12 to 24. First, partnerships between community organizations and AHS were established and service providers (mentees) were identified and agreed to participate. Second, an organization-level needs assessment and individual needs assessment was conducted to determine the learning needs of agency staff. AHS addiction counselor mentors and program coordinators used the information gathered during the needs assessment to tailor the mentorship and knowledge exchange activities specific to the organization s and service provider s needs. The collaboratively determined learning priorities grounded the training and mentorship activities that occurred in Phase Two. Phase Two: Capacity building with service providers The objective of the second phase was to increase capacity to deliver services to youth and young adults by enhancing the substance abuse-related competencies of the service providers identified during Phase One. The target populations associated with this objective were community agency service providers. Knowledge exchange activities were key to building mentee competencies and supporting the ongoing sharing of evidence-informed knowledge. The activities identified for this phase included: 5 See the AHS In Roads Project proposal (July 2008).

20 Final Evaluation - 9 Providing mentorship and facilitating learning; Designing and implementing presentations or workshops; and Developing resource materials to support the mentorship. All service providers engaged in learning sessions/activities to increase their knowledge of substance use and the resources available across the continuum of treatment services in Alberta. Mentees further developed knowledge and skills in substance use, abuse and dependency, basic addiction counseling practices, addiction screening, mental health and early intervention techniques. AHS addiction counselor mentors provided community participants with support, feedback, problem-solving and guidance. Further, the mentors shared learning resources so that mentees could incorporate the evidence-based practices into their day-to-day work with clients. Training, workshops and presentations introduced participants to new and evidence-informed information or Description of the In Roads Learning Areas practices in a Knowledge exchange provided to In Roads participants was based on selected topics from the CCSA Technical Competencies as well as materials used by specific area of AHS Addiction and Mental Health. The stated goal of the In Roads project was interest or identified not to make participants into addictions experts, but rather to assist participants in gaining evidence-based knowledge and skills to support their need. While learning interactions with the youth and young adults that frequent their agencies. In Roads used the foundational and basic levels identified in selected CCSA key was ongoing as part competencies and built curriculum to facilitate agency service providers in of an established those learning areas. The first five learning areas are based on CCSA technical competencies, while the topic of referral and knowledge of addiction mentorship treatment options and resources was added to as a core component of In Roads by program developers. The In Roads program s learning areas are as relationship, follows: workshops and Understanding substance use, abuse and dependency Addiction counseling practices presentations were Addiction screening 1 time-limited (i.e., a half day to two days). Part of this experience included learning how to administer screening tools and participating in workshops on specific topics (e.g. motivational interviewing, harm reduction). It should be noted that both partnered and non-partnered agency staff were invited to participate in these learning events.

21 Final Evaluation - 10 Another activity of this phase included creating resource materials (e.g., curriculum and standard operating procedures) to support learning and incorporate the knowledge and skills into practice within the agency. These resources were intended to increase the service providers range of knowledge and skills, and to function as an ongoing resource allowing participants to share with other agency staff members who did not directly participate in the In Roads initiative. The curriculum designed in the framework of the In Roads project was vetted by the project s Clinical Council. This Council was convened by the project team and composed of key project stakeholders (e.g. project coordinator, community facilitators; AHS Addiction Services and Mental Health supervisors; and, family counsellors) to regulate the standard of the material with Canadian Centre on Substance Abuse (CCSA) competencies and its evidence-informed base. The final product was a 32 module curriculum that will be published and made available to AHS staff. Phase Three: Service delivery to clients The third and final phase of the In Roads project corresponded to the service delivery component of the initiative. The target populations associated with this objective were at-risk youth and young adults. The objective linked to this phase was that the target population was provided information regarding substance use, abuse and dependency, addiction screening (if appropriate) and offered support, early interventions and/or referral for addiction services. The primary activities involved in service delivery include: Substance abuse screening; Using early intervention techniques; and, Providing referrals to treatment or services. It is important to note that although these activities were offered by the partner organizations, it is recognized that not all youth frequenting the agency would require or consent to participating in screening, brief intervention or follow through on recommended referrals. According to the In Roads initiative developers, teaching the mentees to provide early intervention addiction services included capacity building in knowing what services and techniques were effective for an individual client. For example, not all youth and young adults screened would require brief interventions or referral to specialized services. The In Roads project was designed to assist mentees in presenting customized options to youth that best meet the client s needs as well as/or including their readiness to address substance abuse issues.

22 Final Evaluation - 11 Outputs Throughout the implementation of the project, as this report shows, the following outputs were both targeted and achieved: Phase 1: Partnerships between AHS and stakeholder organizations established; Service providers for mentorship/knowledge exchange identified; and, Organization/staff learning needs identified. Phase 2: Mentorship relationships established; Knowledge exchange provided; Presentations and workshops conducted; and, Curriculum developed. Phase 3: Screening, brief intervention and additional treatment or service referrals are provided to clients. Expected Outcomes Across the Three Phases (Including Links to DTFP Program Outcomes) Phases one and two objectives and activities focused on collaborating and building capacity within the partner organization. Both phases contributed to the primary short-term outcome of enhanced collaboration between AHS and partner organizations. This increased collaboration contributed to the secondary short-term outcomes of developed service provider knowledge and skills in substance abuse, screening and early intervention. In addition, it contributed to increased service provider awareness of treatment options for youth and young adults. Together these short-term outcomes facilitated the intermediate outcomes of enhanced capacity to deliver evidence-informed early intervention services and increased access, use of evidence-informed practice information, and increased engagement in early intervention activities within the partner agencies.

23 Final Evaluation - 12 Phase three outcomes were identified as the following: Immediate outcomes: By training mentees to offer early intervention addiction services, the goal was that the youth and young adults they engaged with would have increased knowledge about their substance use, the effects of their use and possible abuse, and the treatment options available to them. Intermediate outcome: It is anticipated that if at-risk youth and young adults have increased awareness of the early intervention services available in the organizations they frequent, they will be more likely to participate in activities that address their substance abuse problems with community service providers or in the specialized treatment system, when needed. Long-term outcome: All three phases contribute to the long-term outcome of increased access to evidence-informed early intervention services by youth and young adults. In Roads linkages to DTFP outcomes The In Roads project s goals and outcomes were aligned with the larger, national DTFP outcomes. As depicted by the shaded areas on the logic model (see Figure 2. Logic Model) the initiative supports DTFP investment areas and has direct links to DTFP immediate, intermediate and longterm outcomes. A primary DTFP investment area was the enhancement of linkages and the exchange of knowledge between people, organization service sectors and jurisdictions. Attached to this investment area was the immediate outcome of enhanced collaboration on responses to treatment system issues among jurisdictions and stakeholders. The In Roads project partnered with youth and young adult-serving community organizations to address gaps in the substance use treatment continuum (from community services to specialized treatment services). This collaboration assisted in strengthening the linkage between community organizations and Alberta Health Services, Addiction and Mental Health. It allowed AHS and community organizations to address collaboratively the treatment service issues directly related to individual agencies, the larger community of service providers and the province of Alberta. In addition to strengthening the link between AHS and partnered agencies, the In Roads project created collaboration and networking

24 Final Evaluation - 13 opportunities among those community agencies participating in the program, as well as with nonpartnered community agencies. Another DTFP investment area closely aligned with the In Roads initiative is that of enhancing the implementation of evidence-informed screening and early intervention information. Within the framework of the In Roads initiative, an evidence-based training curriculum was developed. Guided by the Clinical Council that was convened for this work, the In Roads Program Manual and Curriculum (32 modules) was created and vetted to ensure the document was aligned with CCSA standards. In addition, the project team included consulting psychologists (one for each zone) who met regularly with the mentors (AHS addiction counselors) to facilitate their core training role in the In Roads project. To implement the learning modules, the In Roads project paired AHS addiction counselors with mentees. The mentors provided knowledge and skills on implementing evidence-based screening and early intervention information into the mentees day-to-day work with youth and young adults. It was anticipated that support for, and mentoring of, agency service providers would enhance their capacity to provide early intervention services to the youth and young adults they serve. The DTFP and In Roads project shared the long-term goal of increasing access to sustainable, evidence-informed early intervention treatment services for at-risk youth and young adults. The project foregrounded collaboration with community agencies and service providers. It assisted them in developing their capacity to address the substance use issues of the youth and young adults seeking supports. In Roads anticipated that as a result, those youth and young adults will have greater access to the continuum of addiction services offered throughout the community. Operating Context In Roads began as a project proposal created (2008) by the Alberta Alcohol and Drug Abuse Commission (AADAC) and submitted to Health Canada s Drug Treatment Funding Program (DTFP). AADAC was the organization charged with implementing substance use prevention, treatment and information programs in Alberta. In May of 2008, Alberta s healthcare system was restructured and healthcare-related organizations, including AADAC, were merged into one Alberta Health Services (AHS). As a result of this merger, the former AADAC is now part of the

25 Final Evaluation - 14 larger AHS organization and has been further merged with mental health services, now known as Alberta Health Services, Addiction and Mental Health. Alberta s DTFP project, In Roads, falls under the purview of AHS Addiction and Mental Health. The In Roads project is a community-based program and thus most of the activities included in the program are undertaken within partnered community agencies. As a result, the operating context of the In Roads program varies based on the partnered agency s context, culture and clientele. Project Management/ Governance Structure and Administration The In Roads project was administered by the Community and Specialized initiatives unit within the provincial arm of Alberta Health Services (AHS) Addiction and Mental Health, Primary and Community Care. The project was implemented in three (3) operational zones (Edmonton Zone, Central Zone [Red Deer] and Calgary Zone) and delivery of the project was managed by the addiction and mental health leadership in each of the respective zones. Collaborative by design, decision-making valued consensus building across all entities involved with the initiative. This value flowed down to the individual mentee, who collaborated with their agency lead and the mentor in the design of a curriculum that would meet her/his individual needs. The In Roads project also engaged a Clinical Council that provided input and direction to the knowledge exchange/mentorship component of the project, by supporting and advising on the curriculum, and vetting the document for alignment with CCSA standards. The Council convened by the Provincial Co-ordinator on an as needed basis to provide clinical expertise to the project and included AHS Addiction and Mental Health representatives from across the province. In addition, Consulting Psychologists were retained to offer information and support to the mentors, also on an as needed basis.

26 Final Evaluation - 15 Figure 2, below, presents the governance structure of the In Roads project. Note that the Zone diagram offered is replicated three times, for each of Edmonton, Central, and Calgary. Figure 2 In Roads Governance Structure

27 Final Evaluation - 16 Project Stakeholder Relationships There are various stakeholders who are directly or indirectly involved in developing, supporting and/or delivering the In Roads project. The following are descriptions of the various stakeholders involved in health services in Alberta and the In Roads project. Health Canada s Drug Treatment Funding Program (DTFP): Health Canada provides funding to the In Roads project. HC has overall responsibility to ensure the DTFP partners (e.g. provincial/territorial governments and other key stakeholders) establish common objectives, priorities, and outcomes that are in alignment with the DTFP initiatives (HC, 2008). Alberta Health (AH): is the branch of the Alberta government that works with Alberta Health Services to set the strategic direction for health services, including addiction and mental health. Alberta Health Services (AHS): is the organization responsible for providing health care to all Albertans. Addiction and Mental Health services are part of this larger provincial health system. As such, AHS Addiction and Mental Health provide the support required for the delivery of the In Roads initiative, such as overall project coordination, project operation, project monitoring, as well as project reporting (financial, progress and performance). AHS, Primary & Community Care Addiction & Mental Health: is responsible for the delivery, overall coordination and implementation of the program, evaluation and reporting. This includes planning all program specific responsibilities that align with project goals and objectives. The In Roads team is composed of AHS staff responsible for implementing the provincial direction for the In Roads project. This team includes the program manager, provincial coordinator and the evaluator. In Roads Clinical Council: a group of clinicians from AHS Addiction and Mental Health responsible for providing clinical guidance to the In Roads project, through the creation and vetting of the project curriculum. Community Facilitators: these team members are experienced AHS staff members responsible for building relationships with community agencies and conducting local-level administrative duties (e.g., contract management). Community Facilitators also provide

28 Final Evaluation - 17 knowledge exchange opportunities to community agencies who are not formally partnered with the In Roads project. There are three Community Facilitators, each responsible for one operational zone (Edmonton, Calgary or Red Deer). AHS Mentor Supervisors: these addiction counselor supervisors are responsible for supporting AHS Addiction Counselor Mentor staff in implementing the In Roads project in community agencies. The In Roads program has one locally-based addiction counselor supervisor for each operational zone (Edmonton, Calgary or Red Deer (Central)). AHS Addiction Counselor Mentors: these team members are trained addiction counselors employed by Alberta Health Services to act as mentors to community agency staff members. There are four mentors in the Calgary zone, four mentors in the Edmonton zone and three mentors in Red Deer (Central zone). Consulting Psychologists: registered psychologists are contracted by In Roads to support AHS Mentors in developing their clinical and mentorship skills. There are three consulting psychologists supporting the In Roads program (one per zone). In Roads Partnered Community Agencies: Community agencies (e.g. youth emergency shelters, drop-in centers) who are in a formal partnership with the AHS In Roads project and who are funded to allow staff to engage in the project and are provided cover-off for these service providers to participate in the In Roads mentorship initiative as well as knowledge exchange presentations or events. The partnered community agencies are composed of: Agency Representatives: These individuals act for the community agency that In Roads is partnered with regarding contracting, staffing, scheduling, invoicing, reporting and any other agency-level implementation or decision making. Participants (Mentees): Community agency staff members participating in the In Roads project. Non-Partnered Community Agencies: These community agencies are not formally partnered with the In Roads project, but benefit from the initiative by participating in knowledge exchange presentations or events.

29 Final Evaluation - 18 SECTION 3: EVALUATION SCOPE AND METHODS Evaluation Scope The approach, direction and scope of this evaluation were developed collaboratively with Health Canada. 6 This final evaluation focuses primarily on the contribution achieved towards stated project outcomes (immediate and intermediate), including both the DTFP and AHS identified outcomes. It should be noted that this final evaluation report is an analysis of data collected throughout the In Roads project cycle (August 2010 March 2013). The In Roads project addressed substance abuse through larger, system level change as well as practice changes achieved by individual providers. Many indicators were measured on both an agency (as a whole) and an individual (agency staff member) participant levels in order to capture impacts in each of these domains. Table 3.1 lists the key evaluation questions that the In Roads project has reported on (see the Interim Report, January 2012) and reports on in this final document to Health Canada. The following sections lists outputs, outcomes, indicators and data sources for the three phases of the In Roads project. Evaluation Issues, Questions and Indicators Table 3.1 In Roads Evaluation Issues and Questions Reporting Status Interim Report See: In Roads Program Interim Evaluation (January 2012) Interim Evaluation Issues Evaluation Questions Was the project implemented as intended? Were the expected activities undertaken and outputs delivered? Implementation process challenges and areas for improvement What course corrections were made, why and what is the impact on the project s ability to reach its expected outcomes? What challenges were encountered and how were they addressed? What worked particularly well? What could have been improved? Evaluate progress To what extent was progress made toward the 6 The evaluation results are presented in terms of the progress made on the project performance indicators that were developed collaboratively with Health Canada (In Roads Evaluation Plan, 2010).