Mapping and Evaluating Alberta s FASD Service Networks and Community-Based Initiatives Christopher Cameron Objective Research and Evaluation Inc. Hélène Wirzba Wirzba Consulting Inc. FASDlive 2014 Mapping Our Road to Success Saskatoon, October 8, 2014 Acknowledgements Thank you to: FASD Unit, Disability Services Division, Alberta Human Services FASD Service Networks coordinators and managers / staff of funded agencies, clients and caregivers Alberta Centre for Child, Family and Community Research Barb McLean Susan Abells, Abells Henry Public Affairs Christopher Cameron, Objective Research and Evaluation Inc. Evaluation Steering Committee Overview 1. The Alberta FASD Strategic Plan and service-delivery approach 2. FASD Service Networks 3. Mid-Term Evaluation of the FASD Strategy 4. AB FASD Service Network Evaluation Frameworks 5. AB FASD Strategic and Operational Plan 6. Use of Frameworks during Year-7 evaluation of the Alberta FASD 10-Year Strategy 7. Lessons learned 1
1. The Alberta FASD Strategic Plan and Service-Delivery Approach A cross-ministry, comprehensive, coordinated response to FASD FASD 10-Year Strategic Plan 2007-2017 12 regional FASD Service Networks Ministry Specific FASD initiatives Provincial Councils and Committees Partnerships Alberta s FASD Prevention and Service Cabined-Approved FASD 10-Year Strategy Ministers, DMs and ADMs FASD-Cross-Ministry Committee (CMC) Funded FASD Initiatives Ministry-Funded FASD Initiatives Provincial Network Leadership Team FASD-CMC Sub- Committees and Councils Alberta FASD Service Networks CanFASD Research Network & Other Partners Delivery of Provincially-Funded FASD Initiatives: Awareness, Prevention, Assessment and Diagnosis, Supports to Individuals and Caregivers. Alberta s FASD Prevention and Service Delivery Model (Abells Henry Public Affairs, 2014) 2. Alberta FASD Service Networks A key initiative of the Alberta FASD 10-Year Strategic Plan 12 community-based / communityled partnerships Reflect local priorities and needs Provide a single FASD point of entry in their region Supported by other regional and provincial FASD initiatives www.fasd.ab.ca 2
2. Alberta FASD Service Networks Three service pillars: Universal awareness and prevention, Assessment and diagnosis, Support services for individuals with FASD and their caregivers Client-focused and results-based programming 2007-2012: Foundation and Growth # Networks increased from 8 to 12 # Unique clients increased from 943 to 2,423 Developed governance structures 2013-2017: Focus on continuous improvement 3.Mid-Term Evaluation of the FASD Strategy What gets measured gets done. If you don t measure results, you can t tell success from failure. If you can t see success, you can t reward it. If you can t reward success, you re probably rewarding failure. If you can t see success, you can t learn from it. If you can t recognize failure, you can t correct it. If you can demonstrate results, you can win public support. From Osborn and Gaebler (1992) 3.Mid-Term Evaluation of the FASD Strategy The strategy called for an evaluation in years 5, 7 and 10 Mid-term review of 10-year strategy took place in 2011-12. First attempt to measure client and system outcomes Provided baseline measures against outcomes and targets set in the plan Findings: Major increase in the number of Albertans accessing FASD services in Alberta. Service Network participation in provincial, regional and local levels of strategic planning, networking and information exchange. Cost-effective service model. 3
3.Mid-Term Evaluation of the FASD Strategy 7 recommendations: Provide clients with assessment for intervention and wraparound services supported by a mentor system Define sustainability Clarify outcomes Develop a data collection model Further develop CMC governance structures Improve the CMC funding model Increase access to the Alberta FASD Service Network Program services 4. AB FASD Service Network Evaluation Framework Renewal Initiative Time-frame: July 2012 to June 2013. Community-led, intervention addressing Year-5 recommendations Lead: South Alberta FASD Service Network. Provincial steering committee with representation from various Networks, and the Cross-Ministry Committee. Wide consultation with FASD service providers, Network coordinators, clinicians, policy makers, evaluation experts, intended users Use of evaluation frameworks developed by BC Centre of Excellence for Women s Health (http://www.fasd-evaluation.ca) 4. AB FASD Service Network Evaluation Framework Renewal Initiative Underlying Principles: A set of minimum outcomes and indicators that reflect a common understanding of all Alberta FASD Networks and the FASD Cross-Ministry Committee, and that can be measured and collated provincially, using the common measurement tools. Making use of what is available (existing reporting systems). Incorporating evaluation into case management practices. Value added for service providers. Minimal administrative burden on already overworked staff. 4
4. AB FASD Service Network Evaluation Framework Renewal Initiative Deliverables 5 evaluation frameworks: Assessment and Diagnosis Prevention (PCAP Model) Supports for Individuals and Caregivers Universal Awareness Service Networks System outcomes and client outcomes Indicators for each of the outcomes Measurement Tools 4. AB FASD Service Network Evaluation Framework Renewal Initiative Measurement Tools: Bi-Annual Narrative Reports Surveys, Tracking tools Event and Resource Tracking Templates, Post-Event Survey Stakeholder Survey Client and Caregiver surveys for A&D and support Services Alberta FASD On-Line reporting system Focus Groups with Network Leadership Teams Case Studies 4. AB FASD Service Network Evaluation Framework Renewal Initiative Five Frameworks Program Logic Models and Evaluation Tools available on the Alberta FASD Website, public forum (http://fasd.alberta.ca/forum/) One example: Awareness Framework 5
Layout: Provincial Outcome Service Network Outcomes Client outcomes System outcomes Outputs Activities Target Population Inputs 6
5. AB FASD Strategic and Operational Plan Mid-Term Evaluation 7 recommendations: Provide clients with assessment for intervention and wraparound services supported by a mentor system Define sustainability Clarify outcomes Develop a data collection model Further develop CMC governance structures Improve the CMC funding model Increase access to the Alberta FASD Service Network Program services 5. AB FASD Strategic & Operational Plan Provincial Process started after the Networks Evaluation Framework Initiative Both processes informed each other. An Outcome-Based Management Plan: Results-based (clearly stated and measurable outcomes) Place-based (addressing unique regional needs) Performance management-based (continuous measures and reports to support learning and adaptation) Collaborative (shared responsibility and partnerships) Comprehensively implemented 2007-2017 FASD Strategies FASD Strategic and Operational Plan Pillars Strategy #1: Awareness and Prevention Pillar #1: Awareness (Level 1 Prevention) Pillar #2: Prevention Level 2 Prevention Levels 3 & 4 Prevention Strategy #2: Assessment and Diagnosis Pillar #3: Assessment and Diagnosis Strategy #3: Supports for Individuals Pillar #4: Supports for Individuals and and Caregivers Caregivers Strategy #4: Training and Education Strategy #5: Strategic Planning Strategy #6: Research and Evaluation Strategy #7: Stakeholder Engagement Pillar #5: Organizational Learning Stakeholder Engagement & Strategic Planning Training and Education Research and Evaluation Economic Evaluation 7
Other FASD-CMC Funded Initiatives (Evaluation?) FASD Service Networks (Service Network Evaluation Frameworks) Ministry FASD Initiatives (Evaluation?) FASC-CMC Strategic and Operational Plan Results-Based Budgeting 6. Year-7 Evaluation A logical next step was to test the new Service Network frameworks and the FASD Strategic and Operational Plan during Year 7 evaluation of the Alberta 10-year FASD Strategy. Use identical steering committee and evaluation consultants as in 2012-13 Seek evidence that year-5 Service Network recommendations were met Ensure ability to measure client outcomes and collate key performance indicators on a provincial level for the first time Use findings to refine evaluation framework and measurement tools 6. Year 7 Evaluation Year 7 Service Network Evaluation took place between June 2013 and March 2014. Had good compliance with the evaluation tools: All 12 Networks participated in the Evaluation 12 focus groups 283 stakeholder surveys, 1,169 post-event surveys, 391 client surveys, 464 caregiver surveys 8
6. Year-7 Evaluation Service Network Data constituted bulk of evaluation findings. 1 Provincial Network Evaluation Report & 12 Network-Specific Evaluation Reports Service Network evaluation findings linked well with Provincial performance indicators and outcomes. Testing both levels of evaluation frameworks highlighted need to further align and refine some outcomes and indicators. For example: Service Network Awareness pillar was broader than provincial Awareness pillar; some provincial indicators were not measurable Results and recommendations still in draft form. 7. Lessons Learned Service Network Evaluation Frameworks Development Collaboration was a key to the successful completion of the Service Network Evaluation Frameworks It was a slow process, and dozens of people participated in the process. It was important to find common grounds, agree on a minimum set of activities, outputs and outcomes, and definitions that were relevant for the services to be evaluated. 7. Lessons Learned Provincial FASD Strategic and Operational Plan Re-defining provincial Service Pillars before developing Service Network frameworks would have been helpful Service Network outcomes and indicators informed the development of Provincial outcomes and indicators good linkage Year 7 Evaluation demonstrated that it was possible to collect and compile project data to generate valid provincial information Many of the FASD initiatives in AB still require better linking of their outcomes with Provincial outcomes 9
7. Lessons learned Identify intended users and uses of evaluation. Build understanding and ownership of the evaluation process and findings. Commit time and resources to develop an evaluation plan that meets your needs. Agree on logic models, especially outcomes, select key indicators, and monitoring tools. Verify that data collection tools will generate outcome reports Establish baseline data. Monitor for results, report and use findings. 7. Lessons learned Use findings to inform policies Use findings to inform practices Sustain monitoring and evaluation systems within organizations Strengthen FASD Cross Ministry and Service Network Evaluation Practices Enhance existing data collection platforms to increase standardization and consistency of collection Conduct thorough and comprehensive evaluations of each service pillar Resources Alberta FASD Website: http://fasd.alberta.ca Service Network Evaluation Frameworks: Program Logic Models and Tools Forum Public Assessment and Diagnosis Surveys and Logic Models (Same for Service Networks, Support & Services, Targeted Prevention, and Awareness) FASD Strategic Plan, Yr 5 Evaluation Recommendations, Strategic and Operational Plan Forum Learning and Resources Publications http://www.fasd-evaluation.ca/home/ 10
Questions? Comments? Hélène Wirzba wirzbah@telus.net 403.330.4524 11