Proposal to Evaluate the Aboriginal Youth Leadership Program (AYLP) of the Canada World Youth (CWY)

Transcription

1 One Evaluation Consulting Taking your evaluation in the right direction Proposal to Evaluate the Aboriginal Youth Leadership Program (AYLP) of the Canada World Youth (CWY)

2 The One Evaluation Team Sally Payne Methodology/Stakeholder Liaison Olivia Cleary Program Theory Analyst Ashley Baker Evaluation Advisor/Statistical Analysis Sandra Parsons Program Context/Conflict Mitigation

3 Canada World Youth CWY is a not-for-profit organization that has been offering development opportunities for youth (and in turn, their communities) for four decades, through the vehicle of experiential learning, facilitated by international partnerships

4 Aboriginal Youth Leadership Program Overview A tailored extension of the experiential components of Canada World Youth s core program Asset-based approach Hands-on learning experiences designed to increase knowledge, skills & abilities necessary to pursue education & employment opportunities Sustained community development

5 Aboriginal Leadership Youth Program Key Stakeholders Participants - Aboriginal youth from across Canada aged Community - Partner communities (exhanges) - Inuit communities in Labrador - Internship communities i.e. Nippising First Nation External Organizations - Partner organizations (local, national and international) i.e. UN - Aboriginal organizations i.e. Mi'kmaq Confederation of PEI, National Youth Inuit Council - Academic Centres i.e. McGill ISID - Research Centres i.e. IVCO Staff and Management - CWY Board of Directors - CWY Management Team (including Director of Aboriginal Programs) - CWY Patron (Governor General of Canada) - Project Supervisors - Program Manager - Community Support Persons i.e. mentors, resource persons Funders/Donors - Institiutional Donors i.e. Government of Canada, Saskatechewan Council for International Co-operation - Monthly Gift Donors - Annual Campaign Donors - Various public and private donors and sponsors

6 Program Inputs CWY Board of Directors CWY Management and Staff Volunteers Funding Community Partners International Partners External Organizations

7 Program Activities and Outputs # of Aboriginal youth reached Pre-departure training # of health and safety training sessions Re-entry training United Nations events Internships # of critical learning assessments # of UN events # of internships Youth forums # of youth forums Short exchanges # of short exchanges AYLP Curriculum Strengthen multidisciplinary network # of credits obtained by youth # of multidisciplinary connections

8 Program Outcomes preparation for international affairs ability to engage In international cooperation communication and leadership skills capacity to support community development competencies employment-related knowledge cultural understanding support of Aboriginal youth participation in AYLP Participants develop new skills Participants obtain employment Participants are more integrated into the community Participants are engaged internationally Improved integration of aboriginal youth into the community at large, with increased involvement in academics and employment, leading to sustained community development, and making them exemplars for international change

9 CWY Board of Directors CWY Management and staff Volunteers Funding Community Partners International Partners Pre-departure training Re-entry training United Nations events Internships Youth forums Short exchanges # of Aboriginal youth reached # of health and safety training sessions # of critical learning assessments # of UN events # of internships # of youth forums # of short exchanges preparation for international affairs ability to engage In international cooperation communication and leadership skills capacity to support community development competencies employment-related knowledge cultural understanding Participants develop new skills Participants obtain employment Participants are more integrated into the community Participants are engaged internationally Improved integration of aboriginal youth into the community at large, with increased involvement in academics and employment, leading to sustained community development, and making them exemplars for international change External Organizations AYLP Curriculum Strengthen multidisciplinary network # of credits obtained by youth # of multidisciplinary connections support of Aboriginal youth participation in AYLP Assumption: Aboriginal youth will be reached and will participate in AYLP opportunities. Risk: Aboriginal youth are not aware of program activities. Assumption: Relevant output measurements are accurate and representative of change. Risk: Confounding variables affect output measurements. External Factors: Ability to recruit Aboriginal youth, availability of necessary funding and resources, recruitment of volunteers, and collaboration with multidisciplinary agencies.

10 Evaluation Questions 1.) Rationale/Relevance Evaluation Questions 1.1. Does this program meet the needs of Aboriginal youth and their communities? Do the intended outcomes of the program align with the values and goals of diverse Aboriginal communities? 1.2. Do other programs with similar objectives to the Aboriginal Youth Leadership Program exist within the jurisdiction? Potential Indicators - comparison to existing literature - # youth accessing the program -extent to which the program activities and objectives meet the needs of the communities -evidence that programs overlaps or duplicates efforts of other programs

11 Evaluation Questions 2.) Design and Delivery Evaluation Questions 2.1. To what extent is the program being delivered as intended? Are all intended program activities being carried out as intended? 2.2 To what extent has CWY successfully integrated Aboriginal culture, language, traditional knowledge and identity in the Aboriginal Youth Leadership Program? Potential Indicators -views of program staff -# of community members reporting barriers -types of barriers reported by program participants

12 Evaluation Questions 3.) Impact Evaluation Questions 3.1. What are the quantity and quality of personal and professional relationships formed by Aboriginal youth who participate in the program? 3.2. What impact is the program s Aboriginal grounding having on youth s skill and further educational aspirations? 3.3. To what extent is the program facilitating youth remaining or returning to their Aboriginal communities to become community assets? How can this be improved? Potential Indicators -views of program participants -reported improvement of participants skills -views of program participants -reported improvement of participants skills - pre/post skill assessments -views of program participants

13 Evaluation Questions 4.) Efficiency and Economy Evaluation Questions 4.1. Do the activities of the Aboriginal Youth Leadership Program represent the most cost-effective use of funds? 4.2. Have different program approaches been identified that can more efficiently and effectively achieve the same or similar outcomes? Potential Indicators -comparison to existing similar programs and initiatives -comparison to existing similar programs and initiatives

14 Evaluation Approach Formative Participatory Evaluation Evaluation Steering Committee Contribution Analysis Mixed-Methods Approach

15 Methodology/Analysis Method Jurisdictional Scan Document/Literature Review Interviews Focus Groups Survey Pre/Post Assessment of Skills Analysis Descriptive Analysis/Trend Analysis Trend Analysis Qualitative Analysis, Social Network Analysis Qualitative Analysis, Social Network Analysis Qualitative and Quantitative Analysis Qualitative and Quantitative Analysis

16 Potential Challenges and Mitigation Strategies Challenges Mitigation Strategies Attribution Contribution analysis built into evaluation in the early stages Collection of baseline data to facilitate future summative evaluation Ethical considerations when working with vulnerable populations Ethics approval from IRB Informed consent Sensitivity training for evaluators Engage community leaders to facilitate focus groups

17 Credentialed Evaluator Competencies Technical practice competencies Interpersonal practice competencies Detailed program theory depicted in logic model Careful construction of evaluation questions An appropriate mix of quantitative & qualitative methods to collect data Plan to address anticipated challenges; proactive approach to the attribution challenge Multiple communication methods, based on appropriateness: , telephone, and focus groups. Collaboration and partnership demonstrated through use of community leaders to facilitate focus groups Attendance to issues of diversity and culture: sensitivity training, translation when feasible

18 Cultural Considerations Cultural Sensitivity Training Engaging Community Members/Leaders in Data Collection and Analysis Steering Committee Including Community Members/Leaders

19 Acknowledgements Canadian Evaluation Society and Case Competition Funders Faculty of Science, Memorial University of Newfoundland Monique Campbell Shannon Fisher & Nicole Power

20 Thank you for your consideration. One Evaluation Consulting Taking your evaluation in the right direction