Evaluating Our Advocacy Progress. How do we know if it s working? Julia Coffman Center for Evaluation Innovation March 22,

Transcription

1 Evaluating Our Advocacy Progress How do we know if it s working? Julia Coffman Center for Evaluation Innovation March 22,

2 We ll talk through three big questions. 1. Why evaluate advocacy? 2. What outcomes are most relevant and realistic for advocacy and policy change efforts? 3. What manageable tools can we use to collect data?

3 Be clear about why you re evaluating. Demonstrate Impact Strategic Learning Examining whether a case can be made that an advocacy effort produced its intended results. Providing data that will inform your advocacy efforts while they are happening.

4 What outcomes are relevant for advocacy? 4

5 Focus on interim outcomes and progress. Policy Goal progress

6 Interim outcomes are the changes you expect as you work toward your goal. Think about the changes you will see in your audiences.

7 This framework can be useful. ACTION HOW will they change? OUTCOMES AWARENESS WILL Media Advocacy Public Will Campaigns Political Will Campaigns Communications and Messaging Advocacy Capacity Building Public Forums Leadership Development Voter Outreach Demonstration Programs Public Awareness Campaigns Public Education Litigation Lobbying Community Mobilization Model Legislation Coalition Building Regulatory Feedback Community Organizing Public Polling WHO will change? Influencer Education Policy Analysis/Research Champion Development Policymaker Education PUBLIC INFLUENCERS AUDIENCES DECISION MAKERS 7

8 You can map any advocacy strategy onto it. ACTION Community Mobilization Lobbying Litigation Model Legislation 1 Coalition Building Regulatory Feedback Community Organizing Champion Development WILL Media Advocacy Public Will Campaigns Communications and Messaging Advocacy Capacity Building Political Will Campaigns Public Forums AWARENESS Leadership Development Voter Outreach Demonstration Programs Public Awareness Campaigns Public Polling Policy Analysis/Research Public Education Influencer Education Policymaker Education PUBLIC INFLUENCERS DECISION MAKERS

9 Use it to make sure tactics match expected outcomes. X ACTION Community Mobilization Lobbying Litigation Model Legislation 1 Coalition Building Regulatory Feedback Community Organizing Champion Development WILL Media Advocacy Public Will Campaigns Communications and Messaging Advocacy Capacity Building Political Will Campaigns Public Forums AWARENESS Leadership Development Voter Outreach Demonstration Programs Public Awareness Campaigns Public Polling Policy Analysis/Research Public Education Influencer Education Policymaker Education PUBLIC INFLUENCERS DECISION MAKERS X

10 Where are you positioned? ACTION Community Mobilization Litigation Model Legislation Coalition Building Regulatory Feedback Community Organizing Champion Development OUTCOMES WILL Media Advocacy Public Will Campaigns Communications and Messaging Advocacy Capacity Building Leadership Development Political Will Campaigns Public Forums AWARENESS Voter Outreach Demonstration Programs Public Awareness Campaigns Public Polling Policy Analysis/Research Public Education Influencer Education Policymaker Education PUBLIC INFLUENCERS AUDIENCES DECISION MAKERS

11 Use it to think about interim outcomes. ACTION WILL AWARENESS New Advocates New Donors Leadership Development Voter Outreach Public Education Community Mobilization Constituency Growth Community Organizing Public Will Public Will Campaigns Advocacy Capacity Building Advocacy Capacity Public Awareness Campaigns Coalition Building Media Coverage Media Advocacy Salience Communications and Messaging Issue Reframing Demonstration Programs Attitudes & Beliefs Influencer Education Model Legislation Regulatory Feedback Champion Development Political Will Campaigns Public Forums Public Polling Policy Analysis/Research Knowledge More or Diversified Funding Collaboration & Alignment Political Will Issue Visibility and Recognition Litigation New Champions Lobbying Policymaker Education PUBLIC INFLUENCERS DECISION MAKERS 11

12 Interim Outcomes Awareness Action Advocacy Tactics Increased knowledge Increased issue visibility or recognition Changed attitudes or beliefs Will Increased salience Constituency growth Increased collaboration among advocates Increased media coverage New and active advocates New and active high-profile champions New donors Policy Goals Increased public or political will Increased or diversified funding Increased advocacy capacity

13 What methods or tools are appropriate? 13

14 Both traditional and unique methods can be used. Methods Traditional Interviews Surveys Focus Groups Polling Bellwether Methodology Intense Period Debriefs Critical Incident Timelines Network Mapping Unique Champion Tracking Media Scorecards 360-Degree Critical Incident Reviews Research Panels Case Studies Policymaker Ratings Evidence of Change Journals Social Media Tracking Rapid Response Research

15 Policymaker Ratings Support Influence Confidence 1 = not at all supportive 2 = interested 3 = somewhat supportive 4 = supportive 5 = extremely supportive 1 = not very influential 2 = somewhat influential 3 = Influential 4 = extremely influential 1 = not very confident 2= somewhat confident 3= extremely confident Developed by Harvard Family Research Project

16 Policymaker Ratings Year 1 Year Not at all supportive Somewhat supportive Supportive Extremely supportive Developed by Harvard Family Research Project

17 Policymaker Ratings Del Norte Siskiyou Modoc * Hypothetical Data Low Support Medium Support High Support Trinity Shasta Lassen Humboldt Tehama Plumas Mendocino Glenn Butte Sierra Nevada Colusa Yuba Lake Sutter Placer El Dorado Sonoma Yolo Napa Sacramento Alpine Amador Solano Marin San Calaveras Costa Joaquin Tuolumne Mono Contra San Francisco San Alameda Stanislaus Mariposa Mateo Santa Clara Santa Merced Cruz Madera San Fresno Inyo Benito Tulare Monterey Kings San Luis Obispo Kern San Bernardino Santa Barbara Ventura Los Angeles = Increase in Support Orange Riverside San Diego Imperial Developed by Harvard Family Research Project

18 Policymaker Ratings Low Influence 68 legislators 57% of the legislature Confidence Rating 5% 70% 25% Lo Med Hi 22 legislators 18% of the legislature High Influence Confidence Rating 60% 40% Lo Med Hi Low Support 49% Democrat 51% Republican 64% Democrat 36% Republican 67% Assembly 33% Senate 50% Assembly 50% Senate 17 legislators 14% of the legislature Confidence Rating 40% 60% Lo Med Hi 13 legislators 11% of the legislature Confidence Rating 100% Lo Med Hi High Support 80% Democrat 20% Republican 85% Democrats 15% Republican Developed by Harvard Family Research Project 100% Assembly 75% Assembly 25% Senate

19 Bellwether Methodology Bellwethers are: Influentials in the public and private sectors whose positions require that they are politically informed and that they track a broad range of policy issues Policymakers Administrators Media Other Advocates Funders Business Associations Developed by Harvard Family Research Project

20 Bellwether Methodology What are the top 3 Issues on the children s (ages 0-18) policy agenda? 2005 (n=40) 2008 (n=44) 1. K-12 education 2. Health care 3. Preschool 1. K-12 education 2. Preschool 3. Health care

21 Bellwether Methodology How would you address the achievement gap? Can preschool help address the achievement gap? 2008 FULL SAMPLE (n=72) % 1. K-12 reform 67% 2. Education funding/resources 47% Don't Know 2% No 14% 3. Nonschool supports (health care, family involvement, after school) 4. Supplemental school programs (ELL, literacy programs) 32% 29% Yes 84% 5. Preschool/early care and education 21% 6. Other 13%

22 Bellwether Methodology What is the likelihood that California will increase preschool investments in the next 3 years? Not at all likely 20% Not very likely 50% Very likely 9% Somewhat likely 20%

23 Intense Period Debriefs / After-Action Reviews Level of Advocacy Activity Debrief Debrief Developed by Innovation Network, Inc.

24 Intense Period Debriefs 1. What events triggered this intense period? 2. How was the organization s response determined? Who made the decision? How was it communicated? 3. Which elements of the response worked well? Which didn t? 4. What was the outcome of the intense period? Was it positive or negative? 5. What challenges were faced (anticipated and unanticipated)? How were they addressed? 6. What do you wish you had known going in? How can you get that information next time? 7. What insights do you have and how can they be applied? Developed by Center for Evaluation Innovation 24

25 Wrap Up Advocacy is a great opportunity to use evaluation to support your learning. It s okay to prioritize and focus on what is most important to assess. If you find that your strategies and tactics need to change, change them and your evaluation accordingly! Be creative this is an emerging field. 25

26 Center for Evaluation Innovation 26