Evaluating the impact of institutional funding for CSOs: The case of the UK Civil Society Challenge Fund

Transcription

1 Evaluating the impact of institutional funding for CSOs: The case of the UK Civil Society Challenge Fund Anne-Marie O Riordan Making Impact Evaluation Matter September 2014

2 Research background Two year collaborative research project Triple Line and the Uni. of Bath (UK) Triple Line is a development consultancy specialising in grant management, & design, implementation, M & E and learning Collaboration objective: Triple Line uses robust methodological techniques and systems to report and learn from challenge funds

3 Presentation outline 1. Challenge funds defined 2. DFID s Civil Society Challenge Fund (CSCF) 3. Study of CSCF project evaluations 4. Findings 5. Implications and conclusions

4 1. What is a challenge fund? (1) Provides grants or subsidies (2) With an explicit public purpose (3) Shared between independent agencies, with grant recipients (4) Who are selected competitively (5) On the basis of advertised rules and processes (6) Who retain significant discretion over formulation of their proposals and execution of their project and (7) Who share risks with the grant provider. - O Riordan et al. (2014)

5 1. Impact evaluations of challenge funds: do they exist? But where is the impact and performance evidence to support this explosion of challenge fund programmes? I can t find it - David Elliott (2013) Springfield Centre There are virtually no widely available, credibly reported results of donor partnerships with [development organisations] - Kindornay et al. (2013) North-South Institute There is currently little evidence to show whether they achieve the anticipated development impacts - Adam Kessler (2013) DCED

6 2. Civil Society Challenge Fund Characteristic Civil Society Challenge Fund Grant or subsidy Public intent Inter-agency relationship Eligibility & competition Screening & selection rules Autonomy Risk-sharing Up to 100% grant: 164m committed since ,000: average DFID grant, 98,000: average matched funding Global fund: Empowerment of marginalised people through awareness of their rights to have improved access to better quality services. DFID grant funding mechanism ( ), managed by Triple Line & Crown Agents since 2010, with one UK CSO and at least 1 in-country partner Small UK-based CSO with developing country implementing partner(s) 1,400 proposals: 353 organisations: 526 selected projects- 171 organisations 2 stage process (now closed): Initial concept note submission followed with successful first stage applicants asked to submit a fuller proposal. Capacity building and support role for the FM e.g. logframe development Independent implementation, agreed grantee reporting requirements Proactive management of high risk projects (those in challenging environments or weak systems)

7 2. Example of a CSCF service delivery project Sierra Leone 5 years duration, 290,000 Facilitated improved health care access Training in TB detection and suspect referral and basic palliative care for people living with HIV, AIDS and TB Piloted a palliative care model Community sensitisation campaigns

8 2. Example of a CSCF rights- focused project Peru 3 years duration, 400,000 Enabled farmer leaders to raise public awareness of smallholder rights, advocate for a greater allocation of public investment and services in their communities Ensured that national policy commitments to support growth in the smallholder coffee sector are implemented.

9 2. CSCF monitoring & evaluation of achievements & results Grant set-up Project implementation Project completion YR1 YR2 YR3 YR4 Project logframe Project annual progress reports Project completion report Quality assurance visits External evaluation report Technical team appraisal DFID achievement rating score

10 2. Independent evaluation guidelines Fund manager issued guidelines to assist grantees commission a final project evaluation Outlines the evaluation approach Explains how report will be used Outlines scope Determines report format Outlines the evaluation process Key documents to consult

11 3. Purpose of the study To examine the process, rigour and utility of end-of-project evaluations in the CSCF Are project level independent evaluations useful? How might they contribute to a wider impact evaluation of a fund?

12 3. Research methodology 2 elements to the study 1- Analysis of sample of 10 evaluation reports Content analysis 2- Perception study of stakeholder views Interviews: examine the organisation and utility of the end-of-project evaluation process (grantees, FM technical team, DFID)

13 3. Sample overview Country Thematic Area Primary intended beneficiaries Evaluation budget As % of total budget Guinea Livelihoods Civil society organisations 5, % (CSOs) Bangladesh Social inclusion Deafblind children and young 8, % adults Sierra Leone Health People living with HIV and TB 1, % Ghana Social inclusion Women (in political positions) 10, % Zambia Education Girls and young women 2, % Peru Livelihoods Smallholder coffee producer organisations 7, % Peru, Mexico Livelihoods Informal worker organisations 6, % Cameroon Livelihoods Indigenous people 13, % Kenya Social inclusion Deaf children 7, % Occupied Palestine Territories Social inclusion Disabled young people 9, %

14 4. Findings against pre determined indicators of good quality evaluation (1) Map out the causal chain (2) Understand context (3) Anticipate different types of impact (4) Rigorous evaluation of impact using a credible counterfactual (5) Rigorous factual analysis (6) Use mixed methods/triangulation

15 4. Findings 1. Map out the causal chain All reports strong: guided by logframe Issue as impact may not be felt in the lifetime of the project: e.g. deafblindness project 2. Understand the context All reports relatively strong Dependent on consultant knowledge/ experience

16 4. Findings 3. Anticipate different types of impact Clarity about the target groups - yes Guidelines encouraged discussion on different types of impact yes but bias towards good news 4. Use of the counterfactual Before and after comparisons used in all reports Only two projects referenced control groups

17 4. Findings 5. Rigorous factual analysis Sampling methods varied, lack of depth, transparency issues` Limited detail in how data was analysed 6. Mixed methods Good mix of both qual and quant analysis Consistent use of case studies, testimonies

18 4. Findings Utility Source of independent verification Generally useful in writing the final project report More space to tell the story Half of reports yielded new learning Useful recommendations for all grantees Inadequate budgeting Not a difficult process, information required came from established MEL systems

19 5. Implications & conclusions Stakeholders say: Independent evaluation mechanism of individual projects works because there is a support system (FM-guidelines-local owners) Quality control & assurance issues More scope for improved practice in IE at project level and fund level e.g. improved qualitative research methodologies Can individual evaluations be aggregated to assess fund-level impact?

20 References DFID (2011) How to note- Reviewing and scoring projects, DFID, /HTN-Reviewing-Scoring-Projects.pdf Elliott, D. (2013) Exploding the myth of the challenge fund- a start at least, Springfield Centre, Kessler, A. (2013) Measuring results in challenge funds using the DCED Standard, DCED, Kindornay, S., K. Higgins & M. Olender (2013) Models for Trade-related Private Sector Partnerships for Development, The North-South Institute, ins.ca/wp-content/uploads/2012/11/2012-models-for-trade-related-private- Sector-Partnerships-for-Development1.pdf O Riordan, A., J. Copestake, J. Seibold & D. Smith (2014) What is a challenge fund and when should it be used?, Triple Line Consulting & the University of Bath supported by the ESRC & the UK Technology Strategy Board, Consulting_Practice-Brief_Revised.pdf White, H. (2009) Theory-based Impact Evaluation: Principles and Practices, 3ie,