European Commission: Example Sector Intervention Logics/ Theory of Change Models

Transcription

1 European Commission: Example Sector Intervention Logics/ Theory of Change Models Development Evaluation Day Finnish Ministry of Foreign Affairs 17 December

2 2 Sidney Harris

3 Why this Approach? 1. Managing for Development Results (MfDR) Go beyond inputs and outputs Towards outcomes and impact Contribution of donor to country-led results Based on country-led reporting systems 3

4 Why this Approach? 2. Clear Macro Result Frameworks Don t just focus on project level result frameworks There is also a need for macro level result frameworks (e.g. Country Strategy Papers) Evaluations should avoid building retrospective macro result frameworks 4

5 Why this Approach? 3. Intervention Logic/Theory of Change The need for a Hierarchy of Objectives & Chain of Results Development of an Intervention Logic/ Theory of Change (school of thought) Designed to fill the 'missing middle between outputs and global impact 5

6 Objectives and Results Global Global objectives Intermediate Intermediate Objectives Specific Specific Objectives Outcomes Direct Objectives Outputs Implementatio n Objectives Inputs 6

7 Intervention Logic/Theory of Change Inputs Outputs Outcomes Specific Intermediate Global Activities/ Funding 1 Activities/ Funding 2 Activities/ Funding 3 EQ Effect 1 Effect 4 Effect 2 Effect 5 EQ Effect 7 Effect 9 Effect 8 Effect 10 Effect 11 Activities/ Funding 4 Effect 3 EQ Effect 6 7

8 The Missing Middle The Good Wind Evaluation! Outputs Outcomes Specific Intermediate Global 8

9 Attention 1. This is a highly simplified tool, not a template. Intervention logic must be developed for each country based on its context and existing government strategies 2. Outcomes and are not fully attributable to development interventions; indeed their contribution diminishes and external factors become more and more important as you move up the chain of results 3. Indictors are not standard/fixed but provide ideas on how to set, monitor and evaluate objectives 4. Crosscutting issues should be mainstreamed into relevant indicators (e.g. disaggregation of data by gender) 9

10 10

11 Intervention Logic Output Clusters Institutional Management Better Allocation of al Resources Affordability of Outcomes Quality of Specific Impact Intermediate Family Planning & Health Awareness Economic Growth Global Curricula & Teaching Materials Teacher Recruitment & Training Health Equitable Access to MDG 2 MDG 3 Skills and Learning Enhancement Participation in Society Greater Income Opportunities Poverty Reduction MDG 1 Social Development Facilities Optimal Employment 11

12 Output Clusters Institutional Management Curricula & Teaching Materials Teacher Recruitment & Training Outcomes Better Allocation of al Resources Affordability of Outcome Indicators (1) Indicators 1. Transparent decision making 2. Cost for household Health Facilities Optimal Employment 13. Employment: school/university construction and maintenance 14. Employment: teachers and government workers 12

13 Outcome Indicators (2) Output Clusters Institutional Management Curricula & Teaching Materials Teacher Recruitment & Training Health Facilities Better Allocation of al Resources Affordability of Optimal Employment Results (Outcomes) Quality of Equitable Access to Indicators 3. Absenteeism & Drop Out 4. Teacher ratios/rates 5. Qualification of teachers 6. Teaching material ratios 7. Transition & Repetition 8. Equitable education participation 9. Enrolment rate 10. Completion rate 11. Distance to school/university 12. School and university places 13

14 Specific Impact Indicators Output Clusters Institutional Management Curricula & Teaching Materials Teacher Recruitment & Training Better Allocation of al Resources Affordability of Outcomes Quality of Equitable Access to Specific Impact Skills and Learning Enhancement Indicators 15. Literacy & Numeracy rates 16. Examination results 17. Social views Health Facilities Optimal Employment 14

15 Intermediate Impact Indicators Output Clusters Institutional Management Better Allocation of al Resources Affordability of Outcomes Quality of Specific Impact Intermediate Family Planning & Health Awareness Indicator Curricula & Teaching Materials Skills and Learning Enhancement Teacher Recruitment & Training Equitable Access to Participation in Society Health Greater Income Opportunities 18. Employability Facilities Optimal Employment 15

16 Affordability Outcome Indicators 1. Transparent decision making 11. Distance school/university - x% of students within x kilometre of a school List of Key Indicators (with examples) - Implementation of transparent mechanisms to allocate resources between sectors (e.g. using tools such as average cost ratios per student for different educational levels) 2. Cost for Household - Cost of education (i.e. fees, materials/equipment, lunches etc) as a % of household expenditure; cost per student as a % of income per capita Quality Outcome Indicators 3. Absenteeism and drop out - Teacher/student absenteeism rates; drop out rates 4. Teacher ratios/rates - Student teacher ratio; rate of double shifting 5. Qualification of teachers - Average qualification of teachers at each educational level 6. Teaching material ratios - Student textbook ratio 7. Transition and repetition - Progression rates; repetition rates Access Outcome Indicators 8. Equitable educational participation 12. School and university places - al places as an x% of eligible students Optimal Employment Outcome Indicators 13. Employment: school/university construction and maintenance - Number of people directly and indirectly employed in school/university construction and maintenance 14. Employment: teachers and government workers - Number of people directly & indirectly employed as teachers and government workers linked to education sector Specific Impact Indicators 15. Literacy and numeracy rates - Literacy and numeracy rates by age, sex and social group 16. Examination results - Examination results for various educational levels and subject areas by age, sex & social group 17. Social views - Support for violent conflict; prejudiced views towards other social groups - Ratio of girls to boys in primary, secondary and tertiary education; ratio of other disadvantaged groups compared to x% of population 9. Enrolment rate - Gross and net enrolment rates 10. Completion rate - Gross and net enrolment rates Intermediate Impact Indicators 18. Employability - % of school leavers/graduates gaining paid employment (split by type of employment) 16

17 Mapping Evaluation Questions Output Clusters Institutional Management Curricula & Teaching Materials Teacher Recruitment & Training Health Facilities Better Allocation of al Resources EQ 1 EQ 2 Affordability of Optimal Employment Outcomes Quality of EQ 3 Equitable Access to EQ 4 EQ+ Specific Impact Skills and Learning Enhancement EQ 5 Intermediate Family Planning & Health Awareness Participation in Society Greater Income Opportunities Economic Growth Global Poverty Reduction Social Development EQ s also required for relevance and aid effectiveness issues 17

18 Example Result EQs: Sector (1) EQ 1: To what extend has EC support to education helped to improve education service delivery and resourcing? EQ 2: To what extend has EC support to education helped to strengthen transparency and accountability in the management of education service delivery? 18

19 Example Result EQs: Sector (2) EQ 3: To what extend has EC support to education contributed to improving access and equity related to education? EQ 4: To what extend has EC support to education contributed to improving quality of education? EQ 5: To what extend has EC support to education contributed to enhanced basic education skills, especially literacy and numeracy? 19

20 Detailed Example EQ 3 Quality EQ 4: To what extend has EC support to education contributed to improving quality of education? Judgement Criteria Indicators 4.1. Availability of strategies and resources to enhance quality of learning and teaching - % of non-salary to salary budget allocation - Pupil teacher ratio - Pupil classroom ratio (e.g. disaggregated by region) - Pupil textbook ratio - Strategies for improving quality of learning and teaching (e.g. whole school development) 4.2. Quality related efficiency measures -Decreasing drop out rates -Decreasing % repeaters (e.g. primary school level) 4.3. Qualifications and competencies of teachers and school leaders enhanced 4.4. Provisions made to ensure minimum quality education for children in difficult circumstances (e.g. poor, ethic minorities) -% of teachers certified to teach according to national standards (teacher accreditation) -In-service teacher upgrading institutionalised -Training of school leaders on pedagogic and managerial skills in place and implemented - Upgrading of school buildings and construction of additional classrooms in remote areas - Incentive schemes for teachers in remote and disadvantaged areas operational - Prevalence of bilingual teacher training, curricula-teacher guides and textbooks for multi-lingual environments 20

21 Agriculture & Rural Development 21

22 Agriculture & Rural Development Intervention Logic Output Clusters Outcomes Specific Intermediate Global Efficient Employment Provision of Public Goods MDG s 2, 3, 4, 5, 6 & 7 Enabling Economic & Regulatory Environment Mitigating Agricultural Price Volatility Access Markets Farm Inputs Information Services Capital Better Management of Natural Resources Non-Farm Agri-Labour Productivity Production Quality of Products Profitability of Rural Business Affordability of & Subsistence in Food Trade Higher Rural Income Greater Food Security MDG 1 Environmental Sustainability Economic Growth Poverty Reduction MDG 1 Social Development MDG 7 22

23 Agriculture Based Economies (1) Identified by: Agriculture provides basis for growth (i.e. contributes over 30% of GDP) 70% of poor live in rural areas Typical areas: Sub-Saharan Africa Regions within India & Mexico Intervention focus: Improve smallholder competitiveness Ensure livelihoods and food security 23

24 Agriculture Based Economies (2) Output Clusters Outcomes Specific Intermediate Global Efficient Employment Provision of Public Goods Enabling Economic & Regulatory Environment Mitigating Agricultural Price Volatility Access Markets Farm Inputs Information Services Capital Better Management of Natural Resources Non-Farm Agri-Labour Productivity Production Quality of Products Profitability of Rural Business Affordability of & Subsistence in Food Trade Higher Rural Income Greater Food Security Environmental Sustainability Economic Growth Poverty Reduction Social Development 24

25 Transforming Economies (1) Identified by: Agriculture no longer driver of growth (i.e. 7% of GDP) with rising urban-rural poverty gap 82% of poor live in rural areas Typical areas: South/East Asia (e.g. China, India) Middle East and North Africa (e.g. Morocco) Intervention focus: Comprehensive approach and multiple pathways out of poverty to reduce the urban-rural divide - shifting to high-value agriculture - decentralising non-farm economic activities to rural areas 25

26 Transforming Economies (2) Output Clusters Outcomes Specific Intermediate Global Efficient Employment Provision of Public Goods Enabling Economic & Regulatory Environment Mitigating Agricultural Price Volatility Access Markets Farm Inputs Information Services Capital Better Management of Natural Resources Non-Farm Agri-Labour Productivity Production Quality of Products Profitability of Rural Business Affordability of & Subsistence in Food Trade Higher Rural Income Greater Food Security Environmental Sustainability Economic Growth Poverty Reduction Social Development 26

27 Urban-Based Economies (1) Identified by: Small agriculture sector (i.e. less than 0.5% of GDP) with high urban poverty 45% of poor live in rural areas Typical areas: Latin America Central Asia Intervention focus: Link farmers to modern food markets Improve quality of agri-business and food industry Introduce market for environmental services 27

28 Urban-Based Economies (2) Output Clusters Outcomes Specific Intermediate Global Efficient Employment Provision of Public Goods Enabling Economic & Regulatory Environment Mitigating Agricultural Price Volatility Access Markets Farm Inputs Information Services Capital Better Management of Natural Resources Non-Farm Agri-Labour Productivity Production Quality of Products Profitability of Rural Business Affordability of & Subsistence in Food Trade Higher Rural Income Greater Food Security Environmental Sustainability Economic Growth Poverty Reduction Social Development 28

29 Output Clusters Outcomes Mitigating Agricultural Price Volatility Outcome Indicators (1) Indicators 1. Support for consumers vulnerable to short-term food price changes 2. Support for farmers vulnerable to short-term food price changes Provision of Public Goods Enabling Economic & Regulatory Environment Access Markets Farm Inputs 3. Access to road transportation 4. Access to land 5. Access to water 6. Access to seeds and fertilisers 7. Access to improved breeds of animals/fish Information Services 8. Access to extension services Capital 9. Access to information on market 10. Availability of credit 11. Investment in rural areas 12. Establishing a rural business Better Management of Natural Resources 13. Preservation of natural resources 14. Sustainable management practices 15. Environmental services 29

30 Outcome Indicators (2) Output Clusters Outcomes Indicators Provision of Public Goods Mitigating Agricultural Price Volatility Access Markets Farm Inputs Efficient Employment Non-Farm Agri-Labour Productivity 16. Non-farm employment 17. Agriculture labour employment 18. Land productivity 19. Labour productivity 20. Capital productivity Enabling Economic & Regulatory Environment Information Services Capital Better Management of Natural Resources Production Quality of Products 21. Production 22. Type of production/diversification 23. International marketing standards 30

31 Specific Impact Indicators Output Clusters Outcomes Indicators Efficient Employment Provision of Public Goods Enabling Economic & Regulatory Environment Mitigating Agricultural Price Volatility Access Markets Farm Inputs Information Services Capital Better Management of Natural Resources Non-Farm Agri- Labour Productivity Production Quality of Products Profitability of Rural Business Affordability of & Subsistence in Food Trade 24. Agricultural prices 25. Growth in agricultural businesses 26. Growth in rural non-farm businesses 27. Affordability of food 28. Subsistence in food 29. Agricultural trade 30. Rural non-farm trade 31

32 Intermediate Impact Indicators Output Clusters Outcomes Indicators Provision of Public Goods Enabling Economic & Regulatory Environment Mitigating Agricultural Price Volatility Access Markets Farm Inputs Information Services Capital Better Management of Natural Resources Efficient Employment Non-Farm Agri- Labour Productivity Production Quality of Products Profitability of Rural Business Affordability of & Subsistence in Food Trade Higher Rural Income Greater Food Security Environmental Sustainability 31. Rural Income 32. Undernourishment 33. Land sustainability 34. Sustainable extraction of water 35. Adapting to climate change 32

33 Price Volatility Outcome Indicators Production Outcome Indicators List of Key Indicators (with examples) 1. Consumer vulnerability to short-term price changes 2. Farmer vulnerability to short-term price changes 21. Production 22. Type of production/diversification Access Outcome Indicators 3. Access to road transportation 4. Access to land 5. Access to water 6. Access to seeds and fertilisers 7. Access to improved breeds of animals/fish 8. Access to extension services 9. Access to information on markets 10. Availability of credit 11. Investment in rural areas 12. Establishing a rural business Natural Resource Outcome Indicators 13. Preservation of natural resources 14. Sustainable management practices 15. Environmental services Employment Outcome Indicators 16. Non-farm employment 17. Agricultural labour Productivity Outcome Indicators 18. Land productivity 19. Labour productivity 20. Capital productivity Quality Outcome Indicators 23. International marketing standards Profitability Specific Impact Indicators 24. Agricultural prices 25. Growth in agricultural businesses 26. Growth in rural non-farm businesses Affordability/Subsistence Specific Impact Indicators 27. Affordability of food 28. Subsistence in food Trade Specific Impact Indicators 29. Agricultural trade 30. Rural non-farm trade Rural Income Intermediate Impact Indicators 31. Rural income Food Security Intermediate Impact Indicators 32. Undernourishment Environment Sustainability Intermediate Impact Indicators 33. Land sustainability 34. Sustainable extraction of water 35. Adapting to climate change 33

34 Limitations & Issues with Approach 1. Common Results Frameworks More and better country-led result frameworks (promotion of joint evaluations) Strengthened international agreement on common objectives (i.e. MDGs+) guidance and development of relevant indicators 34

35 Limitations & Issues with Approach 2. DATA, DATA, DATA!!!! This is the problem for obtaining strong evaluation results Data in many developing context is extremely poor There is need to strengthen partner countries in statistical capacity and M&E 35

36 Limitations & Issues with Approach 3. Challenge of Complexity Evaluation of outcomes and impact is very complex (increasing number of external factors) Evaluation methods and tools required that can deal with complexity (e.g. systems thinking) Get out of a project mindset. Look at synergies between sectors 36

37 Thank you! Any Comments/Questions? 37

38 The following 3 Intervention Logics will not be presented in Helsinki. They are only included in case they are required during the discussion after the presentation. 38

39 Road Sector Output Clusters Outcomes Specific Intermediate Global Infrastructure (construction/ maintenance) Institutional Reform Lower Road Transport Time Lower Road Transport Costs Institutional Management of Road System Greater Economic Accessibility Greater Accessibility to Social Services Conditions for Economic Growth Economic Growth Poverty Reduction Economic Environment Social Employment Environmental of Roads Minimised Enhanced Safety & Health Linked to Roads Enhanced Human Capacity Social Development 39

40 Health Sector Output Clusters Outcomes Specific Intermediate Global Essential Medicines & Equipment Regulation of Private Sector Institutional Reform Infrastructure Affordability of Health Care Enhanced Service Availability & Quality Service Utilization & Intervention Coverage Health Behaviour Reduction in Mortality Social Development Poverty Reduction Mitigation of External Risks Health Awareness Reduced Health Risks Reduction in Morbidity & Disability Economic Growth Health Care Worker Recruitment & Training Optimal Employment Stronger Labour Markets 40

41 Water and Sanitation Output Clusters Environment Outcomes Conservation & Preservation of Water Use of Water & Sanitation Specific Intermediate Sustainable National Water Supply Conditions for Economic Growth Global Reduced vulnerability of Ecosystem to climatic events Economic Growth Economic Institutional Reform Infrastructure (construction/ maintenance) Employment Quality of Water & Sanitation Greater Equity in Allocation of Water & Sanitation Affordability of Water & Sanitation Water & Sanitation Health & Hygiene Behavior Access to Water Access to Basic Sanitation Reduced Rate of Water Related Diseases Health Poverty Reduction Social Development 41