AGRICULTURE CASE STUDY Evaluation of Budget Support in Zambia. Dennis Chiwele Martin Fowler Ed Humphrey Alex Hurrell Jack Willis

Transcription

1 AGRICULTURE CASE STUDY Evaluation of Budget Support in Zambia Dennis Chiwele Martin Fowler Ed Humphrey Alex Hurrell Jack Willis

2

3 Preface We would like to express our gratitude to all those who have contributed to the study. In particular, we would like to thank those who gave their time to meet the study team members in Lusaka and to respond to their requests for information. A full list of the people met is included in Annex C. In addition, we would like to express our appreciation thanks to Thom Jayne and Nicky Mason of Michigan State University and the Food Security Research Project, for their invaluable assistance with the econometric analysis. We are particularly grateful for their agreement to run our analysis on the 2008 Supplemental Survey data, which was crucial for obtaining the results detailed in this report, and for sharing the results of their own analyses on maize production and fertiliser use in Zambia. We would also like to thank Ingo Outes-Leon for his advice and guidance in determining the most-appropriate econometric approach and in peer-reviewing the results, Jones Govereh for his inputs into the econometric methodology and Dingi Banda for his suggestions and information on the FSP. The evaluation was commissioned by the Secretariat for Evaluation of the Swedish International Development Cooperation Agency (Sida). The findings are those of the evaluation team, and do not necessarily reflect those of Sida. i

4 Executive summary This study into the impact of general budget support (GBS) on the agricultural sector in Zambia is one of four sector case studies contributing to a wide-ranging evaluation of the impact of budget support in Zambia as a whole. Agriculture was selected due to its importance in providing a strong foundation for national economic growth and the reduction of poverty, particularly in the rural areas where poverty is most prevalent. Overview The objective of the study is to assess both the extent to which budget support has given the Government of the Republic of Zambia (GRZ) the means to implement its national and sector policies, and the impact that these policies have had on national economic development, service delivery and the eradication of poverty since the turn of the century. It is hoped that the study will be used by GRZ policy makers and cooperating partners (CPs). The study is intended to be used to inform ongoing national policy processes, including the drafting of the Sixth National Development Plan and the revision of the Poverty Reduction Budget Support s memorandum of understanding. Information and data were collected and analysed by a small research team, including a series of interviews and meetings with representatives of a wide range of organisations involved with the country s agricultural sector. Information was also obtained from a large collection of reference material on the sector. The analysis included an assessment of the place of the sector in the economy and in the national development strategy, the funding made available to the sector by both the GRZ and the CPs, and the poverty-focus of this expenditure. The influence of the CPs on the formulation of policies for the sector, resulting from both the policy dialogue and conditionality associated with budget support, formed a further area of focus of the study. The terms of reference of the agricultural sector case study identified the Fertiliser Support Programme (FSP) as a focus for a detailed investigation. The FSP is an input subsidy programme for maize production and it has been by far the largest single item on the Ministry of Agriculture and Co-operatives (MACO) budget during the past decade. This involved a quantitative analysis of the effectiveness and impact of the intervention, using econometric techniques to determine which groups benefited from the programme, as well as its impact on the production and income of those beneficiaries. The work was hampered by the fact that it is not possible to attribute, directly, the effect of budget support on livelihoods and incomes due to the fungible nature of the assistance provided by the CPs. This is compounded by the absence of clear counter-factual evidence having a definite understanding of what would have happened to the sector in the absence of budget support. The work also suffered from the limited amount of data that exists on the agricultural sector, and from its poor quality and inconsistency. Agriculture Sector Agriculture is a key economic sector, providing employment for three-quarters of the workforce, and contributing more than 22 per cent of GDP and some 10 per cent of export earnings. In addition, the sector is particularly important in national efforts to reduce poverty, given that some four-fifths of the rural population are classified as poor. Agricultural production is principally low-input/low-output farming by smallholders for home consumption ii

5 with occasional surpluses being marketed. Maize is grown by more than 80 per cent of farmers, with production being highly seasonal. This gives rise to a distinct hunger period. The growth of the sector has been sluggish in recent years, fluctuating significantly from one year to the next (largely in line with rainfall) and it has not performed as well as the rest of the economy. This can be attributed to several factors, including the collapse of key marketing and rural finance institutions, the declining efficiency of agricultural support services, the increasing incidence of HIV/AIDS, the neglect of rural infrastructure (roads, storage, irrigation and electricity) and the decline in farm gate prices of many crops in the aftermath of market liberalisation. Government Inputs The FSP and Food Reserve Agency (FRA) have accounted for some two-thirds of MACO s budget during the past ten years, with final expenditure often significantly exceeding the original budget amounts. The FSP was established to provide fertiliser and improved seeds at subsidised prices to smallholder maize producers in order to increase the productivity and output of the crop. The FRA was established to manage a strategic food reserve, which has mainly involved purchasing maize at subsidised prices from domestic farmers. This pattern of expenditure has, as a result, favoured people in the maize-growing provinces along the line-of-rail. Both programmes have suffered from deep-rooted managerial, administrative and operational weaknesses, with a significant proportion of their respective budgetary resources being unaccounted for each year. Although expenditure by MACO has increased rapidly over the past five years, it has accounted for only between 4.0 per cent and 6.5 per cent of total government expenditure. Much of this increase has been allocated to the two subsidy programmes. The priority given to these programmes compared with other MACO interventions has been at odds with national policy pronouncements, wherein they have been accorded a relatively-low status. At the same time, the proportion of the recurrent budget accounted for by non-wage expenditure has been low, resulting in staff having insufficient operational funds to enable them to work effectively with farmers. CP-funded development projects have accounted for almost all capital expenditure. The current National Agricultural Policy defines GRZ s role for the sector as being to facilitate and support the development of sustainable and competitive agriculture. Agriculture is also seen as a strategic area of focus in the draft Sixth National Development Plan to cover the five-year period from The development of the sector will directly address key problems of poverty, low growth and high rates of unemployment. The Plan notes that the role of the Government will be limited to undertaking key facilitation and regulatory activities aimed at creating the necessary enabling environment for the private sector to flourish. However, experience with the Fifth National Development Plan shows that in practice little use is made of the Plan to prioritise the Ministry s programme of work. The focus of MACO expenditure on the two maize subsidy programmes can be understood from the political economy perspective. The programmes support increases in rural incomes and thereby secure vital rural votes. The policy also seeks to satisfy the urban population by assuring a regular supply of cheap food. Such outcomes clearly outweigh the many technical and economic arguments that highlight the cost of the programmes to the economy and seek a revision in their design. Although standard financial management structures and systems are in place within MACO, recent audit reports have highlighted a number of weaknesses. In spite of a series of iii

6 recommendations being made in each report, there is little systematic follow-up of such audit findings and little, if any, remedial action has taken by MACO. This has been particularly true in the case of the FSP and FRA programmes. In addition, the Ministry has problems with budget management, internal audit and asset management, while its financial reporting to the Treasury tends to be both irregular and late. A series of accounting irregularities involving significant sums of money have been reported by the Auditor General. A CPsupported intervention is seeking to address these challenges by strengthening public financial management processes and capacity. Cooperating Partner Inputs The CPs are closely involved with the agricultural sector through the resources they provide, either directly or indirectly via GBS. Indeed, during the last decade, average expenditure by the CPs on projects has accounted for almost one-third of MACO s total expenditure. However, the significant annual variations in the funds committed have made it difficult for GRZ to manage these inflows effectively, while reporting by the CPs on the funds has been poor. There is little doubt that the provision of GBS has resulted in a more intense policy dialogue than would otherwise have been the case, particularly since the establishment of the Joint Assistance Strategy for Zambia (JASZ) in Agriculture has its own Sector Advisory Group (SAG), which was established in 2008 and which comprises all of the key interest groups in the sector. The SAG meets regularly to discuss general developments in the sector, resolve specific issues and co-ordinate activities. There is also a monthly meeting of all CPs involved with the sector both silent and active at which information and ideas on the sector and on their respective programmes of support are discussed. This interaction has led to a more-harmonised approach of support to the sector, reducing competition and inefficiencies to a minimum. The CPs in the sector appoint three of their members, the Troika, to take the lead and represent the group at all key meetings with the GRZ, including discussions on GBS. This, more-streamlined approach, has undoubtedly reduced the transaction costs for GRZ and improved the quality and coherence of policy dialogue and funding, with respect to agriculture. However, there is no evidence to show that the more-regular and open dialogue has had any greater impact on GRZ s actions than had been the case before such policy dialogue became more formalised under the GBS process. In particular, despite the CPs being increasingly critical of the FSP, they have not been able to stop its rapid expansion in recent years. As an integral part of the budget support process, the GRZ and CPs developed a Performance Assessment Framework (PAF) to assess the performance of general budget support and the progress made in implementing poverty-reduction policies. Three PAF indicators were drawn up for the agricultural sector, based on information presented in the FNDP, namely: (i) consistency of budget releases within the Ministry with the MTEF; (ii) the increased area of land encompassed by new and rehabilitated irrigation schemes; and (iii) enactment of agricultural marketing and credit legislation. Minimal progress has been made by MACO in realising any of the targets; indeed, progress has slowed over time. Work is currently underway to revise the indicators and to reduce their number. Impact Within the time available, it has not been possible to determine the impact of all of MACO s policies on economic development, improved service delivery and the eradication of poverty. iv

7 This study has therefore focused on the largest programme in the MACO portfolio, namely the Fertiliser Support Programme. The beneficiary incidence analysis showed that the recipients of the FSP had higher incomes, more assets and larger farms than the average farmer. This is expected given that the cost after subsidy is still likely to be too high for the poorest farmers. The recipients were more likely to be male headed households, to have larger families and for the head of household to be older. Some statistics, such as the length of time the household had been settled in the area, pointed towards social connections increasing the chances of receiving the FSP. The analysis also showed that the FSP was reaching only 11% of households and has national coverage. Interestingly, there is evidence that the FSP is crowding out private sector fertiliser. The beneficiary incidence analysis showed that of those receiving the FSP in 2006/7, 33.9% had used commercial fertiliser in 1999/2000. Of these farmers, 21.8% no longer used commercial fertiliser now that they received the FSP. In terms of the impact of the FSP itself, the econometric analysis sought to determine the effect of the FSP on a typical recipient in terms of the productivity of maize cultivation; the income from crop cultivation; and the income from crop cultivation less expenditure on fertiliser. The selection bias inherent in the FSP, whereby FSP recipients typically have higher assets, incomes and land than non-fsp recipients, made an accurate assessment of the impact of the FSP problematic. To overcome this, three different methods based on a combined Propensity Score Matching difference-in-difference technique were used. Taken together the results of these three alternative approaches present an ambiguous story. Two methods, which were likely to overstate FSP impact, found a positive impact of the FSP on maize productivity, total crop income and income net of fertiliser costs. One approach, which is likely to underestimate the FSP impact, found no impact on maize productivity, total crop income or income net of fertiliser costs. Drawing on these results and the existing research into the FSP, it is likely that the true impact of the FSP is likely to lie between these conflicting results. As such, it is possible that the FSP is having a positive impact on maize production and crop income, but the magnitude of that impact cannot be effectively determined from this study nor can the extent to which that impact justifies the programme costs. Further analysis has not been possible with the time and data available to this study, but it is understood that FSRP will be producing additional research into the FSP which may shed further light on this issue. Conclusions There are several channels through which budget support may have given GRZ the means to implement national and sector policies. At the most fundamental level, budget support facilitates this process through the resources it provides. The CPs are a significant source of funds both in terms of project aid to the sector and the GBS resource flow that may allocated to agriculture. However, whilst resources to the sector have increased in recent years, including continued CP project support, it is not possible to directly attribute this to the introduction of GBS, due to the absence of a clear counter-factual of the levels of resources would have flowed to GRZ in the form of project aid. In terms of policy dialogue, there has clearly been an improvement in the efficiency of the dialogue that the CPs have had with MACO and other government agencies. However, there is no evidence that this has been any more intense or frequent since general budget support became a permanent feature of the aid landscape. In particular, nor is there evidence that v

8 the effectiveness of this interaction has improved in any way as a result of GBS, with the CPs unable to influence the key policy decisions in the sector. Financial management within MACO remains a problem. Although steps are now being taken to address weaknesses in public financial management through a Government-wide programme supported by most of the budget support CPs and others. It is arguable that the CPs are more concerned about financial management as a result of GBS, but without a clear counter-factual there is no evidence that similar assistance would not have been provided in the absence of GBS. The CPs have continued to provide some technical assistance to the sector. However, again, there is no evidence that the more-recent requests have been any different in terms either the number requested or the expertise sought compared with what might otherwise have been the case in the absence of GBS. The above does not imply that the provision of GBS has had no impact. However, it does imply that there is no clear evidence to conclude that the benefit of GBS was greater than that obtained by the relevant counter-factual. In other words, it cannot be said that fewer resources would have been provided, that the policy dialogue would have been less effective nor that governance would have been weaker had the CPs chosen to provide project aid instead of GBS. vi

9 Table of contents Preface i Executive summary ii List of Tables, Figures and Boxes viii Acronyms x 1 Introduction 1 2 Methodology Objectives of the evaluation Evaluation questions Implementation Limitations to methodology 9 3 Agriculture in Zambia Agricultural sector The Government s principal agricultural programmes Agricultural and rural livelihoods 18 4 Government inputs Resources Ownership and commitment Financial management 35 5 Co-operating Partners inputs Funding Harmonisation and alignment Policy dialogue and conditionality 37 6 Fertiliser Support Programme Beneficiary incidence analysis Econometric analysis of the FSP Qualitative assessment 49 7 Lessons learnt Impact of Budget Support Impact of GRZ Policies 53 8 Conclusions 55 Bibliography 56 Annex A Terms of reference 63 Annex B Econometric analysis of FSP impact 66 Annex C People met 90 vii

10 List of Tables, Figures and Boxes No table of contents entries found. Figure 2.1 Intervention logic of Budget Support 5 Figure 3.1 Share of the total cultivated area under different crops (in per cent) 11 Figure 3.2 Mean scores for household monthly food availability 21 Figure 4.1 Agriculture spending as a proportion of total Government expenditure, 2000 to 2008 (in current Kwacha bn) 25 Figure 4.2 Proportion of MACO expenditure accounted for by the FSP and FRA programmes, 2000 to 2010 (in current Kwacha bn) 28 Figure 4.3 Functional breakdown of MACO expenditure, excluding the FSP and FRA programmes, 2000 to 2010 (in per cent) 29 Figure 4.4 Functional breakdown of MACO s central expenditure, 2000 to 2010 (in per cent) 30 Figure 5.1 Support provided to the agricultural sector by the Co-operating Partners, 2000 to 2008 (in current Kwacha bn) 36 Box 2.1 Approach to counter-factual 7 Box 8.1 Sources of household data on agricultural production 69 viii

11 Acronyms ACF ATE ATT AVCR BMZ CAADP CFS CP CSO FISP FNDP FRA FSP FSRP GBS GDP GRZ HH IOB ITT JASZ LCMS MMD MACO MTEF Agricultural Consultative Forum Average treatment effect Average treatment on the treated Average value cost ratio German Federal Ministry for Economic Co-operation and Development Comprehensive Africa Agricultural Development Programme Crop Forecast Survey Co-operating partner Central Statistical Office Farmer Input Support Programme (from 2009; formerly FSP) Fifth National Development Plan Food Reserve Agency Fertiliser Support Programme Food Security Research Project General budget support Gross domestic product Government of the Republic of Zambia Household Policy and Operations Evaluation Department, Netherlands Ministry of Foreign Affairs Intent to treatment Joint Assistance Strategy for Zambia Living Conditions Monitoring Survey Movement for Multi-party Democracy Ministry of Agriculture and Co-operatives Medium-Term Expenditure Framework ix

12 MSU MVCR NAMBoard OLS OPM PAF PETS PHS PRBS PSM QSDS SAG SEA Sida SNDP SS ToR ZCF Michigan State University Marginal value-cost ratio National Agriculture Marketing Board Ordinary least squares Oxford Policy Management Performance Assessment Framework Public Expenditure Tracking Survey Post Harvest Survey Poverty reduction budget support Propensity Score Matching Qualitative Service Delivery Survey Sector Advisory Group Statistical Enumeration Area Swedish International Development Co-operation Agency Sixth National Development Plan Supplemental survey Terms of reference Zambia Co-operatives Federation x

13 1 Introduction This monograph represents one component of a wider evaluation study to assess both the effectiveness of general budget support (GBS) provided to Zambia by its co-operating partners (CPs) since the end of the 1990s, as well as its contribution to the realisation of the country's development policies. The evaluation examines both the extent to which budget support has given the Government of the Republic of Zambia (GRZ) the means to implement its national and sector policies and the impact that these policies have had on economic development, improved service delivery and the eradication of poverty. The wider study comprises an analysis of both the political economy and the macroeconomic effects of budget support, combined with four sector case studies in agriculture, health, education and infrastructure. The focus of this monograph is agriculture. The growth of the agricultural sector is seen as being critical to the expansion of the overall economy and to the success of any efforts made to reduce poverty, particularly in the rural areas. The agriculture case study follows the methodology applied to all the sector studies as described in the terms of reference, ToR (IOB et al, 2010) and in the inception report (AAID et al, 2010). The sections of the terms of reference relating to the agricultural sector case study are presented in Annex A. The monograph is expected to examine the performance of the sector as a whole and the influence of budget support and other factors on government policies and outcomes. The fungible nature of budget support means that it is not possible to attribute the impact of a particular component of GRZ spending to budget support. However, it remains a useful exercise to evaluate the impact of particular GRZ programmes. The ToR specifically identified the Fertiliser Support Programme (FSP) 1 as the subject for an evaluation, being the largest component of the agriculture budget. The study is intended to be used by GRZ policy makers, both at the central level and in relevant line ministries, as well as by the Zambian Parliament, the Auditor General and CPs, among others. Decision makers in other countries receiving budget support, as well as the CPs providing budget support in those countries, might also form part of the target audience. It is intended that the evaluation will feed into other policy processes in-country, such as the drafting of the Sixth National Development Plan and the revision of the Poverty Reduction Budget Support memorandum of understanding (with its associated Performance Assessment Framework indicators). The wider budget support evaluation has been managed by the Policy and Operations Evaluation Department of the Netherlands Ministry of Foreign Affairs, the Evaluation and Audit Department of the German Federal Ministry for Economic Co-operation and Development and the Secretariat for Evaluation of the Swedish International Development Cooperation Agency (Sida). Each of the evaluation teams has worked closely with staff of the Ministry of Finance and National Planning and other stakeholders in Lusaka. The agricultural sector study was undertaken by a team of three researchers and two econometricians during It involved an extensive series of meetings with representatives of a wide range of organisations working in, or involved with, the country s 1 The FSP has recently been renamed as the Farmer Input Support Programme (FISP). Throughout this paper, the acronym FSP will be used interchangeably to refer to both the FSP and the FISP. 1

14 agricultural sector. The assistance of all those who gave their time to take part in interviews and to provide information and data is gratefully acknowledged. The remainder of this report is structured as follows. o Section 2 provides an overview of the objectives of the evaluation, the evaluation questions, the process for implementation, the main elements of the methodology, and limitations to the methodology. o Section 3 describes the evolution of the agricultural sector, including the livelihoods of the rural population. o Section 4 examines the government resources allocated to the sector, sector policies, ownership and commitment, and the quality of financial management. o Section 5 sets out the resources provided by the CPs to the sector, their influence through policy dialogue and conditionality, and their harmonisation and alignment with GRZ systems. o Section 6 provides an econometric analysis of the largest agricultural programme, the FSP. o Section 7 returns to the evaluation questions to draw some wider lessons on the impact of budget support. o Section 8 offers some final conclusions on the study. 2

15 2 Methodology This section sets out the methodology and approach taken in the evaluation study. It starts by setting out the objectives of the exercise, before proceeding to set out the specific evaluation questions to be addressed and considering the question of attributing any impact specifically to GBS. Finally, the section describes the main elements of the methodology and specific steps taken in the evaluation work, as well as discussing the limitations to methodology, including the quality of existing data. 2.1 Objectives of the evaluation The Inception Report (AAID et al. op.cit.) sets out two objectives of the overall evaluation of GBS. These are: 1. to what extent and under what circumstances budget support in Zambia has successfully given means to the government to implement national and sector policies, related to the objectives of (economic) development and poverty reduction; and 2. the impact of these policies on (economic) development, improved service delivery and the eradication of poverty. Within the context of these objectives, the sector case studies were commissioned with a view to addressing two broad questions: o How did the economy, income and income distribution and livelihood of the people in Zambia develop? o To what extent can changes in the performance of the economy, income and income distribution and livelihood of the people in Zambia be related to government policies and/or to other external or internal factors? The detail of these questions and how they relate to the agricultural sector are discussed further in Section 2.2 below. The ToR specified a dual approach to addressing these questions. First, a qualitative assessment was to be conducted. This involved an analysis of the agricultural sector s place in the national economy and the national development strategy, and an assessment of the contribution of the sector to poverty reduction, including the poverty-focus of expenditure in the Ministry of Agriculture and Co-operatives (MACO). An analysis of the GRZ budget made available to MACO, as well as the funding provided by the principal CPs involved with the sector, was also to be carried out. Of particular importance is an assessment of the influence of the CPs in the formulation and revision of policies for the sector. This influence might result from both policy dialogue and conditionality. Included in this assessment is the choice of the three Performance Assessment Framework (PAF) indicators which have been selected to measure progress in the agricultural sector. Secondly, the case study was to include a quantitative analysis. An econometric analysis of the effectiveness and impact of the FSP would be carried out. This analysis would first involve a beneficiary incidence analysis to determine which groups benefited from the 3

16 programme. The analysis would subsequently determine the impact of the FSP on the production and income of the beneficiaries. The core analysis period is from 2000 to the present day, although data from years prior to this have been used where appropriate. 2.2 Evaluation questions The ToR posed a number of evaluation questions that are common to each of the four sector case studies. This section sets out those questions, considers the issue of attribution and describes the process for obtaining an appropriate counter-factual Questions and sub-questions The two broad evaluation questions, described above, were subsequently broken down into six sub-questions. The questions are repeated below, with slight adjustment to improve their relevance to the agricultural sector 2. Question 1 o How did the sector, income and income distribution and livelihood of the citizens develop? This question can be divided into three sub-questions: o How did the sector perform and interact with the economic and institutional environment? o How did the overall livelihoods, including security and access to services, of the farming population (especially those of the low-income population), change over time? o To what extent have there been changes in the income and income distribution of the farming population, in particular amongst the poorest groups? Question 2 o To what extent can changes in the performance of the sector, income and income distribution and livelihood in rural households be related to the GRZ s policies and/or other external or internal factors? This question can be divided into three sub-questions: o To what extent can changes in sectoral performance be related to changes in economic/fiscal management, changes in agriculture policies/policy processes, or other internal/external factors? o To what extent can changes in overall livelihoods, including citizens security and access to/use of services, be related to changes in GRZ policy/policy processes or to other internal/external factors? o To what extent can changes in income and income distribution be related to changes in GRZ policy/processes or to other internal/external factors? 2 Questions 1 and 2, as presented here, correspond to Questions 4 and 5 in the terms of reference (IOB et al, 2010) and Questions 5 and 6 in the inception report (AAID et al, 2010). 4

17 The study can therefore be thought of as an analysis of the economic and institutional performance of the sector, including its impact on livelihoods, income and income distribution, combined with an assessment of the extent to which this performance is determined by GRZ polices and/or other factors. There is the potential for some overlap between the six questions. For this reason, the report is structured to answer all six questions whilst avoiding any unnecessary repetition Attribution Clearly, within the questions described above, the impact of budget support is central to this evaluation. However, the extent to which one can attribute changes in polices, resources and, ultimately, impact to budget support is one of the most difficult questions to answer. To understand the mechanisms by which this is possible, consider Figure 2.1 which portrays the means by which budget support achieves its impact. Figure 2.1 Intervention logic of Budget Support Source: Faust and Leiderer, 2010 The diagram presents a framework at the national level. At the sectoral level, however, not all of the components in the diagram are relevant to the analysis. For this reason, it is possible to condense the analysis into a set of questions that follow the same logical sequence as Figure 2.1. These questions start with examining the inputs, go on to look at outputs and outcomes and finally consider impact itself. Inputs The first set of questions is related to the inputs associated with budget support. questions might include: These 5

18 o What has been the policy dialogue between the CPs and GRZ associated with budget support? o What conditions were imposed by the CPs on GRZ associated with budget support? 3 o What technical assistance was provided with budget support? o How has accountability by GRZ and the CPs changed as a result of budget support? o How has CPs alignment with government policies and systems changed as a result of budget support? o How has harmonisation between donors changed as a result of budget support? o How has ownership and commitment by GRZ changed as a result of budget support?? Outputs The second set of questions goes on to consider the outputs that arise from the process changes discussed above. These questions might include: o What specific new policies have resulted from: - the policy dialogue between GRZ and the CPs, - the conditionality imposed by the CPs, - technical assistance provided by the CPs, - increased accountability by GRZ and the CPs, - ownership and commitment by GRZ? o What specific efficiencies in GRZ spending have been achieved through: - technical assistance (e.g. strengthening the budget process) - increased alignment - CP harmonisation? Outcomes The outputs (i.e. in the form of new policies and increased efficiency) may result in changes to total GRZ expenditure on the sector, as well as changes in the composition of that spending. This may be difficult to determine in practice, in the absence of a clear counterfactual. However, by tracing through the logical sequence of process and output changes, it may be possible to identify specific outcomes in terms of: o Has budget support increased the resources to the sector in aggregate? o How has budget support affected the allocation of resources within the sector? Impact In terms of the impact of spending on the livelihoods, income and income distribution of the population, it is not possible to attribute specific effects of spending to budget support due to the fungible nature of any CP assistance. Therefore, it is only possible to evaluate the impact of GRZ spending in general in the agricultural sector, without attributing it to budget support. Relevant questions to consider include: o What is the impact of an average $1 of spending in this sector? 3 Conditionality refers to the conditions used by donors for the release of funds. In the case of the agricultural sector, this includes the three specific PAF indicators. This differs from the policy dialogue, which might include a wider range of issues but without specific conditions attached to them. 6

19 o What is the impact of a marginal $1 of spending in this sector? o What is the impact of key programmes in this sector (e.g. FSP)? o Which groups are benefiting from spending in this sector? To retain this logical sequence, the report has been structured to match the components in Figure 2.1, above. Section 4 of the study report considers the GRZ inputs to budget support, as described in the boxes on the left hand side of the diagram. Section 5 considers the CP inputs to budget support, as described in the boxes on the right hand side of the diagram. Section 6 then seeks to determine the impact of the public goods and services (in this case, the FSP) on economic growth and poverty reduction, as described at the bottom of the diagram Counter-factual With the exception of the questions related to impact, each question should be considered against an appropriate counter-factual to determine the contribution from budget support, as opposed to other factors. The approach to counter-factuals is set out in the inception report (AIID et al, 2010) and presented in Box 2.1, below. Box 2.1 Approach to counter-factual In the case of GBS, because it is so complex and the evaluation is many-layered, it would be inappropriate to think in terms of a single overall counter-factual. Rather, it is reasonable to consider what is the appropriate counter-factual for each of the sub-enquiries that make up the overall study. For aggregate flow-of-funds and budgetary effects, it is certainly appropriate to ask whether GBS is additional or a substitute for other forms of aid, and to frame the counter-factual accordingly. (GBS may be a substitution now but considered as a possible addition in future, in which case both alternatives could be considered.) Of course, deciding what is the appropriate counter-factual, in principle, does not mean necessarily that it is practical to model one in econometric detail. For budgetary effects we should ask even if we decide we cannot answer what is the marginal effect of GBS on public expenditure, taking fungibility into account? At the same time, it is important always to consider whether, for the purposes under investigation, GBS is materially different from other forms of aid. (For example, it is not immediately clear, when a macro-economic effect, such as Dutch disease, is at issue, that the effects of more GBS should be considered any differently than the effects of the same amount of aid in a different form). It is important not to stray towards the Herculean task of evaluating aid as such. For assessing relevance and appropriate design of GBS, we have to take account of donor intentions and rationales: if GBS is put forward as a corrective to certain deficiencies in other forms of aid (high transaction costs, for example), then a relevant counter-factual is persistence with those forms of aid, and we have to ask both whether the original diagnosis was correct (did the previous modality have the characteristics identified?) and whether GBS performs better in the relevant dimensions. In doing so we will pay attention, as already indicated, to interactions, both positive and negative, between different forms of aid. Source: IDD and Associates,

20 2.3 Implementation This section sets out the main elements of the methodology and specific steps taken to carry out the evaluation. The work involved three inter-related activities, namely: stakeholder interviews, a literature review and an econometric analysis. The study was undertaken with the supervision of both Sida, who approved the inception report and evaluation framework, and the wider budget support evaluation team, who were presented with a progress report in The Hague on 16 th June Stakeholder interviews Central to the evaluation was an extensive process of stakeholder interviews. Two members of the agricultural sector case study team held initial meetings in Lusaka in March A separate visit to Lusaka, focused on the econometric analysis and the FSP, was undertaken by two team members in May A further visit to Lusaka was made by two members of the team in August 2010, in order to clarify a number of issues with key stakeholders and to obtain the most up-to-date data and documentation on the sector and on the economy as a whole. During each visit, the team held a series of meetings in Lusaka with key stakeholders in the agricultural sector. The stakeholders interviewed included a wide range of the CPs involved with the agricultural sector, appropriate sections of GRZ involved in agriculture, finance and statistics, and members of civil society, the private sector and research bodies including a group of stakeholders assembled by the Agricultural Consultative Forum (ACF). A list of those with whom the team held meetings is provided in Annex C Literature review The agricultural sector and its constituent sub-sectors has been the subject of extensive research in recent years. In particular, the World Bank has recently conducted an evaluation of the FSP (World Bank, 2009) and the lessons and conclusions of that evaluation are repeated here, where appropriate. Other sources used for the literature review include policy and project documents, evaluations, research papers, sector plans/reviews, CPs country strategy papers and related documentation, a functional analysis of MACO, ACF position papers, recent public expenditure reviews of the agricultural sector, monitoring and evaluation reports from MACO and some grey literature. The documents that were used in preparing the monograph are listed in the Bibliography Econometric analysis The approach adopted for the econometric analysis is described in more detail in Section 6.3 and in Annex B. In line with the terms of reference, the analysis was divided into two parts. Firstly, a beneficiary incidence analysis was conducted to determine which groups benefited from the FSP. Secondly, the analysis focused on determining the impact of the FSP on both production and on the income of the beneficiaries. The analysis was conducted using existing data, mainly drawing on the Supplementary Survey (SS) produced by the Food Security Research Project (FSRP) and the Central Statistical Office (CSO). The analysis was conducted by a team of two econometricians, supported by a peer-review team to ensure both its technical quality and its relevance to the local conditions in Zambia. The team co-operated closely with staff of Michigan State 8