Quantitative Methods: Secondary Data

Transcription

1 Assessing Impact of Development Programmes on Food Security Quantitative Methods: Secondary Data Learners Notes FAO, 2011

2 Table of contents Learning objectives... 2 Introduction... 2 When to use secondary data?... 2 Selecting the right datasets... 4 Using secondary data... 6 Summary... 9 If you want to know more... 9 Annex I: Potential data sources Annex II: Example of use of national statistics Annex III: Case study Learners Notes 1

3 Learning objectives Assessing Impact of Development Programmes on Food Security At the end of this lesson you will be able to: identify the possibilities and limitations of using secondary data for assessing impact; understand how to select appropriate datasets to use; identify the main ways of using secondary data. Introduction We can define secondary data as data gathered by a given party for one purpose that is then utilized by another party for a different purpose. Almost all impact assessments will use secondary quantitative data as a source of information, even if primary data are also collected for the specific programme. In this lesson you will see how to use data from secondary sources, ensuring the adequacy and quality of the information they provide. When to use secondary data? Primary data collection for impact assessment may provide the most direct and specific information to assessors, however it can be costly and time consuming. In some cases, good quality data have already been collected, so it may not be necessary to duplicate the effort. In other cases, limited financial resources or time are an issue for the programme. In which case, it may be less expensive to hire an experienced assessor or statistician to identify data sources and carry out an appropriate data analysis, rather than conduct a survey to collect primary data. Note that like all other approaches and methods, the use of secondary data requires specialized skills and competencies, particularly in statistics. This lesson will provide programme managers, M&E staff and external assessors with knowledge about the salient points of using secondary data in impact assessment, including potential data sources. With this knowledge, you can feel more confident about planning to use secondary data in your impact assessment and you will have the necessary elements to help you employ appropriate persons with the necessary skills to successfully assist you. Learners Notes 2

4 The following table compares the strengths, weaknesses, opportunities and constraints of primary and secondary data for assessing programme impact. Collecting primary data Using secondary data Strengths you can control timing of data collection and the location of data to be obtained you can use indicators not routinely collected in other surveys you have information on programme participation status so you can distinguish participants from non participants Weaknesses primary data collection is very expensive much time is required for data collection, analysis and reporting Opportunities strengthens the capacity of programme M&E staff in carrying out primary data collection Constraints may need to set up a new system from scratch to collect data need capacity to design and pre-test questionnaires, train enumerators, supervise data collection, perform analysis and interpret data need to obtain a valid and recent sampling frame for sample selection less expensive than primary data collection no need to set up data collection system in many cases, quality of national surveys is high national surveys may also provide information at sub national levels cannot control the timeliness of data collection and availability of datasets cannot regulate level of disaggregation of data, therefore coverage of project areas cannot be guaranteed existing indicators may not be specific enough for impact assessment of the programme might be difficult to identify programme participants can add key indicators on food security to national surveys such as Household Budget Surveys (HBS), census, Nutrition or Agricultural surveys possibility of using small area statistics to disaggregate national surveys to lower levels than the survey foresaw (combined with recent census data) lack of control over data collection leads to non specific impact assessment of programmes (e.g. timing of existing surveys may not coincide with baseline or end of programme) may require statistical expertise to manipulate data and carry out analyses Learners Notes 3

5 Selecting the right datasets Once you have decided to use secondary data, you will need to know how to go about finding usable datasets and how to decide whether they are of the right quality for carrying out the impact assessment. There are several steps to follow that can guide you in the process of selecting the right datasets: 1. Clearly define your information needs from secondary data sources 2. Locate the potential datasets 3. Assess whether your potential datasets are usable for assessing impact 1. Clearly define your information needs from secondary data sources As a first step, you need to have a clear vision of the information you want from a secondary data source. You should consider the following questions: Which impact indicators do you need to create from secondary data sources? The data you hope to use should include responses to questions from which you can create food security impact indicators for the unit level (individuals, households, communities) that your programme works at. Which background characteristic variables should be included in secondary data sources? You will also need unit level information that describes the population, e.g. demographic characteristics, educational and labour market status, household assets and living arrangements. This assists in identifying an appropriate control group. Do the secondary data sources include information on programme participation status? It will be necessary to distinguish within your potential datasets those individuals, households or communities that participated in the programme from those that did not. 2. Locate the potential datasets The second step is to locate the potential datasets. For national or sector wide surveys that have been carried out by government institutions, two logical places to look for data are: government departments and statistical agencies. National surveys carried out on either a regular or periodic basis are typically the preserve of your country s national statistical organization (NSO) ( Small area surveys and censuses may also be conducted on a sub national basis, in which case it is important to review the archives of regional (states, provinces, local governments) statistical agencies. Learners Notes 4

6 Other appropriate datasets may be identified from a number of sources, such as NGOs, universities or research institutes. Finally, international agencies (including development agencies, donors and United Nations organizations) often play a role in funding data collection efforts and may have access to datasets you could use. Important note about accessing the data Census and survey information, once located, can be requested through inter agency exchanges. For external evaluators, official requests sometimes have to be cleared by senior departmental heads such as permanent secretaries or deputy ministers. Other roadblocks to data acquisition may emerge. For instance, you may face lengthy delays as you await official clearance. Also, NSO staff may be unable to provide the data in a usable format or not have sufficient staff resources to answer requests. You need to be aware of potential constraints in acquiring data from NSOs and other organizations that have archives of usable data. Occasionally, national datasets are available via Internet from international organizations that partnered with government institutions for data collection, such as Demographic and Health Surveys or poverty assessments. Therefore an Internet search is justified when trying to locate datasets that can be used for the impact assessment. Please note that often survey data used to create national indicators are also made available to researchers and other users for re-analysis. 3. Assess whether your potential datasets are usable for assessing impact Once you have identified datasets that match your information needs, you will need to check several data features to judge whether these datasets can be used for your purpose. In particular, you should look at: a. Reference period The data used to assess programme impact correspond to time periods that are relevant to the programme. The data need to have been collected close in time to when programme implementation began and after or near the completion of the programme in order to measure change. b. Population coverage The data were collected in the geographic area targeted by the programme c. Completeness There is little missing data for the variables you need to use; i.e. non responses to questions on household characteristics or questions to be used to create food security indicators d. Accuracy Learners Notes 5

7 the recorded responses are within plausible limits. The questions asked to respondents were carefully formulated to provide valid information. A preliminary (but often overlooked) task is to examine the raw data for errors. These may include duplicated records, invalid responses, inconsistent or contradictory coding, and missing data. Because quantitative impact assessment is so heavily data-reliant, its final quality will vary directly with that of its data input. Probing the data for errors and inconsistencies therefore represents a prudent use of earlystage resources and may lead to eliminating certain datasets from consideration if the problems are excessive. e. Bias The sample was chosen validly to represent the population it is meant to cover; certain sub groups or characteristics are not over- or under- represented in the dataset. Using secondary data When measuring impact with secondary data, you can either: 1. use information from published national survey reports, or 2. re-analyse data. 1. Use of national indicators from published reports Large scale food security programmes are part of national initiatives and policies enacted by Governments to reduce poverty and improve the well-being of their citizens. They contribute towards achievement of international targets such as those indicated in the Millennium Development Goals (MDG) and the World Food Summit (WFS) Plan of Action. Because of this, one option for assessing your programme with secondary data is to use nationallevel indicators found in published reports. These can be used to measure the contribution of your programme towards meeting national and international objectives for reducing food insecurity and poverty. For the purpose of tracking progress towards the achievement of the international targets, countries furnish data to international organizations on a series of indicators that are relevant to food security. It is quite possible to directly use reported indicators, collected at different times at national level, to assess the impact of large scale food security programmes that are applied to the country as a whole. Learners Notes 6

8 This is particularly true when the timing of large scale national surveys corresponds roughly to the beginning and the end of your programme. You can also use reported sub national level indicators when available, for example if your programme is only operating in rural communities. The role of the counterfactual when using national statistics to assess impact When the assumption is that all citizens are intended to benefit from the Programme that it is truly at national scale and that the available national statistics are pertinent to your programme, an assessment of progress can be carried out by tracking these national statistics over time to measure change. In this case, you will not be able to use a control group to create a counterfactual for attributing change to the programme itself. In reality, of course, not everyone would have an equal opportunity to benefit for a number of different reasons, but taking the example of a national agricultural policy governing trade and markets, it might not be possible to identify a valid control group that did not benefit from the policy in order to create the necessary counterfactual. 2. Re-analysis of secondary data In the case that the published national indicators are not specific enough to the type of impact your programme wishes to achieve, it may be necessary to re-analyze secondary data in order to: create outcome and impact indicators, construct a control group, fill in information not collected first hand, or provide baseline levels when it is possible to identify participants from the data. Note that the same survey datasets that generate national statistics are often available to researchers or assessors for re-analysis. Programme-specific indicators can be created from secondary data, provided that the data contain sufficient information to create the indicators and that information is available at the level of observation you are interested in (for example household or community). To do this, you will need to have access to the raw data (i.e. databases with all the data from which you will do the analysis). Take the case of a programme that conducted a post programme survey with both a participant and a control group but did not carry out a baseline survey. Learners Notes 7

9 In this case you may want to recreate a baseline situation for both participants and non participants. However, data collected for another purpose may not allow you to identify your actual programme participants and distinguish them from non participants. This is an important limiting factor of using secondary data when the programme is not at national scale. Therefore, it is important to determine early on whether such a situation exists and if other techniques can be used to resolve the problem. When there is no indication of programme participation status in the data you want to use, you must try to find a way to assign that status by triangulating other sources of information. This reinforces the usefulness of collecting administrative data and using key informants. In cases where programmes have targeted specific individuals or households for participation, household surveys carried out by others in the same geographical location may include a variable that identifies participation status, although this is not very common. With census data, it may be possible to identify communities that you know have taken part in the programme. Secondary data can be re-analyzed using specific statistical techniques to extract households that can be used as a control group. For example, this holds when a programme collected primary data from participant households at baseline and at the end, but: included a control group only for the end-of-programme survey; or did not collect data on a control group at either time period. Secondary data can also be used to fill in missing information that might not have been collected first hand by the programme, such as market or climate data at different times. Learners Notes 8

10 Summary Use of secondary data to assess programme impact is a wise choice when appropriate data can be located or when the programme has limited financial or human resources to collect data directly. National statistics can be used as reported to measure improvement towards meeting national and international goals, for example, the MDGs or the World Food Summit Goals, especially when the scope of the programme is to benefit all citizens. Secondary data may be re-analyzed, using specific statistical techniques, to create a control group, provide baseline levels when it is possible to identify participants from the data, or fill in information not collected first hand. If you want to know more Online resources FAO Deriving Food Security Information from National Household Budget Surveys. Rome. ftp://ftp.fao.org/docrep/fao/011/i0430e/i0430e00.pdf FAO Global Donor Platform for Rural Development, World Bank, Tracking results in agriculture and rural development in less-than-ideal conditions. Rome. FAO Pathways to Success. Rome. Ravallion, Martin The mystery of the vanishing benefits: Ms. Speedy Analyst's introduction to evaluation. World Bank Policy Research Working Paper No Washington, D.C.: World Bank. Wassenich, Paul Data for impact evaluation. Washington, D.C.: World Bank /Rendered/PDF/423800NWP0Doin10Box321452B01PUBLIC1.pdf Online links to national data sources and information on specific national surveys: Demographic and Health Surveys (DHS) website. FEWSNET homepage FAO/WFP Joint Guidelines for Crop and Food Security Assessment Missions. Learners Notes 9

11 Integrated Food Security Phase Classification homepage. International Household Survey Network home page. Multiple Indicator Cluster Survey website. National Household Survey basics _POSTCW06.pdf Smith, L. The Use of Household Expenditure Surveys for the Assessment of Food Security. FAO International Scientific Symposium on Measurement of Food Deprivation and Undernutrition World Bank. Living Standard Measurement Surveys ~pagePK: ~piPK: ~theSitePK: ,00.html World Bank, FAO Tracking results in agriculture and rural development in less-than-ideal conditions. Rome. Version.pdf] World Food Programme. Comprehensive Food Security and Vulnerability Analysis Guidelines FAO World Programme for the Census of Agriculture. Additional Reading Bamberger, M Conducting quality impact evaluations under budget, time and data constraints. Washington, D.C.: World Bank. Carleto, C,, and Morris, S.S Designing methods for the monitoring and evaluation of household food security rural development projects. Technical Guide #10. International Food Policy Research Institute. Washington, D.C.: IFPRI. Habicht, J.P., Pelto G.H., and Lapp, J Methodologies to evaluate the impact of large-scale nutrition projects. Washington, D.C.: World Bank. Ravallion, Martin Evaluation in the practice of development. World Bank Research Observer 24(1): Learners Notes 10

12 Annex I: Potential data sources The following table lists potential sources of data and the types of food security and nutrition information they can contribute towards assessment of large scale food security programmes: Potential sources Household Budget or Income and Expenditure Surveys, Living Standards Measurement Surveys (World Bank) Demographic and Health surveys (USAID) / Multiple Indicator Cluster Surveys (UNICEF) Indicators At household level: income sources, expenditures (of particular interest for food, school, health), poverty indicators, food consumption and expenditures, undernourishment. Maternal and child diet, mortality, nutritional status, knowledge, water/sanitation, gender-specific literacy and school enrolment, access to health care. National Nutritional surveys Agricultural surveys (e.g. WFP/FAO Crop assessments, Agricultural Census, production surveys) Food security and vulnerability surveys, early warning systems, monitoring and information systems (such as: WFP s Comprehensive Food Security and Vulnerability Analysis, Early Warning Systems, Famine Early Warning Network (FEWSNET), Integrated Phase Classification) Individual nutritional status, household and individual diet, household food security indicators. Crop and livestock data, production, productivity, input use, farmer organizations. Individual nutritional status, household dietary diversity, coping strategies, livelihood data, food availability and access, market and trade, shocks and climate. One appropriate source of data on relevant food security indicators is MDG Monitor which also tracks progress towards achieving MDG Goal Number 1: Eradicate Extreme Poverty and Hunger, through specific indicators which could be used to assess improvement in food security, such as prevalence of underweight children under-five years of age and proportion of population below minimum level of dietary energy requirement, or the prevalence of undernourishment ( Learners Notes 11

13 ). Another source of data for tracking a country s progress is through monitoring of its Poverty Reduction Strategies process ( Systems/PRSMonitoringDiagnosticTool_English.pdf). Learners Notes 12

14 Annex II: Example of use of national statistics The following table demonstrates the use of national statistics at different time periods to track progress on improving food security. Food Indicator Time Armenia Brazil Nigeria Vietnam Security Period Dimension Distribution of Wealth Food Availability Food Accessibility Food Utilization (safe food) Outcome (active life) Poverty Gap Ratio (less than USD 1.25 per day, percent) Dietary Energy Supply (kcal per capita) Road Network (km per 1000 rural persons) Share of population with access to clean water, percent Level of undernourishment, percent of the population ,2 44,5 2,2 1, ,9 58,1 2,5 3, Modified from FAO 2009, Pathways to success: Success stories in agricultural production and food security, table p. 7. Learners Notes 13

15 Annex III: Case study Assessing Impact of Development Programmes on Food Security A Case Study of Rural Road Rehabilitation in Trinidad and Tobago in the 1990s Ainsley Charles Road infrastructure has traditionally been regarded as a critical input for the rural economy in general and for the agricultural sector in particular, since it serves to lower transportation costs, to relax constraints to market access, and consequently to boost productivity. Farmers in road-deprived areas cannot easily exploit available market opportunities and so are more likely to choose to produce less risky staple crops over market crops, and to do so using traditional, lower-yield methods. They also tend to under-invest in health and education for themselves and for their children which, in a dynamic setting, lowers their long-term prospects of escaping poverty and food insecurity. One objective of rural road rehabilitation programmes, therefore, is to reverse (or moderate) these adverse outcomes by promoting a higher degree of market integration, the adoption of modern agricultural techniques, and the formation of enhanced human capital. 1. Objective: The objective of this study was to assess whether rehabilitated roads, as compared to unimproved roads that restrict farmers access to agricultural plots, can lead to a) increased school attendance and schooling tenure among school-age children and school leavers, respectively; b) increased use of modern agricultural techniques such as high-yield seed varieties; c) an increase in the variety of crops and in the proportion of marketed harvest; and d) a boost in overall agricultural production in low-income agricultural communities. 2. Beneficiaries: Beneficiaries were farmers and farming households in 50 rural communities throughout Trinidad and Tobago. 3. Programme intervention: Unimproved local roads and tracks that provided access to agricultural plots and a direct link to higher-level national roads were rehabilitated up to a standard of all-weather permanent access. These agricultural access roads, located in 50 programme communities, typically measured between two to three kilometres in length and were selected after having met a series of agriculture ministry-approved technical, economic and environmental criteria. To facilitate the impact assessment, agricultural communities in which access roads were not refurbished were chosen as potential comparison units. 4. Design: The programme was not randomised across communities but was instead targeted to certain communities but not to others. It was therefore necessary to construct a valid counterfactual group from the datasets and minimize differences between targeted farming communities and similar communities selected as controls. To select the best match for targeted communities among all other communities for whom Learners Notes 14

16 information was available in the datasets, a statistical technique called propensity-score matching was employed Data: Only secondary data were available for the impact assessment. Data came from multiple sources but were primarily composed of programme documentation and micro-unit census data. Programme documents were retrieved from the Inter-American Development Bank, a multilateral development institution operating throughout the Americas and the programme s principal funder. Analytical and feasibility studies and administrative and project monitoring reports came from project consultants and government ministries, respectively. Key informants provided additional institutional details and expert knowledge about the programme during a series of in-country interviews. And the national statistical agency provided unit-level data in the form of two population censuses and an agricultural census. 6. Methods: Analysis was conducted at the level of the community, a level of aggregation that was both closest to the actual level of the intervention and identifiable in all three census datasets. First, a model of community programme participation was estimated using maximum-likelihood logit regression which related to a set of key household-, head of household-, labour force-, and community-related characteristics. Using the Propensity Score Matching technique, non programme communities were selected that were most similar to the programme communities. This allowed having two sets of communities (i.e. participating and control) from which it was possible to compare outcome and impact variables to estimate programme impact 7. Outcome measures: For education the two primary outcomes indicators were the school attendance rate and the number of years of schooling (for new labour market entrants). For agriculture, the study used a series of outcomes indicators relating to a mix of agricultural inputs and output. Results: In respect of education, school-age children living in communities with rehabilitated roads attended school in higher proportion compared with children in non-programme comparison communities. Earlychildhood attendance was up to 6.3 percentage points higher, primary school attendance up to 2.0 percentage points higher, and advanced secondary school attendance up to 4.4 percentage points higher. Nineteen- and twenty-four-year-olds from programme communities newly entering the labour market had up to one-half of a year more of completed education compared with similar workers from non-programme communities. In terms of agriculture, farming operators in programme communities were 11 percentage points less likely to retain an informal land tenure status compared with operators in non-programme communities. Also, a lower proportion (-10 percentage points) worked full-time at agriculture, perhaps suggesting a diversification towards other employment options. Within agriculture, programme community farmers used more modern high-yield seed varieties when cultivating pulses (corn) and leafy vegetables 1 The propensity score matching technique, used in this study to match potential control communities with those who benefited from the roads improvement programme, is described in more detail in lesson 3.1. Learners Notes 15

17 (cabbage; lettuce). In addition, they harvested a higher quantity of corn, achieved higher yields of corn and string beans, and marketed a larger share of the total harvest of chive, pigeon peas, and pumpkin. 9. Findings: The study showed that in road-deprived agricultural communities, upgrading roads to permanent-access standard was likely to increase school attendance, extend the number of years spent in school for young labour market entrants, and spur changes in the agricultural input and output mix available to farmers. 10. Lessons: Impact assessment of large-scale, nationally-run programmes is possible, even when primary data are unavailable. In this case, it was possible to use secondary data such as censuses and ongoing survey, providing such data had a minimum number of key features. These features included the ability to identify programme participation, socioeconomic and demographic data for the communities included in the study as well as availability of outcome and impact variables of interest. Learners Notes 16