Impact evaluation using secondary data Experiences from IEG's evaluation of maternal and child health and nutrition in Bangladesh

Transcription

1 Impact evaluation using secondary data Experiences from IEG's evaluation of maternal and child health and nutrition in Bangladesh Howard White Independent Evaluation Group

2 Overview IEG impact evaluations Data sources for impact evaluation Bangladesh Inferences from cross-country data Using DHS data Supplementing project data Limitations of secondary data

3 IEG Impact evaluation Over 80 historically, of which 25 impact evaluation in current meaning of term All ex post, mostly employing single or double difference approach Most recent apply theory-based approach with regression analysis. Contextualized and policy relevant

4 Data sources for IE Projectspecific survey Piggybacking Synchronized survey Secondary data Advantages Geared toward evaluation questions Large sample at low cost Quality control at low cost Cheap Disadvantages Quality of control and data Possibly limited data Long lead time needed Possibly limited data Difficult to arrange Not geared toward evaluation questions

5 Sources of secondary data Existing national surveys (DHS, LSMS) Re-analysis of project data Previous project or research surveys Cross-country data Existing studies

6 Example one: immunization Percent of children Sources: Annual data from WDI. 'Bars' from DHS. Fully immunized (all children) Fully immunized (by 12 months) DPT only (by 12 months)

7 Methods Cross-country regression: fixed effects panel data for 78 developing countries; regressors include income, female literacy, water supply, nutrition and immunization Accounting approach: live-saving approach of each vaccination (Khan and Yoder, 1989) Child-level regression: Hazards models with DHS data, using community, household and child characteristics as control variables; instrument using without-self community mean

8 Cross-country regression: under-five mortality Under five mortality (per 000 live births) s s Income-driven trajectory 50 0 Fitted line GDP per capita (US$)

9 Estimates of child deaths averted by immunization Approach Period Total (millions) Per year (000s) Cross-country regression Accounting-based Child-level regression

10 Example 2: Training TBAs Used Oaxaca decomposition in mortality regressions (i.e. analyze effect of change in input and change in its productivity) (use bivariate probit to allow for endogeneity) See significant increase in productivity (lower mortality) Suggest impact of 5-7deaths averted per 1,000 live births Implication: reconsider abandoning of training

11 Mortality has fallen more rapidly for deliveries attended by trained TBAs Percent of deliveries Neo-natal mortality (per 1,000 live births) Untrained Trained Untrained Trained

12 Example 3: Bangladesh Integrated Nutrition Project (BINP) Growth-monitoring, with nutritional counseling and supplementary feeding Project evaluation found some impact on child nutrition and pregnancy weight gain However, other studies (Save the Children and IMED) found no significant impact

13 IEG study Used project evaluation dataset, but created new control using HKI s NSP data for propensity score matching Also used Save the Children data, including register data Finally, identified project communities in DHS data

14 Outcome data Mid-term End-line single difference in z score HAZ WAZ HAZ WAZ BINP evaluation OED PSM

15 Impact restricted to sub-groups Regression analysis allows for heterogeneous impact: Most malnourished (women and children) Lean season Less educated mothers

16 Weak links in causal chain Mis-targeting of nutritional messages Targeting criteria for supplementary feeding poorly applied Leakage and substitution Knowledge practice gap Pre-pregnancy weight more important than pregnancy weight gain

17 Women s agency Women who can visit health center on own or with children Per cent All women Married in MHH Living with MIL

18 Lessons on using secondary data BINP was a good case, rich data sets available with scope for further and more rigorous analysis DHS (or other large survey) data can establish impact (need to identify beneficiaries) But less able to uncover process Large gap is lack of linked facility data But also lack of demand side factors

19 Bottom line Secondary data can play a valuable role in impact evaluation, especially if intervention (not project) focused. So useful for evaluation at sector level. But should seek to influence survey design.