Relative Effectiveness of Conditional vs. Unconditional Cash Transfers for Schooling in Developing Countries: a systematic review Sarah Baird (George Washington University) Francisco Ferreira (World Bank) Berk Özler (University of Otago/World Bank) Michael Woolcock (World Bank) Not for citation without explicit permission from the authors. 1
Outline Background & objectives Search strategy & selection criteria Data collection & analysis Main Results Authors Conclusions Acknowledgements & Funding 2
BACKGROUND AND OBJECTIVES 3
Background The debate over whether cash transfer programs should have behavioral conditions has been at the forefront of recent global policy discussions. Conditional Cash Transfer (CCT) programs improve the outcomes upon which they are conditioned (Fiszbein and Schady, 2009). Unconditional Cash Transfer (UCT) programs have also been shown to influence the same set of outcomes. Furthermore, the theory that predicts the sign of the relative effects is fairly straightforward and supported by extant evidence. 4
Motivation So, why conduct a systematic review comparing the relative effectiveness of these two types of programs on schooling outcomes? A few reasons: While the sign is obvious, the size of the difference is not Furthermore, the difference may be heterogeneous across settings Few evaluations of truly unconditional cash transfers Microsimulations using structural models of CCTs programs in Mexico and Brazil produced tiny income effects for schooling and child labor. 5
Motivation The systematic review provides: supporting evidence for the existence of income effects on schooling outcomes (OK, mainly enrollment and attendance rather than learning) an unforeseen finding on the heterogeneity of impacts across the intensity of conditions (Morocco s Tayssir was originally designed to test this) more or less a confirmation of the findings from earlier studies. Was it worth it (to the donor that commissioned the review)? There is a fair bit of heterogeneity in effect sizes across studies, especially CCTs, so the SR should not be seen as a substitute for careful consideration of design elements in each setting. However, it can perhaps provide a starting point 6
Background Given that the theoretical default should be to opt for UCTs, why would we attach any conditions? The main argument for UCTs is that the key constraint for poor people is simply lack of money (e.g. because of credit constraints), and thus they are best equipped to decide what to do with the cash (Hanlon, Barrientos and Hulme 2010). Three main arguments for CCTs: market failures that causes suboptimal levels of education; investments in education below socially optimal level; political economy. 7
Three main arguments for the condition 1. The behavior of households is not privately optimal: a. Imperfect information, b. Hyperbolic discounting, c. Intra-household bargaining (principal-agent) problems 2. The behavior of households is not socially optimal, i.e. there are significant externalities. 3. The political economy argument. 8
Figure 1: Theory of change for schooling conditional cash transfers and unconditional cash transfers on schooling outcomes Intervention A Input Immediate Change Unconditional Cash Transfer Cash Income Intermediate Outcomes: Final Outcomes: Intervention B Schooling Conditional Cash Transfer Input Cash IF meet condition Immediate Change Income Relative price of Schooling (subsitution effect) School Attendance School Enrollment Test Scores Moderating Factors: Enforcement of condition (CCT only), transfer size, baseline enrollment rate, transfer recipient, program size 9
SEARCH STRATEGY AND SELECTION CRITERIA 10
Search Strategy Five main strategies were used to identify relevant reports (1) Electronic searches of 37 international databases (concluded on April, 18 2012) (2) contacted researchers working in the area (3) hand searched key journals (4) reviewed websites of relevant organizations (5) given the year delay between the original search and the final edits of the review we updated our references with all new eligible references the study team was aware of as of April 30, 2013. 11
Eligible Reports Report had to either assess the impact of a conditional cash transfer program (CCT), with at least one condition explicitly related to schooling, or evaluate an unconditional cash transfer program (UCT). The report had to include at least one quantifiable measure of enrollment, attendance or test scores. The report had to be published after 1997 The report utilize a randomized control trial or a quasiexperimental design. The report had to take place in a developing country. 12
13
DATA COLLECTION AND ANALYSIS 14
Calculating Effect Sizes Measures of treatment effects come from three different types of studies: CCT vs. control, UCT vs. control, and, for four experimental studies, CCT vs. UCT. For these latter set of studies, a separate effect size for CCT and UCT (each compared with the control group of no intervention) is constructed. We construct odds ratios for effect size measures of enrollment and attendance, and report test score results in standard deviations. Economists typically do not report the ideal level of information, almost exclusively use cluster designs, and there are multiple reports per study, as well as multiple measures per report. 15
Calculating Effect Sizes We define an intervention to be a UCT or a CCT. We define a study to be a different version of a UCT or a CCT (or in a few experiments a UCT and a CCT) implemented in different places For many of these studies, there are multiple publications (journal articles, working papers, technical reports, etc.). We refer to these as reports. In our meta-analysis, the unit of observation is the study. This means that we would like to construct one effect size per study for the overall effect on any of our three outcome variables and for each subgroup (if reported). 16
Calculating Effect Sizes To add to the complexity of these layers, each report may contain multiple estimates for the same outcome. For example: enrollment effects from multiple follow-up surveys (assessing shorter- and longer-term effects); learning effects from multiple achievement tests (such as English, Spanish, and Math); Estimates using multiple estimation techniques (with and without baseline controls, nearest neighbor matching vs. one-to-one matching), and so on Furthermore, some studies report effects only for subgroups (such as by age or urban/rural or grade or sex), but report no overall effect. 17
Calculating Effect Sizes For each subgroup, we construct one effect size by synthesizing and summarizing multiple effect sizes within each report, then again synthesizing and summarizing those combined effect sizes from different reports within a study. We create synthetic effects when the effect sizes are not independent of each other. This is the case when there are multiple effects reported for the same sample of participants. These effects are combined using a simple average of each effect size (ES) and the variance is calculated as the variance of that mean with the correlation coefficient r assumed to be equal to 1 When two or more ES are independent of each other, we create summary effects. To combine these estimates into an overall estimate (or an estimate for a pre-defined subgroup), we utilize a random effects (RE) model. 18
MAIN RESULTS 19
Results of the search 75 reports were included in our review. Table 4: Characteristics of analysis sample Panel A: Reference level characteristics: (N=75) Number % Publication type: Journal article 33 44.00% Working paper 27 36.00% Technical Reports 10 13.33% Dissertation 4 5.33% Unpublished 1 1.33% Reports effects on: Enrollment/Dropout 67 89.33% Attendance 17 22.67% Test Score 12 16.00% 20
Results of the search Panel B: Study level characteristics, binary (N=35) Number % UCT 5 14.29% CCT 26 74.29% UCT/CCT 4 11.43% Regional Distribution Latin America and the Caribbean 19 54.29% Asia 8 22.86% Africa 8 22.86% Female recipient 16 45.71% Pilot Program 9 25.71% Random Assignment 12 34.29% Panel C: Study level characteristics, continuous (N=35) Mean Std Control Follow-up Enrollment Rate 0.785 0.146 # of Reports per Study 2.17 2.360 Transfers per Year 8.24 4.020 Transfer amount (% of HH Income) 5.66 7.890 Annual per Person Cost (USD) 351 414 21
Program Name Country Odds Ratio (95% CI) UCT Social Cash Transfer Scheme Malawi Child Support Grant South Africa CT-OVC Kenya Old Age Pension Program South Africa Old Age Pension Brazil SIHR Malawi Nahouri Cash Transfers Pilot Project Burkino Faso Tayssir Morocco Subtotal (I-squared = 52.2%, p = 0.041). CCT Social Risk Mitigation Project Turkey Program Keluarga Harapan (KPH) Indonesia Bono Juancito Pinto Bolivia Conditional Subsidies for School Attendance Colombia Chile Solidario Chile Ingreso Ciudadano Uruguay Oportunidades Mexico Familias en Accion Colombia Bono de Desarrollo Ecuador Juntos Peru Japan Fund for Poverty Reduction Cambodia Tayssir Morocco Jaring Pengamanan Sosial (JPS) Indonesia PRAF II Honduras Pantawid Pamilyang Pilipino Program Philipines PROGRESA Mexico Nahouri Cash Transfers Pilot Project Burkino Faso Tekopora Paraguay Female Secondary Stipend Program Bangladesh Red de Opportunidades Panama Bolsa Escola Brazil Bolsa Familia Brazil SIHR Malawi CESSP Scholarship Program Cambodia China Pilot China Comunidades Solidarias Rurales El Salvador Red de Proteccion Social Nicaragua Subtotal (I-squared = 86.5%, p = 0.000). Overall (I-squared = 84.5%, p = 0.000) 1.04 (0.82, 1.31) 1.04 (0.53, 2.04) 1.11 (0.84, 1.47) 1.15 (0.82, 1.62) 1.15 (0.96, 1.38) 1.30 (0.96, 1.75) 1.31 (0.94, 1.83) 1.59 (1.38, 1.85) 1.23 (1.08, 1.41) 0.72 (0.47, 1.11) 0.98 (0.95, 1.02) 1.02 (0.92, 1.14) 1.05 (0.96, 1.16) 1.22 (1.00, 1.50) 1.25 (0.87, 1.79) 1.25 (1.09, 1.43) 1.29 (1.06, 1.56) 1.30 (1.07, 1.57) 1.33 (1.16, 1.53) 1.34 (0.95, 1.88) 1.40 (1.20, 1.64) 1.42 (1.19, 1.70) 1.45 (1.20, 1.75) 1.48 (0.80, 2.73) 1.48 (1.27, 1.72) 1.50 (1.03, 2.17) 1.53 (0.72, 3.24) 1.74 (1.10, 2.77) 1.85 (1.23, 2.80) 1.90 (1.01, 3.58) 1.96 (0.82, 4.66) 1.98 (1.53, 2.57) 2.72 (1.92, 3.87) 2.74 (1.18, 6.37) 3.78 (1.62, 8.82) 4.36 (2.08, 9.11) 1.41 (1.27, 1.56) 1.36 (1.24, 1.48).5 1 1.5 2 3 4 intervention reduces enrollment intervention increases enrollment 22
Table 10: Summary of Findings (Enrollment) Odds of Child Being Enrolled in School: Statistically # Effect Comments Significant?* Sizes* CCT vs. UCT Our analysis of enrollment includes 35 Overall (vs. Control) 36% higher Yes 35 UCT (vs. Control) 23% higher Yes 8 CCT (vs. Control) 41% higher Yes 27 CCT (vs. UCT) 15% higher No 35 Condition Enforcement No Schooling Condition (vs. Control) 18% higher Yes 6 Some Schooling Condition (vs. Control) 25% higher Yes 14 Explicit Conditions (vs. Control) 60% higher Yes 15 Intensity of Condition Increases by 7% for each unit increase in intensity of condition. Yes 35 effect sizes from 32 studies. Both CCTs and UCTs significantly increase the odds of a child being enrolled in school, with no significant difference between the two groups. This binary distinction masks considerable heterogeneity in the intensity of the monitoring and enforcement of the condition. When we further categorize the studies, we find a significant increase in the odds of a child being enrolled in school as the intensity of the condition increases. In addition, studies with explicit conditions have significantly larger effects than studies with some or no conditions. Notes: We consider a study to be statistically significant if it is significant at the 90% level or higher. I use the term effect size here instead of study since the studies that directly compare CCTs and UCTs have two effect sizes in the analysis. All other studies have one. 23
Summary of findings (enrollment) Variation due to heterogeneity of effect sizes across studies is much larger for CCTs than UCTs Large range of effect sizes for CCT studies (OR: 0.7 to 4.4) With the exception of Tayssir, the UCT effects on enrollment are tightly bunched together below an OR of 1.3. Recategorizing Ecuador s BDH and Morocco s Tayssir (or excluding these fuzzy studies from the analysis) influenced the conclusion regarding the statistical significance of the comparison of CCT vs. UCT This was less than satisfactory and rather arbitrary Then, we had an idea 24
CCT vs. UCT too simplistic? Could we categorize all programs, and not just the CCTs, in order of the intensity of schooling conditionalities imposed by the administrators? 0. UCT programs unrelated to children or education such as Old Age Pension Programs (2) 1. UCT programs targeted at children with an aim of improving schooling outcomes such as Kenya s CT-OVC or South Africa s Child Support Grant (2) 2. UCTs that are conducted within a rubric of education such as Malawi s SIHR UCT arm or Burkina Faso s Nahouri Cash Transfers Pilot Project UCT arm (3) 3. Explicit conditions on paper and/or encouragement of children s schooling, but no monitoring or enforcement such as Ecuador s BDH or Malawi s SCTS (8) 25
CCT vs. UCT too simplistic? 4. Explicit conditions, (imperfectly) monitored, with minimal enforcement such as Brazil s Bolsa Familia or Mexico s PROGRESA (8) 5. Explicit conditions with monitoring and enforcement of enrollment condition such as Honduras PRAF-II or Cambodia s CESSP Scholarship Program (6) 6. Explicit conditions with monitoring and enforcement of attendance condition such as Malawi's SIHR CCT arm or China s Pilot CCT program (10) 26
Odds Ratio -.5 0.5 1 1.5 2 0 2 4 6 Condition Enforced 27
Program Name Country Odds Ratio (95% CI) No Schooling Conditions Child Support Grant CT-OVC South Africa Kenya Old Age Pension Program Old Age Pension South Africa Brazil SIHR Malawi Nahouri Cash Transfers Pilot Project Burkino Faso Subtotal (I-squared = 0.0%, p = 0.950). Some Schooling Conditions with No Monitoring or Enforcement Social Risk Mitigation Project Turkey Program Keluarga Harapan (KPH) Bono Juancito Pinto Indonesia Bolivia Social Cash Transfer Scheme Malawi Chile Solidario Chile Oportunidades Mexico Bono de Desarrollo Juntos Ecuador Peru PROGRESA Mexico Tekopora Paraguay Tayssir Morocco Female Secondary Stipend Program Bangladesh Bolsa Escola Brazil Bolsa Familia Brazil Subtotal (I-squared = 87.2%, p = 0.000). Explicit Conditions Monitored and Enforced Conditional Subsidies for School Attendance Colombia Ingreso Ciudadano Uruguay Familias en Accion Colombia Japan Fund for Poverty Reduction Cambodia Tayssir Morocco Jaring Pengamanan Sosial (JPS) Indonesia PRAF II Honduras Pantawid Pamilyang Pilipino Program Philipines Nahouri Cash Transfers Pilot Project Burkino Faso Red de Opportunidades Panama SIHR Malawi CESSP Scholarship Program Cambodia China Pilot China Comunidades Solidarias Rurales El Salvador Red de Proteccion Social Nicaragua Subtotal (I-squared = 80.6%, p = 0.000). Overall (I-squared = 84.5%, p = 0.000) 1.04 (0.53, 2.04) 1.11 (0.84, 1.47) 1.15 (0.82, 1.62) 1.15 (0.96, 1.38) 1.30 (0.96, 1.75) 1.31 (0.94, 1.83) 1.18 (1.05, 1.33) 0.72 (0.47, 1.11) 0.98 (0.95, 1.02) 1.02 (0.92, 1.14) 1.04 (0.82, 1.31) 1.22 (1.00, 1.50) 1.25 (1.09, 1.43) 1.30 (1.07, 1.57) 1.33 (1.16, 1.53) 1.48 (1.27, 1.72) 1.53 (0.72, 3.24) 1.59 (1.38, 1.85) 1.74 (1.10, 2.77) 1.90 (1.01, 3.58) 1.96 (0.82, 4.66) 1.25 (1.10, 1.42) 1.05 (0.96, 1.16) 1.25 (0.87, 1.79) 1.29 (1.06, 1.56) 1.34 (0.95, 1.88) 1.40 (1.20, 1.64) 1.42 (1.19, 1.70) 1.45 (1.20, 1.75) 1.48 (0.80, 2.73) 1.50 (1.03, 2.17) 1.85 (1.23, 2.80) 1.98 (1.53, 2.57) 2.72 (1.92, 3.87) 2.74 (1.18, 6.37) 3.78 (1.62, 8.82) 4.36 (2.08, 9.11) 1.60 (1.37, 1.88) 1.36 (1.24, 1.48).5 1 1.5 2 3 6 intervention reduces enrollment intervention increases enrollment 28
29
Summary of findings (intensity of monitoring and enforcement) The 95% confidence intervals of no conditions and explicit conditions do not overlap. Endogeneity (in the form of omitted variable bias) is a possibility, but controlling for obvious moderators leaves the effect unchanged (and, as expected, makes it more precise) None of the other moderators, including transfer size, seems to matter and three quarters of the variation remains unexplained Implications for policymakers designing new programs 30
Table 11: Summary of Findings (attendance and test scores) Panel A: Attendance Odds of Child # Being Enrolled in Statistically Effect Comments School: Significant?* Sizes* Overall (vs. Control) 59% higher Yes 20 A smaller number of studies assess the affect of CCTs UCT (vs. Control 42% higher Yes 5 and UCTs on attendance compared to enrollment. Both CCTs and UCTs have a significant affect on attendance. CCT (vs. Control) 64% higher Yes 15 While the effect size is always positive, we do not detect CCT vs. UCT (regression) 17% higher No 20 significant differences between CCTs and UCTs on Intensity of Conditionality attendance. (regression) Increases by 8% for each unit increase No 20 in intensity of condition. Standard Deviation Increase in Test Scores Panel B: Test Scores # Statistically Effect Comments Significant?* Sizes* Overall (vs. Control) 0.06 Yes 8 There are very few studies that analyze test scores. We have a total of 8 effect sizes measured from 5 studies. UCT (vs. Control 0.04 No 3 CCTs significantly increase test scores, though the size is CCT (vs. Control) 0.08 Yes 5 very small at 0.08 standard deviations. We find no CCT vs. UCT (regression) 0.05 No 8 impact of UCTs on test scores. Additional research on Intensity of Conditionality the impact of CCTs and UCTs on test scores is needed. (regression) In order to include these results in meta-analysis tests should be conducted with the entire sample, and results No 8 presented in terms of standard deviations. Increase of 0.02 standard deviations for each unit increase in intensity of conditions Notes: We consider a study to be statistically significant if it is significant at the 90% level or higher. I use the term effect size here instead of study since the studies that directly compare CCTs and UCTs have two effect sizes in the analysis. All other studies have one. 31
Publication bias? Funnel Plot (CCT enrollment).4 Standard Error of Log Odds Ratio.3.2.1 0 1.41 Odds Ratio 32
Publication bias? The usual interpretation for such a funnel plot is small study bias. But, almost none of the studies in our search fit this criteria Mostly evaluations of large government programs or specially designed experiments There may be legitimate reasons for high variance studies having large effect sizes OR researchers who used existing survey data (not specially designed for evaluation of the programs in question) may have given up after finding null effects The important question is the effect of this issue on the findings of this SR. 33
Publication bias? Cumulative RE Meta-Analysis Program Name Country Odds Ratio (95% CI) Program Keluarga Harapan (KPH) Conditional Subsidies for School Attendance Bono Juancito Pinto Oportunidades Juntos PROGRESA Tayssir Jaring Pengamanan Sosial (JPS) PRAF II Bono de Desarrollo Familias en Accion Chile Solidario SIHR Japan Fund for Poverty Reduction CESSP Scholarship Program Ingreso Ciudadano Nahouri Cash Transfers Pilot Project Red de Opportunidades Social Risk Mitigation Project Female Secondary Stipend Program Pantawid Pamilyang Pilipino Program Bolsa Escola Red de Proteccion Social Tekopora China Pilot Comunidades Solidarias Rurales Bolsa Familia Indonesia Colombia Bolivia Mexico Peru Mexico Morocco Indonesia Honduras Ecuador Colombia Chile Malawi Cambodia Cambodia Uruguay Burkino Faso Panama Turkey Bangladesh Philipines Brazil Nicaragua Paraguay China El Salvador Brazil 0.98 (0.95, 1.02) 1.00 (0.94, 1.07) 1.00 (0.96, 1.04) 1.06 (0.97, 1.16) 1.10 (0.99, 1.23) 1.16 (1.02, 1.31) 1.19 (1.05, 1.35) 1.21 (1.07, 1.37) 1.24 (1.10, 1.39) 1.24 (1.11, 1.39) 1.24 (1.12, 1.39) 1.24 (1.12, 1.38) 1.28 (1.15, 1.43) 1.28 (1.15, 1.42) 1.33 (1.19, 1.49) 1.33 (1.19, 1.48) 1.33 (1.20, 1.48) 1.35 (1.21, 1.50) 1.32 (1.19, 1.47) 1.33 (1.20, 1.48) 1.34 (1.21, 1.48) 1.34 (1.21, 1.49) 1.37 (1.23, 1.52) 1.37 (1.24, 1.52) 1.38 (1.25, 1.54) 1.40 (1.26, 1.56) 1.41 (1.27, 1.56).639 1 1.56 34
AUTHORS CONCLUSIONS 35
Authors Conclusions (1) Our main finding is that both CCTs and UCTs improve the odds of being enrolled in and attending school compared to no cash transfer program. The pooled effect sizes for enrollment and attendance are always larger for CCT programs compared to UCT programs but the difference is not significant. The findings of relative effectiveness on enrollment in this systematic review are also consistent with experiments that contrast CCT and UCT treatments directly. When programs are categorized as having no schooling conditions, having some conditions with minimal monitoring and enforcement, and having explicit conditions that are monitored and enforced, a much clearer pattern emerges. While interventions with no conditions or some conditions that are not monitored have some effect on enrollment rates (18-25% improvement in odds of being enrolled in school), programs that are explicitly conditional, monitor compliance and penalize non-compliance have substantively larger effects (60% improvement in odds of enrollment). 36
Authors Conclusions (2) The effectiveness of cash transfer programs on test scores is small at best. It seems likely that without complementing interventions, cash transfers are unlikely to improve learning substantively. Limitations: Very few rigorous evaluations of UCTs need more research! Study limited to education outcomes Most of the heterogeneity in effect sizes remains unexplained Not much information on cost Researchers: Report relevant data to calculate effect size (i.e. control means at baseline and follow up) Self reports vs. more objective measures. 37
Acknowledgements and Funding Special thanks go to: International Development Coordinating Group of the Campbell Collaboration for their assistance in development of the protocol and draft report. John Eyers and Emily Tanner-Smith as well as anonymous referees for detailed comments that greatly improved the protocol. David Wilson for help with the effect size calculations. Josefine Durazo, Reem Ghoneim, and Pierre Pratley for research assistance. Funding This research has been funded by the Australian Agency for International Development (AusAID). The views expressed in the publication are those of the authors and not necessarily those of the Commonwealth of Australia. The Commonwealth of Australia accepts no responsibility for any loss, damage or injury resulting from reliance on any of the information or views contained in this publication The Institute for International and Economic Policy (IIEP) at George Washington University also assisted with funding for a research assistant. 38
39