Impact Evaluation Methods and Applications to Access to Energy Università Cattolica del Sacro Cuore and Laboratorio Expo (Fondazione Feltrinelli) ExpoSchool, 4-2-2015 acopo Bonan (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to AccessExpoSchool, to Energy 4-2-2015 1 / 43
What you know already Energy poverty is now a recognized global policy issue: Energy access is VITAL for achieving MDGs Quality of life Health care Schooling Time allocation Female empowerment Food safety and security Business development, job creation, income generation and competitiveness (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to AccessExpoSchool, to Energy 4-2-2015 2 / 43
What you know already Energy access at global level (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to AccessExpoSchool, to Energy 4-2-2015 3 / 43
What you know already Energy and Economic Development (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to AccessExpoSchool, to Energy 4-2-2015 4 / 43
What you know already Energy and Human Development (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to AccessExpoSchool, to Energy 4-2-2015 5 / 43
What you know already Energy and Health (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to AccessExpoSchool, to Energy 4-2-2015 6 / 43
Policies for ghting energy poverty Rural electri cation programs: High within-country variability: urban-rural divide Big programs at national level (e.g. Brazil, China, India), with support of international agencies (e.g. World Bank, Sustainable Energy for All Initiative) Di erent technological solutions for production, transmission and distribution Public-private partnerships: need for long-term economic and nancial sustainability, but also a ordability (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to AccessExpoSchool, to Energy 4-2-2015 7 / 43
Policies for ghting energy poverty Adoption of improved cookstoves: Less attention than electricity programs: some large national programs (e.g. China), but not strong involvement by international organizations Large range of technological possibilities, e ciency and impacts on health Global Alliance for Clean Cookstove calls for 100 million homes with clean and e cient cookstoves by 2020 (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to AccessExpoSchool, to Energy 4-2-2015 8 / 43
Why Impact evaluation? Establish causual relationships -> trace the precise relationship between the intervention provided and the outputs and outcomes achieved (the attribution problem) Identify interventions which are e ective and e cient: how much of my outcome I obtain, with 1$ of intervention? Find mechanisms: why something work and something does not? What makes it work? (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to AccessExpoSchool, to Energy 4-2-2015 9 / 43
Impact evaluation methods To determine the impact of the program we need knowledge of counterfactuals, that is, what would have happened in the absence of the program? Problem: the true counterfactual is not observable The fundamental problem of impact evaluation is thus a problem of missing data We don t know what would have happened in the absence of the program (the counterfactual) The key goal of all program/impact evaluation methods is to construct or mimic the counterfactual as best as possible. (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 10 / 43
What we observe: (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 11 / 43
The counterfactual: what would have happened in the absence of the program? (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 12 / 43
The counterfactual: what would have happened in the absence of the program? (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 13 / 43
BUT... We will never have a child both with AND without a bednet at the same time... we will never observe simultaneously a women with AND without electricity So the counterfactual is NEVER observed Possible Solution: Use non-participants as point of comparison = Control Group E.g.: use kids who have not bednets, and women who do not have access to electricity (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 14 / 43
Simple di erence: bene ciaries vs non bene ciaries (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 15 / 43
BUT... If there are di erences in background characteristics between the group of bene ciaries and non-bene ciaries Example: if only kids who are very poor are o ered a bednet, or electricity is distributed in villages which are closer to cities and main roads, therefore better o than more remote ones without electricity Biased comparison: selection bias (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 16 / 43
Selection Bias (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 17 / 43
Selection Bias The outcome di erence that would be observed between bene ciaries and non-bene ciaries if the programme were not implemented; depends on pre-existing di erences between groups Exemple of positive selection bias: parents in schools that received textbooks prioritise education more than those in schools which did not Exemple of negative selection bias: NGO program targeting schools in disadvantaged areas (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 18 / 43
We are interested in the unit level causal e ect. But we cannot observe both potential outcome under treatment and non-treatment at individual level We cannot compare the same indiviual outcome before and after treatment: other factors a ecting the outcome may have changed over time We can only measure the average e ect in the population We need to nd plausible counterfactuals, solving the selection bias (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 19 / 43
How to solve the selection bias? Randomization Di erences-in-di erences Controlling for observables Regression discontinuity design (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 20 / 43
Randomization A treatment randomly assigned ensures that both treated and control groups di er in expectation only through their exposure to the treatment. The selection bias is equal to zero: elimination of pre-existing group di erences because only chance determines which subjects are assigned to which groups Law of large numbers: the larger the sample, the smaller preexisting di erences are likely to be: we compare average e ects: individuals assigned to treatment and control groups di er in expectation only through their exposure to the treatment Researchers test for random assignment by comparing means of various controls across both treated and control groups:.no di erence shows that randomization is ok (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 21 / 43
Problems and concerns with randomization Ethical constraints Financial costs Internal validity: A study is internally valid if we get unbiased estimate of the overall impact of the program in the sample under study excluding all other possible confounding factors. However, this can not be always the case: partial compliance, externalities/spillovers, attrition (non-response bias) External validiy: results are generalizable and replicable? Speci c procedure, treatment, sample. Evidence of good replications helps Hawtorne e ects: changes in behavior among groups (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 22 / 43
What if randomization is not feasable? What if we are in observational settings, when the analyst cannot manipulate the selection process, but can jus tobserve the process? Look at other methods which allow to create comparison groups that are valid under a set of identifying assumptions Identifying assumptions are not directly testable: need to be convincing!! (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 23 / 43
Di erence-in-di erence (DID) Non-experimental setting Outcome variable is observed in two points in time (panel data): before (t = 0): pre-treatment and after the policy intervention (t = 1): post-treatment The di erence in di erence estimator is de ned as the di erence in average outcome in the treatment group before and after treatment minus the di erence in average outcome in thecontrol group before and after treatment Identi cation assumption:the unobserved di erences between treatment and control groups are the same over time (the selection bias is constant over time) (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 24 / 43
(Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 25 / 43
Possible problems Failure of the common trend assumption: what if Y follows a di erent trend for the treatment and control group? BIAS!! Nedd to get more data on other time periods (pre and post intervention), nd other control groups Suppose the program is announced before, what if people manipulate the preprogram characteristics (e.g. migration)? The selection bias is not constant over time anymore! (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 26 / 43
Controlling for observables Non-random context: being bene ciary of a program or not depends on a (self) selection process Assumption: all factors relevant to the outcome variable that enter the selection process are observed for bene ciaries and non-bene ciaries (contained in X ) Find a comparison/control grouop with very similar observable characteristics to the treatment group among the non-bene ciaries Assumption: once we control for (or conditional on) this set of variables X, the selection bias is zero and treatment is as good as random assigned Idea: compare the outcomes of "treated" individuals vs very similar (according to all the characteristics that we assume are driving the selection process) "non-treated" individuals (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 27 / 43
Regression Discontinuity Design The probability of assignement to treatment group is deterministic and depends upon a discontinuous function of one or more observable variables X (when the observed variable exceeds a certain cut-o point): incentives to students above a certain average eligibility for microloans to women living in households with less than one acre of land etc.. Idea: compare individuals just below and just above the threshold. If the individuals cannot manipulate the assignment variable Key assumption: all other factors are continuous with respect to X: all baseline characteristics should have the same distribution just above and below the cuto (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 28 / 43
(Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 29 / 43
Possible problems We cannot test the continuity assumption directly, but we we can test some implications of it: all observed predetermined characteristics should have identical distributions on either side of the cuto The strength of the RD design is its internal validity:arguably the strongest of any quasi-experimental design External validity may be limited What if people can manipulate the level of the underlying variable that determines eligibility? Test for continuity in the density of the assignment variable at the cuto (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 30 / 43
Impact evaluation of access to energy Review of the latest empirical and analytical evidence about impacts of energy access (electricity and improved cookstoves) on di erent outcomes: Labour market, household welfare, business Barriers and drivers to access to modern energy Selection criteria: use of microeconmetric studies and impact evaluation methods (experimental and non-experimental designs) (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 31 / 43
Impacts of access to electricity (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 32 / 43
Impacts of access to electricity (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 33 / 43
Impacts of access to electricity (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 34 / 43
Impacts of access to clean and e cient cookstoves (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 35 / 43
(Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 36 / 43
Barriers and drivers to access to modern energy services Role of women in the family: stronger preference for healthier stoves, but lack the authority to make purchases (Miller and Mobarak, 2013, Bangladesh) Role of networks and opinion leaders in decision to adopt improved cookstoves (Miller and Mobarak, 2013, Bangladesh); electricity connection as social status with consequent bandwagon e ects (Bernard and Torero, 2013, Ethiopia) Informational and nancial constraints (Levine et al. 2013, Uganda) Role of informational constraints and social networks/ peer pressure (see next slides) (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 37 / 43
Research project in Mali Information and social network in the decision to adopt e cient cookstove: an experimental design for Mali Team: P.Battiston (S. S. Sant Anna), J.Bonan (UCSC),J Bleck (Notre Dame Univ, US), P.LeMay-Boucher (Heriot-Watt Univ, GB), B.Sarr (PSE) (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 38 / 43
The context: Malian urban context 95% use solid fuels for cooking only 6% have access to clean fuels less than 0.5% use improved biomass cookstoves The objectives 1 Role of group decisions and social networks: What is the e ect of the group (peer e ects) at the moment of taking decisions on improved technology adoption? Is there evidence of bandwagon e ects? 2 Role of informational barriers: to what extent presenting the product and its advantages increases the uptake? 3 Role of spillover e ects: to what extent observing for longtime people who adopted the new product in uences the decision to purchase? 4 Impact of improved cookstoves on household welfare: what is the e ect of purchase on usage, fuel expenditure, time allocation, health conditions and saving at household level? (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 39 / 43
(Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 40 / 43
Research design Baseline survey of a sample of 960 randomly selected women in Bamako in 32 neighbourhoods Use of Randomized Control Trials (RCT): random assignment to di erent treatments: Demonstration/training session Demonstration + information peer purchase decision Control group Monitor of usage, through installation of Stove Use Monitors on a sub-sample (N=150) of purchasers (January-April 2015) Endline survey (May-June 2015) (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 41 / 43
Results???? Soon!! (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 42 / 43
References On impact evaluation methods J. Angrist and JS Pischke (2009) Mostly Harmless Econometrics: An Empiricist s Companion, Princeton University Press Du o, E., Glennerster, R. and Kremer, M. (2008) Using Randomization in Development Economics Research: a Toolkit. Imbens, G. (2004). Nonparametric Estimation of Average Treatment E ects under Exogeneity: A Review. Review of Economics and Statistics. Rachel Glennerster and Kudzai Takavarasha (2013) Running Randomized Evaluations: A Practical Guide David S. Lee & Thomas Lemieux, 2010. "Regression Discontinuity Designs in Economics," Journal of Economic Literature, vol. 48(2), pages 281-355. Review of impacts on access to energy J.Bonan, S.Pareglio, M.Tavoni (2014). Access to modern energy: a review of impact evaluations, FEEM Working Paper No. 96.2014 (Università Cattolica del SacroImpact CuoreEvaluation and Laboratorio Methods Expo and(fondazione Applications Feltrinelli)) to Access ExpoSchool, to Energy 4-2-2015 43 / 43