Section 2 Randomized and Natural (or Quasi-) Experiments

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "Section 2 Randomized and Natural (or Quasi-) Experiments"

Transcription

1 Section 2 Randomized and Natural (or Quasi-) Experiments

2 Randomized versus natural (or quasi-) experiments Definitions: A (randomized) experiment is designed and implemented consciously by human researchers. It entails conscious use of a treatment and control group with random assignment (e.g. clinical trials of drugs). A natural or quasi-experiment has a source of randomization that is as if randomly assigned, but this variation was not part of a conscious randomized treatment and control design.

3 Randomized Experiments How can randomization solve the evaluation problem? Comparison group selected using randomization device to randomly exclude some fraction of program applicants from program per definition: no selection into treatment (if randomization worked) Main advantage: comparability between program participants and nonparticipants same distribution of observables and unobservables Formally: randomization leads to E(Y1 D=1)=E(Y1 D=0)=E(Y1) and E(Y0 D=0)=E(Y0 D=1)=E(Y0), so ATE = E(Y1)-E(Y0) = E(Y1 D=1)-E(Y0 D=0) (last two exp observed) Note: ATE=E(Y1 D=1)-E(Y0 D=1) = TTE

4 Examples of randomized experiments Large-scale experiments, e.g. in the US/Canada: US National JTPA (Job Training Partnership Act) Study, Tennessee class size experiment (STAR) More recently, randomized experiments in developing countries: Small experiments addressing very specific questions, for example microfinance experiments by Dean Karlan, on education (e.g. schooling inputs) by Michael Kremer and Esther Duflo, etc Example of a large-scale and very successful conditional cash transfer program: Progresa/Oportunidades in Mexico ( )

5 Conditional cash transfer program: Progresa/Oportunidades By 2000, Progresa covered appr. 2.6 mio families (1/3 of rural families or 10% of all families in Mexico) and operates in almost 50,000 rural villages in 31 states. Progresa s budget is about US$ 800 mio or 0.2% of GDP. Significantly improved education and health of children by giving financial incentives ( conditional cash ) to keep children in school, to undertake preventive health care measures (eg vaccinations) and take nutrition additives. Size of the cash transfer is large: on average appr 1/3 of household income given to mother of the family. Served as role model for similar programs all over the world, e.g. Argentina, Colombia, Nicaragua,

6 Conditional cash transfer program: Progresa/Oportunidades Rural Progresa/Oportunidades uses a controlled randomized design: In 1998, 506 of the 50,000 Progresa villages were randomly assigned to control and treatment groups. Eligible (i.e. poor determined through proxy means test) households in treatment villages received benefits immediately, while benefits for eligible households in control villages were postponed for 2 years. T (treatment villages): eligible and ineligible households C (control villages): eligible and ineligible household Households were informed about eligibility and take-up rate was 97%. Timeline of survey (appr 14,500 HH with over 80,000 individuals): Baseline periods: 1997, march 1998 Treatment and control: oct 1998, march 1999, nov 1999 Treatment also for former controls: 2000 Introduction of Progr/Oport in urban areas in 2000 (not randomized due to political opposition)

7 Problems with Experiments in Practice A) Threats to Internal Validity 1. Failure to randomize (or imperfect randomization) 2. Failure to follow treatment protocol ( partial compliance ) Some controls get treatment, some treated dropout of program Differential attrition (e.g. in job training program, controls who find jobs move out of town) 3. Experimental effects Experimenter bias: treatment is associated with extra effort Subject behavior might be affected by being in an experiment (Hawthorne effect) Just as in regression analysis with observational data, threats to internal validity of regression with experimental data implies that Cov(D,U) is not equal to zero, so OLS (difference estimator) is biased.

8 Problems with Experiments in Practice B) Threats to External Validity Nonrepresentative sample Nonrepresentative treatment (that is program or policy) General equilibrium effects (effect of a program can depend on its scale) and peer effects Treatment vs. eligibility effects (what do you want to measure: effect on those who take-up the program or the effect on those eligible)

9 Solutions to drop-out of treatment and contamination bias 1) Can define treatment as intent-to-treat or offer of treatment, in which case dropout is not a problem 2) Alternative solution for dropout and contamination bias Initial random assignment: R=0/1 Decision to participate: D=0/1 Drop out of treatment: R=1 and D=0 Contamination bias: R=0 and D=1 Notation: p0=p(d=1 R=0), p1=p(d=1 R=1) Observe R, D, p0, p1, Y0 if D=0 and Y1 if D=1

10 Notation: p0=p(d=1 R=0) and p1=p(d=1 R=1) E(Y R=0)=E(Y1 R=0)*p0 + E(Y0 R=0)*(1-p0) E(Y R=1)=E(Y1 R=1)*p1 + E(Y0 R=1)*(1-p1) Because of Randomization: E(Y1 R=1)=E(Y1 R=0)=E(Y1) (same for Y0) [Assumption: dropouts are unaffected by being in the treatment group] E(Y R=1) E(Y R=0)= E(Y1)*(p1-p0) E(Y0)*(p1-p0) ATE= E(Y1) E(Y0)= [E(Y R=1)-E(Y R=0)]/(p1-p0) [Wald-Estimator]

11 What to take into account when conducting a randomized experiment? 1) What to randomize on: 1. Randomize eligibility 2. Randomize after acceptance into the program 3. Randomize incentives for take-up for eligibles Ad 2. R=1 if randomized in (treatment group) R=0 if randomized out (control group) D denotes if someone applies to the program and is subject to randomization [here D=1 for all people who are in the randomization] Random assignment implies: For treatment group: E(Y1 X, D=1,R=1) = E(Y1 X, D=1) For control group: E(Y0 X, D=1, R=0) = E(Y0 X, D=1) Experiment gives TTE = E(Y1-Y0 X, D=1)

12 2) Power calculations Def: Power of the design is the probability that, for a given effect size and a given statistical significance level, we will be able to reject the hypothesis of zero effect. Design choices that affect the power of an experiment: Sample size Minimum effect size that the researcher wants to be able to detect Multiple treatment groups Partial compliance Control variables (important to know how much they absorb of the residual variance) Standard software exists for the single-site case Multi-site power analyses get very complicated Need to know the impact variation across sites and the interclass correlations

13 3) Choosing the sites in multi-site experiments External validity: choose sites at random Realistic impacts: choose sites that are not the first to implement the treatment Efficacy: choose sites that will do the best job of implementing the treatment Avoid contamination: choose sites with little or no contact of any sort Examples: Teach for America, PROGRESA/OPORTUNIDADES in Mexico

14 Randomized versus natural (or quasi-) experiments Definitions: A (randomized) experiment is designed and implemented consciously by human researchers. It entails conscious use of a treatment and control group with random assignment (e.g. clinical trials of drugs). A natural or quasi-experiment has a source of randomization that is as if randomly assigned, but this variation was not part of a conscious randomized treatment and control design.

15 Randomized versus natural (or quasi-) experiments Idea for Evaluation of Quasi-Experiments follows that of real randomized experiments: find exogenous source of variation (i.e. variable that affects participation but not directly the outcome)! BUT this is not the case by calling experiments natural : usually not natural but for example changes in laws etc (apart from natural natural experiments such as looking at identical twins, see overview article) and not automatically exogenous Important to understand the source of variation that helps to identify the treatment parameter of interest (see Meyer)

16 How to evaluate these programs? NOTE: the term natural or quasi-experiment does not imply what approach is used to evaluate it! BUT many different approaches of Program Evaluation have been used to evaluate such a quasi-experiment (depending on context of quasi-exp), e.g. Diff-in-Diff or IV. Two types of quasi-experiments: 1. Treatment (D) is as if randomly assigned (perhaps conditional on some control variables) 2. A variable (Z) that influences treatment (D) is as if randomly assigned (Instrumental variable approach)

17 How to evaluate these programs? Randomized experiment In an ideal randomized controlled experiment the treatment level D is randomly assigned: Y=a+b*D+U If D is randomly assigned (for example by computer), then U and D are independently distributed, so E(U D)=0 OLS yields an unbiased estimator of the causal effect b When the treatment is binary, the causal effect b is just the difference in mean outcome (Y) in the treatment vs. the control group (SHOW using counterfactual notation!!). This difference in means is sometimes called the differences estimator.

18 How to evaluate these programs? Randomized experiment Y=a+b*D+U OLS yields an unbiased estimator of the causal effect b Usually one adds other controls to the model: Y=a+b*D+c*X+U Why? 1. Check if randomization worked: if D is randomly assigned, then the OLS estimator with and without the controls, X, should be similar if they aren t, this suggests that D was not randomly assigned Note: to check directly for randomization, regress the treatment indicator, D, on the controls, X, and do a F-test. 2. Increases efficiency: more precise estimator of b (smaller standard errors) 3. Adjust for conditional randomization (apply conditional randomization if interested in treatment effects for different groups)

19 TO DO Preconditions: read Wooldridge Chapter 1 and 2 on causation and on conditional expectations and law of iterated expectations [Read about counterfactual framework: Rubin] Read about randomized experiments: Chapter of Handbook of Development Economics of Duflo et al (sect 1, 2, 6, 8) Read about natural (quasi-) experiments: Meyer this will also be the paper to read for following section difference-in-difference estimation (Rosenzweig/Wolpin on natural natural experiments nice overview)

Instrumental Variables Regression. Instrumental Variables (IV) estimation is used when the model has endogenous s.

Instrumental Variables Regression. Instrumental Variables (IV) estimation is used when the model has endogenous s. Instrumental Variables Regression Instrumental Variables (IV) estimation is used when the model has endogenous s. IV can thus be used to address the following important threats to internal validity: Omitted

More information

Wooldridge, Introductory Econometrics, 4th ed. Chapter 15: Instrumental variables and two stage least squares

Wooldridge, Introductory Econometrics, 4th ed. Chapter 15: Instrumental variables and two stage least squares Wooldridge, Introductory Econometrics, 4th ed. Chapter 15: Instrumental variables and two stage least squares Many economic models involve endogeneity: that is, a theoretical relationship does not fit

More information

The Experimental Method

The Experimental Method The Experimental Method # What is an experiment? How is it different from other methods? Purpose: to demonstrate causation, that A ---> B What are the requirements to demonstrate causality? Correlation

More information

Instrumental Variables & 2SLS

Instrumental Variables & 2SLS Instrumental Variables & 2SLS y 1 = β 0 + β 1 y 2 + β 2 z 1 +... β k z k + u y 2 = π 0 + π 1 z k+1 + π 2 z 1 +... π k z k + v Economics 20 - Prof. Schuetze 1 Why Use Instrumental Variables? Instrumental

More information

The work by Campbell and others on validity and threats to validity within

The work by Campbell and others on validity and threats to validity within Chapter 5 Use a mixed-methods approach: The logic of the comparative advantages of methods The work by Campbell and others on validity and threats to validity within experiments and other types of evaluations

More information

IMPACT EVALUATION: INSTRUMENTAL VARIABLE METHOD

IMPACT EVALUATION: INSTRUMENTAL VARIABLE METHOD REPUBLIC OF SOUTH AFRICA GOVERNMENT-WIDE MONITORING & IMPACT EVALUATION SEMINAR IMPACT EVALUATION: INSTRUMENTAL VARIABLE METHOD SHAHID KHANDKER World Bank June 2006 ORGANIZED BY THE WORLD BANK AFRICA IMPACT

More information

Instrumental Variables & 2SLS

Instrumental Variables & 2SLS Instrumental Variables & 2SLS y 1 = β 0 + β 1 y 2 + β 2 z 1 +... β k z k + u y 2 = π 0 + π 1 z k+1 + π 2 z 1 +... π k z k + v Economics 20 - Prof. Schuetze 1 Why Use Instrumental Variables? Instrumental

More information

How to Randomize. Nassreena Sampaco-Baddiri Innovations for Poverty Action Philippines. 3 rd September, 2014

How to Randomize. Nassreena Sampaco-Baddiri Innovations for Poverty Action Philippines. 3 rd September, 2014 How to Randomize Nassreena Sampaco-Baddiri Innovations for Poverty Action Philippines 3 rd September, 2014 Lecture Overview Unit and method of randomization Real-world constraints Revisiting unit and method

More information

C(t) (1 + y) 4. t=1. For the 4 year bond considered above, assume that the price today is 900$. The yield to maturity will then be the y that solves

C(t) (1 + y) 4. t=1. For the 4 year bond considered above, assume that the price today is 900$. The yield to maturity will then be the y that solves Economics 7344, Spring 2013 Bent E. Sørensen INTEREST RATE THEORY We will cover fixed income securities. The major categories of long-term fixed income securities are federal government bonds, corporate

More information

IAPRI Quantitative Analysis Capacity Building Series. Multiple regression analysis & interpreting results

IAPRI Quantitative Analysis Capacity Building Series. Multiple regression analysis & interpreting results IAPRI Quantitative Analysis Capacity Building Series Multiple regression analysis & interpreting results How important is R-squared? R-squared Published in Agricultural Economics 0.45 Best article of the

More information

Missing data in randomized controlled trials (RCTs) can

Missing data in randomized controlled trials (RCTs) can EVALUATION TECHNICAL ASSISTANCE BRIEF for OAH & ACYF Teenage Pregnancy Prevention Grantees May 2013 Brief 3 Coping with Missing Data in Randomized Controlled Trials Missing data in randomized controlled

More information

The Impact of the Medicare Rural Hospital Flexibility Program on Patient Choice

The Impact of the Medicare Rural Hospital Flexibility Program on Patient Choice The Impact of the Medicare Rural Hospital Flexibility Program on Patient Choice Gautam Gowrisankaran Claudio Lucarelli Philipp Schmidt-Dengler Robert Town January 24, 2011 Abstract This paper seeks to

More information

Methods of Education Research

Methods of Education Research Methods of Education Research Most Common Methods, Controversies, and Ethical Considerations related to Methods and Research Instruments Olatokunbo S. Fashola, Ph.D. Adjunct Research Scientist, Faculty

More information

Planning sample size for randomized evaluations

Planning sample size for randomized evaluations TRANSLATING RESEARCH INTO ACTION Planning sample size for randomized evaluations Simone Schaner Dartmouth College povertyactionlab.org 1 Course Overview Why evaluate? What is evaluation? Outcomes, indicators

More information

Section 14 Simple Linear Regression: Introduction to Least Squares Regression

Section 14 Simple Linear Regression: Introduction to Least Squares Regression Slide 1 Section 14 Simple Linear Regression: Introduction to Least Squares Regression There are several different measures of statistical association used for understanding the quantitative relationship

More information

Empirical Methods in Applied Economics

Empirical Methods in Applied Economics Empirical Methods in Applied Economics Jörn-Ste en Pischke LSE October 2005 1 Observational Studies and Regression 1.1 Conditional Randomization Again When we discussed experiments, we discussed already

More information

Econometric Analysis of Cross Section and Panel Data Second Edition. Jeffrey M. Wooldridge. The MIT Press Cambridge, Massachusetts London, England

Econometric Analysis of Cross Section and Panel Data Second Edition. Jeffrey M. Wooldridge. The MIT Press Cambridge, Massachusetts London, England Econometric Analysis of Cross Section and Panel Data Second Edition Jeffrey M. Wooldridge The MIT Press Cambridge, Massachusetts London, England Preface Acknowledgments xxi xxix I INTRODUCTION AND BACKGROUND

More information

2. Linear regression with multiple regressors

2. Linear regression with multiple regressors 2. Linear regression with multiple regressors Aim of this section: Introduction of the multiple regression model OLS estimation in multiple regression Measures-of-fit in multiple regression Assumptions

More information

Multiple Imputation for Missing Data: A Cautionary Tale

Multiple Imputation for Missing Data: A Cautionary Tale Multiple Imputation for Missing Data: A Cautionary Tale Paul D. Allison University of Pennsylvania Address correspondence to Paul D. Allison, Sociology Department, University of Pennsylvania, 3718 Locust

More information

Case 4: Deworming in Kenya Managing Addressing threats to to experimental integrity

Case 4: Deworming in Kenya Managing Addressing threats to to experimental integrity Case 4: Deworming in Kenya Managing Addressing threats to to experimental integrity This case study is based on Edward Miguel and Michael Kremer, Worms: Identifying Impacts on Education and Health in the

More information

DISCUSSION PAPER SERIES. No. 6059 USING RANDOMIZATION IN DEVELOPMENT ECONOMICS RESEARCH: A TOOLKIT

DISCUSSION PAPER SERIES. No. 6059 USING RANDOMIZATION IN DEVELOPMENT ECONOMICS RESEARCH: A TOOLKIT DISCUSSION PAPER SERIES No. 6059 USING RANDOMIZATION IN DEVELOPMENT ECONOMICS RESEARCH: A TOOLKIT Esther Duflo, Rachel Glennerster and Michael Kremer DEVELOPMENT ECONOMICS ABCD www.cepr.org Available online

More information

The Effect of Housing on Portfolio Choice. July 2009

The Effect of Housing on Portfolio Choice. July 2009 The Effect of Housing on Portfolio Choice Raj Chetty Harvard Univ. Adam Szeidl UC-Berkeley July 2009 Introduction How does homeownership affect financial portfolios? Linkages between housing and financial

More information

Solución del Examen Tipo: 1

Solución del Examen Tipo: 1 Solución del Examen Tipo: 1 Universidad Carlos III de Madrid ECONOMETRICS Academic year 2009/10 FINAL EXAM May 17, 2010 DURATION: 2 HOURS 1. Assume that model (III) verifies the assumptions of the classical

More information

Randomized Controlled Experiments in Education

Randomized Controlled Experiments in Education European Expert Network on Economics of Education (EENEE) Randomized Controlled Experiments in Education EENEE Analytical Report No. 11 Prepared for the European Commission Adrien Bouguen and Marc Gurgand

More information

DEPARTMENT OF ECONOMICS. Unit ECON 12122 Introduction to Econometrics. Notes 4 2. R and F tests

DEPARTMENT OF ECONOMICS. Unit ECON 12122 Introduction to Econometrics. Notes 4 2. R and F tests DEPARTMENT OF ECONOMICS Unit ECON 11 Introduction to Econometrics Notes 4 R and F tests These notes provide a summary of the lectures. They are not a complete account of the unit material. You should also

More information

Impact Assessment of Loans to Female-run Start-ups

Impact Assessment of Loans to Female-run Start-ups Impact Assessment of Loans to Female-run Start-ups ` Farah Said 1 & Mahreen Mahmud 2 September 4, 2014 1. Centre for Research in Economics and Business, Lahore School of Economics, Pakistan 2. University

More information

IMPACT EVALUATION OF BOLSA FAMILIA. September 27th, 2005.

IMPACT EVALUATION OF BOLSA FAMILIA. September 27th, 2005. IMPACT EVALUATION OF BOLSA FAMILIA September 27th, 2005. INTRODUCTION Outline of this Presentation A Brief Description of Bolsa Familia Evaluation Goals Causal Models Caveats on Experiments and quasi-experiments

More information

EBM Cheat Sheet- Measurements Card

EBM Cheat Sheet- Measurements Card EBM Cheat Sheet- Measurements Card Basic terms: Prevalence = Number of existing cases of disease at a point in time / Total population. Notes: Numerator includes old and new cases Prevalence is cross-sectional

More information

UNDERSTANDING ANALYSIS OF COVARIANCE (ANCOVA)

UNDERSTANDING ANALYSIS OF COVARIANCE (ANCOVA) UNDERSTANDING ANALYSIS OF COVARIANCE () In general, research is conducted for the purpose of explaining the effects of the independent variable on the dependent variable, and the purpose of research design

More information

Testing Scientific Explanations (In words slides page 7)

Testing Scientific Explanations (In words slides page 7) Testing Scientific Explanations (In words slides page 7) Most people are curious about the causes for certain things. For example, people wonder whether exercise improves memory and, if so, why? Or, we

More information

Energy Efficiency Program Evaluation

Energy Efficiency Program Evaluation Energy Efficiency Program Evaluation Maximilian Auffhammer George Pardee Jr. Professor University of California at Berkeley National Bureau of Economic Research CESifo Fellow NARUC Webinar May 6th 2015

More information

Making the Leap from Self- Employed to Employer? What matters capital, labor, or training?

Making the Leap from Self- Employed to Employer? What matters capital, labor, or training? Making the Leap from Self- Employed to Employer? What matters capital, labor, or training? David McKenzie, World Bank (with Suresh de Mel and Chris Woodruff) In developing countries there are lots of selfemployed,

More information

Difference in differences and Regression Discontinuity Design

Difference in differences and Regression Discontinuity Design Difference in differences and Regression Discontinuity Design Majeure Economie September 2011 1 Difference in differences Intuition Identification of a causal effect Discussion of the assumption Examples

More information

Rainfall Insurance and Informal Groups

Rainfall Insurance and Informal Groups Rainfall Insurance and Informal Groups Evidence from a Field Experiment in Ethiopia Stefan Dercon 1, Ruth Vargas Hill 2, Ingo Outes-Leon 1, Alebel Bayrou 3, Daniel Clarke 1, Alemayehu Seyoum Taffesse 2

More information

Introduction to Hypothesis Testing. Hypothesis Testing. Step 1: State the Hypotheses

Introduction to Hypothesis Testing. Hypothesis Testing. Step 1: State the Hypotheses Introduction to Hypothesis Testing 1 Hypothesis Testing A hypothesis test is a statistical procedure that uses sample data to evaluate a hypothesis about a population Hypothesis is stated in terms of the

More information

17. SIMPLE LINEAR REGRESSION II

17. SIMPLE LINEAR REGRESSION II 17. SIMPLE LINEAR REGRESSION II The Model In linear regression analysis, we assume that the relationship between X and Y is linear. This does not mean, however, that Y can be perfectly predicted from X.

More information

Missing Data. A Typology Of Missing Data. Missing At Random Or Not Missing At Random

Missing Data. A Typology Of Missing Data. Missing At Random Or Not Missing At Random [Leeuw, Edith D. de, and Joop Hox. (2008). Missing Data. Encyclopedia of Survey Research Methods. Retrieved from http://sage-ereference.com/survey/article_n298.html] Missing Data An important indicator

More information

FIXED EFFECTS AND RELATED ESTIMATORS FOR CORRELATED RANDOM COEFFICIENT AND TREATMENT EFFECT PANEL DATA MODELS

FIXED EFFECTS AND RELATED ESTIMATORS FOR CORRELATED RANDOM COEFFICIENT AND TREATMENT EFFECT PANEL DATA MODELS FIXED EFFECTS AND RELATED ESTIMATORS FOR CORRELATED RANDOM COEFFICIENT AND TREATMENT EFFECT PANEL DATA MODELS Jeffrey M. Wooldridge Department of Economics Michigan State University East Lansing, MI 48824-1038

More information

Wiswall, Labor Economics (Undergraduate), Fall QUIZ. Instructions: Write all answers on the separate answer sheet.

Wiswall, Labor Economics (Undergraduate), Fall QUIZ. Instructions: Write all answers on the separate answer sheet. Wiswall, Labor Economics (Undergraduate), Fall 2005 1 QUIZ Instructions: Write all answers on the separate answer sheet. Make sure you write your name on this answer sheet. (68 points total) Multiple Choice

More information

Issues in Survey Research Design

Issues in Survey Research Design This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike License. Your use of this material constitutes acceptance of that license and the conditions of use of materials on this

More information

Financial Risk Management Exam Sample Questions/Answers

Financial Risk Management Exam Sample Questions/Answers Financial Risk Management Exam Sample Questions/Answers Prepared by Daniel HERLEMONT 1 2 3 4 5 6 Chapter 3 Fundamentals of Statistics FRM-99, Question 4 Random walk assumes that returns from one time period

More information

Lecture 15. Endogeneity & Instrumental Variable Estimation

Lecture 15. Endogeneity & Instrumental Variable Estimation Lecture 15. Endogeneity & Instrumental Variable Estimation Saw that measurement error (on right hand side) means that OLS will be biased (biased toward zero) Potential solution to endogeneity instrumental

More information

Notes 8: Hypothesis Testing

Notes 8: Hypothesis Testing Notes 8: Hypothesis Testing Julio Garín Department of Economics Statistics for Economics Spring 2012 (Stats for Econ) Hypothesis Testing Spring 2012 1 / 44 Introduction Why we conduct surveys? We want

More information

What s New in Econometrics? Lecture 8 Cluster and Stratified Sampling

What s New in Econometrics? Lecture 8 Cluster and Stratified Sampling What s New in Econometrics? Lecture 8 Cluster and Stratified Sampling Jeff Wooldridge NBER Summer Institute, 2007 1. The Linear Model with Cluster Effects 2. Estimation with a Small Number of Groups and

More information

Leaving Nothing to Chance: Randomizing to Measure Impact Citi-FT Financial Education Summit 5-6 November 2014 Kuala Lumpur

Leaving Nothing to Chance: Randomizing to Measure Impact Citi-FT Financial Education Summit 5-6 November 2014 Kuala Lumpur Leaving Nothing to Chance: Randomizing to Measure Impact Citi-FT Financial Education Summit 5-6 November 2014 Kuala Lumpur Pooja Wagh Initiative Coordinator, Global Financial Inclusion Initiative Innovations

More information

I. Cost-effectiveness (Reported numerically)

I. Cost-effectiveness (Reported numerically) Standard for Impact Auditing Version 0.2.14 Release date TBD This is a preliminary draft. Please do not cite or publish. An impact audit is a short-term engagement to assess a nonprofit across four dimensions:

More information

Pre-Analysis Plan: Title of the study

Pre-Analysis Plan: Title of the study Pre-Analysis Plan: Title of the study Author s name Date of latest draft Contents 1 Introduction.................................. 3 1.1 Abstract.................................. 3 1.2 Motivation.................................

More information

Randomized Evaluations of Interventions in Social Service Delivery

Randomized Evaluations of Interventions in Social Service Delivery Randomized Evaluations of Interventions in Social Service Delivery By Esther Duflo, Rachel Glennerster, and Michael Kremer What is the most effective way to increase girls participation in school? How

More information

Chapter 6 Experiment Process

Chapter 6 Experiment Process Chapter 6 Process ation is not simple; we have to prepare, conduct and analyze experiments properly. One of the main advantages of an experiment is the control of, for example, subjects, objects and instrumentation.

More information

Encouraging Education in an Urban School District: Evidence from the Philadelphia Educational Longitudinal Study *

Encouraging Education in an Urban School District: Evidence from the Philadelphia Educational Longitudinal Study * Encouraging Education in an Urban School District: Evidence from the Philadelphia Educational Longitudinal Study * Frank F. Furstenberg, Jr. University of Pennsylvania David Neumark University of California,

More information

Chapter 10: Basic Linear Unobserved Effects Panel Data. Models:

Chapter 10: Basic Linear Unobserved Effects Panel Data. Models: Chapter 10: Basic Linear Unobserved Effects Panel Data Models: Microeconomic Econometrics I Spring 2010 10.1 Motivation: The Omitted Variables Problem We are interested in the partial effects of the observable

More information

Andhra Pradesh School Choice Project Proposal

Andhra Pradesh School Choice Project Proposal Andhra Pradesh School Choice Project Proposal 1. Background: In recent years, access to primary education has expanded tremendously in India and gender gaps have narrowed. Approximately 95% of both boys

More information

Structural Econometric Modeling in Industrial Organization Handout 1

Structural Econometric Modeling in Industrial Organization Handout 1 Structural Econometric Modeling in Industrial Organization Handout 1 Professor Matthijs Wildenbeest 16 May 2011 1 Reading Peter C. Reiss and Frank A. Wolak A. Structural Econometric Modeling: Rationales

More information

Lecture 3: Differences-in-Differences

Lecture 3: Differences-in-Differences Lecture 3: Differences-in-Differences Fabian Waldinger Waldinger () 1 / 55 Topics Covered in Lecture 1 Review of fixed effects regression models. 2 Differences-in-Differences Basics: Card & Krueger (1994).

More information

ECON 142 SKETCH OF SOLUTIONS FOR APPLIED EXERCISE #2

ECON 142 SKETCH OF SOLUTIONS FOR APPLIED EXERCISE #2 University of California, Berkeley Prof. Ken Chay Department of Economics Fall Semester, 005 ECON 14 SKETCH OF SOLUTIONS FOR APPLIED EXERCISE # Question 1: a. Below are the scatter plots of hourly wages

More information

Control Trials. variable Y. Paper Type I - Descriptive. Use estimates from the first two types of. For ex. Some studies find that insurance

Control Trials. variable Y. Paper Type I - Descriptive. Use estimates from the first two types of. For ex. Some studies find that insurance Empirical Research Causality and Randomized Control Trials Three broad types of Empirical papers Paper Type I Descriptive CVD mortality over time Regional differences in medical care How are health insurance

More information

Introduction to Regression and Data Analysis

Introduction to Regression and Data Analysis Statlab Workshop Introduction to Regression and Data Analysis with Dan Campbell and Sherlock Campbell October 28, 2008 I. The basics A. Types of variables Your variables may take several forms, and it

More information

Correlational Research

Correlational Research Correlational Research Chapter Fifteen Correlational Research Chapter Fifteen Bring folder of readings The Nature of Correlational Research Correlational Research is also known as Associational Research.

More information

The Impact of Eyeglasses on the Academic Performance of Primary School Students: Evidence from a Randomized Trial in Rural China

The Impact of Eyeglasses on the Academic Performance of Primary School Students: Evidence from a Randomized Trial in Rural China The Impact of Eyeglasses on the Academic Performance of Primary School Students: Evidence from a Randomized Trial in Rural China Paul Glewwe University of Minnesota Albert Park HKUST Meng Zhao Waseda University

More information

research/scientific includes the following: statistical hypotheses: you have a null and alternative you accept one and reject the other

research/scientific includes the following: statistical hypotheses: you have a null and alternative you accept one and reject the other 1 Hypothesis Testing Richard S. Balkin, Ph.D., LPC-S, NCC 2 Overview When we have questions about the effect of a treatment or intervention or wish to compare groups, we use hypothesis testing Parametric

More information

Study Design and Statistical Analysis

Study Design and Statistical Analysis Study Design and Statistical Analysis Anny H Xiang, PhD Department of Preventive Medicine University of Southern California Outline Designing Clinical Research Studies Statistical Data Analysis Designing

More information

Banking with Agents: Evidence from Senegal. Sinja Buri, IFC Robert Cull, World Bank Xavier Giné, World Bank Sven Harten, IFC

Banking with Agents: Evidence from Senegal. Sinja Buri, IFC Robert Cull, World Bank Xavier Giné, World Bank Sven Harten, IFC Banking with Agents: Evidence from Senegal Sinja Buri, IFC Robert Cull, World Bank Xavier Giné, World Bank Sven Harten, IFC 1 Motivation: 77 percent of adults living on less than $2 a day report not having

More information

ABSORBENCY OF PAPER TOWELS

ABSORBENCY OF PAPER TOWELS ABSORBENCY OF PAPER TOWELS 15. Brief Version of the Case Study 15.1 Problem Formulation 15.2 Selection of Factors 15.3 Obtaining Random Samples of Paper Towels 15.4 How will the Absorbency be measured?

More information

Information and Employee Evaluation: Evidence from a Randomized Intervention in Public Schools. Jonah E. Rockoff 1 Columbia Business School

Information and Employee Evaluation: Evidence from a Randomized Intervention in Public Schools. Jonah E. Rockoff 1 Columbia Business School Preliminary Draft, Please do not cite or circulate without authors permission Information and Employee Evaluation: Evidence from a Randomized Intervention in Public Schools Jonah E. Rockoff 1 Columbia

More information

Types of Specification Errors 1. Omitted Variables. 2. Including an irrelevant variable. 3. Incorrect functional form. 2

Types of Specification Errors 1. Omitted Variables. 2. Including an irrelevant variable. 3. Incorrect functional form. 2 Notes on Model Specification To go with Gujarati, Basic Econometrics, Chapter 13 Copyright - Jonathan Nagler; April 4, 1999 Attributes of a Good Econometric Model A.C. Harvey: (in Gujarati, p. 453-4) ffl

More information

HOW EFFECTIVE IS TARGETED ADVERTISING?

HOW EFFECTIVE IS TARGETED ADVERTISING? HOW EFFECTIVE IS TARGETED ADVERTISING? Ayman Farahat and Michael Bailey Marketplace Architect Yahoo! July 28, 2011 Thanks Randall Lewis, Yahoo! Research Agenda An Introduction to Measuring Effectiveness

More information

Age to Age Factor Selection under Changing Development Chris G. Gross, ACAS, MAAA

Age to Age Factor Selection under Changing Development Chris G. Gross, ACAS, MAAA Age to Age Factor Selection under Changing Development Chris G. Gross, ACAS, MAAA Introduction A common question faced by many actuaries when selecting loss development factors is whether to base the selected

More information

The Impact of Eyeglasses on the Academic Performance of Primary School Students: Evidence from a Randomized Trial in Rural China

The Impact of Eyeglasses on the Academic Performance of Primary School Students: Evidence from a Randomized Trial in Rural China The Impact of Eyeglasses on the Academic Performance of Primary School Students: Evidence from a Randomized Trial in Rural China Paul Glewwe University of Minnesota Albert Park Oxford University Meng Zhao

More information

Chapter 17: Residential Behavior Protocol

Chapter 17: Residential Behavior Protocol Chapter 17: Residential Behavior Protocol The Uniform Methods Project: Methods for Determining Energy Efficiency Savings for Specific Measures Created as part of subcontract with period of performance

More information

Performance Management as Evaluation. Prof. Jeffrey Smith Department of Economics University of Michigan

Performance Management as Evaluation. Prof. Jeffrey Smith Department of Economics University of Michigan Performance Management as Evaluation Prof. Jeffrey Smith Department of Economics University of Michigan Workshop on Market Mechanisms in Active Labor Market Policy October 24, 2008 IAB, Nürnberg Introduction

More information

Econometrics Simple Linear Regression

Econometrics Simple Linear Regression Econometrics Simple Linear Regression Burcu Eke UC3M Linear equations with one variable Recall what a linear equation is: y = b 0 + b 1 x is a linear equation with one variable, or equivalently, a straight

More information

Impact of Business Training on the Success of Start-ups by Women Entrepreneurs in Lebanon

Impact of Business Training on the Success of Start-ups by Women Entrepreneurs in Lebanon Impact of Business Training on the Success of Start-ups by Women Entrepreneurs in Lebanon Khusrav Gaibulloev-UAE Zein Zabaneh- Jordan Benjamin kachero-uganda Mohammed Ben Hafoune- Morocco Rami Allow-Lebanon

More information

Missing Data: Part 1 What to Do? Carol B. Thompson Johns Hopkins Biostatistics Center SON Brown Bag 3/20/13

Missing Data: Part 1 What to Do? Carol B. Thompson Johns Hopkins Biostatistics Center SON Brown Bag 3/20/13 Missing Data: Part 1 What to Do? Carol B. Thompson Johns Hopkins Biostatistics Center SON Brown Bag 3/20/13 Overview Missingness and impact on statistical analysis Missing data assumptions/mechanisms Conventional

More information

World Bank Employment Policy Primer December 2009 No. 12

World Bank Employment Policy Primer December 2009 No. 12 World Bank Employment Policy Primer December 2009 No. 12 A PRACTICIONER S GUIDE TO EVALUATING THE IMPACTS OF LABOR MARKET PROGRAMS * 1. Introduction This note provides an introduction to the impact evaluation

More information

Introduction to Fixed Effects Methods

Introduction to Fixed Effects Methods Introduction to Fixed Effects Methods 1 1.1 The Promise of Fixed Effects for Nonexperimental Research... 1 1.2 The Paired-Comparisons t-test as a Fixed Effects Method... 2 1.3 Costs and Benefits of Fixed

More information

Research Methods & Experimental Design

Research Methods & Experimental Design Research Methods & Experimental Design 16.422 Human Supervisory Control April 2004 Research Methods Qualitative vs. quantitative Understanding the relationship between objectives (research question) and

More information

An Empirical Analysis of Insider Rates vs. Outsider Rates in Bank Lending

An Empirical Analysis of Insider Rates vs. Outsider Rates in Bank Lending An Empirical Analysis of Insider Rates vs. Outsider Rates in Bank Lending Lamont Black* Indiana University Federal Reserve Board of Governors November 2006 ABSTRACT: This paper analyzes empirically the

More information

APPLICATION OF LINEAR REGRESSION MODEL FOR POISSON DISTRIBUTION IN FORECASTING

APPLICATION OF LINEAR REGRESSION MODEL FOR POISSON DISTRIBUTION IN FORECASTING APPLICATION OF LINEAR REGRESSION MODEL FOR POISSON DISTRIBUTION IN FORECASTING Sulaimon Mutiu O. Department of Statistics & Mathematics Moshood Abiola Polytechnic, Abeokuta, Ogun State, Nigeria. Abstract

More information

Premaster Statistics Tutorial 4 Full solutions

Premaster Statistics Tutorial 4 Full solutions Premaster Statistics Tutorial 4 Full solutions Regression analysis Q1 (based on Doane & Seward, 4/E, 12.7) a. Interpret the slope of the fitted regression = 125,000 + 150. b. What is the prediction for

More information

Evaluating the Impact of Conditional Cash Transfer Programs

Evaluating the Impact of Conditional Cash Transfer Programs Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Evaluating the Impact of Conditional Cash Transfer Programs Laura B. Rawlings Gloria

More information

Eradicating Diseases: The Effect of Conditional Cash Transfers on Vaccination Coverage in Rural Nicaragua

Eradicating Diseases: The Effect of Conditional Cash Transfers on Vaccination Coverage in Rural Nicaragua WORKING PAPER Eradicating Diseases: The Effect of Conditional Cash Transfers on Vaccination Coverage in Rural Nicaragua Tania Barham John A. Maluccio April 2008 Health and Society Program HS2008-01 Eradicating

More information

ECON 523 Applied Econometrics I /Masters Level American University, Spring 2008. Description of the course

ECON 523 Applied Econometrics I /Masters Level American University, Spring 2008. Description of the course ECON 523 Applied Econometrics I /Masters Level American University, Spring 2008 Instructor: Maria Heracleous Lectures: M 8:10-10:40 p.m. WARD 202 Office: 221 Roper Phone: 202-885-3758 Office Hours: M W

More information

IPDET Module 6: Descriptive, Normative, and Impact Evaluation Designs

IPDET Module 6: Descriptive, Normative, and Impact Evaluation Designs IPDET Module 6: Descriptive, Normative, and Impact Evaluation Designs Intervention or Policy Evaluation Questions Design Questions Elements Types Key Points Introduction What Is Evaluation Design? Connecting

More information

Repeated Measures Design. Mark Conaway October 11, 1999

Repeated Measures Design. Mark Conaway October 11, 1999 Repeated Measures Design Mark Conaway October 11, 1999 Common design: Repeated measures designs Take measurements on same subject over time or under different conditions. Same basic idea as a randomized

More information

ANNUITY LAPSE RATE MODELING: TOBIT OR NOT TOBIT? 1. INTRODUCTION

ANNUITY LAPSE RATE MODELING: TOBIT OR NOT TOBIT? 1. INTRODUCTION ANNUITY LAPSE RATE MODELING: TOBIT OR NOT TOBIT? SAMUEL H. COX AND YIJIA LIN ABSTRACT. We devise an approach, using tobit models for modeling annuity lapse rates. The approach is based on data provided

More information

Early Bird Catches the Worm: The Causal Impact of Pre-school Participation and Teacher Qualifications on Year 3 NAPLAN Outcomes

Early Bird Catches the Worm: The Causal Impact of Pre-school Participation and Teacher Qualifications on Year 3 NAPLAN Outcomes Early Bird Catches the Worm: The Causal Impact of Pre-school Participation and Teacher Qualifications on Year 3 NAPLAN Outcomes This research looks at the causal impact of attendance at pre-school i in

More information

MICROECONOMETRICS. 27 th of June 1 st of July SUMMER SCHOOL.

MICROECONOMETRICS. 27 th of June 1 st of July SUMMER SCHOOL. 2016 27 th of June 1 st of July SUMMER SCHOOL OVERVIEW The Policy Evaluation programme presents recent developments in the microeconomic analysis of impact evaluation, with two courses taught by experts

More information

Inclusion and Exclusion Criteria

Inclusion and Exclusion Criteria Inclusion and Exclusion Criteria Inclusion criteria = attributes of subjects that are essential for their selection to participate. Inclusion criteria function remove the influence of specific confounding

More information

Impact Evaluation Methods and Applications to Access to Energy

Impact Evaluation Methods and Applications to Access to Energy Impact Evaluation Methods and Applications to Access to Energy Università Cattolica del Sacro Cuore and Laboratorio Expo (Fondazione Feltrinelli) ExpoSchool, 4-2-2015 acopo Bonan (Università Cattolica

More information

Chapter 9 Assessing Studies Based on Multiple Regression

Chapter 9 Assessing Studies Based on Multiple Regression Chapter 9 Assessing Studies Based on Multiple Regression Solutions to Empirical Exercises 1. Age 0.439** (0.030) Age 2 Data from 2004 (1) (2) (3) (4) (5) (6) (7) (8) Dependent Variable AHE ln(ahe) ln(ahe)

More information

1 The Problem: Endogeneity There are two kinds of variables in our models: exogenous variables and endogenous variables. Endogenous Variables: These a

1 The Problem: Endogeneity There are two kinds of variables in our models: exogenous variables and endogenous variables. Endogenous Variables: These a Notes on Simultaneous Equations and Two Stage Least Squares Estimates Copyright - Jonathan Nagler; April 19, 1999 1. Basic Description of 2SLS ffl The endogeneity problem, and the bias of OLS. ffl The

More information

Machine Learning Methods for Causal Effects. Susan Athey, Stanford University Guido Imbens, Stanford University

Machine Learning Methods for Causal Effects. Susan Athey, Stanford University Guido Imbens, Stanford University Machine Learning Methods for Causal Effects Susan Athey, Stanford University Guido Imbens, Stanford University Introduction Supervised Machine Learning v. Econometrics/Statistics Lit. on Causality Supervised

More information

Demand versus Returns?

Demand versus Returns? Public Disclosure Authorized Policy Research Working Paper 6389 WPS6389 Public Disclosure Authorized Public Disclosure Authorized Impact Evaluation Series No. 88 Demand versus Returns? Pro-Poor Targeting

More information

Clinical Study Design and Methods Terminology

Clinical Study Design and Methods Terminology Home College of Veterinary Medicine Washington State University WSU Faculty &Staff Page Page 1 of 5 John Gay, DVM PhD DACVPM AAHP FDIU VCS Clinical Epidemiology & Evidence-Based Medicine Glossary: Clinical

More information

Key Concept. Properties

Key Concept. Properties MAT 155 Statistical Analysis Dr. Claude Moore Cape Fear Community College Chapter 6 Normal Probability Distributions 6 1 Review and Preview 6 2 The Standard Normal Distribution 6 3 Applications of Normal

More information

Supplementary webappendix

Supplementary webappendix Supplementary webappendix This webappendix formed part of the original submission and has been peer reviewed. We post it as supplied by the authors. Supplement to: Basinga P, Gertler PJ, Binagwaho A, Soucat

More information

Response to Critiques of Mortgage Discrimination and FHA Loan Performance

Response to Critiques of Mortgage Discrimination and FHA Loan Performance A Response to Comments Response to Critiques of Mortgage Discrimination and FHA Loan Performance James A. Berkovec Glenn B. Canner Stuart A. Gabriel Timothy H. Hannan Abstract This response discusses the

More information

Chapter Eight: Quantitative Methods

Chapter Eight: Quantitative Methods Chapter Eight: Quantitative Methods RESEARCH DESIGN Qualitative, Quantitative, and Mixed Methods Approaches Third Edition John W. Creswell Chapter Outline Defining Surveys and Experiments Components of

More information

Advanced Forecasting Techniques and Models: ARIMA

Advanced Forecasting Techniques and Models: ARIMA Advanced Forecasting Techniques and Models: ARIMA Short Examples Series using Risk Simulator For more information please visit: www.realoptionsvaluation.com or contact us at: admin@realoptionsvaluation.com

More information

AN INTRODUCTION TO MATCHING METHODS FOR CAUSAL INFERENCE

AN INTRODUCTION TO MATCHING METHODS FOR CAUSAL INFERENCE AN INTRODUCTION TO MATCHING METHODS FOR CAUSAL INFERENCE AND THEIR IMPLEMENTATION IN STATA Barbara Sianesi IFS Stata Users Group Meeting Berlin, June 25, 2010 1 (PS)MATCHING IS EXTREMELY POPULAR 240,000

More information