9. Experiments and quasi-experiments

Transcription

1 9. Experiments and quasi-experiments Up to now: Data sets analyzed were considered as given Now: We may collect data by conducting experiments Our aim is to estimate the causal effects of a treatment (e.g. the effect a policy program, of a job training program) randomized controlled experiment 251

2 Why should we study experiments: Ideal randomized experiments are the benchmark Experiments often are very influential Terminology: Experiment: is designed and implemented by the researcher Quasi-experiment (natural experiment): is based on a randomization that is not the result of a conscious design 252

3 9.1. Potential outcomes, causal effects, and idealized experiments Terminology: The potential outcome is the outcome for an observational unit (individual) under a potential treatment or potential non-treatment For an individual, the causal effect is the difference in potential outcomes, i.e. the difference between the potential outcome with the treatment minus the potential outcome without the treatment 253

4 Remark: Since you cannot observe both potential outcomes on the same individual, the individual causal effects are unobservable Terminology: [continued] The average treatment effect is the population mean value of the individual treatment effects Remark: Distinct individuals may have different treatment effects, but in what follows we assume causal effects that are identical across all individuals 254

5 Regression set-up: Let X i (i = 1,..., n) denote the binary treatment variable defined as { 1 if individual i receives the treatment X i = 0 if individual i does not receive the treatment Assume that individuals are randomly assigned to the treatment, i.e. X i are drawn from simple random sampling (we call X i randomized) Let Y i (i = 1,..., n) be the observed outcome of individual i 255

6 Regression set-up: [continued] We consider the regression model Y i = β 0 + β 1 X i + u i (9.1) Since X i is randomized, it is independent of the error term u i the OLS assumptions hold, and the OLS estimator ˆβ 1 is an unbiased and consistent estimator of the causal effect β 1 It can be shown that the OLS estimator ˆβ 1 can be computed as the difference between the sampling average of the Y i values of the treatment group (X i = 1) and the sampling average of the control (non-treatment) group (X i = 0): (see class) ˆβ 1 = Ȳ treatment Ȳ control (9.2) 256

7 Definition 9.1: (Differences estimator) We call the OLS estimator ˆβ 1 = Ȳ treatment Ȳ control the differences estimator of the causal effect β 1. Differences estimator with additional regressors: We may add a control variable W to Eq. (9.1), so that Y i = β 0 + β 1 X i + β 2 W i + u i (9.3) In general, including W i will result in smaller standard errors of ˆβ 1 257

8 Differences estimator with additional regressors: [continued] Omitted-variable bias is not an issue if X i is randomly assigned But, if the probability of assignment depends on W i, we need to include W i to avoid omitted-variable bias Conditional mean-independence: E(u i X i, W i ) = E(u i W i ) 258

9 Example: Men (W i = 0) and women (W i = 1) are randomly assigned to a course on table manners (X i ) Assume, that women are assigned with a higher probability than men (for whatever reason) Assume further that women have better table manners than men prior to the course Even, if the course has no effect, the treatment group will have better post-course table manners than the control group Reason: the treatment group has a higher fraction of women than the control group 259

10 Example: [continued] If W i is included as control variable, then ˆβ 1 is consistent and unbiased Conditional-mean independence means: Among women, treatment is randomly assigned, so among women, the error term u i is independent of the treatment X i The same applies for men Conclusion: if randomization is based on covariates, then conditionalmean independence holds 260

11 9.2. Threats to validity of experiments Threats to internal validity: Failure to randomize Example: In a job training program individuals are assigned to the treatment group depending on whether their last name falls in the 1st or the 2nd half of the alphabet Because of ethnic differences in last names, ethnicity could differ systematically between the treatment and the control groups Potential ethnic differences in labor market characteristics could imply systematic differences between the treatment and the control group 261

12 Threats to internal validity: [continued] Partial compliance Some members of the control group receive the treatment Some members of the treatment group do not Attrition Example: The most able trainees drop out of the training program due to their acquired job training skills (they get out-oftown jobs) Only the least able members of the treatment group remain 262

13 Threats to internal validity: [continued] Experimental effects In experiments with human subjects, merely because the subjects are in an experiment can change their behavior Hawthorne effect Remarks: A failure to randomize can be detected, if a (binary) regression of X on other variables has explanatory power (e.g. the treatment group has a significantly larger portion of men than the control group) 263

14 Remarks: [continued] Partial compliance is not always a severe problem: if actual treatment (X i ) and the initial (random) assignment (Z i ) are both observable, we can use Z i as an instrument If there is partial compliance, Z i is correlated with X i (relevance) and Z i is exogenous due to random assignment Threats to external validity: Non-representative sample Non-representative treatment General equilibrium effects (scale effects) 264

15 9.3. Experimental estimates of the effect of class size reduction Project STAR: (Student-Teacher-Achievement Ratio) 4-year study When entering the schooling system, a student was randomly assigned to one of three groups: regular class (22 to 25 students), regular class plus aide, small class (13 to 17 students) Regular class students were re-randomized after first year to regular or regular-plus-aide class Grades: Kindergarten, 1st, 2nd, 3rd year Dependent variable Y i : Stanford Achievement Test scores 265

16 Project STAR: [continued] Owing to the two treatment groups (small class, regular class plus aide), the differences regression model specification is where Y i = β 0 + β 1 SmallClass i + β 2 RegAide i + u i SmallClass i = { 1 if student i is in a small class 0 otherwise RegAide i = { 1 if student i is in a regular-plus-aide class 0 otherwise 266

17 Project STAR: [continued] Additional regressors Teacher experience Free lunch eligibility Gender Race Detailed discussion of results: see class 267

18 9.4. Quasi-experiments Definition 9.2: (Quasi-experiment) A quasi-experiment (or natural experiment) has a source of randomization that is as if randomly assigned, but this variation was not the result of an explicit randomized treatment and control design. Two types of quasi-experiments: The treatment X is as if randomly assigned (possibly conditional on control variables) (Type I) 268

19 Two types of quasi-experiments: [continued] A variable Z, which influences receipt of the treatment X, is as if randomly assigned, so that we can use Z as an instrument (Type II) Type I: Notation: Y before i = value of Y for unit i before the experiment Yi after = value of Y for unit i after the experiment Ȳ treatment, before = sample mean of Y of the treatment group before the experiment Ȳ treatment, after = sample mean of Y of the treatment group after the experiment 269

20 Notation: [continued] Ȳ control, before = sample mean of Y of the control group before the experiment Ȳ control, after = sample mean of Y of the control group after the experiment Remarks: If the treatment is as if randomly assigned, conditional on some oberserved variable W, then the treatment effect can in principle be estimated using the differences regression (9.3) on Slide 257 However, some adjustment appears to become necessary due to missing control over randomization differences-in-differences estimator 270

21 Definition 9.3: (Differences-in-differences estimator) The differences-in-differences estimator ˆβ diffs-in-diffs 1 is defined as the average change in Y for those in the treatment group, minus the average change in Y for those in the control group: ˆβ 1 diffs-in-diffs = (Ȳ treatment, after Ȳ treatment, before ) (Ȳ control, after Ȳ control, before ) = Ȳ treatment Ȳ control, where Ȳ treatment is the average change in Y in the treatment group and Ȳ control is the average change in Y in the control group. 271

22 Remarks: If the treatment is randomly assigned, then ˆβ diffs-in-diffs 1 is an unbiased and consistent estimator of the causal effect The differences-in-differences estimator ˆβ diffs-in-diffs 1 is equivalent to the OLS estimator of β 1 in the regression equation Y i = β 0 + β X i + u i, where Y i = Yi after Yi before and X i = 1, if the ith individual is treated (and 0 otherwise) The differences-indifferences estimator allows for systematic differences in pre-treatment characteristics, which may occur in a quasi-experiment, because treatment is not randomly assigned 272

23 The differences-in-differences estimator 273

24 Remarks: If the treatment X is only as if randomly assigned given a control variable W, then conditional mean independence holds E(u i X i, W i ) = E(u i W i ) and the OLS estimator is unbiased and consistent Regression equation Y i = β 0 + β 1 X i + β 2 W i + u i Remember that β 2 has no causal interpretation Clearly, additional regressors W 1i,..., W ri (measuring individual characteristics prior to the experiment) could be included 274

25 Type II: Setting: If a variable Z, which influences treatment X, is as if randomly assigned (conditional on a control variable w), then Z can be used as an instrumental variable for X in an IV regression including the control variable W The instrument Z is exogenous (due to random assignment) The instrument Z is relevant as it is correlated with X 275

26 Example: Does treatment by a specialized stroke unit increase the chance to survive a stroke? Treatment X is non-random (it might depend on patient characteristics and the doctor) But the distance from the next stroke unit is as if randomly assigned and can thus be used as an instrument Remark: A deeper consideration of Type-II quasi-experiments is beyond the scope of this lecture 276

27 Regression discontinuity estimators: If treatment depends on when an observable variable W crosses a threshold value w 0, then you can estimate the treatment effect by comparing individuals just below the threshold (treated) to those just above the threshold (untreated) The treatment effect shows up as the jump in the outcome and the magnitude of this jump estimates the treatment effect In sharp regression discontinuity designs, everyone below the threshold receives treatment In fuzzy regression discontinuity designs, crossing the threshold only affects the probability of treatment 277

28 Regression discontinuity design scatterplot 278

29 Sharp regression discontinuity: Everybody with W < w 0 receives treatment: X i = { 1 if Wi < w 0 0 otherwise Estimate the treatment effect β 1 by OLS of the following regression specification: Y i = β 0 + β 1 X i + β 2 W i + u i If crossing the threshold affects Y i only through the treatment, then E(u i X i, W i ) = E(U i W i ), so that ˆβ 1 is unbiased 279

30 Fuzzy regression discontinuity: Suppose, the probability of treatment (X i = 1) depends on whether Z i = 1 or Z i = 0, where Z i = { 1 if Wi < w 0 0 otherwise Estimate the treatment effect β 1 by OLS of the following regression specification: Y i = β 0 + β 1 X i + β 2 W i + u i If crossing the threshold has no direct effect on Y i (but only affects Y i by influencing the probability of treatment), then Z i is a valid instrument for treatment X i Use IV estimation rather than OLS 280

31 9.5. Potential problems with quasi-experiments Threats to validity: Same threats to internal validity as for ideal experiments (failure to randomize, partial compliance) Note that experimental effects do not occur in quasi-experiments Same threats to external validity as for ideal experiments (non-representative sample or treatment, general equlilibrium effects) 281