Difference in differences and Regression Discontinuity Design

Difference in differences and Regression Discontinuity Design Majeure Economie September 2011

1 Difference in differences Intuition Identification of a causal effect Discussion of the assumption Examples 2 Regression Discontinuity Design Two policy questions Intuition Formal Analysis Discussion of the assumptions Applications 3 Conclusion

Two new impact evaluation techniques Gold standard in impact evaluation = randomized experiments. When it is not possible to run a randomized experiment, three alternative possibilities: instrumental variable Difference in differences Regression discontinuity Contrary to randomized experiments, these methods are not assumption free. Rely on assumptions which cannot be tested => should be carefully discussed.

Intuition Before-After analysis Assume you want to measure the impact of the minimum wage on employment. Not possible to run a randomized experiment. Tough to find out an instrumental variable. You have a data set of 200 american fast-food (FF) located in New Jersey (NJ), with the number of people employed during year N and year N+1. FF highly relevant for this type analysis: they indeed pay the minimum wage to their employees. Minimum wage in NJ changed between N and N+1. You could say that impact of the change of the minimum wage = average number of employees in N+1 - average number of employees in N. Issue = not only the minimum wage changed from the before to the after period (business cycle...)

Intuition Cross-sectional analysis Assume that on top of this data set, you have another with the same data but for FF located in Pennsylvania (Penn) where there was no change in the minimum wage from N to N+1 and where the minimum wage in N and N+1 was similar to that of NJ in N. => you could say that impact of the change of the minimum wage = average number of employees in NJ firms in N+1 - average number of employees in Penn firms in N+1. Issue = NJ and Penn differ on more aspects than just minimum wage. => your estimate might capture more than just the minimum wage effect (maybe fast-food are just bigger in one state than in the other : larger population, larger cities...) => any idea to circumvent these two issues?

Intuition Difference in differences impact of the change of the minimum wage = average number of employees in NJ FF in N+1 - average number of employees in NJ FF in N-(average number of employees in Penn FF in N+1 - average number of employees in Penn FF in N). Difference in differences or double differences. Intuition: you observe a change in FF size from N to N+1 in NJ. Some of it is due to the change in the minimum wage, some of it would have happened anyway. To measure the change that would have happened anyway, you do the same computation but in a State where there has been no change in the minimum wage. => effect of the minimum wage = total change - change that would have happened anyway.

Identification of a causal effect Framework and notations (1/2) Two groups (NJ and Penn, Test and Control group) represented by a dummy G. Two periods of time (before and after) represented by a dummy T. One treatment, represented by a dummy D. D = T G. Only observations from the treatment group in period 1 are treated. Potential outcomes: Y (1): what happens to someone if he receives the treatment. Y (0): what happens to someone if he does not receive receives the treatment. Y = observed outcome

Identification of a causal effect Framework and notations (2/2) For the test group in period 1, is Y equal to Y (1) or to Y (0)? For the control group in period 0? What we want to identify is E(Y (1) Y (0) T = 1, G = 1) = E(Y (1) T = 1, G = 1) E(Y (0) T = 1, G = 1). Which of these two expectations can we easily estimate from the sample? Which one is missing? FF example: average number of employees in NJ FF after the minimum wage increase minus the the average number of employees in NJ FF after the minimum wage increase if actually the minimum wage had not increased.

Identification of a causal effect Summary T = 0 T = 1 G = 0 0% treated 0% treated G = 1 0% treated 100% treated In the minimum wage example, are we indeed facing this situation?

Identification of a causal effect Common trend assumption Assumption: E(Y (0) T = 1, G = 1) E(Y (0) T = 0, G = 1) = E(Y (0) T = 1, G = 0) E(Y (0) T = 0, G = 0). Interpretation of the assumption in the minimum wage example: without the rise of the minimum wage, average number of employees in NJ FF would have followed the same evolution from period 0 to 1 than the evolution observed in Pennsylvania. Under this assumption: E(Y (0) T = 1, G = 1) = E(Y (0) T = 0, G = 1) + E(Y (0) T = 1, G = 0) E(Y (0) T = 0, G = 0) and therefore E(Y (1) Y (0) T = 1, G = 1) = E(Y (1) T = 1, G = 1) E(Y (0) T = 0, G = 1) [E(Y (0) T = 1, G = 0) E(Y (0) T = 0, G = 0)] Double difference or difference in differences (DID).

Discussion of the assumption Interpretation of the common trend assumption Diff in diff is clearly not assumption free: the very strong assumption is the common trend assumption. What it means = without the reform, the trend in fast food employment would have been the same in NJ and Penn. Might not be true: assume for instance that business cycles are not at all the same in NJ and Penn. Maybe the differential trend in FF employment is not due to the reform but just to the fact that the economic conjecture in NJ and Penn did not follow the same trend at all from year N to N+1. The smaller the time periods you look at, the more likely it is that the assumption is verified. If instead of years your units of time are months. Any idea on how one could test the common trend assumption?

Discussion of the assumption Test of the assumption Not testable by a formal test. Graphical test: if you have several years of data, you can test whether the outcome variable followed parallel trends over the period except in the year of the reform. Assume that the minimum wage reform was implemented on 01/01/1993. If you also have data on the same FF in 1991 and 1992, you can compute the same DID but from 1991 to 1992, that is to say over two years when there was no change in the minimum wage. This is what we call a placebo difference in difference. If your common trend assumption is true, do you expect to find that this placebo difference is large or small?

Examples Impact of the minimum wage on employment (1/3) Paper by Krueger and Card, 1994. Increase of the minimum wage in April 1992 in New Jersey, from 4.25$ to 5.05$ per hour. 19% increase. In the meanwhile, minimum wage in the neighboring state of Pennsylvania remained unchanged. Totally changes the wage distribution in restaurants:

Examples Impact of the minimum wage on employment (2/3) Conduct two waves of surveys: one before and one after the increase, and ask to the managers of 200 FF in each state how many employees they have. Increase in minimum wage => increase in employment (+2.76 employees).

Examples Impact of the minimum wage on employment (3/3) Issues with the paper: they do not have several years of data => can not run the placebo tests described above. However, they have employment data on high standards restaurants => compute the diff in diff estimator on them and this is 0 as expected (a rise in the minimum wage should not have any impact on them).

Examples Impact of the 2007 smoking ban in workplaces in France In February 2007, smoking was banned from workplaces in France. Did it give more motivation to working smokers to quit? Data: network of tobacco cessation centers. Comparison of the quitting rate among employed and not employed patients.

Two policy questions Do unemployment benefits increase unemployment duration? Unemployment benefits: Pros: Cons: safety net for those who loose their jobs. Insurance system. no benefits might be counter-productive: forces unemployed to accept whatever type of jobs, even if do not correspond to their qualifications. unemployment benefits = an incentive for unemployed not to look for a job.

Two policy questions Do students learn better in small classrooms? Important question: if the answer = class size has no impact on students performance, you can save a lot of public money setting up classrooms of 50 students => less teachers. Much debated, but in the public debate, analysts tend to compare apples and oranges: first type of analysis = international comparisons. Korean students do better than German ones whereas classroom size is much larger in Korea and in Germany. Makes sense? second type of analysis: univariate regressions in a given country. Among French students, regress their score in a national exam on the number of students in their class. Makes sense? third type of analysis: multivariate regressions. Among French students, regress their score in a national exam on the number of students in their class + control variables (income of their parents...). Makes sense? => need to find an empirical strategy which allows comparing apples and apples.

Intuition Using thresholds to identify the effect of a policy In Israël schools, there cannot be a class with more than 40 students. => if you have a school with 40 children enrolled in grade 1: only one grade 1 class. If 41 students => 2 grade 1 classes (one has 21 students, the other one has 20 students). => to measure the impact of class size on students achievement, maybe you can compare schools with 40 grade 1 students to schools with 41 grade 1 students. Intuition: for sure, big schools and small schools are not similar (urban / rural etc...) => school size is not independant from student s performance. However, passing from 40 to 41 students is something which can be regarded as random: schools with 40 grade 1 students should be similar to those with 41, but very strong difference in class size. => to identify the effect of class size on students performance, compare achievement of students in schools with 40 students to achievement of students in schools with 41 students.

Intuition Sharp or fuzzy design? Sharp design: the rule is deterministic. If you are more than 50 years old then you get larger unemployment benefits. Fuzzy design: the rule is not deterministic. some headmasters will decide to open a new class from 35 students onwards (parent s pressure, size of the classroom). However, the rule has some impact on the actual class size.

Formal Analysis Identification of the treatment effect (1/2) To simplify, we assume that the treatment is binary (receive unemployment benefits over a long period or over a short period according to your age. In Austria, in some areas, if you are unemployed and more than 50, you get unemployment benefits for a long time (2 years) whereas you get them for only 6 months if you are below 50: T = 1 iif A 50. We are in the sharp design situation. Outcome of interest: whether those unemployed find a job in less than months. Each individual has once more two binary outcomes: Y 1 and Y 0. For individuals strictly below 50 which outcome do we observe? For individuals above 50 which outcome do we observe? We observe Y = Y 0 + 1 {A 50} (Y 1 Y 0 ) What we assume is that E(Y 1 A) and E(Y 0 A) are continuous functions of A. We acknowledge the fact that the probability to find a job is a function of age (not the same for old than young unemployed). But we are just assuming that it changes continuously with age. => how to estimate a causal impact of the treatment with these assumptions?

Formal Analysis Identification of the treatment effect (sharp design) (2/2) By comparing the probability of finding a job of individuals slightly below 50 to the same probability but for individuals slightly above 50. lim A 50,A 50 E(Y A) lim A 50,A<50 E(Y A) = lim A 50,A 50 E(Y 1 A) lim A 50,A<50 E(Y 0 A) = E(Y 1 A = 50) E(Y 0 A = 50) = E(Y 1 Y 0 A = 50), thanks to the continuity assumption.

Formal Analysis Estimation of the treatment effect in the sharp design case An interesting quantity (E(Y 1 Y 0 A = 50)) is equal to something we can estimate from the sample: lim A 50,A 50 E(Y A) lim A 50,A<50 E(Y A). to estimate lim A 50,A 50 E(Y A) lim A 50,A<50 E(Y A), run the following regression: Y = α + β 1 (A 50) + β 2 (A 50) 2 +... + β k (A 50) k +β 1 (A 50)1 {A 50} + β 2 (A 50)2 1 {A 50} +... + β k (A 50)k 1 {A 50} + γ1 {A 50} + ε You can check that under this model, γ = lim A 50,A 50 E(Y A) lim A 50,A<50 E(Y A).

Formal Analysis Intuition Estimate Y as a continuous (polynomial) function in the left hand side of the threshold, and in the right hand side, and see whether these two functions connect at the threshold.

Discussion of the assumptions Informal tests No formal test of the continuity assumption. In the class size example, you can plot for instance some students characteristics (X =parents income...) as a function of the size of the school. What you expect = no big jump at the 40 students threshold. Intuition: Y jumps at the threshold, and so does CS. If those things = the two only things which jump, then we can attribute the jump in Y to the jump in CS (people comparable with every respect in the left and in the right of the threshold except with respect to CS). But if other things jump at the threshold, such as parents income, then we do not know whether jump in Y due to jump in CS or to jump in parents income. Example of micro-credit program in Mexico: eligibility rule = having a land smaller than 1 ha (10 000 m 2 ). => some people temporarily sold part of their land to benefit from the program. => now, people in the left and in the right of the threshold no longer comparable. In the left: people whose land = really below 1 ha + dishonest people with land above 1ha. In the right, only honest people whose land is above 1ha. => you are comparing apples and bananas again!

Applications Graphical analysis in the class size example

Applications OLS results

Applications Instrumental variable results

Your evaluation (1/2) Start indicating: How many sessions of the course you attended. Whether your initial background in maths was strong or weak. This will help me interpreting your comments. Please indicate whether you think the following objectives of the course were reached: Give you some basic knowledge about econometric theory (OLS) Make you realize that measuring causality is both important and difficult

Your evaluation (2/2) You are business school students. But Empirical articles we discussed in class were mostly policy oriented. You are business school students => those papers might be of little interest to you. => last question = do you think that the empirical papers we studied in class were interesting? Very interesting Somewhat interesting Not very interesting Not interesting at all

Useful quantitative methods for your professional life Methods to help you to forecast future outcomes based on information available today: regressions. time series model to predict future sales... logistic regression to predict whether a customer will default or not... Method to assess the efficacy of a marketing, sales policy: randomized experiment.

Useful tools to get a clearer vision on some policy issues (1/2) Impact evaluation = very important question, very difficult to answer: what would have happened if people had not benefited from the program? Hundreds of studies try to answer this question on all kinds of policies. Results often contradictory. Quality of the results = very unequal according to the methodology of the study => all of them should not deserve the same degree of attention. Important to draw a hierarchy. One key principle to assess the value of the study in assessing the impact of a policy: in trying to reconstruct the hypothetical scenario of what would have happened to people if they had not received the program, do they compare apples to apples or apples to bananas?

Useful tools to get a clearer vision on some policy issues (2/2) International comparisons: average class size is lower in the US or in France than in Eastern Europe, however results of students in standardized scores lower => small class size = detrimental. But this kind of study compare things that can not be compared. France and Eastern Europe educational system differ in much more aspects than class size (% of a cohort enrolled, role of education in society etc...). Intra-country comparisons with OLS regression. Can be misleading if you forget the right controls: you are not yet comparing comparable things. Can yield better results when many controls added (the things you are comparing are now comparable with respect to some dimensions). Quasi-experimental (IV, RDD, diff-in-diff) methods are already better. Gold standard = randomized experiments.