Control Trials. variable Y. Paper Type I - Descriptive. Use estimates from the first two types of. For ex. Some studies find that insurance

Transcription

1 Empirical Research Causality and Randomized Control Trials Three broad types of Empirical papers Paper Type I Descriptive CVD mortality over time Regional differences in medical care How are health insurance premiums changing g over time? These papers generally DON T TRY AND SAY WHY the trend might be changing over time Although there is likely to be some speculation 2 Empirical Research (cont.) Paper Type II Relate variable X to variable Y Effect of Price on the quantity of Medical Care Effect of race on Income/Health Effect of hypertension on risk of CVD These papers are making a causal argument The strength of which is up to the reader to evaluate Empirical Research (cont.) Paper Type III Use estimates from the first two types of papers to make policy recommendations For ex. Some studies find that insurance generosity affects the use of IVF services Because of limited opportunities, individuals maximize the chance of having at least one child 3 4

2 Empirical Research (cont.) One unintended consequence of this is multiple births Multiple births result in higher costs and lower infant health Using estimates from the IVF papers, someone else might write a paper about the optimal level of insurance benefit Policy Relevance We are going to focus on the Second Paper Type All three types of papers influence policy But paper type II is generally of most interest to policy researchers because it provides magnitudes for the phenomena of interest Magnitudes aid policy makers in the decision to allocate resources 5 6 Causation What do we mean by causation? We are asking a WHAT IF question What if instead of X happening, Z happened? How would that change the outcome of Interest? Thus one must always state the alternative The what if scenario is also called a COUNTERFACTUAL Some Notation Following Folland (1986) Some units Uwhere U can be a person, city, school Assume for simplicity two treatments T and C TTreatment and CControl Treatment can be a variety of things Drug, education, income, textbooks, copays Y represents outcome from receiving treatment So Y T (u) And Y C (u) Units are assigned to treatment and control and outcomes measured at a point in time after the assignment has taken place 7 8

3 Fundamental Problem of Causation CANNOT observe the effect of treatment and control for the same person Unless Temporal Stability AND Causal Transience are observed Temporal Stability (TS) Effect of T on U is same now and the future Causal Transience (CT) Effect of T on U doesn t change once U is exposed to T Or Unit homogeneity is observed Y t (U 1 )=Y t (U 2 ) and Y c (U 1 )=Y c (U 2 ) Fundamental Problem of Causation (cont.) Because of Temporal Stability and Causal Transience we can only estimate average treatment effects Average treatment effect equals [E(Y t (U)) E(Y c (U))] This is simply the mean difference of the outcome across the treatment and control groups 9 10 Paper Type IICausality Observational Studies Most are crosssectional Some type of statistical procedure that relates variables X and Y Ordinary Least Squares, Logistic Regression, Propensity Scores Quasi Experimental/Natural Experiments Regression Discontinuity Difference in Difference Instrumental Variables Randomized Control Trial (RCT) Gold Standard Observational Studies I Difficult to show causation purely from observational data, why? An example: Researchers are interested in whether income is related to health Direct effects Can buy more medical care Indirect effects Able to afford health insurance Some researchers believe health insurance affects health 11 12

4 Observational Studies I (cont.) Money can affect level of education Education might help you get better information Education might help you process information faster Observational Studies I (cont.) Take data from the cross section (point in time) Selfreported health as the dependent variable and Income as the independent variable Also adjust for a variables such as education, insurance,,geography, g age, sex, race, income, family education etc. and identify an effect Can we say this is the true effect of income on health? Observational Studies II Magnitudes from observational studies are generally biased upwards Especially from crosssectional studies There are some examples where estimates from observational studies are biased downwardd These are rare cases in the universe of all published studies Can you think of an association that is biased downward? I.e. An RCT would increase the size of your coefficient 15 Observational Studies II (cont.) In some studies the bias is hard to sign For example a researcher is interested in whether having fire insurance leads to more fire accidents relative to not having fire insurance What is the IDEA? Fire insurance lowers the cost of having your place burn down Thus individuals have less of an incentive to be careful, which in turn increases probability of a fire (also called ExAnte Moral Hazard) 16

5 Observational Studies II (cont.) Look at the Correlation between purchasing insurance and Having a fire in the next 5 years? In observational dataindividuals for whom fire insurance is more valuable (more likely to have a fire) will be more likely to buy fire insurance, How does this affect the coefficient? Not adjusting for this biases the coefficient i upwards In observational data individuals who are more cautious might also be more likely to buy fire insurance. Cautious people might have fewer fires than risky people Not adjusting for this will bias the coefficient downward Observational Studies II (cont.) Conclusion: Apriori impossible to tell whether relationship obtained from observational data is above or below the true effect of having fire insurance on having a fire Observational Studies III Given the above examples, Observational studies primarily show associations We will talk more about research designs with observational data that get us closer to causality Randomized Control Trial Randomization is a process used to assign a treatment to either treatment or control Randomization guarantees independence between treatment and all the other variables that might affect outcomes of interest A simple procedure for randomization coin flipping If randomizations is done correctly the mean difference across treatment and control groups E(Y t (U)) E(Y c (U)) is said to be unbiased How can we test whether randomization worked? 19 20

6 RCT (cont.) ) Without randomization it is very difficult to guarantee that it is truly the treatment that is responsible for the outcome Most nonexperimental procedures are aimed at finding a control group that is similar to the treatment group RCT (cont.) If its such a good idea why aren t there more RCTs? Ethical Problems Smoking is a good example Costs RCTs cost a lot of money The Rand HI experiment cost 280 Million 2004 dollars This was to randomize 7,791 people and to follow them for approximately 8 years RCT (cont.) Costs also impact the duration of the experiment Rand Health Insurance experiment only ran for 8 years Attrition can be high This is also a problem with nonexperimental designs Importantly people who drop out of the experiment are likely different from people who stay in the experiment Treatment effects could be different for the two groups RCT (cont.) i.e. Conjecture that treatment effect is higher for the group that stays in the experiment If you only used people who stayed in the experiment there would again be a upward bias to the measured treatment effect Even though there is attrition, one strategy is to estimate the effect as if there was no attrition 23 24

7 RCT (cont.) Keep everyone in the sample even if some people p are not longer taking the treatment This is called intent to treat analysis Intent t to treat t might dilute the true effect since not all individuals in treatment are taking the drug But this preserves the experiment and any estimates are still valid In a later lecture we will consider another solution to the attrition problem RCT (cont.) Treatment becomes Controls Different from Attrition Difficult to generalize from location to location Will experiment in location A reveal the same effect if done in location B Hawthorne Effects Observation makes people behave differently Thus results might not apply to nonobserved setting RCT (cont.) Finally Some things are not easily Manipulated How does one randomize Sex? How about race? Lets come back to this Example (I)Discrimination What is the effect of Sex (Race) on Employment (tenure)? Many studies show differences across the groups on a variety of outcomes 27 28

8 What Does Theory Say? Two Theories Statistical Discrimination Employers have limited resources to get information about any single individual, but know something about group averages They use information on the group average to make an inference about a specific individual What Does Theory Say (cont.) Wide applicability Physician decision making, Product selection, Speeding tickets This is Profiling TasteBased Discrimination Employers do not like to employ individuals from a specific group What Does Theory Say (cont.) ) Two types of discrimination have very different policy implication In a competitive market, firms will bear the cost of tastebased discrimination Statistical discrimination will likely never be competed away Why? Because using information about the group solves a problem that the profiler faces Testing for Discrimination I How do we test whether there is discrimination and second if so what type of discrimination? One idea is to simply compare mean wages across different groups from real world data What are the problems with this method? Employer observes something that you as a researcher do not (experience, good looks) Cannot separate out two theories with this method 31 32

9 Testing for Discrimination I (cont.) Let s take a step back How would one design an experiment to determine whether there is discrimination? In the RCT framework this question amounts to, How does one randomize race? Seems difficult to do Falls into one of these characteristics that cannot be manipulated Testing for Discrimination II Audit Studies Send in hispanics, african americans and whites for job interviews Two Problems: Auditors are matched on some observables except race» height, weight, age, dialect, dressing style and hairdo, Is that enough? Testing for Discrimination III Hard to manipulate race in life, but EASY to manipulate race on paper Which name doesn t belong? Chow Yun Phat, Yao Ming, Srikanth Kadiyala Correct answer is clearly Srikanth because he is not rich and famous Racial groups can have very different sounding names Testing for Discrimination III (cont.) Manipulate the resume so only difference is a african american sounding name vs. a white sounding name Emily Walsh vs. Jamal Jones Greg Baker vs. Lakisha Washington Find some real employers from the newspapers Two markets: Chicago/Boston 1300 Ads 35 36

10 Testing for Discrimination III (cont.) They vary not only the name (two resumes) but also type of resume More experience and Skills vs. Less experience and Skills Typically 4 different types of resumes to each job advertisement Measure call cack rate Researchers set up fake tel. #s to receive call backs Results Testing for Discrimination III (cont.) African Americans need to send 15 resumes to get 1 call back Whites need to send 10 resumes to get 1 call back 50% gap in call back between two groups Whites with high h quality resume receive nearly 30% more callbacks vs. whites with low quality resumes African american with high quality resumes don t experience the same benefit Amazing fact, experience and some other skills not being rewarded in the marketplace for african americans Separating Theories Does this method separate Statistical from Taste Based Discrimination? YES, Why? This study is superior to Audit studies, why? Perfect Matching on Treatment and Control This study has quite a few positives in the Realm of RCTs, What are they?: No attrition! No mixing of treatment and control! Cheap! Example IIPSA Screening Trials (I) What is the effect of the use of the PSA test on prostate cancer mortality? Two trials European Trial 162,000 asymptomatic men aged 5569 years U.S. Trial76,693 receive annual screening U.S. Trial 85% use of the PSA test in treatment group 52% use of the PSA test in the control group 39 40

11 Example IIPSA Screening Trials (II) U.S. Trial (cont.) After 7 years 2820 cancers detected in treatment group 2322 cancers detected in the control group 50 deaths due to prostate cancer in treatment group 44 deaths due to prostate cancer in control group No statistical difference in mortality Example IIPSA Screening Trials (III) E.U. Trial Average followup 8.8 years 82% screened in the treatment group 20% screened in the control group 5990 cancers detected in the treatment group 4307 cancers detected in the control group 214 deaths in screening group and 326 in control group 20% reduction in mortality Some Common NonExperimental Designs Designs without control groups X 01 Observe only data from post treatment (X) treatment 01 X 02 Observe data from pre and post treatment period X 03 Observe data from pre and post; observe a longer pre period Some Common Problems with NonExperimental Design Ambiguous Temporal Precedence For crosssectional data History Events occurring concurrently with intervention affect results Maturation Naturally occurring changes over time confused with intervention Regression to the mean 43 44

12 Some Common NonExperimental Designs Designs without control groups X 01 No control group Causality impossible to show 01X 02 or X 03 No true control group, History, maturation are problems Arguments can be made against regression to the mean since you have longer time period Most important thing to remember Treatment Timing might not be random 45 Cites Free For All? Lessons from the Rand Health Insurance Experiment, Joe Newhouse Statistics and Causal Inference, Journal of American Statistical Association, Vol. 81, no. 396, Dec. 1986, pp , Paul Holland Are Emily and Greg More Employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination, American Economic Review, Vol. 94, no. 4, Sept. 2004, pp , Marianne Bertrand, Sendhil Mullainathan 46