Correlation and Model Fit

Transcription

1 Correlation and Model Fit Prof. Jacob M. Montgomery Quantitative Political Methodology (L32 363) November 9, 2016 Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

2 Some class business Poster projects Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

3 Some class business Poster projects Poster files due on 12/7 at 10am. (29 days from now) Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

4 Some class business Poster projects Poster files due on 12/7 at 10am. (29 days from now) Problem set is long. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

5 Some class business Poster projects Poster files due on 12/7 at 10am. (29 days from now) Problem set is long. This lecture is pretty abstract with a lot of equations. PLEASE ask questions. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

6 Overview Last time: Inference with regression A bit on interpreting regression output Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

7 Overview Last time: Inference with regression A bit on interpreting regression output This time: A quick reversion to correlation (r) Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

8 Overview Last time: Inference with regression A bit on interpreting regression output This time: A quick reversion to correlation (r) Understanding the rest of the stuff on R-output Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

9 Overview Last time: Inference with regression A bit on interpreting regression output This time: A quick reversion to correlation (r) Understanding the rest of the stuff on R-output RMSE and Model fit (r 2 ) Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

10 Overview Last time: Inference with regression A bit on interpreting regression output This time: A quick reversion to correlation (r) Understanding the rest of the stuff on R-output RMSE and Model fit (r 2 ) F-tests Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

11 Alternative measure of correlation Pearson s r Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

12 Alternative measure of correlation Pearson s r (Standardized slope) S Y = (Y i Ȳ ) 2 n 1 Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

13 Alternative measure of correlation Pearson s r (Standardized slope) S Y = S X = (Y i Ȳ ) 2 n 1 (X i X ) 2 n 1 Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

14 Alternative measure of correlation Pearson s r (Standardized slope) S Y = S X = (Y i Ȳ ) 2 n 1 (X i X ) 2 n 1 r = ( Sx S Y ) ˆβ Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

15 Reminder: These are the main parameters ˆβ = n i=1 ( ) (X i X )(Y i Ȳ ) n i=1(x i X ) 2 ˆα = Ȳ ˆβ X Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

16 How good is our model?: Thinking about variance Unconditional Variance: Estimate of total variance in the population S 2 = ˆσ 2 Y = (Y i Ȳ ) 2 n 1 S = ˆσ Y = (Y i Ȳ ) 2 n 1 Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

17 How good is our model?: Thinking about variance Unconditional Variance: Estimate of total variance in the population S 2 = ˆσ 2 Y = (Y i Ȳ ) 2 n 1 S = ˆσ Y = (Y i Ȳ ) 2 n 1 Sum of Squared Error: A measure of is spread around the line SSE = n i=1 (Y i Ŷ i ) 2 = n i=1 (Y i ˆα ˆβX i ) 2 Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

18 How good is our model?: Thinking about variance Unconditional Variance: Estimate of total variance in the population S 2 = ˆσ 2 Y = (Y i Ȳ ) 2 n 1 S = ˆσ Y = (Y i Ȳ ) 2 n 1 Sum of Squared Error: A measure of is spread around the line SSE = n i=1 (Y i Ŷ i ) 2 = n i=1 (Y i ˆα ˆβX i ) 2 Conditional Variance: Estimate of variance around line in population ˆσ 2 = SSE n 2 = (Y i Ŷ i ) 2 SSE n 2 ˆσ = n 2 = (Y i Ŷ i ) 2 n 2 ˆσ 2 is sometimes called Mean squared error (MSE) and ˆσ is Root mean squared error (RMSE) or Residual standard error (in R) or Standard error of the estimate (in SPSS). Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

19 A bit of visualization Histogram of Y Regression of X and Y Histogram of Y when X=2 Frequency Y Frequency Y X Y[X == 2] A really, really good line will have small conditional variance. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

20 How good is our model?: Thinking about variance Hold onto some basic ideas: (Y i Ȳ ) 2 Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

21 How good is our model?: Thinking about variance Hold onto some basic ideas: (Y i Ȳ ) 2 Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

22 How good is our model?: Thinking about variance Hold onto some basic ideas: (Y i Ȳ ) 2 = Total Sum of Squares Unconditional variance: S 2 = (Y i Ȳ ) 2 n 1 Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

23 How good is our model?: Thinking about variance Hold onto some basic ideas: (Y i Ȳ ) 2 = Total Sum of Squares Unconditional variance: S 2 = (Y i Ȳ ) 2 n 1 (Y i Ŷ i ) 2 Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

24 How good is our model?: Thinking about variance Hold onto some basic ideas: (Y i Ȳ ) 2 = Total Sum of Squares Unconditional variance: S 2 = (Y i Ȳ ) 2 n 1 (Y i Ŷ i ) 2 Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

25 How good is our model?: Thinking about variance Hold onto some basic ideas: (Y i Ȳ ) 2 = Total Sum of Squares Unconditional variance: S 2 = (Y i Ȳ ) 2 n 1 (Y i Ŷ i ) 2 = Sum of Squared Error Conditional variance: ˆσ 2 = SSE n 2 = (Y i Ŷ i ) 2 n 2 Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

26 How good is our model?: Thinking about variance Hold onto some basic ideas: (Y i Ȳ ) 2 = Total Sum of Squares Unconditional variance: S 2 = (Y i Ȳ ) 2 n 1 (Y i Ŷ i ) 2 = Sum of Squared Error Conditional variance: ˆσ 2 = SSE n 2 = (Y i Ŷ i ) 2 n 2 We are going to say that IF we have a really good model, ˆσ 2 should be a lot smaller than S 2. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

27 Regression between presidential vote outcomes and election outcomes Two-party presidential vote share Q2 GDP growth rate Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

28 We draw the line that reduces SSE Residuals e i = (Y i Ŷ i ) = (y i ˆα ˆβX i ) Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

29 We draw the line that reduces SSE Residuals e i = (Y i Ŷ i ) = (y i ˆα ˆβX i ) Incumbent party share of vote Residuals for presidential regression nd quarter GDP Growth Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

30 How good is our model? : r 2 Define some preliminary terms: TSS = (Y i Ȳ ) 2 Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

31 How good is our model? : r 2 Define some preliminary terms: TSS = (Y i Ȳ ) 2 SSE = (Y i Ŷ i ) 2 Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

32 How good is our model? : r 2 Define some preliminary terms: TSS = (Y i Ȳ ) 2 SSE = (Y i Ŷ i ) 2 Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

33 How good is our model? : r 2 Define some preliminary terms: TSS = (Y i Ȳ ) 2 SSE = (Y i Ŷ i ) 2 R-squared r 2 = Explained Variance Total Variance = Total Variance - Unexplained Variance Total Variance Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

34 How good is our model? : r 2 Define some preliminary terms: TSS = (Y i Ȳ ) 2 SSE = (Y i Ŷ i ) 2 R-squared r 2 = Explained Variance Total Variance = Total Variance - Unexplained Variance Total Variance = TSS SSE TSS Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

35 Some notes on r 2 R-squared r 2 = TSS SSE TSS SSE=0 (Perfect fit) r 2 = 1 Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

36 Some notes on r 2 R-squared r 2 = TSS SSE TSS SSE=0 (Perfect fit) r 2 = 1 SSE = TSS (No fit) r 2 = 0 Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

37 Some notes on r 2 R-squared r 2 = TSS SSE TSS SSE=0 (Perfect fit) r 2 = 1 SSE = TSS (No fit) r 2 = 0 Does not depend on units of measurement Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

38 Some notes on r 2 R-squared r 2 = TSS SSE TSS SSE=0 (Perfect fit) r 2 = 1 SSE = TSS (No fit) r 2 = 0 Does not depend on units of measurement Sometimes called the coefficient of determination Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

39 Some notes on r 2 R-squared r 2 = TSS SSE TSS SSE=0 (Perfect fit) r 2 = 1 SSE = TSS (No fit) r 2 = 0 Does not depend on units of measurement Sometimes called the coefficient of determination It does not penalize for model complexity. Often adjusted R-squared is used. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

40 Some notes on r 2 R-squared r 2 = TSS SSE TSS SSE=0 (Perfect fit) r 2 = 1 SSE = TSS (No fit) r 2 = 0 Does not depend on units of measurement Sometimes called the coefficient of determination It does not penalize for model complexity. Often adjusted R-squared is used. In R-output this is labeled Multiple R-squared Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

41 Some notes on r 2 R-squared r 2 = TSS SSE TSS SSE=0 (Perfect fit) r 2 = 1 SSE = TSS (No fit) r 2 = 0 Does not depend on units of measurement Sometimes called the coefficient of determination It does not penalize for model complexity. Often adjusted R-squared is used. In R-output this is labeled Multiple R-squared Why do we use it? Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

42 Some notes on r 2 R-squared r 2 = TSS SSE TSS SSE=0 (Perfect fit) r 2 = 1 SSE = TSS (No fit) r 2 = 0 Does not depend on units of measurement Sometimes called the coefficient of determination It does not penalize for model complexity. Often adjusted R-squared is used. In R-output this is labeled Multiple R-squared Why do we use it? Gives us an overall impression for how well our model is doing. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

43 Some notes on r 2 R-squared r 2 = TSS SSE TSS SSE=0 (Perfect fit) r 2 = 1 SSE = TSS (No fit) r 2 = 0 Does not depend on units of measurement Sometimes called the coefficient of determination It does not penalize for model complexity. Often adjusted R-squared is used. In R-output this is labeled Multiple R-squared Why do we use it? Gives us an overall impression for how well our model is doing. We can informally compare models. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

44 R output Call: lm(formula = vote ~ q2gdp, data = Abram) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-15 *** q2gdp * --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 15 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 1 and 15 DF, p-value: Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

45 Example: Explaining congressional elections Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

46 A real quick primer on the F-statistic for Regression Before we said that r 2 is intuitively r 2 Explained Variance =. It makes Total Variance sense then that (1 r 2 ) is the percent of variance we haven t explained. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

47 A real quick primer on the F-statistic for Regression Before we said that r 2 is intuitively r 2 Explained Variance =. It makes Total Variance sense then that (1 r 2 ) is the percent of variance we haven t explained. It turns out that: F-statistic for regression F = r 2 /p (1 r 2 )/[n (p + 1)] Here p is the number of covariates (gdp, incumbent, etc.), and n is the number of observations. This will be distributed according to the F-distribution with df 1 = p, and df 2 = n (p + 1). Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

48 Example F-Distributions And now you understand (almost) everything on a regression table. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

49 Hypothesis testing with F-statistics Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

50 Interpreting the F-test Is our model any good? This is a formalized way of asking whether our model is any good. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

51 Interpreting the F-test Is our model any good? This is a formalized way of asking whether our model is any good. We compare the amount of variance explained by the regression to the amount unexplained. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

52 Interpreting the F-test Is our model any good? This is a formalized way of asking whether our model is any good. We compare the amount of variance explained by the regression to the amount unexplained. This is essentially a comparison of the following two models: Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

53 Interpreting the F-test Is our model any good? This is a formalized way of asking whether our model is any good. We compare the amount of variance explained by the regression to the amount unexplained. This is essentially a comparison of the following two models: H 0 : Y i = α + ɛ i Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

54 Interpreting the F-test Is our model any good? This is a formalized way of asking whether our model is any good. We compare the amount of variance explained by the regression to the amount unexplained. This is essentially a comparison of the following two models: H 0 : Y i = α + ɛ i H a : Y i = α + X i β + ɛ i Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

55 Interpreting the F-test Is our model any good? This is a formalized way of asking whether our model is any good. We compare the amount of variance explained by the regression to the amount unexplained. This is essentially a comparison of the following two models: H 0 : Y i = α + ɛ i H a : Y i = α + X i β + ɛ i This is more useful in multivariate regression: Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

56 Interpreting the F-test Is our model any good? This is a formalized way of asking whether our model is any good. We compare the amount of variance explained by the regression to the amount unexplained. This is essentially a comparison of the following two models: H 0 : Y i = α + ɛ i H a : Y i = α + X i β + ɛ i This is more useful in multivariate regression: H 0 : Y i = α + ɛ i Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

57 Interpreting the F-test Is our model any good? This is a formalized way of asking whether our model is any good. We compare the amount of variance explained by the regression to the amount unexplained. This is essentially a comparison of the following two models: H 0 : Y i = α + ɛ i H a : Y i = α + X i β + ɛ i This is more useful in multivariate regression: H 0 : Y i = α + ɛ i H a : Y i = α + X i1 β 1 + X i2 β 2 + X i3 β ɛ i Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

58 R output Call: lm(formula = vote ~ q2gdp, data = Abram) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) e-15 *** q2gdp * --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 15 degrees of freedom Multiple R-squared: , Adjusted R-squared: F-statistic: on 1 and 15 DF, p-value: Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

59 Posters: General guidelines If you gather data, no sensitive questions. No time series data You need to have a plan to deal with endogeneity Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

60 The first thing you need to do... 1 Think of a research question! This step should always come first. You, as the researcher, need to formulate some sort of question that you want answered. You should care about this question and its answer! Not in an I want a good grade way, but in an I really care about the answer to this question kind of way. If you don t care about the question itself, then the project will be miserable to complete. Make it easier on yourself and research something you re actually interested in. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

61 Once you have a question... 1 Once you have settled on a question, the next step is to come up with a hypothesis pertaining to that question. 2 The hypothesis needs to be testable with the tools that you have learned (or will learn) in this course. This generally precludes giant questions: Why do Americans vote? What makes people make environmentally friendly choices? Does President Obama have an American birth certificate? 3 However, smaller questions are interesting too! Do yard signs make people more likely to vote? Can door knocking campaigns lead to higher levels of recycling? Are racial attitudes related to beliefs about whether President Obama is not a citizen? Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

62 Once you have a question... 1 This hypothesis does not have to ultimately be supported. Most hypotheses are wrong especially interesting ones. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

63 Once you have a question... 1 Your question must be testable with data, meaning that you have to find a proper data source for your research project. 2 If you cannot find data that helps answer your question, then, for the purposes of this class, the question is unanswerable. If this applies to your research question, then it s time to find a new one. 3 Fortunately, the internet is full of data sources. If you can think of a data set you may need, there is a good chance that someone has collected it. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

64 Finding Data 1 There are many data sources. And you can use any data source you want for your project, as long as the data provides evidence for an answer to your question. Do you wonder if you re data is appropriate? Ask yourself if the data you have collected would be able to convince someone else of your answer. If there is any question as to whether or not this is the case, you should search for more data. While having data is better than not having data, there are differences in the quality of data across data sets. Some data is very credible while other data should certainly be scrutinized. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

65 Good Data Sources 1 For political science data, the best place to begin looking for data is This website is a new project by Harvard s Institute for Quantitative Social Sciences that is meant to direct undergraduate students on how to do a quantitative social science research project. The link above is to the data section, which contains a database of credible data sources. If you want to do something in the political science realm, this is the place to begin looking for the data set that best suits your question. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

66 Specific Data Sources 1 For many questions regarding American political attitudes and behavior, the American National Election Study (ANES) is the best survey to use. Survey is done every two years before and after an election. Asks many sociological and political questions. Attempts to capture behavior through a large set of questions. The survey has a relatively large sample size. However, not all respondents are asked the same battery of questions. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

67 Specific Data Sources 1 For questions more of the sociological nature and questions that the ANES just doesn t seem to work well for, the General Social Survey (GSS) may suit your needs better. The GSS is done every one or two years and has a very large sample of respondents. Asks many sociological questions and questions about attitudes towards issues. Only samples respondents from the United States. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

68 Other Data Sources 1 For questions that could be categorized in the realm of international politics, you can use the Correlates of War surveys and its various offshoots. Another good source is the V-Dem project ( 2 In general, any data set that you find a link for on the IQSS website is going to be credible and rather informative. Start early and spend a good amount of time finding the right data set. 3 Many scholars post their datasets on their websites. Many journals archive related datasets on their websites. If you find a good paper written in the last 5 years, it might just be online waiting for you. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

69 Other Data Sources 1 Does your question require an obscure data source that cannot be found on the IQSS website? Then, searching the ICPSR data set archives might be the best option for you. ICPSR is an initiative based at the University of Michigan that houses a huge data base of political science data sets. Data sets contained in this archive range from Iowa Census data from 1908 to the results of every U.S. House election from the 1940 s to the present. This database is less user friendly than the IQSS student one, but if you can t find what you need on that website, then this may be your next best option. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

70 Other Data Sources 1 The IQSS also has another, albeit less user friendly, database of data sources: Like the ICPSR website, there is a very large set of data sets here. However, it is less user friendly than the IQSS student website. Finding the right data may take a long time to find on this website. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

71 Non-Political Science Data 1 The question for your project does not have to be political science related! You can choose to do a project on any question that you are interested in answering. If you re interested in sports, then the following website is a database of sports related data sources: Another database of data sets can be found on http: //rss.acs.unt.edu/rdoc/library/ecdat/html/00index.html. These data sets, which are actually a part of an R package vary on subject from tobacco budgets to crime in North Carolina to Heating and Cooling system choices. Before you use any of these data sets, though, make sure that there is a valid codebook which contains information about the survey method, sampling procedures, etc. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

72 Other Options 1 Are none of these data sources good enough for you? Then maybe you should collect your own data. Remember, we discussed sampling schemes at the very beginning of the course, so we expect you to tell us why your sampling scheme is good if you choose to collect your own data. Keep in mind that you need a decent sample size. However, if you can pull it off you can often get much better data to answer your question of interest. However, if you choose to collect your own data, start early and talk to your group, the TAs, etc. The more opinions about your project you get, the better it will be. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

73 Rules 1 Above all else, if you choose to collect your own data it has to be appropriate! In research, we have a board that approves human research projects called the IRB. If you think your data is questionable by appropriateness standards, read the IRB website to see if it would meet their standards. In short, this eliminates surveys involving minors, the homeless population, prisoners, the mentally handicapped, and any other population that is in a position that might cause them to feel obligated to take part in your survey. In addition to these requirements, your data cannot involve questions about alcohol or drug usage. Use your common sense! Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

74 How to get a good grade. 1 Once you have your data and have done all of the statistical analysis that you can possibly do, you and your team are tasked to make a poster telling everybody about the awesome work you have done. Your findings may seem minor to you, but they are probably much more interesting than you give yourself credit for. The data is new to everyone else, so they will probably find it very interesting. 2 The best way to make sure that your poster gets a good grade from the QPM team is to follow the instructions on the last page of your syllabus! The rubric for grading your poster is very descriptive and thorough. The examples on blackboard all scored an A based on this rubric. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36

75 Some notes about the posters 1 We need documented R code for everything you do to make your poster! And this R code should be full of comments! Lots and lots of comments! Lots of comments eliminates guess work if we aren t sure what you are doing in your R code. Less guess work almost absolutely leads to a better grade. 2 Spend a good amount of time on the statistical analysis as well as the poster presentation. We like well thought out projects as well as well presented projects. Both of these contribute to your overall grade. 3 Finally, don t wait until the last minute for this project. This project is a pretty significant portion of your grade and should be treated as such. Waiting until the last minute to do this project will surely lead to a poor grade. Lecture 19 (QPM 2016) R, R-squared and F November 9, / 36