Unit 9 Multiple Regression: Chapter 11 in IPS

Transcription

1 Unit 9 Multiple Regression: Chapter 11 in IPS The `mathematics of multiple regression is a direct extension of simple linear regression Predictions use the regression equation t-statistics have same interpretation R-squared has the same interpretation Residuals help evaluate model fit. Some things need more care Interpretation of a model with more than one predictor No model can be sensibly interpreted without the context in which the data were gathered. 1

2 Chicago Housing Price Data Will examine a data set with the following variables: Response variable: selling price in thousands of dollars for small sample of houses in Chicago, (price) Predictor variables: Number of bedrooms (bdr) Floor space in square feet (flr) Front footage (in feet) of lot (lot) Number of bathrooms (bth) Data set is small (26 observations) but provides nice illustration of concepts of regression and interplay of variables. Easy to imagine that selling price depends on each of the predictors. Want to learn how it depends on all simultaneously 2

3 Excel version of a subset of the data We begin at the end of the story, an estimated model, then backtrack to the statistical model and some aspects of model building 3

4 Correlation between response and predictors Correlations capture association in only pairs of variables What if we wanted too look at the simultaneous effect of predictors? 4

5 Multiple Regression Multiple regression is a straightforward extension of the linear regression we have been using Multiple regression arises when we wish to examine the association of a response variable Y with several predictors simultaneously. Instead of using a model of the form Y = β0 + β1x + ε We use Y = β0 + β1x1 + β2x2 + β3x3 + + βpxp + ε Rather than outline the theory, best to begin with an example As in simple regression, parameters in the model are estimated by minimizing sum of squared distances between observed and model predicted responses 5

6 Some additional detail, interpretation Systematic part of the model Random part, N(0, σ) Observed values of Y have mean E(Y) and sd σ 6

7 Stata output for a regression with price as response (Y), and bdr, flr, lot, bth as predictors 7

8 From the text, p 611, 6th ed. similar graphic, p 687, 5th ed. 8

9 Important concepts Interpretation of regression coefficients: A coefficient for any one of the variables estimates how much the mean of the response variable changes if That variable changes by one unit All other variables are held constant. The `effect of each variable on the mean response (when other variables are held constant) is linear. Each t-ratio provides a test statistic for the hypothesis that the coefficient for the corresponding variable is zero The null hypothesis that the predictor variables (as a complete set) are not associated with the response cannot be tested using the individual t-ratios. The F-statistic is used for this, explanation to come later 9

10 Concepts Predicted Y values Predicted Y values now depend on all predictors, but are easy to calculate Two and three bedrooms house with 1 bath, 800 sq ft of floor space and 30 ft wide lot would be predicted to cost (2 (-3.83)) + (1 7.70) + ( ) + ( ) = 49.1 ($49,160) (3 (-3.83)) + (1 7.70) + ( ) + ( ) = ($45,330) R-squared (R 2 ) has a surprisingly simple interpretation: R-squared is the ratio of the variance of the predicted Y values to the variance of the observed Y values 10

11 What can be learned from multiple regression? Regression provides information about the joint behavior of variables that might be quite different than behavior of pairs of variables (price vs bdr, or price vs bath, for instance). What is happening with the regression variable? 11

12 Regression of price on bdr Full regression 12

13 Data are sorted on floor space. In clusters of houses with similar floor space, what happens when bdr increases? 13

14 Regression and Analysis of Variance Multiple regression is often accompanied by a Analysis of Variance (ANOVA) table (below for house price data The construction of the table is motivated by the relationships of sums of squares shown on the next slide Subsequent slide has the ANOVA table, definition of the F-statistic 14

15 ANOVA and the F-statistic Both are based on a decomposition of sums of squared distances that is not easy to prove, but easy to believe In words, Total sum of squared distances of y-values from mean = Sum of squared distances of (regression) model predicted values from the mean + Sum of squared distances of y-values from (regression) model predicted y-values Total SS = Model SS + Error SS SST = SSM + SSE SST SSM SSE 15

16 ANOVA table in regression Source SS DF MS F Model Residual SSM SSE DFM = p DFE = n p -1 MSM = SSM/ DFM MSE = SSE/DFE MSM/MSE Total SST DFT = n - 1 MST The F-ratio is a test of the null hypothesis that ALL the regression coefficients (not counting the intercept) are 0. That is, H 0 : β 1 = β 2 = = β p = 0 16

17 Logic of the F-test The model helps explain the data if the sum of squares due to the model (SSM) is a `large part of the total sum of squares (SST) For that to happen SSM will be large and SSE will be small (relative to SSM) The mean squares (MSM and MST) adjust for the number of observations in the data set. Adjustment needed since SSM will increase whenever the number of observations increases. 17

18 F-distribution The F-distribution is specified by a numerator and a denominator degrees of freedom. (Table E in IPS) To get the p-value associated with the observed value of F, look up the tail area for the F-distribution with p and n- p -1 degrees of freedom for numerator and denominator. A value of for an F(4,21) distribution is highly significant (p<.001). 18

19 TABLE E F critical values Probability p Table entry for p is the critical value F with probability p lying to its right. F* Degrees of freedom in the numerator p Degrees of freedom in the denominator

20 F critical values (continued) Relevant part of table E Degrees of freedom in the numerator p minator

21 Interpretation of binary predictor variables Suppose a predictor variable takes on only two values, 0 and 1 Called a binary predictor. What is the interpretation of regression in this setting? Use the housing data as a example, with a variable recoding three_plus_bdr takes on value 0 if 2 bedrooms in house Remember there are no 1 bdr homes in data set value 1 if 3 or more bedrooms in house i.e., more than 2 bedrooms 21

22 Regression of price on three_plus_bdr price Coef. Std. Err. t P> t [95% Conf. Interval] three_plus~r _cons A 1 unit change in three_plus_bdr corresponds to moving from 2 or fewer bedrooms to more than 3 bedrooms Houses with three or more bedrooms are associated with a selling price increase of $5, (keeping the $.22 here is a bit silly) 22

23 Using binary predictors: the mathematical interpretation 23

24 Using three_plus_bdr as a grouping variable for a t-test. ttest price, by(three_plus_bdr) Two-sample t test with equal variances Group Obs Mean Std. Err. Std. Dev. [95% Conf. Interval] combined diff diff = mean(0) - mean(1) t = Ho: diff = 0 degrees of freedom = 24 Ha: diff < 0 Ha: diff!= 0 Ha: diff > 0 Pr(T < t) = Pr( T > t ) = Pr(T > t) = tabulate bdr, summarize(price) Three bedroom houses are on average more expensive Summary of price bdr Mean Std. Dev. Freq Total

25 Using three_plus_bdr in the regression. regress price flr lot bth three_plus_bdr Source SS df MS Number of obs = 26 F( 4, 21) = 9.50 Model Prob > F = Residual R-squared = Adj R-squared = Total Root MSE = price Coef. Std. Err. t P> t [95% Conf. Interval] flr lot bth three_plus~r _cons In the multiple regression, having 3+ bedrooms is negatively associated with price. 25

26 More complicated uses of predictors Using binary variables when predictors are categorical, taking on more than two values, but with no natural ordering The General Social Survey shows an example Looking for interactions in regression Interactions occur when the effect of one variable depends upon the value of another specific example coming later 26

27 The National General Social Survey The General Social Survey is conducted every two years by the National Opinion Research Center to measure social attitudes and items that may be associated with those attitudes. A complete description of the survey may be found at including the interviews used to gather the data we will examine. The data used here is a subset of the items asked during the interview. Some of the variables on the data used in class are `constructed variables 27

28 28

29 General Social Survey 1998 General Social Survey (survey of 2832 people age 18+, conducted by the National Opinion Research Center). Available variables include ABORTION Attitude toward abortion (0 = least permissive to 7 = most permissive) COHABTN Attitude toward unmarried cohabitation (2 = least permissive to 10 = most permissive) INCOME Annual income (1 = under $1,000 to 23 = $110,000 or over) CONSERV Political conservatism (1 = extremely liberal to 7 = extremely conservative) EDUCAT Years of formal education (range: 0 20) MALE Dummy: 1 = male, 0 = femal MAED Mother s education (range: 0 20) PAED Father s education (range: 0 20) MARITAL Marital status (1 = married, 2 = widowed,3 = divorced, 4 = separated, 5 = never married) PARTNRS Number of sex partners in past 5 years (0 = none to 8 = more than 100) RESPAGE Age in years (range: 18 89) RACE Race (1 = white, 2 = black, 3 = other) RELOSITY Religiosity scale (sum of standardized items with range: ) RELIGION Religion (1 = Protestant, 2 =Catholic, 3 = Jewish, 4 = None, 5 = other) 29

30 Using binary predictors, categorical variables with more than 2 levels 30

31 Recoding categorical variables In the regression on the next slide, the following codes are used: black = 1 if the participant self-report on the survey was `black (i.e., race coded 2), and 0 for any other response. othrace = 1 if the self-report was other (race coded 3), and 0 for any other response to race. So the coefficients of black and othrace represent differences in average income of these groups compared to respondents who self-reported white 31

32 Summary statistics and the regression 32

33 More complicated uses of predictors Using binary variables when predictors are categorical, taking on more than two values, but with no natural ordering The General Social Survey shows an example Looking for interactions in regression Interactions occur when the effect (or association) of one variable depends upon the value of another 33

34 Interactions, or non-additivity, in regression Perhaps best understood in terms of a question in the context of the General Social Survey The data shows that African-American respondents reported a lower income than whites and will show that blacks report lower income than whites and `other combined. The data will show that reported income increases with increasing education (coming, but expected) Both of the above remain true in a multiple regression model with both variables. Does the data provide any information about whether the effect of education on income is the same for blacks as it is for whites? 34

35 Income vs education 35

36 Income vs education and race category 36

37 Key for regression lines on next slide All regressions include only subjects with no missing data on any of the variables Red: all cases included Green: only blacks in the regression Orange: only other race in the regression Light blue: only whites in the regression Note that the background scatterplot is not changing, but the points included in the regression lines are 37

38 38

39 Red: all cases included Green: only blacks in the regression Orange: only other race in the regression Light blue: only whites in the regression 39

40 Modeling a possible interaction 40

41 Modeling an interaction The change in the mean of Y induced by changing X 1 depends upon the value of X 2. This behavior is called an interaction effect. Examine this in the GSS using Stata 41

42 Adding a black race by education interaction 42

43 Adding another interaction with other race 43

44 Topics not covered Using residuals to examine quality of the fit of the model to the data Read pp in text R 2 vs adjusted R 2 in the Stata output. Transformations of the predictors 44

45 Main points from multiple regression Regression coefficients estimate the association of each predictor with response, when that predictor changes and all other predictors are held constant Individual t-tests and confidence intervals are used to study the significance of an individual predictor The F-statistic is used to assess the evidence for the association of a group of predictors with the response Many interesting issues in model interpretation Interrelationships among variables Coding binary predictors to study `group effects Interactions, on non-additivity 45