Multiple Regression. Frank Wood. November 28, 2011

Transcription

1 Multiple Regression Frank Wood November 28, 2011

2 Review Regression Estimation We can solve this equation X Xb = X y (if the inverse of X X exists) by the following (X X) 1 X Xb = (X X) 1 X y and since we have (X X) 1 X X = I b = (X X) 1 X y

3 Least Square Solution The matrix normal equations can be derived directly from the minimization of w.r.t to β Q = (y Xβ) (y Xβ)

4 Fitted Values and Residuals Let the vector of the fitted values are ŷ 1 ŷ 2 ŷ =... ŷ n in matrix notation we then have ŷ = Xb

5 Hat Matrix-Puts hat on y We can also directly express the fitted values in terms of X and y matrices ŷ = X(X X) 1 X y and we can further define H, the hat matrix ŷ = Hy H = X(X X) 1 X The hat matrix plans an important role in diagnostics for regression analysis.

6 Hat Matrix Properties 1. the hat matrix is symmetric 2. the hat matrix is idempotent, i.e. HH = H Important idempotent matrix property For a symmetric and idempotent matrix A, rank(a) = trace(a), the number of non-zero eigenvalues of A.

7 Residuals The residuals, like the fitted value ŷ can be expressed as linear combinations of the response variable observations Y i e = y ŷ = y Hy = (I H)y also, remember e = y ŷ = y Xb these are equivalent.

8 Covariance of Residuals Starting with e = (I H)y we see that σ 2 {e} = (I H)σ 2 {y}(i H) but σ 2 {y} = σ 2 {ɛ} = σ 2 I which means that σ 2 {e} = σ 2 (I H)I(I H) = σ 2 (I H)(I H) and since I H is idempotent (check) we have σ 2 {e} = σ 2 (I H)

9 ANOVA We can express the ANOVA results in matrix form as well, starting with where SSTO = (Y i Ȳ ) 2 = Y 2 i (P Y i ) 2 n leaving y y = Y 2 i ( P Y i ) 2 n = 1 n y Jy SSTO = y y 1 n y Jy

10 SSE Remember SSE = e 2 i = (Y i Ŷ i ) 2 In matrix form this is SSE = e e = (y Xb) (y Xb) = y y 2b X y + b X Xb = y y 2b X y + b X X(X X) 1 X y = y y 2b X y + b IX y Which when simplified yields SSE = y y b X y or, remembering that b = (X X) 1 X y yields SSE = y y y X(X X) 1 X y

11 SSR We know that SSR = SSTO SSE, where SSTO = y y 1 n y Jy and SSE = y y b X y From this SSR = b X y 1 n y Jy and replacing b like before SSR = y X(X X) 1 X y 1 n y Jy

12 Quadratic forms The ANOVA sums of squares can be interpretted as quadratic forms. An example of a quadratic form is given by 5Y Y 1Y 2 + 4Y 2 2 Note that this can be expressed in matrix notation as (where A is always (in the case of a quadratic form) a symmetric matrix) ( ) ( ) ( ) 5 3 Y1 Y1 Y = y Ay The off diagonal terms must both equal half the coefficient of the cross-product because multiplication is associative. Y 2

13 Quadratic Forms In general, a quadratic form is defined by y Ay = i j a ijy i Y j where a ij = a ji with A the matrix of the quadratic form. The ANOVA sums SSTO,SSE and SSR can all be arranged into quadratic forms. SSTO = y (I 1 n J)y SSE = y (I H)y SSR = y (H 1 n J)y

14 Quadratic Forms Cochran s Theorem Let X 1, X 2,..., X n be independent, N(0, σ 2 )-distributed random variables, and suppose that n i=1 X 2 i = Q 1 + Q Q k, where Q 1, Q 2,..., Q k are nonnegative-definite quadratic forms in the random variables X 1, X 2,..., X n, with rank(a i ) = r i, i = 1, 2,..., k. namely, If r 1 + r r k = n, then Q i = X A i X, i = 1, 2,..., k. 1. Q 1, Q 2,..., Q k are independent; and 2. Q i σ 2 χ 2 (r i ), i = 1, 2,..., k

15 Tests and Inference The ANOVA tests and inferences we can perform are the same as before Only the algebraic method of getting the quantities changes Matrix notation is a writing short-cut, not a computational shortcut

16 Inference We can derive the sampling variance of the β vector estimator by remembering that b = (X X) 1 X y = Ay where A is a constant matrix A = (X X) 1 X A = X(X X) 1 Using the standard matrix covariance operator we see that σ 2 {b} = Aσ 2 {y}a

17 Variance of b Since σ 2 {y} = σ 2 I we can write σ 2 {b} = (X X) 1 X σ 2 IX(X X) 1 = σ 2 (X X) 1 X X(X X) 1 = σ 2 (X X) 1 I = σ 2 (X X) 1 Of course E(b) = E((X X) 1 X y) = (X X) 1 X E(y) = (X X) 1 X Xβ = β

18 Variance of b Of course this assumes that we know σ 2. If we don t, as usual, replace it with MSE. σ 2 {b} = ( σ 2 n σ2 X 2 + P X σ 2 (Xi X ) 2 P (Xi X ) 2 P X σ 2 σ 2 (Xi X ) 2 P (Xi X ) 2 ) s 2 {b} = MSE(X X) 1 = ( MSE n + X 2 MSE P (Xi X ) 2 X MSE P (Xi X ) 2 P X MSE (Xi X ) 2 P MSE (Xi X ) 2 )

19 Mean Response To estimate the mean response we can create the following matrix X h = ( 1 X h ) The prediction is then Ŷ h = X h b Ŷ h = X h b = ( ) ( ) b 1 X 0 h = ( ) b b 0 + b 1 X h 1

20 Variance of Mean Response Is given by σ 2 {Ŷ h } = σ 2 X h (X X) 1 X h and is arrived at in the same way as for the variance of β Similarly the estimated variance in matrix notation is given by s 2 {Ŷ h } = MSE(X h (X X) 1 X h )

21 Wrap-Up Expectation and variance of random vector and matrices Simple linear regression in matrix form Next: multiple regression

22 Multiple regression One of the most widely used tools in statistical analysis Matrix expressions for multiple regression are the same as for simple linear regression

23 Need for Several Predictor Variables Often the response is best understood as being a function of multiple input quantities Examples Spam filtering-regress the probability of an being a spam message against thousands of input variables Football prediction - regress the probability of a goal in some short time span against the current state of the game.

24 First-Order with Two Predictor Variables When there are two predictor variables X 1 and X 2 the regression model Y i = β 0 + β 1 X i1 + β 2 X i2 + ɛ i is called a first-order model with two predictor variables. A first order model is linear in the predictor variables. X i1 and X i2 are the values of the two predictor variables in the i th trial.

25 Functional Form of Regression Surface Assuming noise equal to zero in expectation E(Y ) = β 0 + β 1 X 1 + β 2 X 2 The form of this regression function is of a plane -e.g. E(Y ) = X1 + 5X 2

26 Loess example

27 Meaning of Regression Coefficients β 0 is the intercept when both X 1 and X 2 are zero; β 1 indicates the change in the mean response E(Y ) per unit increase in X 1 when X 2 is held constant β 2 -vice versa Example: fix X 2 = 2 E(Y ) = X 1 + 5(2) = X 1 X 2 = 2 intercept changes but clearly linear In other words, all one dimensional restrictions of the regression surface are lines.

28 Terminology 1. When the effect of X 1 on the mean response does not depend on the level X 2 (and vice versa) the two predictor variables are said to have additive effects or not to interact. 2. The parameters β 1 and β 2 are sometimes called partial regression coefficients.

29 Comments 1. A planar response surface may not always be appropriate, but even when not it is often a good approximate descriptor of the regression function in local regions of the input space 2. The meaning of the parameters can be determined by taking partials of the regression function w.r.t. to each.

30 First order model with > 2 predictor variables Let there be p 1 predictor variables, then Y i = β 0 + β 1 X i1 + β 2 X i β + p 1X i,p 1 + ɛ i which can also be written as p 1 Y i = β 0 + β k X ik + ɛ i k=1 and if X i0 = 1 is also can be written as where X i0 = 1 p 1 Y i = β k X ik + ɛ i k=1

31 Geometry of response surface In this setting the response surface is a hyperplane This is difficult to visualize but the same intuitions hold Fixing all but one input variables, each β p tells how much the response variable will grow or decrease according to that one input variable

32 General Linear Regression Model We have arrived at the general regression model. In general the X 1,..., X p 1 variables in the regression model do not have to represent different predictor variables, nor do they have to all be quantitative(continuous). The general model is p 1 Y i = β k X ik + ɛ i where X i0 = 1 k=1 with response function when E(ɛ i )=0 is E(Y ) = β 0 + β 1 X β p 1 X p 1