OLS Geometry. Walter Sosa-Escudero. February 3, Econ 507. Econometric Analysis. Spring 2009

Transcription

1 Econ 507. Econometric Analysis. Spring 2009 February 3, 2009

2 Vector Space Geometry A vector space S is a set along with an addition and a scalar multiplication on S that satisfies some properties: conmutativity, associativity, etc. The euclidean space R n is the vector space formed by all vectors in R n with the usual definition of sum of vectors and scalar multiplication. Actually we will impose more structure than the requirements to form a vector space.

3 Some Definitions and Notation Inner product: < x, y > x y Norm: x (x x) 1/2 = ( n i=1 x2 i )1/2. Orthogonality: x and y are orthogonal iff < x, y >= x y = 0 Linear dependence: x 1,..., x k are linearly dependent if there exists x j, 1 j k and coefficients c i such that x j = i j c ix i

4 Vector geometry in R 2 Vector representation Vector addition. Scalar multiplication Angles, perpendicular and parallel vectors.

5 A vector in R 2

6 Vector addition: parallelogram s rule

7 Subspaces of the Euclidean Space A vector subspace is any subset of a vector space that is itself a vector space. Span: S(x 1,..., x k ) { z E n z = k i=1 b ix i, b i R } is the euclidean vector subspace spanned by x 1,..., x k, that is the set of all liner combinations of (x 1,..., x k ). Alternatively X = [x 1 x k ], S(X) {z E n z = Xγ} is the subspace generated by the columns of X (the span of X). All vectors that can be formed as linear combinations of the columns of X.

8 Orthogonal complement: S (X) { w E n w z = 0 for all z S(x) }. All vectors that are orthogonal to the columns of X. Basis: a basis of V is a list of linearly independent vectors that spans V. Dimension: # of vectors of any basis. Note dim S(X) ρ(x) Result: X n k with dim S(X) = k dim S (X) = n k

9 X is a vector in R 2. S(X) is the subspace spanned by X, S (X) is its orthogonal complement, each of dimension 1.

10 Variables and observations in the axis The goal is to represent the data and the OLS estimator. We need to change our notion of point. A scatter plot takes every observation as a point. Now we need to think of Y and the columns of X as K + 1 points in R n. Each column is a point

11 Source: Bring, J., 1996, A Geometric Approach to Compare Variables in a Regression Model, The American Statistician, 50,1, pp What do you expect to happen with this picture if we add a third person? A fourth?

12 Vector Spaces By definition, any point in S(X) can be expressed as Xβ, β R k. Least squares: given X and Y, find the point in S(X) that is the closest as possible to Y.

13 The problem: min β y xβ min β y xβ 2. Define: ˆβ (solution to the problem), Ŷ = X ˆβ, e = Y Ŷ Some properties: e is orthogonal to any point in S(X), in particular, to X or X ˆβ. ˆβ = (X X) 1 X Y. From the orthogonality condition X (Y ˆβ) = 0.

14

15

16 Projections Vector Spaces A projection is a mapping that takes any point in E n into a point in a subspace of E n. An orthogonal projection maps any point into the point of the subspace that is closest to it. Ŷ = X ˆβ = X(X X) 1 X Y = P X Y is the orthogonal projection of Y on S(X). P X = X(X X) 1 X is the projection matrix that projects Y orthogonally on to S(X). e = Y Ŷ = Y X ˆβ = (I X(X X) 1 X )Y = M X Y is the projection of Y on to the orthogonal complement of S(X), that is, S (X). M X I P X = I X(X X) 1 X. is the projecton matrix that projects Y orthogonally on to S (X).

17 Properties: easy to check algebraically, better to understand them geometrically M X and P X are symmetric matrices. M X + P X = I. This suggests the orthogonal decomposition Y = M X Y + P X Y

18 P X and M X are idempotent: P X P X = P X, M X M X = M X. Intuition: if a vector is already in S(X), further projecting it in S(X) has no effect. P X M X = 0. Think about what you get of doing fisrt one projection and then the other (in any order). P X and M X anihilate each other. 0 is the only point that belongs to both S(X) and S (X). M X anihilates any point in S(X), that is M X Xβ = 0 P X anihilates any point in S (X) : P X Xβ = 0 CHECK If A is a non-singular matrix K K, P XA = P X. ρ(x) = ρ(p X )

19 Goodness of fit Vector Spaces From the orthogonal decomposition Then Y = P Y + MY Y Y = Y P Y + Y MY (1) = Y P P Y + Y M MY (2) Y 2 = P Y 2 + MY 2 (3) In R 2 this is simply Pythagoras theorem. Then R 2 = P Y 2 Y 2 = cos 2 θ where θ is the angle formed by Y and P Y. Actually this is the uncentered R 2.

20

21 The Frisch-Waugh-Lovell Theorem Consider the linear model: Y = Xβ + u And partition it as follows: Y = X 1 β 1 + X 2 β 2 + u X 1, X 2 matrices of k 1 and k 2 explanatory variables. Then, X = [X 1 X 2 ], β = (β 1 β 2 ) and k = k 1 + k 2. M 1 I X 1 (X 1 X 1) 1 X 1, projects any vector in Rn in the orthogonal complement of the span of X 1. Y M 1 Y, X 2 M 1X 2, respectively, OLS residuals of regressing Y on X 1, and all columns of X 2 on X 1.

22 Suppose that we are interested in estimating β 2, and consider the following alternative methods: Method 1: Proceed as usual and regress Y on X obtaining the OLS estimator ˆβ = ( ˆβ 1 ˆβ 2 ) = (X X) 1 X Y. ˆβ2 would be the desired estimate. Method 2: Regress Y on X2 and obtain as estimate β 2 = (X2 X 2 ) 1 X2 Y Let e 1 and e 2 be the residuals vectors of the regressions in Method 1 and 2, respectively. Theorem (Frisch and Waugh, 1933, Lovell, 1963): ˆβ2 = β 2 (first part) and e 1 = e 2 (second part).

23 Proof (boring): Start point with the orthogonal decomposition: Y = P Y + MY = X 1 ˆβ1 + X 2 ˆβ2 + MY To prove the first part, multiply by X 2 M 1 to get: X 2M 1 Y = X 2M 1 X 1 ˆβ1 + X 2M 1 X 2 ˆβ2 + X 2M 1 MY M 1 X 1 = 0, why? X 2 M 1M = X 2 M X 2 P 1M = 0 (same reasons as before) Then: X 2 M 1Y = X 2 M 1X 2 ˆβ2 So: ˆβ2 = (X 2 M 1X 2 ) 1 X 2 M 1Y

24 To prove the second part multiply the orthogonal decomposition by M 1 and obtain: M 1 Y = M 1 X 1 ˆβ1 + M 1 X 2 ˆβ2 + M 1 MY Again, M 1 X 1 = 0 MY belongs to the orthogonal complement of [X 1 X 2 ], so further projecting it on the orthogonal complement of X 1 (which is what premultiplying by M 1 would do) has no effect, hence M 1 MY = MY. This leaves: M 1 Y M 1 X 2 ˆβ2 = MY Y X 2 ˆβ 2 = MY e 2 = e 1

25 Geometric Illustration of FWLT

26 Geometric Illustration of FWLT

27 Comments and Intuitions Idea of controling for X 1 : either put it in the model, or first get rid of it by extracting its effect. What if X 1 and X 2 are orthogonal?

28 of the FWLT Deviations from means. Detrending Seasonal effects Later on: multicolinearity, omitted variable bias, panel-data fixed-effects estimation, instrumental variables.

29 Deviation from means Simple model with intercept Y = Xβ + u = β [X 2 X 3 X K ] β 1, 1 (1, 1,..., 1), β 1 = (β 2, β 3,..., β K ), and X k, k = 2,..., K are the corresponding columns of X. Two methods of estimating β 1 Method 1: Regress Y on X = [1 X 2 X K ]. Method 2: Get residuals of projecting X k, k = 2,..., K on 1, call them X k. Do the same with Y, and call them Y.

30 Note P 1 = 1(1 1) 1 1 = n 1 J, J is an n n matrix of 1 s. Then P 1 X k = 1 n JX k = ( X k, X k,..., X k ) so X k = M 1X k = (I P 1 )X k = X k ( X k, X k,..., X k ), an n 1 vector with typical element X ik = X ik X k So the second method consists in: 1 Reexpress all varaibles as deviations from their sample means. 2 Run the standard regression of these residuals without intercept. Question: what happens if we forget to reexpress Y as deviations from its means. Generalize this result