Lecture 8. Econ August 19

Transcription

1 Lecture 8 Econ August 19

2 Lecture 8 Outline 1 Eigenvectors and eigenvalues 2 Diagonalization 3 Quadratic Forms 4 Definiteness of Quadratic Forms 5 Uniqure representation of vectors

3 Eigenvectors and Eigenvalues Definition An eigenvalue of the square matrix A is a number λ such that A λi is singular. If λ is an eigenvalue of A, then any x 0 such that (A λi)x = 0 is called an eigenvector of A associated with the eigenvalue λ. Therefore: 1 Eigenvalues solve the equation det(a λi) = 0. 2 Eigenvectors are non trivial solutions to the equation Ax = λx Why do we care about eigenvalues and their corresponding eigenvectors? They enable one to relate complicated matrices to simple ones. They play a role in the study of stability of difference and differential equations. They make certain computations easy. They let us to define a way for matrices to be positive or negative and that matters for calculus and optimization.

4 Definition Characteristic Equation If A is an n n matrix, the characteristic equation is defined as α 11 λ α 12 α 1n α 21 α 22 λ α 2n f (λ) = det(a λi) = 0 or f (λ) = det... α m1 α m2 α mn λ = This is a polinomial equation in λ. Example For a two by two matrix: ( A λ I = a11 λ a a 21 a 22 λ Hence the characteristic equation is Which typically has two solutions. ) det A = (a 11 λ)(a 22 λ) a 12 a 21 λ 2 (a 11 + a 22 )λ + a 11 a 22 a 12 a 21 = 0

5 Characteristic Polynomial The characteristic equation f (λ) = det(a λi) = 0 is a polynomial degree n. By the Fundamental Theorem of Algebra, it has n roots (not necessarily distinct and not necessarily real). That is f (λ) = (λ c 1 )(λ c 2 ) (λ c n ) where c 1,..., c n C (the set of complex numbers) and the c i s are not necessarily distinct. Notice that f (λ) = 0 if and only if λ {c 1,..., c n}, so the roots are all the solutions of the equation f (λ) = 0. if λ = c i R, there is a corresponding eigenvector in R n. if λ = c i R, the corresponding eigenvectors are in C n \ R n. Another way to write the characteristic polynomial is P(λ) = (λ r 1 ) m1 (λ r k ) m k, where r 1, r 2..., r k are distinct roots (r i r j when i j) and m i are positive integers summing to n. m i is called the multiplicity of root r i.

6 FACT Distinct Eigenvectors The eigenvectors corresponding to distinct eigenvalues are distinct. Proof By Contradiction. Let λ 1,..., λ k be distinct eigenvalues and x 1,..., x k the associated eigenvectors. Suppose these vectors are linearly dependent (why is this a contradiction?). WLOG, let the first k 1 vectors be linearly independent, while x k is a linear combination of the others. Thus, α i i = 1,..., k 1 not all zero such that: k 1 i=1 αi xi = x k Multiply both sides by A and use the eigenvalue property (Ax = λx): k 1 α i λ i x i = λ k x k i=1 Multiply the first equation by λ k and subtract it from the second: k 1 α i (λ i λ k )x i = 0 i=1 Since the eigenvalues are distinct λ i λ k for all i; hence, we have a non-trivial linear combination of the first k 1 eigenvectors equal to 0, contradicting their linear independence.

7 Diagonalization Definition We say B is diagonalizable if we can find matrices P and D, with D diagonal, such that P 1 BP = D If a matrix is diagonalizable or where D is a diagonal matrix. Why do we care about this? PP 1 BPP 1 = PDP 1 B = PDP 1 We can use simple (i.e. diagonal) matrices to represent more complicated ones. This property is handy in many applications. An example follows: linear difference equations.

8 Difference Equations Detour A difference equation is an equation in which discrete time is one of the independent variables. For example, the value of x today depends linearly on its value yesterday: x t = ax t 1 t = 0, 1, 2, 3,... This is a fairly common relationship in time series data and macro. Given some initial condition x 0, this equation is fairly easy to solve using x 1 = ax 0 x 2 = ax 1 = a 2 x 0 recursion : Hence:. x t 1 = ax t 2 = a t 1 x 0 x t = ax t 1 = a t x 0 x t = a t x 0 t = 0, 1, 2, 3,...

9 Difference Equations Detour Continued Consider now a two-dimensional linear difference equation: ( ) ( ) ( ) ct+1 b11 b = 12 ct t = 0, 1, 2, 3,... k t+1 b 21 b 22 k t given some initial condition c 0, k 0. ( ) ( ) ct b11 b Set y t = t and B = 12 and rewrite this more k t b 21 b 22 compactly as y t+1 = By t t = 0, 1, 2, 3,... where b ij R each i, j. We want to find a solution y t, t = 1, 2, 3,... given the initial condition y 0. Such a dynamical system can arise as a characterization of the solution to a standard optimal growth problem (you will see this in macro). This is hard to solve since the two variables interact with each other as time goes on. Things would be much easier if there were no interactions (b 12 = 0 = b 21) because in that case the two equations would evolve independently.

10 Difference Equations Detour: The End We want to solve y t+1 = By t t = 0, 1, 2, 3,... If B is diagonalizable, there exist an invertible 2 2 matrix P and a diagonal 2 2 matrix D such that ( ) P 1 d1 0 BP = D = 0 d 2 Then y t+1 = By t t P 1 y t+1 = P 1 By t t P 1 y t+1 = P 1 BPP 1 y t t ^y t+1 = D^y t t where we defined ^y t = P 1 y t t (this is just a change of variable). Since D is diagonal, after this change of variable to ^y t we now have to solve two independent linear univariate difference equations ^y it = d t i ^y 0 t for i = 1, 2 which is easy because we can use recursion.

11 Diagonalization Theorem Theorem If A is an n n matrix that either has n distinct eigenvalues or is symmetric, then there exists an invertible n n matrix P and a diagonal matrix D such that A = PDP 1 Moveover, the diagonal entries of D are the eigenvalues of A, and the columns of P are the corresponding eigenvectors. Note Premultiply by P and postmultiply by P 1, the theorem says: Definition P 1 AP = D Two square matrices A and B are similar if A = P 1 BP for some invertible matrix P. The theorem says that some square matrices are similar to diagonal matrices that have eigenvalues on the diagonal.

12 Diagonalization Theorem: Idea of Proof We want to show that for a given A there exist a matrix P and a diagonal matrix D such that A = PDP 1, where the diagonal entries of D are the eigenvalues of A and the columns of P are the corresponding eigenvectors. Idea of Proof (a real proof is way too diffi cult for me) Suppose λ is an eigenvalue of A and x is an eigenvector. Thus Ax = λx. If P is a matrix with column j equal to the eigenvector associated with λ i, it follows that AP = PD. The result would then follow if one could guarantee that P is invertible. The proof works by showing that when A is symmetric, A only has real eigenvalues, one can find n linearly independent eigenvectors even if the eigenvalues are not distinct (these results use properties of complex numbers). See a book for details.

13 A Few Computational Facts For You To Prove Facts det AB = det BA = det A det B If D is a diagonal matrix, then det D is equal to the product of its diagonal elements. det A is equal to the product of the eigenvalues of A. Definition The trace of a square matrix A is given by the sum of its diagonal elements. That is, tr (A) = n i=1 a ii. Fact tr (A) = n λ i, where λ i is the ith eigenvalue of A (eigenvalues counted with multiplicity). i=1

14 Unitary Matrices Remember A t is the transpose of A: the (i, j) th entry of A t is the (j, i) th entry of A. Definition An n n matrix A is unitary if A t = A 1. REMARK By definition every unitary matrix is invertible.

15 Unitary Matrices Notation A basis V = {v 1,..., v n } of R n is orthonormal 1 if each basis element has unit length (v i v i = 1 i), and 2 distinct basis elements are orthogonal (v i v j = 0 for i j). Compactly, this can be written as v i v j = δ ij = { 1 if i = j 0 if i j. Theorem An n n matrix A is unitary if and only if the columns of A are orthonormal. Proof. Let v j denote the j th column of A. A t = A 1 A t A = I v i v j = δ ij {v 1,..., v n } is orthonormal

16 Symmetric Matrices Have Orthonormal Eigenvectors Remember A is symmetric if a ij = a ji for all i, j, where a ij is the (i, j) th entry of A. Theorem If A is symmetric, then the eigenvalues of A are all real and there is orthonormal basis V = {v 1,..., v n } of R n consisting of the eigenvectors of A. In this case, P in the diagonalization theorem is unitary and therefore: A = PDP t The proof is also beyond my ability (uses the linear algebra of complex vector spaces).

17 Quadratic Forms Think of a second-degree polynomial that has no constant term: n f (x 1,..., x n ) = α ii xi 2 + β ij x i x j i=1 i<j Let Then, α ij = { βij 2 if i < j β ji 2 if i > j α 11 α 1n and A =..... α n1 α nn f (x) = x t Ax This inspires the idea of a quadratic form.

18 Quadratic Forms Definition A quadratic form in n variables is a function Q : R n R that can be written as Q(x) = x t Ax where A is a symmetric n n matrix. Why do we care about quadratic forms? They show up in many places. For example, think about a function from R n to R. We will see that the first derivative of this function is a vector (we take the derivative one component at a time). Thus, the second derivative is a matrix (we take the derivative of the first derivative one component at a time). This matrix can be thought of as a quadratic form. Thus, a function s shape could be related to the sign of a quadratic form. What is the sign of a quadratic form anyhow?

19 Examples of Quadratic Forms Example When n = 1 a quadratic form is a function of the form ax 2. Example When n = 2 it is a function of the form a 11 x a 12 x 1 x 2 + a 22 x 2 2 (remember a 12 = a 21 by symmetry). Example When n = 3, it is a function of the form a 11 x a 22 x a 33 x a 12 x 1 x 2 + 2a 13 x 1 x 3 + 2a 23 x 2 x 3

20 Sign of A Quadratic Form Definition A quadratic form Q(x) = x t Ax is 1 positive definite if Q(x) > 0 for all x 0. 2 positive semi definite if Q(x) 0 for all x. 3 negative definite if Q(x) < 0 for all x 0. 4 negative semi definite if Q(x) 0 for all x. 5 indefinite if there exists x and y such that Q(x) > 0 > Q(y). In most cases, quadratic forms are indefinite. What does all this mean? Hard to tell, but we can try to look at special cases to get some intuition.

21 Positive and Negative Definiteness Idea Think of positive and negative definiteness as a way one applies to matrices the idea of positive and negative. In the one-variable case, Q(x) = ax 2 and definiteness follows the sign of a. Obviously, there are lots of indefinite matrices when n > 1. Diagonal matrices also help with intuition. When A is diagonal: n Q(x) = x t Ax = a ii xi 2. therefore the quadratic form is: positive definite if and only if a ii > 0 for all i, positive semi definite if and only if a ii 0 for all i negative definite if and only if a ii < 0 for all i, negative semi definite if and only if a ii 0 for all i, and indefinite if A has both negative and positive diagonal entries. i=1

22 Quadratic Forms and Diagonalization For symmetric matrices, definiteness relates to the diagonalization theorem. Assume A is symmetric. By the diagonalization theorem: A = R t DR, where D is a diagonal matrix with (real) eigenvalues on the diagonal and R is an orthogonal matrix. For any quadratic form Q(x) = x t Ax, by definition, A is symmetric. Then we have Q(x) = x t Ax = x t R t DRx = (Rx) t D (Rx). The definiteness of A is thus equivalent to the definiteness of its diagonal matrix of eigenvalues, D. Think about why.

23 Quadratic Forms and Diagonalization: Analysis A quadratic form is a function Q(x) = x t Ax where A is symmetric. Since A is symmetric, its eigenvalues λ 1,..., λ n are all real. Let V = {v 1,..., v n } be an orthonormal basis of eigenvectors of A with corresponding eigenvalues λ 1,..., λ n. By an earlier theorem, the P in the diagonalization theorem is unitary. λ Then: A = U t 0 λ 2 0 DU where D =..... and U is unitary. 0 0 λ n We know that any x R n can be written as x = n i=1 γ i v i. Then, one can rewrite a quadratic form as follows: ( ) ( ) t ( ) ( ) t ( ) Q(x) = Q γi v i = γi v i A γi v i = γi v i U t DU γi v i = (U ) t γ i v i D (U ) ( ) t ( ) γ i v i = γi Uv i D γi Uv i = (γ 1,..., γ n )D γ 1. γ n = λ i γ 2 i

24 Quadratic Forms and Diagonalization The algebra on the previous slide yields the following result. Theorem The quadratic form Q(x) = x t Ax is 1 positive definite if λ i > 0 for all i. 2 positive semi definite if λ i 0 for all i. 3 negative definite if λ i < 0 for all i. 4 negative semi definite if λ i 0 for all i. 5 indefinite if there exists j and k such that λ j > 0 > λ k. REMARK We can check definiteness of a quadratic form using the eigenvalues of A.

25 Principal Minors Definition A principal submatrix of a square matrix A is the matrix obtained by deleting any k rows and the corresponding k columns. Definition The determinant of a principal submatrix is called the principal minor of A. Definition The leading principal submatrix of order k of an n n matrix is obtained by deleting the last n k rows and column of the matrix. Definition The determinant of a leading principal submatrix is called the leading principal minor of A. Principal minors can be used in definitess tests.

26 Another Definiteness Test Theorem A matrix is 1 positive definite if and only if all its leading principal minors are positive. 2 negative definite if and only if its odd principal minors are negative and its even principal minors are positive. 3 indefinite if one of its kth order leading principal minors is negative for an even k or if there are two odd leading principal minors that have different signs. This classifies definiteness of quadratic forms without finding the eigenvalues of the corresponding matrices. Think about these conditions when applied to diagonal matrices and see if they make sense in that case.

27 Back to Linear Algebra Definitions All vectors below are elements of X (a vector space) and all scalars are real numbers. The linear combination of x 1,..., x n with coeffi cients α 1,..., α n : n y = α i x i The set of all linear combinations of elements of V = {v 1,..., v k } k spanv = {x X : x = λ i v i with v V } A set V X spans X if spanv = X. A set V X is linearly dependent if i=1 i=1 v 1,..., v n V and α 1,..., α n not all zero such that i=1 A set V X is linearly independent if it is not linearly dependent. Thus, V X is linearly independent if and only if: n α i v i = 0 with each v i V α i = 0 i i=1 n α i v i = 0. A basis of X is a linearly independent set of vectors in X that spans X.

28 Vectors and Basis Any vector can be uniquely written as a finite linear combination of the elements of some basis of the vector space to which it belongs. Theorem Let V be a basis for a vector space X over R. Every vector x X has a unique representation as a linear combination of a finite number of elements of V (with all coeffi cients nonzero). Haven t we proved this yet? Not at this level of generality. The unique representation of 0 is 0 = i α i v i.

29 Any vector has a unique representation as linear combination of finitely many elements of a basis. Proof. Since V spans X, any x X can be written as a linear combination of elements of V. We need to show this linear combination is unique. Let x = s S 1 α s v s and x = s S 2 β s v s where S 1 is finite, α s R, α s 0, and v s V for each s S 1 and where S 2 is finite, β s R, β s 0, and v s V for each s S 2. Define S = S 1 S 2, Then α s = 0 for s S 2 \ S 1 and β s = 0 for s S 1 \ S 2 0 = x x = s S 1 α s v s s S 2 β s v s = s S α s v s s S β s v s = s S (α s β s )v s Since V is linearly independent, we must have α s β s = 0, so α s = β s, for all s S. s S 1 α s 0 β s 0 s S 2 So S 1 = S 2 and α s = β s for s S 1 = S 2, and the representation is unique.

30 Theorem Every vector space has a (Hamel) basis. A Basis Always Exists This follows from the axiom of choice (did we talk about this?). An equivalent result says that if a linearly independent set is not a basis, one can always add to it to get a basis. Theorem If X is a vector space and V X is a linearly independent set, then V can be extended to a basis for X. That is, there exists a linearly independent set W X such that V W spanw = X There can be many bases for the same vector space, but they all have the same number of elements. Theorem Any two Hamel bases of a vector space X have the same cardinality (are numerically equivalent).

31 Standard Basis Definition The standard basis for R n consists of the set of N vectors e i, i = 1,..., N, where e i is the vector with component 1 in the ith position and zero in all other positions. e 1 = e 2 = e n 1 = e n = A standard basis is a linearly independent set that spans R n. 2 Elements of the standard basis are mutually orthogonal. When this happens, we say that the basis is orthogonal. 3 Each basis element has unit length. When this also happens, we say that the basis is orthonormal. Verify all these.

32 Orthonormal Bases We know an orthonormal basis exists for R n (the standard basis). Fact One can always find an orthonormal basis for a vector space. Fact If {v 1,..., v k } is an orthonormal basis for V then for all x V, k k x = α i v i = (x v i ) v i i=1 i=1 This follows from the properties on the previous slide (check it).

33 Dimension and Basis Definition The dimension of a vector space X, denoted dim X, is the cardinality of any basis of X. Notation Reminder For V X, V denotes the cardinality of the set V. Fact M m n, the set of all m n real-valued matrices, is a vector space over R. A basis is given by { 1 if k = i and l = j {E ij : 1 i m, 1 j n} where (E ij ) kl = 0 otherwise The dimension of the vector space of m n matrices is mn. Proving this is an exercise.

34 Dimension and Dependence Theorem Suppose dim X = n N. If A X and A > n, then A is linearly dependent. Proof. If not, A is linearly independent and can be extended to a basis V of X : a contradiction A V V A > n Intuitively, if A s dimension is larger than the dimension of X there must be some lineraly dependent elements in it. Theorem Suppose dim X = n N, V X, and V = n. If V is linearly independent, then V spans X, so V is a basis. If V spans X, then V is linearly independent, so V is a basis. Prove this as part of Problem Set 8.

35 Tomorrow We illustrate the formal connection between linear functions and matrices. Then we move to some useful geometry. 1 Linear Functions 2 Linear Functions and Matrices 3 Analytic Geometry in R n