Lecture 16: Cauchy-Schwarz, triangle inequality, orthogonal projection, and Gram-Schmidt orthogonalization (1) Travis Schedler Thurs, Nov 4, 2010 (version: Thurs, Nov 4, 4:40 PM)
Goals (2) Corrected change-of-basis formula! The Cauchy-Schwarz inequality Triangle inequality Orthogonal projection and least-squares approximations Orthonormal bases Gram-Schmidt orthogonalization
Corrected change-of-basis formula! (3) Take the linear transformation T := L A L(Mat(2, 1, F)) of left-multiplication by the matrix ( ) 1 2 A :=. 3 4 Now, in the standard basis (v 1 (, v ) 2 ), M(T ( ) = ) A. 1 0 Consider the new basis v 1 :=, v 1 2 :=. Then we get 1 Av 1 = ( ) 3 = 3v 1 + 4v 7 2, Av 2 = ( ) 2 = 2v 1 + 2v 4 2. Thus, M (v 1,v 2 ) (T ) = ( ) 3 2. 4 2
Change-of-basis continued (4) ( ) v Incorrect change-of-basis matrix: 1 v 2 = S ( ) 1 1 S =. Then 0 1 ( v1 v 2 ), i.e., ( ) ( ) ( ) S 1 1 1 1 2 1 1 M (v1,v 2 )(T )S = 0 1 3 4 0 1 ( ) ( ) ( ) 2 2 1 1 2 4 = = M 3 4 0 1 3 7 (v 1,v 2 ) (T ). Correct one is the transpose: ( v 1 v 2) = ( v1 v 2 ) S. Then ( ) ( ) ( ) S 1 1 0 1 2 1 0 M (v1,v 2 )(T )S = 1 1 3 4 1 1 ( ) ( ) ( ) 1 2 1 0 3 2 = = = M 2 2 1 1 4 2 (v 1,v 2 ) (T ).
Correct change-of-basis and triangular changes-of-basis (5) Correct: Change of basis matrix from (v i ) to (v i ) is: S = M (v i ),(v i )(I ) = put on i-th column the coefficients of v i in terms of v 1,..., v n. (cf. Homework 5.) I.e. S = (s ij ) with v i = s 1i v 1 + + s ni v n. If T L(Mat(n, 1, F)) and (v i ) = standard basis, then the change-of-basis is S = (v 1 v n): put columns together. Corollary (similar to Proposition 5.12) The change-of-basis matrix S = (s ij ) from (v i ) to (v i ) is upper-triangular if and only if v i = s 1i v 1 + + s ii v i for all i, i.e., s ji = 0 for j > i. Corollary (cf. Proposition 5.12) Let S be an upper-triangular change of basis. Then M (vi )(T ) is upper-triangular iff M (v i )(T ) is upper-triangular. Proof: If A is upper triangular, so is S 1 AS, and vice-versa.
The Cauchy-Schwarz inequality (6) Theorem (6.6: Cauchy-Schwarz inequality) u, v u v. Moreover, this is an equality iff one of u and v is a scalar multiple of the other. Proof. It is enough to suppose v 0. Write u = proj v (u) + w. Then u 2 = proj v (u) + w 2 = proj v (u) 2 + w 2 u,v v,v v 2 = u,v 2 v,v v, v = u,v 2. v 2 Hence, u 2 v 2 u, v 2 as desired. Equality holds iff w = 0, i.e., proj v (u) = u, i.e., u is a multiple of v.
Proof of triangle inequality (7) Theorem: u + v u + v. Equality holds iff one of u, v is a positive multiple of the other. u + v 2 = u + v, u + v = u, u + v, v + ( u, v + v, u ) = u 2 + v 2 + 2 Re u, v u 2 + v 2 + 2 u, v u 2 + v 2 + 2 u v = ( u + v ) 2. Here, Re is the real part: Re(a + bi) := a for a, b R. Thus 2 Re(z) = z + z. Equality holds in the first inequality iff u, v is nonnegative. Equality holds in the second inequality iff one of u, v is a multiple of the other. For both equalities to hold: say u = λv and u, v 0: then λ v, v 0, i.e., λ 0.
Orthogonal complement (8) Definition Let U V.Then, U := {u V : u, u = 0, u U}. Theorem (Theorem 6.29) U is a complement to U, i.e., V = U U. Proof. First, u U U implies u, u = 0, i.e., u = 0. So U U = {0}. Let (u 1,..., u k ) be a basis of U. Consider the map T : V F k given by T (v) = ( v, u 1,..., v, u k ). Then, U = null T. Since U U = 0, T U is injective. Since dim U = k = dim F k, T U is also surjective. By rank-nullity, dim V = dim U + dim U. So V = U U.
Orthogonal projection (9) Let U V. Then P U,U Proposition (Proposition 6.36) is the orthogonal projection. Let v V. Then, for all u U, v P U,U (v) v u. Thus, u := P U,U (v) is the best approximation of v by an element of U. It is also called the least-squares approximation because, when V = F n, it minimizes v u = n i=1 v i u i 2. Proof. Write v u = (v P U,U (v)) + (P U,U (v) u). By definition, v P U,U (v) U, and P U,U (v) u U. By the Pythagorean theorem, v u 2 = v P U,U (v) 2 + P U,U (v) u 2. Note: equality holds iff u = P U,U (v).
Orthonormal bases (10) Lemma Suppose that v 1,..., v k is a set of nonzero vectors such that v i v j for all i j. Then v 1,..., v k are linearly independent. Proof. Observe v i, λ 1 v 1 + + λ k v k = λ i v i, v i. If λ i 0, this is nonzero. Hence λ 1 v 1 + + λ k v k 0. Thus, there can be no linear dependence. Definition An orthonormal basis v 1,..., v n of (V,, ) is a basis of V such that v i, v j = δ ij. Definition: A list (v 1,..., v k ) is orthonormal if v i, v j = δ ij. Corollary If dim V = n, then any orthonormal list of length n is an orthonormal basis.
Formula for orthogonal projection (11) Proposition (6.35) Let e 1,..., e k be an orthonormal basis of U. Then, P U,U (v) = v, e 1 e 1 + + v, e k e k = proj e1 v + + proj ek v. Corollary (Theorem 6.17): when U = V and dim V = n, v = v, e 1 e 1 + + v, e n e n. Proof. Let u be the RHS. Then v u, e i = v, e i v, e i = 0 for all i. Hence, v u U. Since u U and v = u + (v u), we deduce u = P U,U (v).
Gram-Schmidt orthogonalization (12) Theorem (6.20: Gram-Schmidt) Given a basis (v 1,..., v n ) of V, there is an upper-triangular change of basis e i := s 1i v 1 + + s ii v i so that (e 1,..., e n ) is orthonormal. Moreover, the s ij are computable by an effective algorithm! Proof. By induction on n = dim V ; base case dim V = 0. Let U := Span(v 1,..., v n 1 ). Inductively, form the orthonormal basis (e 1,..., e n 1 ) of U via upper-triangular change of basis. It suffices to find e n such that e n, e j = δ nj (then automatically upper-triangular). Set e n := v n P U,U (v n ) U. So e n, e j = 0 for j n. e n is nonzero since v n / U. Set e n := e n/ e n. Then e n, e j = δ nj.
Corollaries (13) Corollary (Corollary 6.24) Every finite-dimensional inner-product space has an orthonormal basis. Proof: Apply Gram-Schmidt to an arbitrary basis. Corollary (Corollary 6.25) Every orthonormal list of vectors in V can be extended to an orthonormal basis of V. Proof: Extend to an arbitrary basis and perform Gram-Schmidt. Corollary (Corollary 6.27) Suppose that M(T ) is (block) upper-triangular in some basis. Then there exists an orthonormal basis where it is still (block) upper triangular. Proof: Pick an arbitrary basis in which the matrix is as desired, and then apply Gram-Schmidt. It is an upper-triangular change of basis, so it preserves (block) upper-triangularity.
Further corollaries (14) Corollary (Corollary 6.27+) Over C, there exists an orthonormal basis in which M(T ) is upper-triangular. Over R, there exists an orthonormal basis in which it is block upper-triangular with diagonal blocks of size 1 1 or 2 2. Corollary (Corollary 6.33) (U ) = U. Proof: Clearly U (U ). They are both complements to U, so they have the same dimension. Hence they are equal.
Norm formulas (15) Proposition (Proposition 6.15) Let (e 1,..., e k ) be an orthonormal list. Then a 1 e 1 + + a k e k 2 = a 1 2 + + a k 2. Proof: Repeated application of the Pythagorean theorem, since a i e i, a i e i = a i 2. Corollary Let (e 1,..., e k ) be an orthonormal basis of U. Then P U,U (v) = v, e 1 2 + + v, e k 2. Proof: Apply the previous corollary and P U,U (v) = v, e 1 e 1 + + v, e k e k. When U = V, we get v 2 = v, e 1 2 + + v, e n 2 (cf. Theorem 6.17).