Lecture 16: Cauchy-Schwarz, triangle inequality, orthogonal projection, and Gram-Schmidt orthogonalization (1)

Transcription

1 Lecture 6: Cauchy-Schwarz, triangle inequality, orthogonal projection, and Gram-Schmidt orthogonalization () Travis Schedler Thurs, Nov 4, 200 (version: Thurs, Nov 4, 4:40 PM) Goals (2) Corrected change-of-basis formula! The Cauchy-Schwarz inequality Triangle inequality Orthogonal projection and least-squares approximations Orthonormal bases Gram-Schmidt orthogonalization Corrected change-of-basis formula! (3) Take the linear transformation T : L A L(Mat(2,, F)) of left-multiplication by the matrix ( ) 2 A :. 3 4 Now, ( in ) the standard ( ) basis (v, v 2 ), M(T ) A. 0 v :, v 2 :. Then we get ( ( Av 3 3v 7) + 4v 2, Av 2 2 2v 4) + 2v 2. Consider the new basis Thus, M (v,v 2 ) (T ) ( )

2 Change-of-basis continued (4) ( ) v Incorrect change-of-basis matrix: v 2 ( ) v S, i.e., S v 2 ( ). Then 0 S 2 M (v,v 2)(T )S M (v,v 2 ) (T ). Correct one is the transpose: ( ( ) v v 2) v v 2 S. Then S M (v,v 2)(T )S M (v,v 2 ) (T ). Correct change-of-basis and triangular changes-of-basis (5) Correct: Change of basis matrix from (v i ) to (v i ) is: S M (v i ),(vi) (I) put on i-th column the coefficients of v i in terms of v,..., v n. (cf. Homework 5.) I.e. S (s ij ) with v i s iv + + s ni v n. If T L(Mat(n,, F)) and (v i ) standard basis, then the change-of-basis is S (v v n): put columns together. Corollary (similar to Proposition 5.2). The change-of-basis matrix S (s ij ) from (v i ) to (v i ) is upper-triangular if and only if v i s iv + + s ii v i for all i, i.e., s ji 0 for j > i. Corollary 2 (cf. Proposition 5.2). Let S be an upper-triangular change of basis. Then M (vi)(t ) is upper-triangular iff M (v i )(T ) is upper-triangular. Proof: If A is upper triangular, so is S AS, and vice-versa. The Cauchy-Schwarz inequality (6) Theorem 3 (6.6: Cauchy-Schwarz inequality). u, v u v. Moreover, this is an equality iff one of u and v is a scalar multiple of the other. Proof. It is enough to suppose v 0. Write u proj v (u) + w. Then u 2 proj v (u)+w 2 proj v (u) 2 + w 2 u,v v,v v 2 u,v 2 v 2. Hence, u 2 v 2 u, v 2 as desired. u,v v,v Equality holds iff w 0, i.e., proj v (u) u, i.e., u is a multiple of v. 2 v, v 2

3 Proof of triangle inequality (7) Theorem: u + v u + v. Equality holds iff one of u, v is a positive multiple of the other. u + v 2 u + v, u + v u, u + v, v + ( u, v + v, u ) u 2 + v Re u, v u 2 + v u, v u 2 + v u v ( u + v ) 2. Here, Re is the real part: Re(a + bi) : a for a, b R. Thus 2 Re(z) z + z. Equality holds in the first inequality iff u, v is nonnegative. Equality holds in the second inequality iff one of u, v is a multiple of the other. For both equalities to hold: say u λv and u, v 0: then λ v, v 0, i.e., λ 0. Orthogonal complement (8) Definition 4. Let U V.Then, U : {u V : u, u 0, u U}. Theorem 5 (Theorem 6.29). U is a complement to U, i.e., V U U. Proof. First, u U U implies u, u 0, i.e., u 0. So U U {0}. Let (u,..., u k ) be a basis of U. Consider the map T : V F k given by T (v) ( v, u,..., v, u k ). Then, U null T. Since U U 0, T U is injective. Since dim U k dim F k, T U is also surjective. By rank-nullity, dim V dim U + dim U. So V U U. Orthogonal projection (9) Let U V. Then P U,U is the orthogonal projection. Proposition 0. (Proposition 6.36). Let v V. Then, for all u U, v P U,U (v) v u. Thus, u : P U,U (v) is the best approximation of v by an element of U. It is also called the least-squares approximation because, when V F n, it minimizes v u n i v i u i 2. Proof. Write v u (v P U,U (v)) + (P U,U (v) u). By definition, v P U,U (v) U, and P U,U (v) u U. By the Pythagorean theorem, v u 2 v P U,U (v) 2 + P U,U (v) u 2. Note: equality holds iff u P U,U (v). 3

4 Orthonormal bases (0) Lemma 6. Suppose that v,..., v k is a set of nonzero vectors such that v i v j for all i j. Then v,..., v k are linearly independent. Proof. Observe v i, λ v + + λ k v k λ i v i, v i. If λ i 0, this is nonzero. Hence λ v + + λ k v k 0. Thus, there can be no linear dependence. Definition 7. An orthonormal basis v,..., v n of (V,, ) is a basis of V such that v i, v j δ ij. Definition: A list (v,..., v k ) is orthonormal if v i, v j δ ij. Corollary 8. If dim V n, then any orthonormal list of length n is an orthonormal basis. Formula for orthogonal projection () Proposition 0.2 (6.35). Let e,..., e k be an orthonormal basis of U. Then, P U,U (v) v, e e + + v, e k e k proj e v + + proj ek v. Corollary (Theorem 6.7): when U V and dim V n, v v, e e + + v, e n e n. Proof. Let u be the RHS. Then v u, e i v, e i v, e i 0 for all i. Hence, v u U. u P U,U (v). Since u U and v u + (v u), we deduce Gram-Schmidt orthogonalization (2) Theorem 9 (6.20: Gram-Schmidt). Given a basis (v,..., v n ) of V, there is an upper-triangular change of basis e i : s i v + + s ii v i so that (e,..., e n ) is orthonormal. Moreover, the s ij are computable by an effective algorithm! Proof. By induction on n dim V ; base case dim V 0. Let U : Span(v,..., v n ). Inductively, form the orthonormal basis (e,..., e n ) of U via upper-triangular change of basis. It suffices to find e n such that e n, e j δ nj (then automatically uppertriangular). Set e n : v n P U,U (v n ) U. So e n, e j 0 for j n. e n is nonzero since v n / U. Set e n : e n/ e n. Then e n, e j δ nj. 4

5 Corollaries (3) Corollary 0 (Corollary 6.24). Every finite-dimensional inner-product space has an orthonormal basis. Proof: Apply Gram-Schmidt to an arbitrary basis. Corollary (Corollary 6.25). Every orthonormal list of vectors in V can be extended to an orthonormal basis of V. Proof: Extend to an arbitrary basis and perform Gram-Schmidt. Corollary 2 (Corollary 6.27). Suppose that M(T ) is (block) upper-triangular in some basis. Then there exists an orthonormal basis where it is still (block) upper triangular. Proof: Pick an arbitrary basis in which the matrix is as desired, and then apply Gram-Schmidt. It is an upper-triangular change of basis, so it preserves (block) upper-triangularity. Further corollaries (4) Corollary 3 (Corollary 6.27+). Over C, there exists an orthonormal basis in which M(T ) is upper-triangular. Over R, there exists an orthonormal basis in which it is block upper-triangular with diagonal blocks of size or 2 2. Corollary 4 (Corollary 6.33). (U ) U. Proof: Clearly U (U ). They are both complements to U, so they have the same dimension. Hence they are equal. Norm formulas (5) Proposition 0.3 (Proposition 6.5). Let (e,..., e k ) be an orthonormal list. Then a e + + a k e k 2 a a k 2. Proof: Repeated application of the Pythagorean theorem, since a i e i, a i e i a i 2. Corollary 5. Let (e,..., e k ) be an orthonormal basis of U. Then P U,U (v) v, e v, e k 2. Proof: Apply the previous corollary and P U,U (v) v, e e + + v, e k e k. When U V, we get v 2 v, e v, e n 2 (cf. Theorem 6.7). 5