INNER PRODUCTS. 1. Real Inner Products Definition 1. An inner product on a real vector space V is a function (u, v) u, v from V V to R satisfying

Transcription

1 INNER PRODUCTS 1. Real Inner Products Definition 1. An inner product on a real vector space V is a function (u, v) u, v from V V to R satisfying (1) αu + βv, w = α u, w + β v, s for all u, v, w V and all α, β R; (2) u, v = v, u for all u, v V ; (3) v, v 0 for all v V, with equality if and only if v = 0. A real vector space together with an inner product is called a real inner product space. Item 1 asserts that the map (u, v) u, v is linear in the first variable. In view of 2, it is also linear in the second variable. Example 2. The usual dot product defines an inner product on R n. u, v = u v = u i v i Example 3. The set C([a, b]) of all continuous real valued functions on the interval [0, 1] forms a real vector space under pointwise addition and scaling. We can define an inner product on C([a, b]) by f, g = b a f(x)g(x) dx. Definition 4. Let V be a real vector space with inner product,. For any v V we define the norm of v by v = v, v 1/2. Note that the square root on the right is defined as a non-negative real number by 3. Lemma 5. Let V be a real vector space with inner product,. (1) (Positivity) v 0, with equality if and only if v = 0. (2) (Homogeneity) αv = α v for every α R and every v V. (3) (Cauchy-Schwarz Inequality) u, v u v, with equality if and only if u and v are parallel. (4) (Triangle Inequality) u + v u + v. Proof. Positivity is immediate from item 3 in the definition of inner product. For homogeneity, we have αv 2 = αv, αv = α 2 v, v = α 2 v 2. The result follows by taking square roots. 1

2 INNER PRODUCTS 2 We now turn to the Cauchy-Schwarz Inequality. If u and v are parallel, then one is a scalar multiple of the other, say v = αu for some α R. Therefore u, v = u, αu = α u, u = α u 2 = u αu = u v. On the other hand, if u and v are not parallel, then u + αv 0 for every α R, so, by positivity, 0 < u + αv 2 = u + αv, u + αv = u u, v α + v 2 α 2 In particular, taking α = u,v v 2 gives 0 < u 2 u, v 2 v 2, from which the Cauchy-Schwarz Inequality follows. Finally, we turn to the proof of the Triangle Inequality. We have u + v 2 = u + v, u + v = u u, v + v 2 u u, v + v 2 u 2 + u v + v 2 = ( u + v ) 2. Taking square roots gives the desired result. Definition 6. Two vectors u and v in a real inner product space are said to be orthogonal if u, v = 0. Theorem 7 (Pythagorean Theorem). If u and v are orthogonal vectors in a real inner product space, then Proof. u + v 2 = u 2 + v 2. u + v 2 = u + v, u + v = u u, v + v 2 = u 2 + v Exercises. (1) When does equality hold in the Triangle Inequality? (2) For any non-negative integer n, define c n, s n C([0, 2π]) by c n (x) = cos nx, s n (x) = sin nx. Using the inner product of Example 3: (a) Calculate c n and s n. (b) Show that c n and s m are orthogonal for every m, n Z +. (c) Show that c n is orthogonal to c m and s n is orthogonal to x m whenever n m. (3) Prove the converse of the Pythagorean Theorem: In a real inner product space, if u + v 2 = u 2 + v 2, then u and v are orthogonal.

3 INNER PRODUCTS 3 2. The Complex Case It is tempting to define an inner product on a complex vector space by simply transcribing the definition of a real inner product, but allowing the scalars to be complex. Unfortunately, this does not lead to anything useful: There would be no inner products on any complex vector space containing a non-zero vector (see Exercise 1). A slight refinement of the second item in the definition is needed. Definition 8. An inner product on a complex vector space V is a function (u, v) u, v from V V to C satisfying (1) αu + βv, w = α u, w + β v, s for all u, v, w V and all α, β C; (2) u, v = v, u for all u, v V ; (3) v, v 0 for all v V, with equality if and only if v = 0. A complex vector space together with an inner product is called a complex inner product space. We will use to phrase inner product space to refer to either a real or complex inner product space. Note that a complex inner product is linear in the first variable, but not in the second. In fact, we have u, αv + βw = αv + βw, u = α v, u + β w, u = α v, u + β w, u = α u, v + β u, w. We say that, is conjugate linear in the second variable. Example 9. The Hermitian inner product on C n is defined by z, w = z i w i. As in the real case, we define v = v, v 1/2, and we say u and v are orthogonal if u, v = 0. Complex variants of Lemma 5 and Theorem 7 can be established as in the real case. We state them here for completeness. The proofs are left to the exercises. Lemma 10. Let V be a complex vector space with inner product,. (1) (Positivity) v 0, with equality if and only if v = 0. (2) (Homogeneity) αv = α v for every α C and every v V. (3) (Cauchy-Schwarz Inequality) u, v u v, with equality if and only if u and v are parallel. (4) (Triangle Inequality) u + v u + v. Theorem 11 (Pythagorean Theorem). If u and v are orthogonal vectors in a complex inner product space, then u + v 2 = u 2 + v 2.

4 INNER PRODUCTS Exercises. (1) Show that if the conjugation on the right side of 2 were omitted no inner products would exist on any complex vector space containing a non-zero vector. (2) Let V be a complex vector space. We may view V as a real vector space by simply ignoring non-real scalars. Now suppose that, is a complex inner product on V, and define u, v = Re u, v. (a) Show that this defines a real inner product. (b) Show that the real inner product, and the complex inner product, define the same norm. (c) Show that orthogonality with respect to the complex inner product implies orthogonality with respect to the real inner product, but not conversely. It follows that the real Pythagorean Theorem is stronger than the complex version. (3) Let, be an inner product on a complex vector space V. (a) Show that for every u, v V and every α C we have u + αv = u Re (α u, v ) + α 2 v 2 (b) Prove the Cauchy-Schwarz Inequality for complex inner products. Hint: Take α = u,v v Orthonormal Sets Definition 12. A set of vectors {e 1,..., e n } in an inner product space is said to be orthogonal if e i, e j = 0 whenever i j. If, in addition, e i = 1 for each i, we call the set orthonormal. An orthonormal set which is also a basis for V is an orthonormal basis. Theorem 13. If {e 1,..., e n } is an orthonormal basis for V, then for every v V we have v = v, e i e i. Corollary 14. If {e 1,..., e n } is an orthonormal basis for V, then for any u, v V we have u, v = u, e i v, e i. Note that the corollary asserts that the inner product of two vectors in a real or complex finite dimensional inner product space is the Euclidean or, respectively, Hermitian inner product of their coordinate vectors in R n or C n. Corollary 15. Let T be a linear transformation from a finite dimensional inner product space V to a finite dimensional inner product space W. If A = [a ij ] is the matrix of T with respect to ordered orthonormal bases (e 1,..., e n ) and (f 1,..., f m ), respectively, then a ij = T e j, e i. Proof. The jth column of is the coordinate vector of T e j with respect to the basis (f 1,..., f m ). The result follows from Corollary 13.

5 INNER PRODUCTS 5 Theorem 16 (Bessel s Inequality). If {e 1,..., e n } is orthonormal in V, then for every v V we have v, e i 2 v 2. Moreover, we have equality for every v V if and only if the set is an orthonormal basis. Theorem 17. Let {v 1,..., v n } be linearly independent. There is an orthonormal set {e 1,..., e n } such that for 1 k n we have span{e 1,..., e k } = span{v 1,..., v k }. Proof (Gram-Schmidt Process): Let V k = span{v 1,..., v k }. We first construct, by induction on k, an orthogonal set {w 1,..., w n } such that span{w 1,..., w k } = V k. The required orthonormal set is then obtained by setting e k = w k w k. We begin by setting w 1 = v 1. For 1 < k n, suppose by induction that we have an orthogonal set {w 1,..., w k 1 } which spans V k 1. Since dim V k 1 = k 1, it follows that {w 1,..., w k 1 } is a basis for V k 1, and in particular, w i 0 for 1 i k 1. Let k 1 w k = v k v k, w i w i 2 w i. It follows from the orthogonality of {w 1,..., w k 1 } that w k, w i = 0 for 1 i k 1, and further that w k 0, since v k V k 1. Moreover, w k V k, so {w 1,..., w k } is a linearly independent set in V k. Since dim V k = k, it must be a basis for V k. This completes the proof. Corollary 18. Let V be an inner product space of finite dimension n, and let {e 1,..., e k } be orthonormal in V. There are e k+1,..., e n V such that {e 1,..., e n } is an orthonormal basis. In particular, every finite dimensional inner product space has an orthonormal basis. 4. Orthogonal Projections Definition 19. Let V be an inner product space and let E V. The orthogonal complement of E is Lemma 20. E is a subspace of V. E = {v V : v, w = 0 for every w E}. Theorem 21 (Projection Theorem). Let V be an inner product space and let W be a finite dimensional subspace. For every v V there are unique v 1, v 2 V such that (1) v 1 W and v 2 W ; (2) v = v 1 + v 2. Proof. Let {e 1,..., e k } be an orthonormal basis for W. If v 1 and v 2 are as advertised, then v, e i = v 1 + v 2, e i = v 1, e i + v 2, e i = v 1, e i so k k v, e i e i = v 1, e i e i = v 1

6 INNER PRODUCTS 6 so v 1, and hence also v 2, is uniquely determined. To complete the proof, let v V, and define k (1) v 1 = v, e i e i and v 2 = v v 1. Clearly v 1 W and v = v 1 + v 2. It only remains to check that v 2 W. It is straightforward to check that v 2 is orthogonal to each e i, and therefore to any linear combination of {e 1,..., e k }. Since {e 1,..., e k } is a basis for W, it follows that v 2 is orthogonal to each member of W, which completes the proof. Remark 22. The vector v 1 in the Projection Theorem is the orthogonal projection of v on the subspace W. The proof shows that v 1 is given by (1), where {e 1,..., e k } can be any orthonormal basis for the subspace W. The linear operator defined by k (2) P v = v, e i e i is the orthogonal projector of V onto W. The uniqueness assertion in the Projection Theorem assures us that the operator P defined by (2) does not depend on the choice of an orthonormal basis for W. Corollary 23. If W is a subspace of a finite dimensional inner product space V, then dim W + dim W = dim V. Proof. If {e 1,..., e m } is an orthonormal basis for W and {f 1,..., f n } is an orthonormal basis for W, then {e 1,..., e m, f 1,..., f n } is an orthonormal set which spans V, and is therefore an orthonormal basis for V. Definition 24. A linear functional on a real (or complex) vector space V is a linear transformation from V into R (or C). Theorem 25 (Representation Theorem for Linear Functionals). Let λ be a linear functional on a finite dimensional inner product space V. There is a unique w V such that λ(v) = v, w for every v V. Proof. We first establish uniqueness. Suppose that v, w 1 = v, w 2 for every v V. Then v, w 1 w 2 = 0 for every v V. In particular, taking v = w 1 w 2 we obtain w 1 w 2 2 = 0, and hence w 1 = w 2, establishing uniqueness. For existence, since V is finite dimensional, it has an orthonormal basis {e 1,..., e n }. For every v V we have v = v, e i e i so Λv = v, e i Λe i = v, (Λe i )e i = v, w

7 INNER PRODUCTS 7 with w = (Λe i )e i. 5. Adjoints Let V and W be finite dimensional inner product spaces over the same scalar field (R or C), and let T : V W be linear. Our immediate goal is to define a transposed transformation T going in the reverse direction, from W to V. Lemma 26. Let V, W, and T be as above. For each w W there is a unique w V such that for every v V we have T v, w = v, w. Proof. Apply the Representation Theorem to the linear functional λ(v) = T v, w. Definition 27. With V, W, and T as above, we define the adjoint mapping T : W V by defining T w to be the unique w V such that T v, w = v, w for every v V. Thus the map T is characterized by the identity for all v V and w W. Lemma 28. T is linear. T v, w = v, T w Proof. Let w 1, w 2 W and let α 1, α 2 be scalars. Then for every v V we have v, T (α 1 w 1 + α 2 w 2 ) = T v, α 1 w 1 + α 2 w 2 Since this holds for every v V, it follows that Theorem 29. (1) (T ) = T (2) (S + T ) = S + T (3) (αt ) = αt (4) (ST ) = T S = α 1 T v, w 1 + α 2 T v, w 2 = α 1 v, T w 1 + α 2 v, T w 2 = v, α 1 T w 1 + α 2 T w 2 T (α 1 w 1 + α 2 w 2 ) = α 1 T w 1 + α 2 T w 2. Theorem 30. Let V and W be finite dimensional inner product spaces and let T be a linear transformation from V to W. Let A be the matrix of T with respect to ordered orthonormal bases. Then the matrix of T is A T. This is an immediate consequence of Corollary 15 and the definition of the adjoint transformation. Of course, in the real case, the bar over A can be omitted.

8 INNER PRODUCTS 8 Definition 31. A linear operator T on a finite dimensional inner product space is self-adjoint if T = T. A real n n matrix that defines a self adjoint operator on R n is called real symmetric. A complex n n matrix that defines a self-adjoint operator on C n is Hermitian. Note that a real n n matrix is real symmetric if and only if A T = A. A complex n n matrix is Hermitian if and only if A T = A. Lemma 32. All eigenvalues of a self-adjoint operator are real. Proof. Let α be an eigenvalue of a self-adjoint operator with eigenvector v. Then so α = α. α v 2 = α v, v = αv, v = T v, v = v, T v = v, αv = α v 2 Note that we do not (yet) assert that eigenvalues exist. Corollary 33. Every Hermitian matrix has a real eigenvalue. Proof. An n n Hermitian matrix defines a self adjoint operator on C n, which has a complex eigenvalue by the Fundamental Theorem of Algebra. By the previous lemma, this complex eigenvalue must be real. Corollary 34. Every self-adjoint operator has a real eigenvalue. Proof. In the complex case, this is immediate from Lemma 32 and the Fundamental Theorem of Algebra. For the real case, it suffices to show that the matrix A of the transformation with respect to some orthonormal basis has a real eigenvalue. But the matrix is real symmetric, and hence also Hermitian, so, by the previous corollary, it has a real eigenvalue. 6. Orthogonal and Unitary Operators Theorem 35 (Polarization Identities). (1) If V is a real inner product space, then for every u, v V we have u, v = 1 ( u + v 2 u 2 v 2). 2 (2) If V is a complex inner product space, then for every u, v V we have u, v = 1 ( u + v 2 + i u + iv 2 u v 2 i u iv 2). 4 Proof. For any scalar α we have (3) u + αv 2 = u + αv, u + αv = u 2 + α u, v + α v, u + α 2 v 2 In the real case, we have v, u = u, v, and the result follows by taking α = 1. For the complex case, note that when α = 1, (3) becomes, upon multiplication by α α u + αv 2 = α u 2 + u, v + α 2 u, v + α v 2. In particular, taking α = 1, 1, i, and i gives u + v 2 = u 2 + u, v + v, u + v 2 u + v 2 = u 2 u, v + v, u v 2 i u + iv 2 = i u 2 + u, v v, u + i v 2 i u iv 2 = i u 2 + u, v v, u i v 2

9 INNER PRODUCTS 9 Adding the four identities above gives the desired result. Theorem 36. Let T be a linear operator on a finite dimensional vector space V. The following are equivalent. (1) For every u, v V we have T u, T v = u, v. (2) For every v V we have T v = v. (3) For every orthonormal basis {e 1,..., e n } for V, the set {T e 1,..., T e n } is also an orthonormal basis for V. (4) For some orthonormal basis {e 1,..., e n } for V, the set {T e 1,..., T e n } is also an orthonormal basis for V. (5) T T is the identity operator on V. (6) T T is the identity operator on V. Proof. The implication 1 = 2 is trivial, and the reverse implication is immediate from the Polarization Identity. The implications 1 = 3 = 4 are also trivial. 4 = 1: Let {e 1,..., e n } be an orthonormal basis. For u, v V we can write so u = α 1 e α n e n v = β 1 e β n e n T u = α 1 T e α n T e n T v = β 1 T e β n T e n. Since {T e 1,..., T e n } is an orthonormal basis, we have T u, T v = α i β i = u, v. We have now established the equivalence of 1 2. The equivalence of 5 and 6 is a consequence of the general fact that one sided inverses for linear operators on finite dimensional vector spaces are two sided inverses. We complete the proof by establishing equivalence of 5 and 1. 1 = 5: Let u V. Then for any v V we have T T u, v = T u, T v = u, v Since this is true for every v V, it follows that T T u = u. Since u V is arbitrary, it follows that T T is the identity operator on V. 5 = 1: If T T is the identity operator, then for any u, v V we have T u, T v = T T u, v = u, v. A linear operator on a real inner product space satisfying one, and hence all, of the conditions of Theorem 36 is called orthogonal. A linear operator on a complex inner product space satisfying one, and hence all, of the conditions of Theorem 36 is called unitary.

10 INNER PRODUCTS Spectral Theorem In this section, we identify the linear operators on a finite dimensional inner product space V which can be diagonalized by an orthogonal (in the real case) or unitary (in the complex case) transformation. Equivalently, we want to identify the linear the linear operators T on V such that V has an orthonormal basis consisting of eigenvectors for T. Suppose that T is a linear transformation on an inner product space V, and that {e 1,..., e n } is an orthonormal basis for V consisting of eigenvectors of T, with corresponding eigenvalues λ 1,..., λ n. Then T e j, e i = e j, T e i = e j, λ i e i = λ i δ ij. It follows that T e j = λ j e j. Therefore, T T e j = T T e j = λ j 2 e j. Since the operators T T and T T agree on a basis, we have T T = T T. Definition 37. A linear operator T on a finite dimensional inner product space is normal if T T = T T. For example, orthogonal, unitary, symmetric, and Hermitian operators are all normal. We have established the following. Lemma 38. If T is a linear operator on an inner product space V which admits an orthonormal basis consisting of eigenvectors for T, then T is normal. The Spectral Theorem asserts that the converse holds for linear operators on complex inner product spaces. Theorem 39 (Spectral Theorem). Let T be a normal operator on a finite dimensional complex inner product space V. Then V has an orthonormal basis consisting of eigenvectors for T. The proof of the Spectral Theorem requires some preliminary lemmas. Lemma 40. If T is a normal operator on V, then T v = T v for every v V. Proof. T v 2 = T v, T v = T T v, v = T T v, v = T v, T v = T v 2. Corollary 41. If T is normal then T and T have the same null space. Corollary 42. Let T be a normal operator with eigenvalue λ, and corresponding eigenvector v. Then λ is an eigenvalue for T with the same eigenvector v. Proof. Since T is normal, so is T λi, so T λi and (T λi) = T λi have the same null space. Lemma 43. Let T be a normal operator on V with eigenvalue λ. Let W be the corresponding eigenspace. Then W is invariant under T. Proof. Let v W. Then for any w W we have so T v W. T v, w = v, T w = v, λw = λ v, w = 0

11 INNER PRODUCTS 11 Proof of the Spectral Theorem. We use induction on the dimension n of V. If n = 1, then any unit vector in V forms an orthonormal basis. For n > 1, suppose by induction that the conclusion holds for every complex inner product space of dimension less than n. By the Fundamental Theorem of Algebra, the characteristic polynomial of T has a complex root λ, which must be an eigenvalue of T. Let W be the corresponding eigenspace, and let {e 1,..., e k } be an orthonormal basis for W. By Lemma 43, V 0 = W is invariant under T, so the restriction T 0 of T to V 0 is a linear operator on V 0. Further, the restriction, 0 of the inner product to V 0 V 0 is an inner product on V 0. One easily verifies that T0 is the restriction of T to V 0, and so T 0 is normal. Since the dimension of V 0 is less than n, it follows from our induction hypothesis that V 0 has an orthonormal basis {f 1,..., f l } consisting of eigenvalues for T 0. It follows that {e 1,..., e k, f 1,..., f l } is an orthonormal basis for V The real case. The conclusion of the Spectral Theorem may fail for normal operators on real inner product spaces for the simple reason that such operators need not have eigenvalues. For example, a rotation of the plane through an angle which is not a multiple of pi is normal but has no eigenvalues. Lemma 44. Let T be a linear operator on a finite dimensional real inner product space V. If V has an orthonormal basis consisting of eigenvectors of T, then T is self-adjoint. Proof. Let {e 1,..., e n } be an orthonormal basis of V consisting of eigenvectors for T, with corresponding eigenvalues λ 1,..., λ n. Arguing as in the proof of Lemma 38, we obtain T e j = λ n e j = T e j, so T and T agree on a bases, and are therefore equal. The real version of the Spectral Theorem asserts the converse. Theorem 45 (Real Spectral Theorem). If T is a self adjoint operator on a finite dimensional real inner product space V, then V has an orthonormal basis consisting of eigenvectors for T. The proof of the real version of the Spectral Theorem is essentially the same as the proof of the complex version, appealing to Corollary 34 instead of the Fundamental Theorem of Algebra for the existence of eigenvalues.