The Four Fundamental Subspaces: 4 Lines

Transcription

1 he Four Fundamental Subspaces: Lines Gilbert Strang, Massachusetts Institute of echnology. Introduction. he epression Four Fundamental Subspaces has become familiar to thousands of linear algebra students. hose subspaces are the column space and the nullspace of A and A. hey lift the understanding of A D b to a higher level a subspace level. he first step sees A (matri times vector) as a combination of the columns of A. hose vectors A fill the column space C.A/. When we move from one combination to all combinations (by allowing every ), a subspace appears. A D b has a solution eactly when b is in the column space of A. he net section of this note will introduce all four subspaces. hey are connected by the Fundamental heorem of Linear Algebra. A perceptive reader may recognize the Singular Value Decomposition, when Part of this theorem provides perfect bases for the four subspaces. he three parts are well separated in a linear algebra course! he first part goes as far as the dimensions of the subspaces, using the rank. he second part is their orthogonality two subspaces in R n and two in R m. he third part needs eigenvalues and eigenvectors of A A to find the best bases. Figure will show the big picture of linear algebra, with the four bases added in Figure. he main purpose of this paper is to see that theorem in action. We choose a matri of rank one, A D y. When m D n D, all four fundamental subspaces are lines in R. he big picture is particularly clear, and some would say the four lines are trivial. But the angle between and y decides the eigenvalues of A and its Jordan form those go beyond the Fundamental heorem. We are seeing the orthogonal geometry that comes from singular vectors and the skew geometry that comes from eigenvectors. One leads to singular values and the other leads to eigenvalues. Eamples are amazingly powerful. I hope this family of by matrices fills a space between working with a specific numerical eample and an arbitrary matri.. he Four Subspaces. Figure shows the fundamental subspaces for an m by n matri of rank r. It is useful to fi ideas with a by matri of rank : A D : hat matri is in row reduced echelon form and it shows what elimination can accomplish. he column space of A and the nullspace of A have very simple bases: and span C.A/; spans N.A /: After transposing, the first two rows of A are a basis for the row space and they also tell us a basis for the nullspace:

2 and span C.A /; and span N.A/: he last two vectors are orthogonal to the first two. But these are not orthogonal bases. Elimination is enough to give Part of the Fundamental heorem: Part he column space and row space have equal dimension r D rank he nullspace N.A/ has dimension n r; N.A / has dimension m r hat counting of basis vectors is obvious for the row reduced rref.a/. his matri has r nonzero rows and r pivot columns. he proof of Part is in the reversibility of every elimination step to confirm that linear independence and dimension are not changed. C.A / dim r dim n r Row space all A y R n in Nullspace A D N.A/ Perpendicular.A y/ D Perpendicular y.a/ D Column space all A R m C.A/ dim r y in dim m r Nullspace of A A y D N.A / Figure : Dimensions and orthogonality for any m by n matri A of rank r. Part says that the row space and nullspace are orthogonal complements. he orthogonality comes directly from the equation A D. Each in the nullspace is orthogonal to each row: A D.row /.row m/ D is orthogonal to row is orthogonal to row m he dimensions of C.A / and N.A/ add to n. Every vector in R n is accounted for, by separating into row C null. For the 9 ı angle on the right side of Figure, change A to A. Every vector b D A in the column space is orthogonal to every solution of A y D. More briefly,.a/ y D.A y/ D. Part C.A / D N.A/? Orthogonal complements in R n N.A / D C.A/? Orthogonal complements in R m

3 Part of the Fundamental heorem creates orthonormal bases for the four subspaces. More than that, the matri is diagonal with respect to those bases u ; : : : ; u n and v ; : : : ; v m. From row space to column space this is Av i D i u i for i D ; : : : ; r. he other basis vectors are in the nullspaces: Av i D and A u i D for i > r. When the u s and v s are columns of orthogonal matrices U and V, we have the Singular Value Decomposition A D U V : : Part AV D A v v r v n D u u r u m r D U : he v s are orthonormal eigenvectors of A A, with eigenvalue i. hen the eigenvector matri V diagonalizes A A D.V U /.U V / D V. /V. Similarly U diagonalizes AA. When matrices are not symmetric or square, it is A A and AA that make things right. his summary is completed by one more matri: the pseudoinverse. his matri A C inverts A where that is possible, from column space back to row space. It has the same nullspace as A. It gives the shortest solution to A D b, because A C b is the particular solution in the row space: AA C b D b. Every matri is invertible from row space to column space, and A C provides the inverse: Pseudoinverse A C u i D v i i for i D ; : : : ; r: Figure : Orthonormal bases that diagonalize A ( by ) and A C ( by ).

4 Figure shows the four subspaces with orthonormal bases and the action of A and A C. he product A C A is the orthogonal projection of R n onto the row space as near to the identity matri as possible. Certainly A C is A when that inverse eists.. Matrices of Rank One. Our goal is a full understanding of rank one matrices A D y. he columns of A are multiples of, so the column space C.A/ is a line. he rows of A are multiples of y, so the row space C.A / is the line through y (column vector convention). Let and y be unit vectors to make the scaling attractive. Since all the action concerns those two vectors, we stay in R : A D y y D y : he trace is y y y C y D y: he nullspace of A is the line orthogonal to y. It is in the direction of y? D.y ; y /. he algebra gives Ay? D.y /y? D and the geometry is on the left side of Figure. he good basis vectors are y and y?. On the right side, the bases for the column space of A and the nullspace of A D y are the orthogonal unit vectors and?. C.A / y? y? N.A / N.A/ C.A/ Figure : he fundamental subspaces for A D y are four lines in R.. Eigenvalues of y. he eigenvalues of A were not mentioned in the Fundamental heorem. Eigenvectors are not normally orthogonal. hey belong to the column space and the nullspace, not a natural pair of subspaces. One subspace is in R m, one is in R n, and they are comparable (but usually not orthogonal) only when m D n. he eigenvectors of the singular by matri A D y are and y? : Eigenvectors A D.y / D.y / and Ay? D.y /y? D : he new and crucial number is that first eigenvalue D y D cos. his is the trace since D. he angle between row space and column space decides the orientation in Figure. he etreme cases D and D = produce matrices of the best kind and the worst kind: Best cos D when D y: hen A D is symmetric with D ; Worst cos D when D y? : hen A D y? y has trace zero with D ; Worst is a short form of nondiagonalizable. he eigenvalue D is repeated and the two eigenvectors and y? coincide when the trace y is zero. At that point A D y

5 cannot be similar to the diagonal matri of its eigenvalues (because this will be the zero matri). he right choice of Q AQ will produce the Jordan form in this etreme case when and y are orthonormal: J D y y y D y D Jordan chose the best basis ( and y) to put y in that famous form, with an offdiagonal to signal a missing eigenvector. he SVD will choose two different orthonormal bases to put y in its diagonal form.. Factorizations of A D y. By bringing together three important ways to factor this matri, you can see the end result of each approach and how that goal is reached. We still have kk D kyk D and y D cos. he end results are ; ƒ, and. A. Singular Value Decomposition U AV D D B. Diagonalization by eigenvectors S AS D cos D ƒ cos sin C. Orthogonal triangularization Q AQ D D he columns of U; V; S, and Q will be ; y;?, and y?. hey come in different orders! A. In the SVD, the columns of U and V are orthonormal bases for the four subspaces. Figure shows u D in the column space and v D y in the row space. hen Ay D.y /y correctly gives with D. he nullspace bases are u D? and v D y?. Notice the different bases in U and V, from the reversal of and y: U AV D? y y y? D? D D he pseudoinverse of y is y. he norm of A is D. B. In diagonalization, the eigenvectors of A D y are and y?. hose are the columns of the eigenvector matri S, and its determinant is y D cos. he eigenvectors of A D y are y and?, which go into the rows of S (after division by cos ). S A S D cos y? y y? D y D cos D ƒ his diagonalization fails when cos D and S is singular. he Jordan form jumps from ƒ to J, as that off-diagonal suddenly appears. C. One of the many useful discoveries of Isaac Schur is that every square matri is unitarily similar to a triangular matri: Q AQ D with Q Q D I:

6 His construction starts with the unit eigenvector in the first column of Q. In our by case, the construction ends immediately with? in the second column: Q y AQ D? y? cos sin D? D D his triangular matri still has norm, since Q is unitary. Numerically is far more stable than the diagonal form ƒ. In fact survives in the limit cos D of coincident eigenvectors, when it becomes J. Note he triangular form is not so widely used, but it gives an elementary proof of a seemingly obvious fact: A random small perturbation of any square matri is almost sure to produce distinct eigenvalues. What is the best proof? More controversially, I wonder if Schur can be regarded as the greatest linear algebraist of all time? Summary he four fundamental subspaces, coming from A D y and from A D y, are four lines in R. heir directions are given by ;? ; y; and y?. he eigenvectors of A and A are the same four vectors. But there is a crucial crossover in the pictures of Figures --. he eigenvectors of A lie in its column space and nullspace, not a natural pair. he dimensions of the spaces add to n D, but the spaces are not orthogonal and they could even coincide. he better picture is the orthogonal one that leads to the SVD. References hese are among the tetbooks that present the four subspaces.. David Lay, Linear Algebra and Its Applications, hird edition, Addison-Wesley ().. Peter Olver and Chehrzad Shakiban, Applied Linear Algebra, Pearson Prentice-Hall ().. heodore Shifrin and Malcolm Adams, Linear Algebra: A Geometric Approach, Freeman ().. Gilbert Strang, Linear Algebra and Its Applications, Fourth edition, Cengage (previously Brooks/Cole) ().. Gilbert Strang, Introduction to Linear Algebra, hird edition, Wellesley-Cambridge Press ().