Generalized Inverses: How to Invert a Non-Invertible Matrix

Transcription

1 Generlized Inverses: How to Invert Non-Invertible Mtrix S. Swyer September 7, 2006 rev August 6, Introduction nd Definition. Let A be generl m n mtrix. Then nturl question is when we cn solve Ax y for x R m, given y R n. If A is squre mtrix m n nd A hs n inverse, then. holds if nd only if x A y. This gives complete nswer if A is invertible. However, A my be m n with m n, or A my be squre mtrix tht is not invertible. If A is not invertible, then eqution. my hve no solutions tht is, y my be not be in the rnge of A, nd if there re solutions, then there my be mny different solutions. 2 0 For exmple, ssume A. Then A, so tht 6 0 A is not invertible. It would be useful to hve chrcteriztion of those y R 2 for which it is possible to find solution of Ax y, nd, if Ax y is solution, to find ll possible solutions. It is esy to nswer these questions directly for 2 2 mtrix, but not if A were 8 or 0 0. A solution of these questions cn be found in generl from the notion of generlized inverse of mtrix: Definition. If A is n m n mtrix, then G is generlized inverse of A if G is n n m mtrix with AGA A.2 If A hs n inverse in the usul sense, tht is if A is n n nd hs two-sided inverse A, then while by.2 A AGAA A A G AA G A AA A AA A Thus, if A exists in the usul sense, then G A. This justifies the term generlized inverse. We will see lter tht ny m n mtrix A hs t lest one generlized inverse G. However, unless A is n n nd A is invertible,

2 Generlized Inverses there re mny different generlized inverses G, so tht G is not unique. Generlized inverses re unique is you impose more conditions on G; see Section below. One consequence of.2 is tht AGAGAG nd GAGAGA. In generl, squre mtrix P tht stisfies P 2 P is clled projection mtrix. Thus both AG nd GA re projection mtrices. Since A is m n nd G is n m, AG is n m m projection mtrix nd GA is n n. In generl if P is projection mtrix, then P P 2 implies P y P P y nd P z z for for ll z P y in the rnge of P. Tht is, if P is n n, P moves ny x R n into V {P x : x R n } the rnge of P nd then keeps it t the sme plce. If x R n, then y P x nd z x P x I P x stisfies x y + z, P y y, nd P z P x P x P x P 2 x 0. Since then P x P y + z y, we cn sy tht P projects R n onto its rnge V long the spce W {x : P x 0}. The two projections AG nd GA both pper in the next result, which shows how generlized inverses cn be used to solve mtrix equtions. Theorem.. Let A by n m n mtrix nd ssume tht G is generlized inverse of A tht is, AGA A. Then, for ny fixed y R m, i the eqution Ax y, x R n. hs solution x R n if nd only if AGy y tht is, if nd only if y is in the rnge of the projection AG. ii If Ax y hs ny solutions, then x is solution of Ax y if nd only if x Gy + I GAz for some z R n.4 Remrk. If we wnt prticulr solution of Ax y for y in the rnge of A, we cn tke x Gy. Proof of Theorem.. All of the prts of the theorem re esy to prove, but some involve somewht unintuitive mnipultions of mtrices. Proof of prt i: If y is in the rnge of the projection AG, tht is if AGy y, then AGy y nd x Gy is solution of Ax y. Conversely, if Ax y, then GAx Gy nd AGAx AGy AGy, while AGA A implies AGAx Ax y. Thus AGy y. Thus, if Ax y hs ny solutions for given y R m, then x Gy is prticulr solution. Proof of prt ii: This hs two prts: First, if AGy y, then ll of the vectors in.4 re solutions of Ax y. Second, tht.4 contins ll possible solutions of Ax y.

3 Generlized Inverses If AGy y nd x Gy + I GAz, then Ax AGy + AI GAz y + A AGAz y, so tht ny x R n tht stisfies.4 with AGy y is solution of Ax y. Conversely, if Ax y, let z x. Then the right-hnd side of.4 is Gy +I GAx Gy +x GAx Gy +x Gy x, so tht ny solution x of Ax y is given by.4 with z x. 0 Exmple. Let A s before. Set G. Then AGA A 6 so tht G is generlized inverse of A. The two projections ppering in Theorem. re 0 AG nd GA 0 In this cse A x y 6 x x + 2y y x + 6y Thus Ax y hs solution x only if y c AG x y 0 0 x y x x x + 2y x. On the other hnd, so tht the rnge of the projection AG is exctly the set of vectors { c The theorem then sys tht if y c }., then the set of solutions of Ax y is exctly z x Gc + I GA z z c + 0 z z 2 c + c + z 0 2 z 2.5 It is esy to check tht Ax y c for ll x in.5, nd, with some extr work, tht these re ll solutions.

4 Generlized Inverses The ABCD-Theorem nd Generlized Inverses of Arbitrry Mtrices. Let A be n rbitrry m n mtrix; tht is, with n columns nd m rows. Then we cn write 2... n n A v v 2... v n m m2... mn w w w m where v i re the columns of A nd w j re the rows. In generl, the column rnk of A cll it r c is the dimension of the vector spce in R m tht is spnned by the columns {v i } of A, nd the row rnk of A cll it r r is the dimension of the vector spce in R n tht is spnned by the rows {w j } of A. Tht is, r c is the lrgest number of linerly-independent columns v i in R m, nd r r is the the lrgest number of linerly-independent rows w j in R n. Then r c m, since the lrgest number of linerly independent vectors in R m is m, nd r c n since there re only n columns to begin with. Thus r c min{m, n}. By the sme rguments, r r min{m, n}. It cn be shown tht r c r r for ny m n mtrix, so tht the row rnk nd the column rnk of n rbitrry mtrix A re the sme. The common vlue r r c r r min{m, n} is clled the rnk of A. Let A be n m n mtrix with rnka r min{m, n} s bove. Then, one cn show tht, fter suitble rerrngement of rows nd columns, A cn be written in prtitioned form s A 2. c d where is r r nd invertible, b is r n r, c is m r r, nd d is m r n r. In fct, we cn prove the following representtion theorem for generl mtrices: Theorem 2.. Let A is n m n mtrix with rnk r rnka. Then the rows nd columns cn be permuted so tht it cn be written in the prtitioned form 2. where is r r nd invertible. In tht cse d c b, so tht A c c.2 b Note tht, b, c, d in 2. nd 2.2 re mtrices, not numbers. Some of the entries b, c, d in 2. my be empty, in which cse they do not pper, for exmple if m n nd A is invertible.

5 Generlized Inverses Remrks. If A is 2 2 mtrix of numbers with > 0 but c d r rnka, then deta d bc 0. This implies d bc/. We cnnot write bc/ for mtrices, but 2.2 with d b c is the pproprite generliztion for mtrices. The mtrix d c b is lwys defined nd is m r n r, since c is m r r, is r r, nd b is r n r. Exmple. Let A xy be the outer product of vectors x R m nd y R n, so tht A is m n. Assume x 0 nd y 0. Then rnka since every row of A is multiple of y nd every column of A is multiple of x. In this cse, we cn write A xy y x y x y 2... y n x 2... x 2y 2... x 2 y n x m x m y 2... x m y n This is in the form 2.2 where b x y 2... y n is n row vector, c y x 2... x m is n m column vector, nd d b c y x 2... x m x y x y 2... y n x 2... x m y 2... y n is the outer product of n m -dimensionl vector nd n n - dimensionl vector. Remrk. A Note tht 2.2 cn lso be written Ir c c I c r b This cn be viewed s generliztion of the representtion A uv for n outer product of two vectors u, v. Proof of Theorem 2.. If the first r rows of A re linerly independent nd rnka rnk r in 2., then the lst m r rows of A re liner combintions of the first r rows. This mens tht we cn write the lst m r rows of A s c d i r T ij j for i m r j

6 Generlized Inverses where T ij i m r, j r re numbers. In terms of mtrices, c d T T T b 2. where T is m r r. The reltion 2. implies c T nd hence T c. This implies T b c b d in 2., which completes the proof of Theorem 2.. Theorem 2.2. Let A c d c c b 2.4 be n m n mtrix with r rnka where is r r nd invertible, s in Theorem 2.. Let 0 G 2.5 where the 0 s in 2.5 represent mtrices of zeroes of dimension sufficient to mke G n n m mtrix. Then G is generlized inverse of A. Proof. By 2.4 nd 2.5 c 0 d c Ir 0 c 0 c d d c c b where I r is the r r unit mtrix. This implies AGA A since d c b by 2.4, so tht G is generlized inverse of A. The two projections in this cse re AG Ir 0 c 0 nd GA Theorem. then sys tht Ax y y 2 R m r if nd only if AGy Ir 0 y c 0 y 2 y y 2 Ir b y c y cn be solved for y R r, y y 2

7 Generlized Inverses Tht is, if nd only if y 2 c y. In tht cse, the generl solution of Ax y for x R n is x x Gy + I x m GAz 2 0 y 0 + b z y 2 0 I m r z 2 y + bz 2 0 for rbitrry z 2 R m r. Remrk. This shows tht ny m n mtrix A hs t lest one generlized inverse G of the form 2.5. Since often mny different linerly-independent sets of r rows cn be permuted to the upper r rows nd mny different linerly-independent sets of r columns cn be permuted into the first r column positions, mtrix A with rnka r < n cn hve mny different generlized inverses of this form.. The Penrose Inverse. In generl, n m n mtrix A hs mny different generlized inverses unless m n nd A is invertible. It is possible, however, to dd conditions to the definition of generlized inverse so tht there is lwys unique generlized inverse under the dditionl conditions. Definition. G is clled Penrose inverse of the m n mtrix A if G is n n m mtrix tht stisfies the four conditions i AGA A ii GAG G iii AG AG is n orthogonl projection in R m iv GA GA is n orthogonl projection in R n Condition ii sys tht A is generlized inverse of G, in ddition to G being generlized inverse of A. The fct thn n rbitrry m n mtrix A hs unique n m Penrose inverse follows from the Singulr Vlue Decomposition theorem in mtrix lgebr. Some generlized inverses tht re nturl to use in prctice re Penrose inverses nd some re not. The next section gives n exmple of Penrose inverse. 4. Fitted Vlues in Sttistics. Let X be n n r mtrix with r < n nd rnkx r. Then X X is invertible. If observed vlues Y R n cn be exctly fit by the prmeters β R r, then z 2 Y Xβ, Y R n, β R r 4.

8 Generlized Inverses The mtrix X cnnot be invertible, since r < n. However, suppose tht we wnt generl procedure to choose n rbitrry β in terms of Y, in the hopes tht lter we cn find justifiction for this procedure other thn it gives definite nswer. In tht cse, we cn consider generlized inverse of X. Specificlly, G will be generlized inverse of X if G is r n nd XGX X Since X X is invertible, n obvious choice is G X X X 4.2 since then XGX XX X X X X. The two projections XG nd GX re GX X X X X I r nd XG XX X X H 4. Note tht both projections re symmetric: Tht is, I r I r nd H H. In ddition GXG X X X XX X X X X X G Tht is, G is the unique Penrose inverse of the n r mtrix X. Theorem. now sys tht Y Xβ cn be solved exctly if nd only if XGY HY Y ; tht is, if nd only if Y is in the rnge of the n n projection H. Moreover, if HY Y, then every solution of Xβ Y is of the form β GY X X X Y + I r GXz, z R r X X X Y 4.4 since GX I r by 4.. In other words, if Y Xβ for some vector β, then the only solution β of Xβ Y for given Y is given by 4.4. Indeed, it follows directly from 4. tht X must be one-one: Tht is, if Xβ Xβ 2, then GXβ GXβ 2 β β 2. There is etter motivtion for the solution β GY for G in 4.2 thn rbitrriness or orneriness. Suppose tht we view Y s Xβ tht re observed with errors. Tht is, s Y Xβ + e 4.5

9 Generlized Inverses where e {e i } re independent errors. Then we cn consider the vlue of β tht minimizes the sum of errors min β n 2 n Yi Xβ i Yi X β 2 i 4.6 i i If we set This implies n 2 n Yi Xβ i Y i β j β i j i n r 2 Y i X i β X ij 0 i r 2 X i β r n X ij X i β i n X ij Y i, i j r which cn be written in more compct form s X Xβ X Y 4.7 Since we re ssuming tht X X is invertible, 4.7 implies β X X X Y GY for G in 4.2. Tht is, the lest-squres solution of 4.6 for β is given by β β GY, where G is the Penrose inverse of the n r mtrix X.