In the rth step, the computaton of the Householder matrx H r requres only the n ; r last elements of the rth column of A T r;1a r;1 snce we donothave

What are the relatve errors?
What values of the bdagonalzaton scheme produces?
What is the name of the matrces A = DX?

Transcription

1 An accurate bdagonal reducton for the computaton of sngular values Ru M. S. Ralha Abstract We present a new bdagonalzaton technque whch s compettve wth the standard bdagonalzaton method and analyse the numercal accuracy of such technque. We show that, for some matrces, our algorthm produces sngular values wth low relatve errors. 1 Introducton In [10] we proposed a new algorthm for reducng a rectangular matrx A to bdagonal form from where the sngular values of A can be computed teratvely usng the classcal method due to Golub and Kahan or any of the alternatve algorthms gven n [6], [9] and [12]. In [13] we presented results of numercal tests carred out on a CRAY Y-MP EL and compared the speed of our code (n Fortran77) to the speed of: the CRAY's lbrary module for SVD, the NAG's routne where R-bdagonalzaton as proposed n [4] s mplemented, and the LAPACK's routne these results haveshown that our algorthm s very compettve n terms of performance and we clamed that t s a serous alternatve for computng the sngular values of large matrces. However, at the tme we had not fully understood the behavour of our method for matrces wth very small sngular values, more precsely we were not able to explan the errors n the smallest sngular values of almost rank decent matrces. We have done sgncant progress n ths matter snce we have dented the cause of such errors and know howtomprove the orgnal method n order to produce accurate small sngular values. Furthermore, we found that for matrces A = DX where D s dagonal and X s well condtoned, our bdagonalzaton scheme, just lke the one-sded Jacob algorthm, produces sngular values wth low relatve errors. 2 A new bdagonalzaton method Our bdagonalzaton strategy conssts of two stages. In the rst one (dubbed \trorthogonalzaton") we perform n ; 2 Householder transformatons A r := A r;1 H r r =1 n; 2 (1) where A 0 represents the ntal m-by-n matrx A. The basc dea s to select the approprate Householder matrces H r n (1) such that the resultng rectangular matrx A n;2 s \trorthogonal",.e.the columns a of A n;2 satsfy a T a j =0 for j ; jj > 1 (2) Notng that (2) s equvalent to say that the n-by-n symmetrc matrx A T n;2 A n;2 s trdagonal, we conclude that to produce n (1) a rectangular \trorthogonal" matrx, we just need to use the Householder matrces that, f appled on both sdes of the symmetrc matrx A T A would produce a smlar trdagonal form T n the well-known reducton T := H n;2 H 1 ; AT A H 1 H n;2 =(AH 1 H n;2 ) T (AH 1 H n;2 ) (3) Centre of Mathematcs, Unversty of Mnho, Braga, Portugal. emal: r ralha@math.umnho.pt 1

2 In the rth step, the computaton of the Householder matrx H r requres only the n ; r last elements of the rth column of A T r;1a r;1 snce we donothave ths product explctly formed, we just need to compute the n ; r dot products wth the approprate columns of A r;1. The second stage of our algorthm s a varant of the moded Gram-Schmdt orthogonalzaton method (MGS) and produces an upper bdagonal matrx B n the decomposton A n;2 = QB, where Q s orthogonal. Such decomposton requres only O(mn) flops snce the property (2) holds. In the followng we wll refer to the procedure that mplements ths decomposton as TRIMGS. 3 Accuracy of the sngular values of A n;2 Snce the reducton of the ntal matrx to \trorthogonal" form nvolves n ; 2 Householder transformatons, we can expect the process to be numercally stable. In fact, we have, wth beng the machne precson, Theorem 1 The \trorthogonalzaton" algorthm s backward stable,.e. for the computed \trorthogonal" matrx e A n;2 := A e H 1 e H2 e H n;2 we have ea n;2 =(A + E) H 1 H 2 H n;2 wth kek 2 =(n ; 2):O(): kak 2 (4) Proof. To prove ths result, we smply need to adapt the proof gven n [14], p. 124, for a sequence of orthogonal transformatons appled to both sdes of A From (4) we conclude, usng a well-kown perturbaton theorem, that the sngular values e of the \trorthogonal" matrx A e n;2 are close to the sngular values of A n the sense that j ; e j(n ; 2):O(): kak 2 (5) holds for =1 n. Ths s a satsfactory error bound for large sngular values (those near kak 2 ) snce most of the computed dgts of e wll concde wth the correct ones. However, such bound s not so good for sngular values much smaller than kak 2 whch may exhbt very large relatve errors. It s known that the one-sded Jacob method can compute all sngular values to hgh relatve accuracy for matrces A = DX, where D s dagonal and X s well-condtoned (see [14], pp ). The relatve errors n the sngular values b of the matrx b G obtaned from A wth post-multplcaton of m successve Gvens rotatons, satsfy the followng bound j ; b j O(m") 2 (X) (6) Interestngly, the proof of ths result can be adapted to our \trorthogonalzaton" method, snce t can be shown that such proof, gven n [14], can be extended to any one-sded orthogonal transformaton. Therefore, the bound (6) holds, wth m = n ; 2, for the sngular values of the \trorthogonal" matrx e A n;2. 4 Loss of orthogonalty and errors In practce, we have experenced that very small sngular values of the bdagonal form produced wth our method, let us call t ORTHOGSVD, may exhbt large absolute errors. Is ths because we are usng MGS that s known to produce a matrx whch may be far from orthogonal n the case of ll-condtoned matrces? Snce Bjorck has shown [8] that MGS produces a trangular matrx whch s numercally as good as that from the Householder QR factorzaton,we conclude that the sngular values of the trangular matrx are close to those of the ntal matrx. However, our procedure TRIMGS bulds an upper bdagonal form, assumng that a complete QR decomposton would produce a trangular matrx R wth neglgble elements r j for j>+1. Unfortunately, we can not guarantee ths, n general, snce for ll-condtoned matrces there 2

3 may appear large jr j j above the bdagonal form for nstance, for the \trorthogonal" matrx obtaned from the Hlbert matrx of order n =11 the last four columns of R are gven n Table 1: ;2:7493e ; 17 4:4252e ; 18 ;5:9938e ; 18 1:1687e ; 17 ;7:3356e ; 17 1:3068e ; 17 ;1:8988e ; 17 3:7100e ; 17 2:9805e ; 16 ;5:3282e ; 17 7:8085e ; 17 ;1:5086e ; 16 1:3420e ; 15 ;2:3995e ; 16 3:5153e ; 16 ;6:7919e ; 16 ;6:7391e ; 15 1:2050e ; 15 ;1:7654e ; 15 3:4108e ; 15 3:8096e ; 14 ;6:8119e ; 15 9:9794e ; 15 ;1:9281e ; 14 ;4:0374e ; 08 4:3893e ; 14 ;6:4303e ; 14 1:2424e ; 13 5:3952e ; 09 7:4512e ; 10 ;4:8120e ; 13 9:2972e ; 13 8:3168e ; 11 ;2:8831e ; 12 ;8:3296e ; 12 3:0252e ; 13 7:2449e ; 13 8:8244e ; 15 Table 1: Last four columns of R produced by the MGS method appled to the \trorthogonal" matrx obtaned from the Hlbert matrx of order n =11. We notce that there are elements above the upper bdagonal form of absolute value as large as 10 ;12 and 10 ;13, therefore errors of ths sze may aect the sngular values of the bdagonal form produced wth TRIMGS. Ths actually happens snce the two smallest sngular values computed wth ORTHOGSVD are 5:7089e ; 013 and 8:4136e ; 012 whereas the correct values, wth ve sgncant dgts, are 3:3932e ; 015 and 7:8071e ; 013: The problem s that two columns, say a (1) and a (1),ofA j n;2 may be far from orthogonal, even f a (1)T a (1), when at least one of the norms jja (1) jj or jja (1) jj s very small n other j j words, A n;2 s truly \trorthogonal", to workng precson, only f the quanttes T c (1) := a(1) a (1) j j jja (1) jj jja (1) jj j (7) are close to the machne precson, for j>+1. We have nvestgated how large values jc (1) j ntroduce mportant errors n the bdagonal j form produced wth TRIMGS. For the elements r 1j of the rst row ofr we have r 1j := jja (1) jj:c (1) j =2 ::: n (8) j 1j Therefore, jr 1j j can be much smaller than jc (1) j provded jja(1) jj s small,.e. loss of orthogonalty 1j j of the columns a (1), j>2 relatvely to the rst column a (1) j 1, s harmless to the accuracy of the computed R snce such loss of accuracy occurs gradually wth decreasng jja (1) jj. For ths reason, j we expect to have n all cases jr 1j jkak, j =3 ::: n (9) Representng by a (j) the th column of the matrx under transformaton, after makng t orthogonal to q j;1,wehave proved that, for each j =3 n jr j j t jr ;1 j j jja(;1) jj =2 jja () j ; 2 (10) jj Ths gves an estmate of the growth of the sze of the elements r 2 j r j;2 j n the jth column of R, startng wth r 1j : In practcal tests, we found ths estmate to be qute accurate. 3

4 5 Reorthogonalzaton of columns From the analyss carred out n the prevous secton, t becomes clear that for the bdagonal form produced wth TRIMGS to have accurate sngular values, the elements r j have to be neglgble, for j = 3 n and = 1 j ; 2: Ths wll happen f A n;2 s \trorthogonal" to workng precson and n some cases ths may requre reorthogonalzaton of the columns of A n;2.e. we apply the same procedure twce, the rst tme to produce A n;2 and the second tme to mprove the \trorthogonalty" of ths matrx. Havng completed ths procedure wth the Hlbert matrx of order n =11 the MGS method appled to the resultng \trorthogonal" matrx produces an upper trangular matrx whose last four columns are gven n Table 2: ;5:9847e ; 025 8:3919e ; 027 4:3511e ; 030 ;6:2775e ; 029 ;1:2327e ; 024 1:5856e ; 026 1:6832e ; 028 ;1:6234e ; 028 ;2:8172e ; 024 7:6665e ; 026 ;8:9748e ; 028 2:4253e ; 028 ;7:6227e ; 024 ;6:5261e ; 026 ;4:6777e ; 027 1:1409e ; 027 4:5097e ; 023 ;2:8395e ; 025 2:4386e ; 026 ;5:8864e ; 027 ;2:4042e ; 022 1:3565e ; 024 ;1:3398e ; 025 3:4580e ; 026 4:0374e ; 008 ;8:7048e ; 024 8:7033e ; 025 ;2:1923e ; 025 5:3952e ; 009 7:4513e ; 010 6:5099e ; 024 ;1:6409e ; 024 8:3159e ; 011 ;8:8968e ; 012 1:4707e ; 023 7:8320e ; 013 5:6476e ; 014 3:4089e ; 015 Table 2: Last four columns of R produced by the MGS method appled to the trorthogonal matrx (wth reorthogonalzaton) obtaned from the Hlbert matrx of order n = 11. These values are to be compared wth those gven n Table 1 and show that the elements r j above the bdagonal form are neglgble. Therefore, we expect the bdagonal form computed wth the procedure TRIMGS to be accurate (n fact, ts elements concde wth those of the correspondng dagonals of R up to the machne precsson). Of course, the use of reorthogonalzaton doubles the arthmetc complexty of the process of producng a \trorthogonal" matrx. We have not devsed so far an efectve sheme to mplement selectve reorthogonalzaton n ths procedure. It must be stressed out that not all ll-condtoned matrces requre the use of reorthogonalzaton. In the next secton we gve some examples of such matrces. 6 Numercal results In ths secton we dscuss the numercal results obtaned wth our procedure n the case of selected matrces. Lauchl matrces ALauchl matrx s a (n +1) n matrx of the type L(n ) = The sngular values of L(n ) are 1 =,ofmultplcty n ; 1, and 2 = p n + 2. For small values of, L(n ) s ll-condtoned, therefore we are nterested n testng ORTHOGSVD for 4

5 matrces of ths type. We found that, n most cases, the method produces sngular values wth very low relatve errors, even for very small values. For nstance, wth = p almost all sngular values exhbt a relatve error of the order of magntude of although for certan szes n one of the sngular values s aected by a larger relatve error. In Table 3, the maxmum relatve error produced wth ORTHOGSVD s shown for some values n: Wth = agan all but one of the sngular values have relatve errors close to but one sngular value exhbts a very large relatve error, even for small szes n: For nstance, for n =7 we get one sngular value equal to 5:4390e ; 016 whch has not a sngle sgncant dgt correct, snce t 2:2204e ; 016. The gan n accuracy due to the use of reorthogonalzaton s dramatc n the case of = as t can be apprecated n Table 4. From Table 3, we can also conclude that ORTHOGSVD wth reorthogonalzaton s more accurate than the procedure svd of Matlab (for = the relatve errors of the sngular values produced wth svd are essentally the same as for = p ). n (a) 5.5e e e e e e-13 max j;ej (b) 8.8e e e e e e-15 (c) 4.2e e e e e e-13 Table 3: Maxmum relatve error n the sngular values of L(n p ), computed wth ORTHOGSV D, wth (a) and wthout reorthogonalzaton (b) and wth the functon svd of Matlab (c). n max j;ej (a) 5.5e e e e e e+01 (b) 8.8e e e e e e-15 Table 4: Maxmum relatve error n the sngular values of L(n ), computed wth ORTHOGSVD, wthout (a) and wth reorthogonalzaton (b). 6.1 Random matrces produced wth RANDSVD The Matlab's functon randsvd [7] generates a random matrx wth pre-assgned sngular values. Used n the form A = randsvd(n k 1), t produces a square matrx of order n, wth a sngle sngular value equal to one and n ; 1 sngular values equal to 1=k: In Table 5 the maxmum absolute error n the smallest n ; 1 sngular values of matrces of ths type, produced wth k =10 7 s gven (wth k =10 15 we obtaned smlar absolute errors). max 2n n j ; e j (a) 5.5e e e e e e-17 (b) 5.5e e e e e e-17 Table 5: Maxmum absolute error n the smallest n ;1 sngular values of A = randsvd(n ), computed wth ORTHOGSVD, wthout reorthogonalzaton (a), and wth the functon svd of Matlab (b). 6.2 A=DX Fnally, we consder the example used n [14], pp , to llustrate the hablty of the onesded Jacob algorthm to compute the sngular values of A wth small relatve errors, accordng to (6), when A = DX, whered s dagonal and X s well-condtoned. The matrx G

6 wth = 10 ;20 has the sngular values (to at least 16 dgts) p 3, p 3,, : As shown by Demmel, the classcal reducton to bdagonal form produces a matrx whose three smallest sngular values are very naccurate approxmatons of the true values. In constrast, the onesded Jacob method produces accurate sngular values. We nowshow, that n the case of ths matrx our method also produces very accurate sngular values. A straghtforward Matlab's mplementaton of our method appled to matrx G produces a matrx B whch has the followng sngular values (computed wth MAPLE usng 30 decmal dgts arthmetc) 1 =1: =1: ;20 3 =9: ;21 4 =9: ;21 whch concde wth p 3, p 3,,, respectvely, to at least 16 decmal dgts. References [1] J.K.H.Wlknson, The Algebrac Egenvalue Problem, The Oxford Unversty Press, 1965 [2] A. Bjorck, Solvng Lnear Least Squares Problems by Gram-Schmdt Orthogonalzaton, BIT 7:1-21 (1967). [3] B.N.Parlett, The Symmetrc Egenvalue Problem, Prentce-Hall, Inc., [4] T. F. Chan, An Improved Algorthm for Computng the Sngular Value Decomposton, ACM Trans. on Math. Software, vol.8: (1982) [5] G. H. Golub, C. F. Van Loan, Matrx Computatons, 2nd ed., The Johns Hopkns Unversty Press, [6] J. Demmel, W. Kahan, Accurate Sngular Values of Bdagonal Matrces, SIAM J. Sc. Sta Comput. 11: (1990). [7] N. Hgham, A Collecton of Test Matrces n Matlab, ACM Trans. on Math. Software, vol.17, N.3: (1991). [8] A. Bjorck, Numercs of Gram-Schmdt Orthogonalzaton, Lnear Algebra and ts Applcatons 197, 198: (1994). [9] K. Vnce Fernando, Beresford N. Parlett, Accurate Sngular Values and Derental QD Algorthms, Numersche Mathematk, vol.67: (1994). [10] R. M. S. Ralha, A New Algorthm for Sngular Value Decompostons, Proc. of the II Euromcro Work. on Parallel and Dstrbuted Processng, , IEEE Computer Socety Press (1994). [11] C. L. Lawson, R. J. Hanson, Solvng Least Squares Problems, Classcs n Appled Mathematcs, SIAM, [12] T. Y. L, Noah H. Rhee, Zhonggang Zeng, An Ecent and Accurate Parallel Algorthm for the Sngular Value Problem of Bdagonal Matrces, Numersche Mathematk, vol.69: (1995). [13] R. M. S. Ralha, A. Mackewcz, An Ecent Algorthm for the Computaton of Sngular Values, Proc. of the III Internatonal Congress of Numercal Methods n Engneerng, M. Doblare, J.M. Correas, E. Alarcon, L. Gavete and M. Pastor (Eds.), , SEMNI (1996). [14] J. Demmel, Appled Numercal Lnear Algebra, SIAM (1997). 6