Eigenvalue Solvers. Derivation of Methods

Transcription

1 Eigenvalue Solvers Derivation of Methods

2 Similarity Transformation Similarity transformation for A: BAB ; this can be done with any nonsingular B. Let Ax = λx, then BAB Bx BAx λbx = =. BAB has the same eigenvalues as A, and eigenvectors Bx where x is an eigenvector of A. Although any nonsingular B possible, most stable and accurate algorithms with orthogonal (unitary) matrix. For example used in the QR algorithm. 2

3 Similarity Transformation Orthogonal similarity transformation for A: * where QQ= I. * QAQ, If * QAQ L F = 0 L 2, then ( A) = ( L ) ( L ) L L L. 2 If we can find Q QQ 2 that yields such a decomposition we have reduced the problem to two smaller problems. Moreover, AQ QL range Q is invariant subspace. Eigenpair Lz = λz gives eigenpair AQz = QLz = λq z. = and ( ) 3

4 Approximation over Search Space For large matrices we cannot use full transformations. Often we do not need all eigenvalues/vectors. Look for proper basis Q that captures relevant eigenpairs. We do not need Q 2. Approximations over subspace range( Q ): L = Q AQ * When is an approximation good (enough)? We will rarely find AQ QL = 0 unless we do huge amount of work. Not necessary. We are working with approximations and we must deal with numerical error anyway. 4

5 Approximation over Search Space Let AQ QL = R with R small relative to A. Now, ( ) A RQ Q QL AQ R QL * = = 0. range( Q ) is exact invariant subspace of perturbed matrix, Â. * Â= A RQ and Â A A = R A If R A sufficiently small, then Q is acceptable. In fact, this is as good as we can expect (unless we're lucky). Any numerical operation involves perturbed operands! Note that we cannot always say that Q is accurate. 5

6 Power method So, what kind of methods lead to good Q? In many cases matrices are sparse and/or it is possible to do the matrix-vector product,ax, very cheap. First consider very small subspace: vector Well-known method is the Power method. ( k ) ( k) ( k) v + = Av / Av, ( ) ( ) ( ) k k k λ + + vj / vj = until convergence. Converges to eigenpair with largest absolute eigenvalue, if unique. May converge very slowly if dominant eigenvalue not well-separated (next one is close). 6

7 Power method Assume A diagonalizable: Decompose v ( 0) along eigenvectors: v ( ) A= XΛX and λ > λj = x α 0 j j j k v ( k) A k v ( 0) x k k x x λ j = = j jλα j j = λ α j j α + > j λ So, ( k ) k k v x since λj / λ 0. If λ > λ2 λj, then λ λ Convergence may be slow if 2 determines rate of convergence. λ λ 2, and linearly at best. 7

8 Work with search space Improve on the Power method by keeping all iterates. m 2 m Krylov space: K ( A, v) = span { v, Av, A v,, A v} Let Qm = qq2 q m m give orthonormal basis for (, ) Approximate eigenpair ( Qw), m * m μ m m m m μ is called Ritz pair if AQ w Q w Q Q AQ w μw = 0 K A v. We find Ritz pairs from eigenpairs ( μ,w) of H Q AQ. * m m This is a small eigenvalue problem that we can solve using standard software (e.g., QR algorithm in LAPACK). 8

9 Arnoldi Generate an orthogonal basis for the Krylov subspace: q = v (0) /æv (0) æ 2 ; for k = :m, q = Aq k ; for j = :k, h j,k = q & k q ; q = q q k h k,j ; h k+,k = æq æ 2 ; q k+ = q /h k+,k ; end Accurate orthogonalization required: (partial) reorthogonalization AQ m = Q m+ H m, where Q m+ = [q q 2 q m+ ] and H m = [ h j,k ] j=:m+,k=:m & with Q m+ Q m+ = I m+ and range(q m+ )=K m+ (A, v (0) ). 9

10 Arnoldi Compute m approximate eigenpairs from H m = Q & m AQ m : H m y = y which gives approximate pairs: (,Q m y) of A. To reduce costs we select after m steps kapproximate eigenvectors, get new Q k (k < m), and discard the rest. All Ritz vectors (Krylov space) have same residual:. q m+ AQ m y = Q m+ H m y = Q m H m y + q m+ e m T yh m+,m = Q m y m + q m+ y m h m+,m We can use to continue generating vectors from the Krylov q m+ space. 0

11 Lanczos The Lanzcos algorithm is Arnoldi applied to a symmetric system using short recurrence (two previous vectors). AQ m = Q m+ T m, where T m = [ t j,k ] j=:m+,k=j :j+ is tridiagonal and symmetric (even cheaper) Use T m = Q & m AQ m to compute approximate eigenpairs. T m y = y which gives approximate pairs: (, Q m y) of A. If we only need eigenvalues (and do not re-orthogonalize) we can discard the vectors and restarting is not necessary. Special care required to avoid spurious copies of eigenvalues. Otherwise, techniques for Arnoldi can be used.

12 IRA We need to restart from time to time to save memory and CPU time. This also holds for Lanczos if we want to compute eigenvectors or reorthogonalize (part.) for accuracy. Efficient and accurate implementation is so-called Implicitly Restarted Arnoldi (Lanczos). This method discards unwanted Ritz pairs using polynomial filter applied to Hessenberg matrix (tridiagonal). 2

13 IRA Discard unwanted eigenpairs with polynomial filter. (Sorensen 92) The remaining vectors span new Krylov subspace with different starting vector closer to wanted invariant subspace. Method builds search space of dimension m, discards m k vectors, and extends again to m vectors using the residual of the Ritz pairs (same vector; w m + fromawm = Wm + H m) ARPACK from user guide from SIAM (Lehoucq, Sorensen, and Yang 98) 3

14 Inverse Iteration What if we want the smallest (magnitude) eigenvalue or any other one that is not the largest in magnitude? Inverse iteration: x k = A x k w solve Ax k = x k or x k =(A si) x k w solve (A si)x k = x k Inverse iteration with A converges to smallest (in magnitude) eigenvalue if unique, and inverse iteration with (A si) converges to eigenvalue/vector closest to s. By choosing s more accurate we can improve convergence speed. (A si) has eigenvalues ( j s) and so, assuming we want to approximate the eigenpair with (simple) eigenvalue the rate of convergence is determined by max j!k k s j s k This can be made arbitrarily small if we know k sufficiently accurate. 4

15 Rayleigh Quotient Iteration Let x be an approximate eigenvector for a real matrix A. Then approximate eigenvalue is given by n % linear least squares problem: x ˆ l Ax e x T x ˆ = x T Ax e ˆ = xt Ax x T x (Rayleigh quotient) Let Av = v and A be symmetric then æx væ = O( ) e ˆ = O( 2 ) Let x = v + p, where vz p, ævæ 2 = æpæ 2 = x T Ax x T x = (v+ p) T A(v+ p) (v+ p) T (v+ p) = v T Av+2 p T Av+ 2 p T Ap v T v+2 p T v+ 2 p T p = + 2 p T Ap + 2 l ( + 2 p T Ap)( 2 )= + O( 2 ) 5

16 Rayleigh Quotient Iteration So given O( ) approximation to v we get O( 2 ) approximation to. Cunning plan: Improve eigenvalue estimate s every step using Rayleigh quotient. x k =(A r k I) x k w solve (A r k I) x k = x k with r k = x k This is called Rayleigh Quotient Iteration For symmetrix matrices, close to solution, we have cubic convergence. Let x = v k + p = v k + j!k v j j, where v z p, ævæ 2 = æpæ 2 = T Ax k x T k x k x =(A ˆI) x =( k ˆ) v k + j! v j j ( j ˆ) normalization gives x( k ˆ)=v k +( k ˆ) j! v j j ( j ˆ) and the norm of the error now gives æx( k ˆ) v kæ 2 = æ( k ˆ) j! v j j ( j ˆ) æ 2 [ æ( k ˆ) j! v j jæ = O( 3 ) 6

17 Rayleigh Quotient Iteration The Rayleigh Quotient iteration r k = x k T x k Ax k T x k solve (A si)x k = x k requires repeatedly solving a (new) linear system. We can make this cheaper by first reducing A to tridiagonal or Hessenberg form using unitary similarity transformations: A = Q T AQ This gives T = Q T AQ is tridiagonal if A is symmetric, and H = Q T AQ is (upper) Hessenberg if A is nonsymmetric. Solving a linear tridiagonal system (order n) takes only O(n) operations. Solving an upper Hessenberg system (order n) takes O(n 2 ) operations, for n Givens rotations and backsubstitution O(n 2 ). 7

18 (Approximate) Shift Invert Methods Inverse iteration with shift and Rayleigh quotient iteration are examples of shift-invert methods Very fast convergence, but require inverse of matrix Expensive for large matrix, especially if shift changes every iteration Hence, approximate shift invert methods (really, shift, approximate invert...) Approximate solution by one step approximation or few steps of iterative method Davidson s method, Jacobi-Davidson method 8

19 Davidson Davidson s method also uses a search space and Ritz values/vectors from that space, but extension of search space is different. Let H y m n m * *, such that QQ m m = I, QmAQm = Hm and = λy. We have residual r= AQy m λqy m. Q m Then extend search space as follows: Solve( DA λi) t = r where D A is diagonal ofa. * ( m m) t = I Q Q t (orthog) and set Qm+ = Qm t. Compute H m + and its eigenpairs, continue. 9

20 Davidson Very effective for outer most (esp.) dominant eigenvalues. Various explanations for this have been given. In many cases D λi A λi has only few outlying the matrix ( ) ( ) A eigenvalues (). Another interprets Davidson as an improvement to an old method by Jacobi (2). This method discards unwanted Ritz pairs using polynomial filter applied to Hessenberg matrix (tridiagonal). 20

21 Jacobi-Davidson Linear eigenvalue problem: Ax = λx * Orthon. basis for search space: Q k, Hk = QkAQk ( λ, x), Ritz pair: ( λ,u) where u = Qky and residual r k = Au λu Correction equation: I uu * A λi I uu * t = r, t u Solve: ( )( )( ) * * t = ( I QQ k k) t/ ( I QQ k k) t Qk+ = Qk t, * Hk+ = Qk+ AQk+ Continue Discard unwanted eigenpairs with polynomial filter or using Schur decomposition of H k +. (Sleijpen and van der Vorst 96) 2

22 Some literature.r.b. Morgan, Generalizations of Davidson s method for computing eigenvalues of large nonsymmetric matrices, J. Comput. Physics 0, pp , Sleijpen and van der Vorst, A Jacobi-Davidson iteration method for linear eigenvalue problems, SIAM Review 42(2), pp , 2000 (extension of SIMAX 996). (2) Provides many good references 22

23 Quick Comparison Implicitly restarted Arnoldi (IRA) and Davidson work well for eigenvalues on the boundary of the spectrum. Jacobi-Davidson also works for these, but may be less efficient. Jacobi-Davidson (JD) is superior for interior eigenvalues. Probably, because it is an approximate shift and invert technique and more effectively so than Davidson. To improve the 'invert' part preconditioning can be used in the linear solver. This may improve the effectiveness dramatically. (One can also use various linear solvers, e.g. multigrid to improve convergence) IRA has some advantage in looking for several eigenvalues at once. 23