Krylov subspace methods

Transcription

1 Program Lecture 3 Uppsala, April 2014 Krylov subspace methods Krylov basis & Hessenberg matrices Arnoldi s expansion Selecting approximate eigenvalues Convergence Stability issues in Arnoldi s decomposition Lanczos expansion Gerard Sleijpen Department of Mathematics sleij101/ (Krylov) Subspace methods Extraction en selection strategies Ritz values and harmonic Ritz values 1 2 Orthonormal basis K k (A,u 0 ) Recall V k [v 1,...,v k ]. Suppose v 1,...,v k is an orthonormal Krylov basis K k (A,u 0 ). Compute v k+1 by orthogonalising Av k against V k : Expand: w = Av k, Orthogonalize: ṽ = w V kh k with h k = V k w, Normalise: v k+1 = ṽ/ν k with ν k = ṽ 2. Theorem. Orthogonalising Av j against V j for j = 1,...,k leads to AV k = V k+1 H k, with V k orthonormal, spanning K k (A,v 1 ), H k is (k +1) k upper Hessenberg. Note. The matrix H k comes for free in the orthogonalization process. Note. With h k ( h T k,ν k) T, we have Av k = w = V k h h k +v k+1 ν k = [V k,v k+1 ][ k ν k Assemble A[V k 1,v k ] = [V k,v k+1 ] ] H k 1 h k ν k = V k+1 hk Find y k such that AV k y k ϑ k V k y k Find y k such that V k+1 H k y k V k (ϑ k y k ) if v 1 = ρ 0 u 0 Find y k such that H k y k ϑ k ( y k T,0)T Details later. 3 4

2 Hessenberg and Krylov Orthogonalisation Hessenberg matrices and Krylov subspaces are intimately related. Theorem. Consider the relation AV k = V k+1 H k, where V k+1 = [V k,v k+1 ] is n (k +1), and H k is (k +1) k. Terminology. If V is an n k orthonormal matrix and w is an n vector, then, with orthonormalise w against V, we mean: construct an n-vector v and a (k +1)-vector h such that v V, v 2 = 1, w = [V,v] h Then, v 1,...,v k form a Krylov basis for K k (A,v 1 ) Notation. [v, h] = Orth(V,w) i.e., V j spans K j (A,v 1 ) for all j = 1,...,k, H k is Hessenberg. Use a stable variant of Gram Schmidt. In Arnoldi s decomposition, V k is selected to be orthonormal (to ease computations and to enhance stability). Arnoldi s method: orthonormalise Av k against V k to obtain v k+1 all k. 5 Note that the last coordinate of h is 0 if w is in the span of V: in such a case (and if k < n), we select v to be a (random) normalised vector orthogonal to V (we insist on expanding to avoid stagnation in subsequent steps). 6 Arnoldi s decomposition Eigenvalues and Arnoldi s decomposition AV k 1 = V k H k 1, Ax = λx. with V k n k orthonormal, H k 1 k (k 1) Hessenberg. Find a normalised u k K k (A,r 0 ) such that Expand the decomposition to AV k = V k+1 H k. with ϑ k u k Au k, r k Au k ϑ k u k Notation. [V k+1, H k ] = ArnStep(A,V k,h k 1 ) is small in some sense and ϑ k almost has the desired properties. w = Av k Arnoldi s decomposition: AV k = V k+1 H k. [v k+1, h k ] = Orth(V k,w) V k+1 = [V k,v k+1 ] H k = H k 1 0 k 1, H k [H k, h k ] Note r k = V k+1 (H k y k ϑ k y k ), ϑ k = y k H k y k. r k 2 = H k y k ϑ k y k 2 7 The computation of r k 2 and ϑ k is in k-space! 8

3 Arnoldi s method [Arnoldi 52] Convergence Proposition. With ux k V k y k and r k Au k ϑ k u k, y k solves H k y k = ϑ k y k r k K k (A,u 0 ) Select k max and tol Set ρ 0 = 1, V 1 = [ 1 u u ], H 0 = [] for k = 1,...,k max do Break if ρ k < tol [V k+1,h k ] = ArnStep(A,V k,h k 1 ) Solve H k y = ϑ y for k eigenpairs (ϑ, y). Select a pair, say, (ϑ k, y k ), y k y k / y k 2 ρ k = h k+1,k e k y k end for x = V k y k, λ = ϑ k. 9 The space K k (A,b) = span(v k ) contains all vectors that can be computed with k 1 steps of (shifted) power method, and also the vectors computed with a k 1-degree polynomial filter. faster convergence than any polynomial filter method. Achieving better convergence depends on how the approximate eigenpairs are extracted from the search subspaces span(v k ). Using Ritz-Galerkin, for extremal eigenvalues (selecting extremal Ritz values) Arnoldi:shifted power GMRES:Richardson 10 Gram Schmidt orthogonalisation Gram Schmidt orthogonalisation [v, h] = Orth(V,w), then v V, v 2 = 1, w = [V,v] h Classical Gram Schmidt [v, h] = Orth(V,w), then v V, v 2 = 1, w = [V,v] h Modified Gram Schmidt Loss of stability. h = V w, v = w V h ν = v 2, h ( h T,ν) T, v v/ν Sensitive to perturbations on V. DOTs and AXPYs introduce rounding errors. Scaling by ν amplifies rounding errors if tan( (w,span(v)) = ν h 2 1. Note. Costs of computing h 2 are negligible (wrt v 2 ) (wrt costs computing v 2 ). 11 v = w for j = 1,...,k do h j = v j v, v v v jh j end for ν = v 2, h = (h 1,h 2,...,h k,ν) T, v v/ν Loss of stability. Sensitive to perturbations on V Smaller rounding errors from AXPYs. Scaling by ν amplifies rounding errors if ν h More stable. Harder to parallel. 12

4 Gram Schmidt orthogonalisation Stability of the Gram Schmidt variants [v, h] = Orth(V,w), then v V, v 2 = 1, w = [V,v] h Repeated Gram Schmidt with DGKS criterion Loss of stability. h = V w, v = w V h ν = v 2, µ = h 2 while ν τµ g = V v, v v V g ν = v 2, µ = g 2, h h+ g end while h ( h T,ν) T, v v/ν Not sensitive to perturbations on V Smaller rounding errors from AXPYs. Scaling by ν amplifies rounding errors if ν h Orthogonalisation recursively applied to the columns of an n k matrix W leads to computed V and R such that W+ = V R for some n k perturbation matrix with with R is k k upper triangular, F 4k 2 u W F, Loss of orthogonality: V V I k 2 κu(c 2 (W)) l ClassGS: κ of order kn, ModGS: κ of order kn, l = 1. l = 2 (conjecture). RepGS: κ may depend on 1 τk (rarely), l = 0. Householder QR: κ = 0, l = Gram Schmidt and Arnoldi Theorem. Modified Gram Schmidt is sufficiently stable for solving linear systems. Eigenvalue computations requires more stability. Proof. In Arnoldi, the n (k +1) matrix W is [ ] 1 W = [v 1,Av 1,...,Av k ] and R = 0 H k. Arnoldi s decomposition based methods 1) Use recursive expansion for building a Krylov basis V k (involves high dimensional operations) 2) Consider a projected problem as: AV k y k ϑ k V k y k V k, or AV k y k ϑ k V k y k AV k (For theoretical analysis) 3) Form a projected matrix, as H k = V k AV k. (high dim) Hence, when solving Ax = r 0 with x k = V k y k, we have min Wz 2 z 2 r k 2 r 0 2 (take z = ( r 0 2, y T k )T ). 4) Use the projected matrix to solve the projected problem for y k in k-space (only k-dimensional operations) 5) Assemble u k V k y k. (high dim) Therefore, we have the (sharp) estimate C 2 (W) A 2 r 0 2 r k Note. When recursively using Gram-Schmidt to compute the component of Av k that is orthonormal to V k, the projected matrix H k comes for free. 16

5 Krylov subspace methods Krylov subspace methods Krylov subspace methods search for approximate solutions in a Krylov subspace: the search subspace is a Krylov subspace. Stages. Expansion. Expand a Krylov basis v 1,...,v k recursively Extraction. Extract an approximate solution from span(v k ) If space becomes too large Shrinking. (Restart) For some l < k, select a Krylov basis ṽ 1,...,ṽ l in the space span(v k ) such that span(ṽl) contains promising approximations. 17 Why searching for approximations in Krylov subspaces? 1) Convergence based on polynomial approximation theory (better than Richardson, Power method, etc.) 2) Krylov structure can be exploited to enhance efficiency. For instance, with Arnoldi s method, the Hessenberg matrix (projected matrix) comes for free. if A is Hermitian then expansion vectors can efficiently be computed (as in Lanczos, CG,...) 18 Ax = b Ax = λx Extraction strategies Subspace methods Iterate until sufficiently accurate: Expansion. Expand the search subspace V k = span(v k ). Restart if dim(v k ) is too large. Extraction. Extract an appropriate approximate solution (ϑ, u) from the search subspace. Example. Krylov subspace methods as GMRES, CG, Arnoldi, Lanczos: expansion by t k = Av k Goal. Expansion. (x,v k+1 ) (x,v k ) Extraction. Find u = V k y k s.t. (x,u) (x,v k+1 ) 19 Let V span(v) be a search subspace. Find u Vy V such that (Ritz )Galerkin. Au ϑu V Ritz values Orthogonal residuals Au b V for solving Ax = b Petrov Galerkin. Au ϑu AV harmonic Ritz values. Minimal residuals for solving Ax = b: u = minargz Az b 2 Au b AV Refined Ritz. For a given approximate eigenvalue ϑ, u minargũ V Aũ ϑũ 2 20

6 Selection Ritz values Ritz Galerkin and Petrov Galerkin lead to k Ritz pairs (ϑ i,u i ), Petrov pairs, respectively (i = 1,...,k). Select the most promising one as approximate eigenpair. Most promising : 1) Formulate a property that, among all eigenpairs, characterises the wanted eigenpair Example. λ = max(re(λ j )), λ = min λ j, λ = min λ j τ,... 2) Select among all Ritz pairs the one with this property. Example. ϑ = max(re(ϑ i )), ϑ = min ϑ i, ϑ = min ϑ i τ,... Warning. May lead to a wrong selection One wrong selection = one useless iteration step. Proposition. u = Vy. Ritz values are Rayleigh quotients: Au ϑu V ϑ = ρ(u) u Au u u. Proposition. For a given approximate eigenvector u, the Rayleigh quotient is best approximate eigenvalue, i.e., gives the smallest residual: Au ϑu 2 Au ϑu 2 ( ϑ C) ϑ = ρ(u). Proof. Au ϑu V Au ϑu Vy = u ϑ = ρ(u). Au ϑu 2 Au ϑu 2 ( ϑ C) Au ϑu u. One wrong selection at restart may spoil convergence Ritz values Ritz values For ease of discussion, assume AX = XΛ with X X = I, where X = [x 1,...,x n ], Λ = diag(λ 1,...,λ n ): Ax i = λ i x i (i = 1,...,n), the eigenvectors x i form an orthonormal basis of C n. Terminology. A has an orthonormal basis X of eigenvectors. Note. A is normal iff A A = AA. Hermitian and unitary matrices are normal. A is normal A has an orthonormal basis of eigenvectors. For ease of discussion, assume AX = XΛ with X X = I. u approximate eigenvector, u 2 = 1, ϑ = ρ(u). u = β i x i with i β i 2 = 1, ϑ = ρ(u) = i β i 2 λ i. Proposition. If A is normal, then any Ritz value is a convex mean (i.e., weighted averages) of eigenvalues. Proposition. Ritz values form a safe selection for finding extremal eigenvalues, an unsafe selection for interior eigenvalues

7 For ease of discussion, Harmonic Ritz values assume AX = XΛ with X X = I. Assume we are interested in eigenvalue λ closest to 0, 0 is in the interior of the spectrum, λ 0. Note that A 1 x = 1 λ x and 1 λ extremal in {1 λ i } With respect to W, find x Wy st A 1 x µ x W: largest µ forms a safe selection ( λ 1 µ, x x) Select W = AV. Then, with u Vy, we have x = Au A 1 x µ x W µ 1 u Au AV For ease of discussion, Harmonic Ritz values assume AX = XΛ with X X = I. Assume we are interested in eigenvalue λ closest to 0, 0 is in the interior of the spectrum, λ 0. Strategy using harmonic Ritz values 1) Solve Au ϑu AV 2) Select ϑ closest to 0. Proposition. If A is normal, then harmonic Ritz values are harmonic means of the eigenvalues. Proposition. Harmonic Ritz values form a safe selection for finding eigenvalues in the interior (close to 0). Harmonic Ritz relates to Ritz for an inverted matrix