Efficient Iterative Methods for Large Scale Inverse Problems

Transcription

1 Efficient Iterative Methods for Large Scale Inverse Problems Joint work with: James Nagy Emory University Atlanta, GA, USA Julianne Chung (University of Maryland) Veronica Mejia-Bustamante (Emory)

2 Inverse Problems in Imaging Imaging problems are often modeled as: b = Ax + e where A - large, ill-conditioned matrix b - known, measured (image) data e - noise, statistical properties may be known Goal: Compute approximation of image x

3 Application: Medical Imaging PET motion correction for brain imaging: Head moves during data acquisition reconstructed brain image, b is distorted by motion blur.

4 Application: Medical Imaging PET motion correction for brain imaging: Head moves during data acquisition reconstructed brain image, b is distorted by motion blur. Attach cap with fixed markers to patient head.

5 Application: Medical Imaging PET motion correction for brain imaging: Head moves during data acquisition reconstructed brain image, b is distorted by motion blur. Attach cap with fixed markers to patient head. Motion detection camera records position of patient head.

6 Application: Medical Imaging PET motion correction for brain imaging: Head moves during data acquisition reconstructed brain image, b is distorted by motion blur. Attach cap with fixed markers to patient head. Motion detection camera records position of patient head. Construct large, sparse matrix A from position information. Solve linear inverse problem, b = Ax + e.

7 Application: Medical Imaging Original reconstruction Improved reconstruction Collaborators: Sarah Knepper, Piotr Wendykier (Math/CS) John Votaw, Niv Raghunath, Tracy Faber (Radiology)

8 Inverse Problems in Imaging A more challenging problem: where b = A(y) x + e A(y) - large, ill-conditioned matrix b - known, measured (image) data e - noise, statistical properties may be known y - parameters defining A, usually approximated Goal: Compute approximation of image x and improve estimate of parameters y

9 Application: Space Situational Awareness Multi-Frame Blind Deconvolution: Given images, b:

10 Application: Space Situational Awareness Multi-Frame Blind Deconvolution: Given images, b: Solve nonlinear inverse problem b = A(y) x + e

11 Outline Linear Inverse Problem: b = Ax + e 1 Linear Inverse Problem: b = Ax + e Golub-Kahan Bidiagonalization Golub-Kahan Based Hybrid Methods 2 Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples 3

12 Linear Inverse Problem Golub-Kahan Bidiagonalization Golub-Kahan Based Hybrid Methods Assume A = A(y) is known exactly. We are given A and b, where b = Ax + e A is an ill-conditioned matrix, and we do not know e. We want to compute an approximation of x. Bad idea: e is small, so ignore it, and use x inv A 1 b = x + A 1 e

13 Linear Inverse Problem Golub-Kahan Bidiagonalization Golub-Kahan Based Hybrid Methods Assume A = A(y) is known exactly. We are given A and b, where b = Ax + e A is an ill-conditioned matrix, and we do not know e. We want to compute an approximation of x. Bad idea: e is small, so ignore it, and use x inv A 1 b = x + A 1 e

14 Inverse Solution Golub-Kahan Bidiagonalization Golub-Kahan Based Hybrid Methods Original image

15 Inverse Solution Golub-Kahan Bidiagonalization Golub-Kahan Based Hybrid Methods Original image Inverse solution

16 Regularization Linear Inverse Problem: b = Ax + e Golub-Kahan Bidiagonalization Golub-Kahan Based Hybrid Methods Basic Idea: Instead of computing x inv = A 1 b, use: so that where x reg = A 1 regb ˆx = A 1 regb = A 1 reg (Ax + e) = A 1 regax + A 1 rege A 1 regax x and A 1 rege is not too large

17 Regularized Solution Golub-Kahan Bidiagonalization Golub-Kahan Based Hybrid Methods Original image Inverse solution

18 Regularized Solution Golub-Kahan Bidiagonalization Golub-Kahan Based Hybrid Methods Original image Regularized solution

19 An Example: Tikhonov Regularization Golub-Kahan Bidiagonalization Golub-Kahan Based Hybrid Methods { min b Ax 2 x 2 + λ 2 x 2 } 2 min x [ b 0 ] [ A λi ] x 2 2

20 An Example: Tikhonov Regularization Golub-Kahan Bidiagonalization Golub-Kahan Based Hybrid Methods { min b Ax 2 x 2 + λ 2 x 2 } 2 min x [ b 0 ] [ A λi ] x 2 2 Choose λ to minimize GCV(λ) = n (I AA 1 reg)b 2 2 ( trace(i AA 1 reg) ) 2 where A 1 reg = (A T A + λ 2 I) 1 A T

21 Computational Approaches Golub-Kahan Bidiagonalization Golub-Kahan Based Hybrid Methods Computational approaches: For small matrices, can use SVD. For large matrices, computing SVD is expensive. SVD algorithms do not readily simplify for structured or sparse matrices. Alternative for large scale problems: LSQR iteration (Paige and Saunders, ACM TOMS, 1982)

22 Golub-Kahan Bidiagonalization Golub-Kahan Based Hybrid Methods Golub-Kahan (Lanczos) Bidiagonalization Given A and b, for k = 1, 2,..., compute U k+1 = [ ] u 1 u 2 u k u k+1, u1 = b/ b V k = [ ] v 1 v 2 v k α 1 β 2 α 2 B k = β k α k β k+1 where U k+1 and V k have orthonormal columns, and A T U k+1 = V k B T k + α k+1v k+1 e T k+1 AV k = U k+1 B k

23 GKBD and LSQR Golub-Kahan Bidiagonalization Golub-Kahan Based Hybrid Methods At kth GKBD iteration, use QR to solve projected LS problem: where x k = V k f min b x R(V k ) Ax 2 2 = min βe 1 B k f 2 2 f

24 GKBD and LSQR Golub-Kahan Bidiagonalization Golub-Kahan Based Hybrid Methods At kth GKBD iteration, use QR to solve projected LS problem: where x k = V k f min b x R(V k ) Ax 2 2 = min βe 1 B k f 2 2 f For our ill-posed inverse problems: Singular values of B k converge to k largest sing. values of A. Thus, x k is in a subspace that approximates a subspace spanned by the large singular components of A. For k < n, x k is a regularized solution. x n = x inv = A 1 b (bad approximation)

25 Golub-Kahan Based Hybrid Methods Golub-Kahan Bidiagonalization Golub-Kahan Based Hybrid Methods To avoid noisy reconstructions, embed regularization in GKBD: O Leary and Simmons, SISSC, Björck, BIT Björck, Grimme, and Van Dooren, BIT, Larsen, PhD Thesis, Hanke, BIT Kilmer and O Leary, SIMAX, Kilmer, Hansen, Español, SISC Chung, N, O Leary, ETNA 2007 (HyBR Implementation)

26 Golub-Kahan Bidiagonalization Golub-Kahan Based Hybrid Methods Regularize the Projected Least Squares Problem To stabilize convergence, regularize the projected problem: min f [ βe1 0 ] [ Bk λi Note: B k is very small compared to A, so ] f Can use expensive methods to choose λ (e.g., GCV) Very little regularization is needed in early iterations. GCV tends to choose too large λ for bidiagonal system. Our remedy: Use a weighted GCV (Chung, N, O Leary, 2007) Can also use WGCV information to estimate stopping iteration (approach similar to Björck, Grimme, and Van Dooren, BIT, 1994). 2 2

27 Nonlinear Inverse Problem Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples We want to find x and y so that b = A(y)x + e With Tikhonov regularization, solve min x,y [ A(y) λi ] [ b x 0 ] 2 As with linear problem, choosing a good regularization parameter λ is important. Problem is linear in x, nonlinear in y. y R p, x R n, with p n. 2

28 General Mathematical Model Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples Assume where b = A(y) x + e b is measured, noisy data (image) Parametric model for A(y) is known. e.g., can implement function: A = param2matrix(y) We use iterative methods, so: Do not need A explicitly. Just need to be able to compute matrix-vector products with A(y) and A(y) T

29 Solution Scheme Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples We use Tikhonov regularization, and solve nonlinear least squares (NLLS) problem: [ { min A(y) x b 2 x,y 2 + λ 2 x 2 } 2 = min A(y) x,y λi Goals: Exploit structure of NLLS problem. Exploit structure of A(y) Use parameter choice method to choose λ. ] [ b x 0 ] 2 2

30 Standard Gauss-Newton Approach ([ ]) Define f(z) = f xy = Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples [ A(y) λi Consider min ψ(z) = min f(z) 2 z z 2 [ f(x, y) Jacobian is given by: J ψ = x Basic Gauss-Newton iteration:» x0 choose initial z 0 = for l = 0, 1, 2,...» b r l = 0 end y 0» A(yl ) λi d l = arg min d J ψ d r l 2 z l+1 = z l + d l ] x x l [ b 0 ] f(x, y) y ]

31 General Gauss-Newton Method Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples Difficulties with the general Gauss-Newton approach: Constructing and solving linear systems with J ψ can be very expensive. Effective preconditioners may be difficult to find. Requires either specifying a priori regularization parameter λ or estimating it within a nonlinear iterative scheme. Do not take algorithmic advantage of fact that the problem is strongly convex in x. May take small steps due to the nonlinearity induced by y.

32 Separable Nonlinear Least Squares Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples Variable Projection Method: Implicitly eliminate linear term. Optimize over nonlinear term. Some general references: Golub and Pereyra, SINUM 1973 (also IP 2003) Kaufman, BIT 1975 Osborne, SINUM 1975 (also ETNA 2007) Ruhe and Wedin, SIREV, 1980 How to apply to inverse problems?

33 Variable Projection Method Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples Exploit following properties of our problem: ([ ]) ψ(z) = ψ xy is linear in x. y contains relatively few parameters compared to x. Implicitly eliminate linear parameters x: x(y) = arg min x ψ(x, y) = arg min x [ A(y) λi ] [ b x 0 ] 2. 2 Obtain reduced cost functional: ρ(y) ψ(x(y), y), Use Gauss-Newton to minimize reduced cost functional ρ(y)

34 Variable Projection Approach Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples Consider ρ(y) ψ(x(y), y) = f(x(y), y) 2 2 [ [ A(y)x ] Jacobian is given by: J ρ = f y = y 0 Reduced Gauss-Newton iteration: ] = [ Ĵρ 0 ] choose initial y 0 for l = 0, 1, 2,... x l = arg min x» A(yl ) λ l I» b x 0 2 ˆr l = b A(y l ) x l d l = arg min d b Jρd ˆr l 2 y l+1 = y l + d l end

35 Variable Projection Approach Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples Consider ρ(y) ψ(x(y), y) = f(x(y), y) 2 2 [ [ A(y)x ] Jacobian is given by: J ρ = f y = y 0 Reduced Gauss-Newton iteration: choose initial y 0 for l = 0, 1, 2,... [x l, λ l ] = HyBR(A(y l ), b) ] = [ Ĵρ 0 ] end ˆr l = b A(y l ) x l d l = arg min d b Jρd ˆr l 2 y l+1 = y l + d l

36 Example 1: Blind Deconvolution Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples A(y) = A(p(y)) where A is , with entries given by p. p is PSF, with entries: ( (i k) 2 s2 2 p ij = exp (j l)2 s1 2 ) + 2(i k)(j l)ρ2 2s1 2s2 2 2ρ4 (k, l) is the PSF center (location of point source) y vector of unknown parameters: s 1 y = s 2 ρ

37 Example 1: Blind Deconvolution Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples Can get analytical formula for Jacobian: where x = vec(x). Ĵ ρ = { A( p(y) ) x } y = { A( p(y) ) x } p = A(X) y { p(y) } y { p(y) } Though in this example, finite difference approximation of Ĵρ works very well.

38 Example 1: Blind Deconvolution Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples Gauss-Newton Iteration History G-N Iteration y x HyBR λ HyBR its

39 Example 1: Blind Deconvolution Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples Blurred image Initial reconstruction Final reconstruction

40 Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples Example 2: Multi-Frame Blind Deconvolution Similar setup as previous problem, except: Use 3 different blurred images (frames). y has 9 parameters (3 for each PSF) Goal: Find approximations of 3 PSFs and true image.

41 Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples Example 2: Multi-Frame Blind Deconvolution Similar setup as previous problem, except: Use 3 different blurred images (frames). y has 9 parameters (3 for each PSF) Goal: Find approximations of 3 PSFs and true image.

42 Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples Example 2: Multi-Frame Blind Deconvolution Convergence results: relative error (y) relative error (x) GN iteration GN iteration

43 Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples Example 2: Multi-Frame Blind Deconvolution blurred images initial reconstruction final reconstruction

44 Nonlinear Least Squares Formulation Regularized Variable Projection Method Solving the linear subproblem Examples Example 2: Multi-Frame Blind Deconvolution blurred images initial reconstruction final reconstruction

45 Regularized variable projection method works well for challenging inverse problems in image processing. Exploits high level structure. Can exploit low level structure in linear system solves. HyBR works well for the linear system solves Automatic estimation of regularization parameter. Automatic estimation of stopping iteration. Can incorporate preconditioning. Some work to do: Stopping rule for the Gauss-Newton iteration. Incorporating other regularization schemes, and constraints (nonnegativity). Geometric models for nonlinear distortions and synthetic boundary conditions: Daniel Fan, Emory University