Introduction to Inverse Problems (2 lectures)

Transcription

1 Introduction to Inverse Problems (2 lectures) Summary Direct and inverse problems Examples of direct (forward) problems Deterministic and statistical points of view Ill-posed and ill-conditioned problems An illustrative example: The deconvolution problem Truncated Fourier decomposition (TFD); Tikhonov regularization Generalized Tikhonov regularization; Bayesian perspective. Iterative optimization. IP, José Bioucas Dias, 2007, IST 1

2 Direct/Inverse problems Direct (forward) problem Causes Inverse problem Effects Example: Direct problem: the computation of the trajectories of bodies from the knowledge of the forces. Inverse problem: determination of the forces from the knowledge of the trajectories Newton solved the first direct/inverse problem: the determintion of the gravitation force from the Kepler laws describing the trajectories of planets 2

3 An example: a linear time invariant (LTI) system Direct problem: Fourier domain Inverse problem: Source of difficulties: is unbounded A perturbation on leads to a perturbation on given by high frequencies of the perturbation are amplified, degrading the estimate of f 3

4 Image deblurring Observation model in (linear) image restoration/reconstruction observed image noise Linear operator original image (e.g., blur, tomography, MRI,...) Goal: estimate f from g 4

5 Image deblurring via regularization original Blurred, 9x9 uniform restored 5

6 MRI example Hydrogen density 2D frequency samples (9.4%) 6

7 Compressed Sensing (sparse representation) Sparse vector f Random matrix Observed data y Compressed Sensing N=1000 M=

8 Classes of direct problems Deterministic observation mechanism Original data (image) Operator + Observed data (image) perturbation 8

9 Classes of direct problems (deterministic) Linear space-invariant imaging systems Blur (motion, out-of-focus, Diffraction-limited imaging atmospheric) Near field acoustic holography Channel equalization Parameter identification Linear space-variant imaging systems (first kind Fredholm equation) X-ray tomography MR imaging Radar imaging Sonar imaging Inverse diffraction Inverse source Linear regression 9

10 Classes of direct problems Statistical observation mechanism Original data (image) Observed data (image) Ex: Linear/nonlinear observations in additive Gaussian noise + 10

11 Classes of direct problems (statistic) Linear/nonlinear observation driven by non-additive noise Parameters of a distribution Random signal/ image Rayleigh noise in coherent imaging Poisson noise in photo-electric conversion SPET (single photon emission tomography) PET (positron emission tomography) Ex: Amplitude in a coherent imaging system (radar, ultrasound) Terrain reflectance Inphase/quadrature backscattered signal 11

12 Well-posed/ill-posed inverse problems [Hadamard, 1923] Definition: Let be a (possible nonlinear) operator The inverse problem of solving Hadamard sense if: is well-posed in the 1) A solution exists for any in the observed data space 2) The solution is unique 3) The inverse mapping is continuous An inverse problem that is not well-posed is termed ill-posed The operator A of an inverse well/ill-posed problem is termed well/ill-posed 12

13 Finite/Infinite dimensional linear operators Linear Operators: The linear inverse problem equivalently, is well-posed if 1) and 2) holds or, and If is finite-dimensional, the corresponding inverse problem is well-posed iif either one of the properties 1) and 2) holds Example: In infinite-dimensional spaces Consider A defined on, If a solution of exists, it is unique since However, there are elements not in Thus, A is ill-posed (point 1 of the Hadamard conditions does not hold) Stability is also lacking: Take Then, does not converge when 13

14 Ill-conditioned inverse problems lll-posed lll-conditioned Many well-posed inverse problems are ill-conditioned, in the sense that For linear operators (tight bound) 14

15 Example: Discrete deconvolution Cyclic convolution Matrix notation N-periodic funtions A is cyclic Toeplitz 15

16 Example: Discrete deconvolution 16

17 Eigen-decomposition of cyclic matrices (unitary) Eigenvector (Fourier) matrix Eigenvalue matrix (diagonal) is the DFT of at frequency 17

18 Example: Discrete deconvolution (geometric viewpoint) Action of A on f Note: 18

19 Example: Discrete deconvolution (inferring f) Assume that Then is invertible and Thus, assuming the direct model We have error 19

20 Example: cyclic convolution with a Gaussian kernel What went wrong? 20

21 Example: Discrete deconvolution (estimation error) Size of the error Assume that Thus Which is a set enclosed by an ellipsoid with radii 21

22 Example: Discrete deconvolution (estimation error) The estimation error is the vector The components satisfy 22

23 Cyclic convolution with a Gaussian kernel (cont.) (unit impulse function) Noise dominates at high frequencies and is amplified by 23

24 Example: Discrete deconvolution (A is ill-posed) Assume now that is not invertible and it may happen that i.e, some are zero Least-squares solution Projection error 24

25 Example: Discrete deconvolution (A is ill-posed) Least-squares approach Orthogonal components 25

26 Example: Discrete deconvolution (A is ill-posed) Invisible objects is the minimum norm solution (related to the Moore-Penrose inverse) 26

27 Example: Discrete deconvolution (Regularization) A is ill-conditioned A is ill-posed In both cases small eigenvalues are sources of instabilities Often, the smaller the eigenvalue the more oscilating the corresponding eigenvector (high frequences) Regularization by filtereing: shrink/threshold large values of i.e, multiply the eigenvalues by a regularizer filter such that as 27

28 Example: Discrete deconvolution Regularization by filtering (frequency multiplication time convolution) Such that 1) as 2 ) The larger eigenvalues are retained Truncated Fourier Decomposition (TFD) Tikhonov (Wiener) filter 28

29 Example: Discrete deconvolution (Regularization by filtering) TFD Tikhonov Tikhonov regularization Thus Solution of the variational problem 29

30 Example: Discrete deconvolution (1D example) Gaussian shaped of standard deviation = f g frequency

31 Example: Discrete deconvolution (1D example -TFD) f f f f 1 f f f f 1 f f 1 f f

32 Example: Discrete deconvolution (2D example-fd) uniform 32

33 Example: Discrete deconvolution (2D example-tfd) 33

34 Curing Ill-posed/Ill-conditioned inverse problems Golden rule for solving ill-posed/ill-conditioned inverse problems Search for solutions which: are compatible with the observed data satisfy additional constraints (a priori or prior information) coming from the (physics) problem 34

35 Generalized Tikhonov regularization Tikhonov and TFD regularization are not well suited to deal with data Nonhomogeneities, such as edges Generalized Tikhonov regularization Bayesian viewpoint Data Discrepancy term Penalty/ Regularization term Negative loglikelihood Negative logprior 35

36 Dominating approaches to regularization 1) 2) 3) 4) In given circumstances 2), 3), and 4) are equivalent 36

37 Example: Discrete deconvolution (Nonquadratic regularization) penalize oscillatory solutions discontinuity preserving (robust) regularization is nonconvex hard optimization problem non-discontinuity preserving regularization is convex treatable optimization problem 37

38 Optimization - Quadratic Linear system of equations Large systems require iterative methods - Non-quadratic and smooth Methods: Steepest descent, nonlinear conjugate gradient, Newton, trust regions, - Non-quadratic and nonsmooth Constrained optimization (Linear, quadratic, second-order cone programs) Methods: Iterative Shrinkage/Thesholding; Coordinate Subspace Optimization; forward-backward splitting; Primal-dual Newton Majorization Minimizaton (MM) class 38

39 Majorization Minorization (MM) Framework Let Majorization Minorization algorithm:...with equality if and only if Easy to prove monotonicity: Notes: should be easy to maximize EM is an algorithm of this type. 39

40 Example: Discrete deconvolution (1D example NQ Regula.) Tikhonov f f f f f f f f f f f f

41 Example: Discrete deconvolution (2D example-total Variation) Total variation regularization (TV) where TV regularizer penalizes highly oscilatory solutions, while it preserves the edges 41

42 Bibliography [Ch1. RB2, Ch1. L1] Important topics Euclidian and Hilbert spaces of functions [App. A, RB2] Linear operators in function spaces [App. B, RB2] Euclidian vector spaces and matrices [App. C, RB2] Properties of the DFT and the FFT algorithm [App. B, RB2] Matlab scripts TFD_regularization_1D.m TFD_regularization_2D.m TFD_Error_1D.m TV_regulatization_1D.m 42