Numerical Methods For Image Restoration

Transcription

1 Numerical Methods For Image Restoration CIRAM Alessandro Lanza University of Bologna, Italy Faculty of Engineering CIRAM

2 Outline 1. Image Restoration as an inverse problem 2. Image degradation models: noises, blurs 3. Ill-conditioning -> Regularization 4. Variational MAP approaches 5. Blind Deblurring and Inpainting 6. Motion Deblurring 7. Deblurring of particular images

3 Image Restoration AIM: Restore Degraded Images 3

4 Image Restoration AIM: Restore Degraded Images

5 Restoration Problem Degradation model: u 0 = k u + n = + Observed image Known/Unknown Point Spread Function Unknown true image Known/Unknown noise Goal: Given u 0, recover u Inverse Problem

6 Input (cause) x Problems Model (rule) Φ(x) p( x) Output (effect) y

7 Input (cause) x Problems Model (rule) Φ(x) p( x) Output (effect) y Direct Problem (Degradation: blur + noise)

8 Input (cause) x Problems Model (rule) Φ(x) p( x) Output (effect) y Identification Problem (Degradation model estimation)

9 Input (cause) x Problems Model (rule) Φ(x) p( x) Output (effect) y Inverse Problem (Restoration: deblurring + denoising)

10 Input (cause) x Problems Model (rule) Φ(x) p( x) Output (effect) y Inverse + Identification Problem (Degradation model estimation + Restoration: blind deblurring + denoising)

11 Inverse problems are everywhere Reconstruct the density distribution within the human body by means of Xray projections Inferring seismic properties of the Earth s interior from surface observations Invert the observed data in order to derive the internal properties of the Sun

12 Image Restoration

13 Degradation model Continuous degradation model: u ( x) k( x, y) u( y) dy n( x) x R 0 2 Perturbed observed image Blur and noise-free image Data noise Point Spread Function: Linear space-variant blur

14 Degradation model Continuous degradation model: u ( x) k( x y) u( y) dy n( x) x R 0 2 Perturbed observed image Blur and noise-free image Data noise Point Spread Function: Linear space-invariant blur Integral equation can be expressed as u k u n 0 convolution

15 Noise Original salt and pepper speckle Gauss-add Gauss-molt independent identically distributed (i.i.d.)

16 Blur PSFs with sharp edges: PSFs with smooth transitions

17 Space variant - space invariant blur Two causes for motion blur Blur is the same Blur is different Hand shaking Object motion

18 Degradation model Continuous degradation model: u ( x) k( x y) u( y) dy n( x) x R 0 u k u n 0 Discretization yields: u = Au 0 with matrix A block Toeplitz with Toeplitz blocks 2 severely ill-conditioned regularization

19 Degradation model Discrete degradation model: u = Au 0 with matrix A block Toeplitz with Toeplitz blocks severely ill-conditioned Deblurring: - Invert low-pass filtering - Backward diffusion process - Decrease entropy regularization - Basic method: TSVD (Truncated Singular Value Decomposition) - Frequency domain methods: Wiener, Richardson-Lucy, wavelets

20 Boundary conditions Zero Dirichlet Periodic Reflexive (Neumann) Antireflexive: S.Serra-Capizzano, M. Donatelli, et al. Synthetic Boundary Conditions: Y. Wai, J. Nagy

21 Ill-conditioning effect u=a -1 b Solution Au=b: add 0.1% noise to rhs b b e u A ( b e) A b A e u A e u=a -1 b

22 Variational approaches Minimize the energy functional: û arg min E ( u;k ) 1 ( k u u ) ( u ) d Data term: enforces the match between the sought image and the observed image via the blur model Smoothness term: brings in regularity assumptions about the unknown image ( s ) s 2 Tikhonov ( s ) s TV ( s ) ln( 1 s / ) Perona Malik

23 Variational approaches as MAP estimation Maximize the posterior probability: û arg max p( u u ;k ) Minimize the Logarithmic Energy: û arg min E( u u 0;k ) arg min ( log p( u u 0;k )) But (from Bayes rule): p( u u ;k ) p( u u;k ) p( u;k ) p( u u;k ) p( u ) Hence: û arg min ( E( u u 0;k ) E( u 0 u;k ) E( u )) i.i.d Gaussain noise + exponential prior for image gradients: û arg min E ( u;k ) ( k u u ) ( u ) d 2 0 0

24 General model for variational deblurring û arg min E ( u;k ) 1 ( k u u ) ( u ) d arg min G( x, y,u, u ) d where u (u,u ) Euler-Lagrange equations (can be linear or non-linear): dg u 0 in Gu G x u G y u 0 ( x,y), 0 on Ω du x y n u t E ( u;k ) k ( k u u ) D( u ) u D( u ) ( '( u ) ) u 0 0 x y

25 General model for variational deblurring u t 0 0 E ( u;k ) k ( k u u ) D( u ) in u 0 on Ω n Second order non-linear elliptic PDEs u D( u ) ( '( u ) ) u 2 ( u ) u D( u ) u Tikhonov linear u ( u ) u D( u ) ( ) TV non linear u

26 General model for variational deblurring t u u( t 0 ) u in Ω t 0 0 u E ( u;k ) k ( k u u ) D( u ) in 0 Gradient descent u 0 on Ω n parabolic PDEs 2 u D( u ) ( '( u ) ) u ( u ) u D( u ) u Tikhonov linear u ( u ) u D( u ) ( ) TV non linear u

27 Tikhonov and TV variational deblurring t u u( t 0 ) u in Ω t 0 0 u E ( u;k ) k ( k u u ) D( u ) in 0 Gradient descent u 0 on Ω n parabolic PDEs u D( u ) ( '( u ) ) u Tikhonov: linear diffusion-reaction PDE...classical FD schemes TV: non-linear diffusion-reaction PDE...ad-hoc num. schemes

28 Comparison (only blur) Tikhonov Total Variation

29 Comparison (blur and noise) Tikhonov Total Variation

30 TV Regularization I Deg of Blur II Deg of Blur III Deg of Blur Blurred and Noisy Images Total Variation Regularization

31 Edge-enhanched regularization

32 PDE-based approaches Linear (forward parabolic): smoothing [Witkin-Koenderink 83-84, heat equation = Gaussian filter] ut u ut G u0 2t Scale-space u 0

33 PDE-based approaches Non-Linear (forward-backward parabolic): smoothing-enhancing [Perona Malik 87,Cattè Lions Morel 92, non-linear diffusion ] u ( g( u ) u ) t 2 2 g( u ) 1 1 u k 2 2 g(s) 2 u g( u ) exp( ) 2 k 2 s

34 PDE-based approaches Anisotropic Non-linear diffusion: smoothing-enhancing [Weickert, 98, tensor diffusion] u ( D( J ( u )) u ) t Edge-enhancing Coherence-enhancing

35 PDE-based approaches Non-Linear (hyperbolic): enhancing [Osher Rudin 90, shock filters ] u t u x sign( u xx )

36 Blind Deblurring MAP estimation Maximize the posterior probability: û arg max p( u,k u ) Minimize the Logarithmic Energy: û arg min E( u,k u 0 ) arg min ( log p( u,k u 0 )) But (from Bayes rule): p( u,k u ) p( u u,k ) p( u,k ) p( u u,k ) p( u )p( k ) Hence: û arg min ( E( u,k u 0 ) E( u 0 u,k ) E( u ) E( k )) i.i.d Gaussain noise + exponential prior for image gradients: û arg min ( k u u ) ( u ) ' ( k ) d 2, ' 0 0

37 TV Variational Blind Deblurring 2 0, ' û arg min ( k u u ) u ' k d ), ( ), ( 1, ), ( 0,, y x k y x k dxdy y x k k u ), ( min ), ( ), ( min ), ( k u F k u F k u F k u F n k n n n u n n Alternating Minimization Algorithm: Globally convergent with H 1 regularization. (C. and Wong (IEEE TIP, 1998))

38 TV Variational Blind/Non-Blind Deblurring Blind non-blind Recovered Image PSF l = 210 6, l = An out-of-focus blur is recovered automatically Recovered blind deconvolution images almost as good as non-blind Edges well-recovered in image and PSF 38

39 TV (Blind) Deblurring, Denoising, Inpainintg Image degraded by blur, noise and missing regions - Scratches - Occlusions - Defects in films/sensors

40 TV (Blind) Deblurring E u u k u u k 2 ( ) 0 ' R R R Out-of-focus blur Gaussian blur (harder: zero-crossing in the frequency domain)

41 TV Inpainintg E ( u) u u u ED E 0 2 Scratch Removal Graffiti Removal

42 TV (Blind) Deblurring and Inpainting = Problems with inpaint then deblur: Original Signal PSF Blurred Signal Blurred + Occluded = Original Signal PSF Blurred Signal Blurred + Occluded = Inpaint first reduce plausible solutions Should pick the solution using more information

43 TV (Blind) Deblurring and Inpainting Problems with deblur then inpaint: Original Occluded Support of PSF Dirichlet Neumann Inpainting Different BC s correspond to different intensities in inpaint regions. Most local BC s do not respect global geometric structures

44 TV (Blind) Deblurring and Inpainting E u u k u u k ( ) 2 0 ' ED E E

45 TV (Blind) Deblurring and Inpainting Inpaint then deblur (many ringings) Deblur then inpaint (many artifacts)

46 TV (Blind) Deblurring and Inpainting the vertical strip is completed the vertical strip is not completed use higher order inpainting methods E.g. Euler s elastica, curvature driven diffusion

47 (Blind) Motion Deblurring Q. Shan et al., High-quality Motion Deblurring from a Single Image, SIGGRAPH2008

50 (Blind) Motion Deblurring Q. Shan et al., High-quality Motion Deblurring from a Single Image, SIGGRAPH2008 iterations

51 (Blind) Motion Deblurring Q. Shan et al., High-quality Motion Deblurring from a Single Image, SIGGRAPH2008 Hand-held camera

57 (Blind) Motion Deblurring Q. Shan et al., High-quality Motion Deblurring from a Single Image, SIGGRAPH2008 Hand-held camera: non space-invariant blur (slight camera rotation, motion parallax)

58 (Blind) Deblurring of 1-D Bar Codes S. Esedoglu, Blind deconvolution of bar code signals, Inverse Problems, 2004 i.i.d additive Gaussian noise u = α G σ u 0 + n scalar gain Spatially-invariant Gaussian blur

59 (Blind) Deblurring of 1-D Bar Codes S. Esedoglu, Blind deconvolution of bar code signals, Inverse Problems, 2004 TV regularization (no a priori information on the number of bars) min E λ u, α, σ = u R + λ α G σ u u 0 2 R Euler-Lagrange equations Gradient descent

60 (Blind) Deblurring of 1-D Bar Codes S. Esedoglu, Blind deconvolution of bar code signals, Inverse Problems, 2004

61 Deblurring of 2-D Bar Codes R. Choksi et al., Anisotropic total variation regularized L 1 -approximation and denoising/deblurring of 2D bar codes, Inverse Problems and Imaging, 2011 u = k u 0 + n Standard isotropic TV: rounding off of corners anisotropic TV L2 fidelity: loss of contrast L1 fidelity min E λ u; k = u x + u y + λ k u u 0 R 2 R 2 approximated as finite dimensional convex optimization problem discretization by using standard forward finite differences and quadrature Linear program reformulation

62 Deblurring of 2-D Bar Codes R. Choksi et al., Anisotropic total variation regularized L 1 -approximation and denoising/deblurring of 2D bar codes, Inverse Problems and Imaging, 2011

63 Blind Deblurring

64 Work in progress Variational constraint on noise whiteness Fractional-order diffusion

65 Thanks for your attention!