Numerical Methods For Image Restoration CIRAM Alessandro Lanza University of Bologna, Italy Faculty of Engineering CIRAM
Outline 1. Image Restoration as an inverse problem 2. Image degradation models: noises, blurs 3. Ill-conditioning -> Regularization 4. Variational MAP approaches 5. Blind Deblurring and Inpainting 6. Motion Deblurring 7. Deblurring of particular images
Image Restoration AIM: Restore Degraded Images 3
Image Restoration AIM: Restore Degraded Images
Restoration Problem Degradation model: u 0 = k u + n = + Observed image Known/Unknown Point Spread Function Unknown true image Known/Unknown noise Goal: Given u 0, recover u Inverse Problem
Input (cause) x Problems Model (rule) Φ(x) p( x) Output (effect) y
Input (cause) x Problems Model (rule) Φ(x) p( x) Output (effect) y Direct Problem (Degradation: blur + noise)
Input (cause) x Problems Model (rule) Φ(x) p( x) Output (effect) y Identification Problem (Degradation model estimation)
Input (cause) x Problems Model (rule) Φ(x) p( x) Output (effect) y Inverse Problem (Restoration: deblurring + denoising)
Input (cause) x Problems Model (rule) Φ(x) p( x) Output (effect) y Inverse + Identification Problem (Degradation model estimation + Restoration: blind deblurring + denoising)
Inverse problems are everywhere Reconstruct the density distribution within the human body by means of Xray projections Inferring seismic properties of the Earth s interior from surface observations Invert the observed data in order to derive the internal properties of the Sun
Image Restoration
Degradation model Continuous degradation model: u ( x) k( x, y) u( y) dy n( x) x R 0 2 Perturbed observed image Blur and noise-free image Data noise Point Spread Function: Linear space-variant blur
Degradation model Continuous degradation model: u ( x) k( x y) u( y) dy n( x) x R 0 2 Perturbed observed image Blur and noise-free image Data noise Point Spread Function: Linear space-invariant blur Integral equation can be expressed as u k u n 0 convolution
Noise Original salt and pepper speckle Gauss-add Gauss-molt independent identically distributed (i.i.d.)
Blur PSFs with sharp edges: PSFs with smooth transitions
Space variant - space invariant blur Two causes for motion blur Blur is the same Blur is different Hand shaking Object motion
Degradation model Continuous degradation model: u ( x) k( x y) u( y) dy n( x) x R 0 u k u n 0 Discretization yields: u = Au 0 with matrix A block Toeplitz with Toeplitz blocks 2 severely ill-conditioned regularization
Degradation model Discrete degradation model: u = Au 0 with matrix A block Toeplitz with Toeplitz blocks severely ill-conditioned Deblurring: - Invert low-pass filtering - Backward diffusion process - Decrease entropy regularization - Basic method: TSVD (Truncated Singular Value Decomposition) - Frequency domain methods: Wiener, Richardson-Lucy, wavelets
Boundary conditions Zero Dirichlet Periodic Reflexive (Neumann) Antireflexive: S.Serra-Capizzano, M. Donatelli, et al. Synthetic Boundary Conditions: Y. Wai, J. Nagy
Ill-conditioning effect u=a -1 b Solution Au=b: add 0.1% noise to rhs b b e 1 1 1 1 u A ( b e) A b A e u A e u=a -1 b
Variational approaches Minimize the energy functional: û arg min E ( u;k ) 1 ( k u u ) ( u ) d 2 0 2 Data term: enforces the match between the sought image and the observed image via the blur model Smoothness term: brings in regularity assumptions about the unknown image ( s ) s 2 Tikhonov ( s ) s 2 2 2 2 2 TV ( s ) ln( 1 s / ) Perona Malik
Variational approaches as MAP estimation Maximize the posterior probability: û arg max p( u u ;k ) Minimize the Logarithmic Energy: û arg min E( u u 0;k ) arg min ( log p( u u 0;k )) But (from Bayes rule): p( u u ;k ) p( u u;k ) p( u;k ) p( u u;k ) p( u ) 0 0 0 Hence: û arg min ( E( u u 0;k ) E( u 0 u;k ) E( u )) i.i.d Gaussain noise + exponential prior for image gradients: û arg min E ( u;k ) ( k u u ) ( u ) d 2 0 0
General model for variational deblurring û arg min E ( u;k ) 1 ( k u u ) ( u ) d 2 0 2 arg min G( x, y,u, u ) d where u (u,u ) Euler-Lagrange equations (can be linear or non-linear): dg u 0 in Gu G x u G y u 0 ( x,y), 0 on Ω du x y n u t E ( u;k ) k ( k u u ) D( u ) u D( u ) ( '( u ) ) u 0 0 x y
General model for variational deblurring u t 0 0 E ( u;k ) k ( k u u ) D( u ) in u 0 on Ω n Second order non-linear elliptic PDEs u D( u ) ( '( u ) ) u 2 ( u ) u D( u ) u Tikhonov linear u ( u ) u D( u ) ( ) TV non linear u
General model for variational deblurring t u u( t 0 ) u in Ω t 0 0 u E ( u;k ) k ( k u u ) D( u ) in 0 Gradient descent u 0 on Ω n parabolic PDEs 2 u D( u ) ( '( u ) ) u ( u ) u D( u ) u Tikhonov linear u ( u ) u D( u ) ( ) TV non linear u
Tikhonov and TV variational deblurring t u u( t 0 ) u in Ω t 0 0 u E ( u;k ) k ( k u u ) D( u ) in 0 Gradient descent u 0 on Ω n parabolic PDEs u D( u ) ( '( u ) ) u Tikhonov: linear diffusion-reaction PDE...classical FD schemes TV: non-linear diffusion-reaction PDE...ad-hoc num. schemes
Comparison (only blur) Tikhonov Total Variation
Comparison (blur and noise) Tikhonov Total Variation
TV Regularization I Deg of Blur II Deg of Blur III Deg of Blur Blurred and Noisy Images Total Variation Regularization
Edge-enhanched regularization
PDE-based approaches Linear (forward parabolic): smoothing [Witkin-Koenderink 83-84, heat equation = Gaussian filter] ut u ut G u0 2t Scale-space u 0
PDE-based approaches Non-Linear (forward-backward parabolic): smoothing-enhancing [Perona Malik 87,Cattè Lions Morel 92, non-linear diffusion ] u ( g( u ) u ) t 2 2 g( u ) 1 1 u k 2 2 g(s) 2 u g( u ) exp( ) 2 k 2 s
PDE-based approaches Anisotropic Non-linear diffusion: smoothing-enhancing [Weickert, 98, tensor diffusion] u ( D( J ( u )) u ) t Edge-enhancing Coherence-enhancing
PDE-based approaches Non-Linear (hyperbolic): enhancing [Osher Rudin 90, shock filters ] u t u x sign( u xx )
Blind Deblurring MAP estimation Maximize the posterior probability: û arg max p( u,k u ) Minimize the Logarithmic Energy: û arg min E( u,k u 0 ) arg min ( log p( u,k u 0 )) But (from Bayes rule): p( u,k u ) p( u u,k ) p( u,k ) p( u u,k ) p( u )p( k ) 0 0 0 Hence: û arg min ( E( u,k u 0 ) E( u 0 u,k ) E( u ) E( k )) i.i.d Gaussain noise + exponential prior for image gradients: û arg min ( k u u ) ( u ) ' ( k ) d 2, ' 0 0
TV Variational Blind Deblurring 2 0, ' û arg min ( k u u ) u ' k d ), ( ), ( 1, ), ( 0,, y x k y x k dxdy y x k k u ), ( min ), ( ), ( min ), ( 1 1 1 1 k u F k u F k u F k u F n k n n n u n n Alternating Minimization Algorithm: Globally convergent with H 1 regularization. (C. and Wong (IEEE TIP, 1998))
TV Variational Blind/Non-Blind Deblurring Blind non-blind Recovered Image PSF l = 210 6, l = 1.510 5 An out-of-focus blur is recovered automatically Recovered blind deconvolution images almost as good as non-blind Edges well-recovered in image and PSF 38
TV (Blind) Deblurring, Denoising, Inpainintg Image degraded by blur, noise and missing regions - Scratches - Occlusions - Defects in films/sensors
TV (Blind) Deblurring E u u k u u k 2 ( ) 0 ' 2 2 2 R R R Out-of-focus blur Gaussian blur (harder: zero-crossing in the frequency domain)
TV Inpainintg E ( u) u u u ED E 0 2 Scratch Removal Graffiti Removal
TV (Blind) Deblurring and Inpainting = Problems with inpaint then deblur: Original Signal PSF Blurred Signal Blurred + Occluded = Original Signal PSF Blurred Signal Blurred + Occluded = Inpaint first reduce plausible solutions Should pick the solution using more information
TV (Blind) Deblurring and Inpainting Problems with deblur then inpaint: Original Occluded Support of PSF Dirichlet Neumann Inpainting Different BC s correspond to different intensities in inpaint regions. Most local BC s do not respect global geometric structures
TV (Blind) Deblurring and Inpainting E u u k u u k ( ) 2 0 ' ED E E
TV (Blind) Deblurring and Inpainting Inpaint then deblur (many ringings) Deblur then inpaint (many artifacts)
TV (Blind) Deblurring and Inpainting the vertical strip is completed the vertical strip is not completed use higher order inpainting methods E.g. Euler s elastica, curvature driven diffusion
(Blind) Motion Deblurring Q. Shan et al., High-quality Motion Deblurring from a Single Image, SIGGRAPH2008
(Blind) Motion Deblurring Q. Shan et al., High-quality Motion Deblurring from a Single Image, SIGGRAPH2008
(Blind) Motion Deblurring Q. Shan et al., High-quality Motion Deblurring from a Single Image, SIGGRAPH2008
(Blind) Motion Deblurring Q. Shan et al., High-quality Motion Deblurring from a Single Image, SIGGRAPH2008 iterations
(Blind) Motion Deblurring Q. Shan et al., High-quality Motion Deblurring from a Single Image, SIGGRAPH2008 Hand-held camera
(Blind) Motion Deblurring Q. Shan et al., High-quality Motion Deblurring from a Single Image, SIGGRAPH2008 Hand-held camera
(Blind) Motion Deblurring Q. Shan et al., High-quality Motion Deblurring from a Single Image, SIGGRAPH2008 Hand-held camera
(Blind) Motion Deblurring Q. Shan et al., High-quality Motion Deblurring from a Single Image, SIGGRAPH2008 Hand-held camera
(Blind) Motion Deblurring Q. Shan et al., High-quality Motion Deblurring from a Single Image, SIGGRAPH2008 Hand-held camera
(Blind) Motion Deblurring Q. Shan et al., High-quality Motion Deblurring from a Single Image, SIGGRAPH2008 Hand-held camera
(Blind) Motion Deblurring Q. Shan et al., High-quality Motion Deblurring from a Single Image, SIGGRAPH2008 Hand-held camera: non space-invariant blur (slight camera rotation, motion parallax)
(Blind) Deblurring of 1-D Bar Codes S. Esedoglu, Blind deconvolution of bar code signals, Inverse Problems, 2004 i.i.d additive Gaussian noise u = α G σ u 0 + n scalar gain Spatially-invariant Gaussian blur
(Blind) Deblurring of 1-D Bar Codes S. Esedoglu, Blind deconvolution of bar code signals, Inverse Problems, 2004 TV regularization (no a priori information on the number of bars) min E λ u, α, σ = u R + λ α G σ u u 0 2 R Euler-Lagrange equations Gradient descent
(Blind) Deblurring of 1-D Bar Codes S. Esedoglu, Blind deconvolution of bar code signals, Inverse Problems, 2004
Deblurring of 2-D Bar Codes R. Choksi et al., Anisotropic total variation regularized L 1 -approximation and denoising/deblurring of 2D bar codes, Inverse Problems and Imaging, 2011 u = k u 0 + n Standard isotropic TV: rounding off of corners anisotropic TV L2 fidelity: loss of contrast L1 fidelity min E λ u; k = u x + u y + λ k u u 0 R 2 R 2 approximated as finite dimensional convex optimization problem discretization by using standard forward finite differences and quadrature Linear program reformulation
Deblurring of 2-D Bar Codes R. Choksi et al., Anisotropic total variation regularized L 1 -approximation and denoising/deblurring of 2D bar codes, Inverse Problems and Imaging, 2011
Blind Deblurring
Work in progress Variational constraint on noise whiteness Fractional-order diffusion
Thanks for your attention!