Single Depth Image Super Resolution and Denoising Using Coupled Dictionary Learning with Local Constraints and Shock Filtering

Transcription

1 Single Depth Image Super Resolution and Denoising Using Coupled Dictionary Learning with Local Constraints and Shock Filtering Jun Xie 1, Cheng-Chuan Chou 2, Rogerio Feris 3, Ming-Ting Sun 1 1 University of Washington, 2 Industrial Technology Research Institute (ITRI), Taiwan, 3 IBM T. J. Watson Research Center Hawthorne, U.S.

2 Outline Introduction Our contribution Simulation results Conclusion

3 Motivation Depth images often are low-resolution and noisy which affects the quality of the applications Human are sensitive to 3D noises and jagged edges 2D patch 3D patch

4 Objective Input: Single noisy, low-resolution depth map Output: A clean, increased resolution depth map

5 Related Work on Depth Super Resolution Fusion of multiple depth images Use a guiding high resolution color image However, multiple depth maps or guiding color images at the target resolution often are unavailable. Q. Yang et al., Spatial-depth Super Resolution for Range Images, CVPR 2007.

6 Related Work Learning-based single Image super resolution J. Yang, J. Wright, T. Huang, Y. Ma, Image Super-resolution as Sparse Representation of Raw Image Patches, CVPR, 2008.

7 Problems from the Properties of Low-Resolution Depth Maps Lack of texture -> Overfitting Noisy and jagged edges

8 Our Contribution Propose a dictionary learning based algorithm by Adding local constraints into the coupled dictionary learning process To prevent the dictionary from over-fitting Incorporating an adaptively regularized Shock filter To tackle the jagged edges and noises in the depth map

9 Our Coupled Dictionary Learning Training set - Divide images into patches Feature Extraction Low-res Images: [Gx, Gy, Gxx, Gyy] High-res Images: f h =y h -y l (y l is the bilinear interpolation result of y l )

10 Our Coupled Dictionary Learning Impose a local constraint Objective: Given training feature patches x, learn a dictionary d such that: Linear Combination of dictionary bases Local Constraint dc, 2 j min ( x d c d x c ) i i i j i i j c: weighting coefficient vector 2

11 For each low resolution patch, only the dictionary bases which are most similar to it are selected, effectively preventing the overfitting problem Preserve the manifold assumption in the feature space and keep the locality constraint

12 Sparse Reconstruction Based on the Learned Coupled Dictionary i ' 2 i min l l i.. i c c s d c s t c L i Shared coeffs. 0 Linear combination of s d c i ' h h i high-res dictionary bases d contains 10% of dictionary atoms with closest distances to the low-resolutions patches

13 Edge Denoising Based on Adaptively Regularized Shock Filter Why Shock filter? Edge preserving Remove jagged noises Good smoothing of depth images which have less texture

14 Edge Denoising Based on Regularized Shock Filter Shock term for edge enhancement Smoothing in the tangent direction 2 I I t arctan( a I m ( )) I I I Smoothing in the gradient direction G. Gilboa, N. Sochen, Y. Y. Zeevi, Image Enhancement and Denoising by Complex Diffusion Processes, PAMI, vol. 26, issue 8, pp , 2004.

15 Adaptively Regularized Denoising Shock Filter Adaptive weight Small beta Corners Plain Region (little Smoothing) Edges (Smoothing along tangent direction) Quad Tree Large beta 2 I I t arctan( a I m ( )) I I I

16 Edge Denoising Based on Adaptively Regularized Shock Filter Filtering result

17 Edge Denoising Based on Adaptively Regularized Shock Filter Filtering result Low-res Result of [1] Ours [1] J. Yang et al., Image Super-resolution as Sparse Representation of Raw Image Patches, CVPR, 2008.

18 Quantitative Results RMSE COMPARISON SCALED *3 RMSE COMPARISON SCALED *4 Cones Venus Teddy Tsukuba Cones Venus Teddy Tsukuba Nearest Neighbor Sparse coding [1] K-SVD based [2] Aodha et. al [3] Tsai et. al in [4] Hornacek. et. al [5] Our (w/o Shock filter) Our (with Shock filter) [1] J. Yang et al., Image Super-resolution as Sparse Representation of Raw Image Patches, CVPR, [2] R. Zeyde et al., On Single Image Scale-up using Sparse Representations, Curves and Surfaces, [3] O. M. Aodha et al., Patch based Synthesis for Single Depth Image Super-resolution, ECCV, [4] C. Tsai et al., Context-aware Single Image Super-resolution Using Locality-constrained Group Sparse Representation, VCIP, [5] M. Hornacek, et. al, Depth super resolution by rigid body self-similarity in 3d, CVPR, 2013.

19 Visual Results Ours Nearest Neighbor (NN) [1] [2] Ours [1] J. Yang, J. Wright, T. Huang, and Y. Ma, Image Super-resolution as Sparse Representation of Raw Image Patches, CVPR, [2] O. M. Aodha, N. D. Campbell, A. Nair, and G. J. Brostow, Patch based Synthesis for Single Depth Image Super-resolution, ECCV, 2012.

20 Visual Results NN [1] [2] Ours [1] R. Zeyde, M. Elad, M. Protter, On Single Image Scale-up using Sparse Representations, in Curves and Surfaces, [2] O. M. Aodha, N. D. Campbell, A. Nair, and G. J. Brostow, Patch based Synthesis for Single Depth Image Super-resolution, in ECCV, 2012.

21 3D Visual Results NN [1] [2] Ours [1] R. Zeyde, M. Elad, M. Protter, On Single Image Scale-up using Sparse Representations, Curves and Surfaces, [2] O. M. Aodha, N. D. Campbell, A. Nair, and G. J. Brostow, Patch based Synthesis for Single Depth Image Super-resolution, ECCV, 2012.

22 View Synthesis Results GT O. M. Aodha, et. al Patch based Synthesis for Single Depth Image Super-resolution, ECCV, C. Tsai et. al Context-aware Single Image Superresolution Using Locality-constrained Group Sparse Representation, VCIP, Ours

23 Conclusion Propose a dictionary learning based algorithm by - Adding local constraints to prevent the dictionary from over-fitting and improve the result - Incorporate an adaptively regularized Shock filter to tackle the jagged edges and noises in the depth map Simulation results confirm the effectiveness of the proposed algorithm

24 Questions? Thanks!