Extracting Depth and Matte using a Color-Filtered Aperture Yosuke Bando TOSHIBA + The University of Tokyo Bing-Yu Chen National Taiwan University Tomoyuki Nishita The University of Tokyo
Outline Background Related Work Our Method Results Conclusion 2
Computational Cameras Capture various scene properties High dynamic range, high resolution, Large field of view, reflectance, depth,... and more With elaborate imaging devices Camera arrays Additional optical elements [Wilburn 2005] [Nayar 1997] 3
Compact Computational Cameras Small devices Simple optical elements [Ng 2005] [Levin 2007] 4
Our Goal Capture scene properties With minimal modification to the camera 5
Our Goal Capture scene properties Depth maps Alpha mattes With minimal modification to the camera Captured image Depth Matte 6
Our Goal Capture scene properties Depth maps Alpha mattes With minimal modification to the camera Put color filters in a camera lens aperture Captured image Depth Matte Lens with color filters 7
Our Goal Capture scene properties Depth maps Alpha mattes With minimal modification to the camera Put color filters in a camera lens aperture This idea itself is not new Captured image Depth Matte Lens with color filters 8
Contents Background Related Work Our Method Results Conclusion 9
Previous Color-Filter Methods Extract (only) depth maps With low precision Or, a specialized flashbulb is used Spoils the visual quality of captured images [Amari 1992] [Chang 2002] 10
Coded Aperture Patterned mask in the aperture Changes frequency characteristics of defocus Facilitates blur identification/removal [Levin 2007] [Veeraraghavan 2007] Lens with a mask Captured image Amount of defocus blur ( depth) 11
Single-Lens Multi-View Capture Records light rays separately depending on their incident angle Enables light field rendering [Ng 2005] [Veeraraghavan 2007] [Adelson 1992] [Georgeiv 2006] [Liang 2008] 12
Matting Automatic matting by multiple cameras [McGuire 2005] 3 cameras with half mirrors Video matting [Joshi 2006] Array of 8 cameras Video matting 13
Our Method Features Automatic depth and matte extraction Single hand-held camera Single shot Contributions 1. Improved depth estimation 2. Novel matting algorithm for images captured thru a color-filtered aperture 14
Outline Background Related Work Our Method Results Conclusion 15
Our Method Color-filtered aperture Depth estimation Matting 16
Our Method Color-filtered aperture Depth estimation Matting 17
Our Prototype Camera Lens Canon EF 50mm f/1.8 II lens 18
Our Prototype Camera Lens Canon EF 50mm f/1.8 II lens Aperture part of the disassembled lens 19
Our Prototype Camera Lens Canon EF 50mm f/1.8 II lens Fujifilter SC-58, BPB-53, and BPB-45 20
Our Prototype Camera Lens Canon EF 50mm f/1.8 II lens Fujifilter SC-58, BPB-53, and BPB-45 Our prototype lens with color-filters 21
Our Prototype Camera Lens Took me just a few hours to fabricate Using a micro-screwdriver and a box cutter Canon EF 50mm f/1.8 II lens Fujifilter SC-58, BPB-53, and BPB-45 Our prototype lens with color-filters 22
Captured Image 23
Red Plane 24
Green Plane 25
Blue Plane 26
Captured Image Has depth-dependent color-misalignment NOT due to chromatic aberration 27
Why Do Colors Get Misaligned? Color filters Background Foreground object Lens Image sensor 28
Why Do Colors Get Misaligned? Color filters Background Foreground object Lens Image sensor 29
Why Do Colors Get Misaligned? Color filters Background Foreground object Lens Image sensor Color filters Background Lens Image sensor 30
Why Do Colors Get Misaligned? Color filters Background Foreground object Lens Image sensor Color filters Background Lens Image sensor 31
Our Method Color-filtered aperture Depth estimation Matting 32
Depth Estimation Our camera captures 3 views in the RGB planes Stereo reconstruction problem Red plane Green plane Blue plane 33
Depth Estimation Our camera captures 3 views in the RGB planes Stereo reconstruction problem However, their intensities donʼt match Contribution 1: improved correspondence measure between the RGB planes Red plane Green plane Blue plane 34
Original Image 35
Disparity = 1 36
Disparity = 2 37
Disparity = 3 38
Disparity = 4 39
Disparity = 5 40
Disparity = 6 41
When Is The Color Aligned? Disparity 42
Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] 43
Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] 44
Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] 45
Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] 1 B 0.5 1 0 0 R 0.5 1 0 0.5 G 46
Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] Misalign by 1 pixel 1 B Disparity = 0 Disparity = 1 0.5 1 0 0 R 0.5 1 0 0.5 G 47
Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] Misalign by 1 pixel Disparity = 0 Disparity = 1 B B 1 1 0.5 0.5 0 0 R 0.5 1 0 0.5 1 G 0 0 R 0.5 1 0 0.5 1 G 48
Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] Misalign by 3 pixels Disparity = 0 Disparity = 1 Disparity = 3 B B 1 1 0.5 0.5 0 0 R 0.5 1 0 0.5 1 G 0 0 R 0.5 1 0 0.5 1 G 49
Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] Misalign by 3 pixels Disparity = 0 Disparity = 1 Disparity = 3 B B B 1 1 1 0.5 0.5 0.5 0 0 R 0.5 1 0 0.5 1 G 0 0 R 0.5 1 0 0.5 1 G 0 0 R 0.5 1 0 0.5 1 G 50
Color-Alignment Measure L = σ λ 0λ1λ 2 2 r σ 2 g σ 2 b Variances along the principal axes (eigenvalues) Variances along the RGB axes Disparity = 0 Disparity = 1 Disparity = 3 B B B 1 1 1 0.5 0.5 0.5 0 0 R 0.5 1 0 0.5 1 G 0 0 R 0.5 1 0 0.5 1 G 0 0 R 0.5 1 0 0.5 1 G 51
Color-Alignment Measure L = σ λ 0λ1λ 2 2 r σ 2 g σ 2 b Variances along the principal axes (eigenvalues) Variances along the RGB axes Disparity = 0 Disparity = 1 Disparity = 3 B L = 0.003 BL = 0.11 BL = 0.39 1 1 1 0.5 0.5 0.5 0 0 R 0.5 1 0 0.5 1 G 0 0 R 0.5 1 0 0.5 1 G 0 0 R 0.5 1 0 0.5 1 G 52
Depth Estimation Solve for the disparity that makes the color-alignment measure minimum Captured image Pixel-wise estimates (intensity depth) 53
Depth Estimation Solve for the disparity that makes the color-alignment measure minimum With smoothness constraints Graph-cut optimization [Boykov 2001] Captured image Pixel-wise estimates (intensity depth) After graph-cuts 54
Our Method Color-filtered aperture Depth estimation Matting 55
Matting Problem of estimating foreground opacity I( x, y) = α( x, y) F( x, y) + (1 α( x, y)) B( x, y) Input image Matte Foreground color Background color 56
Trimap Assigns each pixel to one of 3 labels Strictly foreground (α = 1) Strictly background (α = 0) Unknown (α to be computed) strictly background un k nown Captured image strictly foreground Trimap 57
Trimap Assigns each pixel to one of 3 labels Strictly foreground (α = 1) Strictly background (α = 0) Unknown (α to be computed) Generated from the depth map strictly background un k nown strictly foreground Captured image Trimap 58
Trimap-Based Matting Captured image Trimap 59
Trimap-Based Matting Errors remain where the foreground and background colors are similar Captured image Trimap [Levin 2006] 60
Trimap-Based Matting Errors remain where the foreground and background colors are similar Contribution 2: matte error correction using color misalignment cues Captured image Trimap [Levin 2006] 61
Basic Idea Estimate foreground and background colors based on the current matte Captured image Current matte 62
Basic Idea Estimate foreground and background colors based on the current matte Captured image Estimated foreground color Current matte Estimated background color 63
Basic Idea Estimate foreground and background colors based on the current matte Detect inconsistent color misalignments Captured image Estimated foreground color Current matte Estimated background color 64
Synthetic Toy Example Synthesized input image Ground truth matte 65
Synthetic Toy Example Background Synthesized input image Ground truth matte Camera 66
Synthetic Toy Example Background Synthesized input image Ground truth matte Foreground Camera 67
Synthetic Toy Example Hard example Similar foreground and background colors Background Foreground Foreground Background 68
Synthetic Toy Example Hard example Foreground Background Similar foreground and background colors But solvable Color misalignment cues from ʻxʼ textures Foreground Background 69
Trimap-Based Matting Background Foreground Input image Trimap 70
Trimap-Based Matting Background Foreground Input image Estimated matte Trimap 71
Trimap-Based Matting Background Foreground Input image Estimated matte Trimap Ground truth 72
Matting Algorithm Background Foreground Current matte Input image 73
Matting Algorithm Background Foreground Current matte Estimated foreground color Input image Estimated background color 74
Matting Algorithm Current matte Estimated foreground color Detected color-misalignment Input image Estimated background color 75
Matting Algorithm Current matte Estimated foreground color Detected color-misalignment Input image Estimated background color Detected color-alignment 76
Matting Algorithm Current matte Estimated foreground color Detected color-misalignment Input image Estimated background color Detected color-alignment 77
Matting Algorithm Background Foreground Final matte Input image 78
Matting Algorithm Background Foreground Final matte Ground truth Input image 79
Outline Background Related Work Our Method Results Conclusion 80
Results of Depth & Matte Extraction Captured image Depth Matte 81
Comparison: Depth Estimation With the previous color-filter methods Local estimation to show raw performance Captured Captured [Amari 1992] [Chang 2002] Our method 82
Comparison: Matting With the trimap-based matting methods The trimaps were generated by our method Trimap Trimap [Levin 2006] [Wang 2007] Our method 83
Comparison with Ground Truth Mattes Ground truth Composite image (color-aligned) Composite image (color-misaligned) [Levin 2006] [Wang 2007] Our method 84
Image Editing Image composition Color-alignment reconstruction Novel view synthesis Refocusing Video matting 85
Example 1: Composition Matte Different Background Matte Captured image Different Background Composite 86
Example 2: Color-Alignment Reconst. Cancel the estimated misalignment Foreground color Matte Re-compose Background color Cancel the estimated misalignment 87
Reconstructed Image 88
Captured Image 89
Examples 3 & 4: View/Focus Synthesis 90
Example 5: Video Matting 91
Outline Background Related Work Our Method Results Conclusion 92
Conclusion Automatic depth and matte extraction using a color-filtered aperture Improved depth estimation Novel matting algorithm 93
Conclusion Automatic depth and matte extraction using a color-filtered aperture Improved depth estimation Novel matting algorithm Easy-to-use computational photography Put color filters in a camera lens Take a single photo with a hand-held camera 94
Conclusion Automatic depth and matte extraction using a color-filtered aperture Improved depth estimation Novel matting algorithm Easy-to-use computational photography Put color filters in a camera lens Take a single photo with a hand-held camera Limitation Entirely red objects cannot be handled 95
Thank You!! Any questions? Acknowledgments Takeshi Naemura Yusuke Iguchi Takuya Saito Johanna Wolf Zoltan Szego Paulo Silva Saori Horiuchi 96
Other Possible Filter Arrangements Our prototype More lightefficient Even more lightefficient but with lower depth resolution Higher depth resolution Simple horizontal arrangement Arbitrary shapes Combination with coded aperture Multi-aperture 97