Extracting Depth and Matte using a Color-Filtered Aperture

Transcription

1 Extracting Depth and Matte using a Color-Filtered Aperture Yosuke Bando TOSHIBA + The University of Tokyo Bing-Yu Chen National Taiwan University Tomoyuki Nishita The University of Tokyo

2 Outline Background Related Work Our Method Results Conclusion 2

3 Computational Cameras Capture various scene properties High dynamic range, high resolution, Large field of view, reflectance, depth,... and more With elaborate imaging devices Camera arrays Additional optical elements [Wilburn 2005] [Nayar 1997] 3

4 Compact Computational Cameras Small devices Simple optical elements [Ng 2005] [Levin 2007] 4

5 Our Goal Capture scene properties With minimal modification to the camera 5

6 Our Goal Capture scene properties Depth maps Alpha mattes With minimal modification to the camera Captured image Depth Matte 6

7 Our Goal Capture scene properties Depth maps Alpha mattes With minimal modification to the camera Put color filters in a camera lens aperture Captured image Depth Matte Lens with color filters 7

8 Our Goal Capture scene properties Depth maps Alpha mattes With minimal modification to the camera Put color filters in a camera lens aperture This idea itself is not new Captured image Depth Matte Lens with color filters 8

9 Contents Background Related Work Our Method Results Conclusion 9

10 Previous Color-Filter Methods Extract (only) depth maps With low precision Or, a specialized flashbulb is used Spoils the visual quality of captured images [Amari 1992] [Chang 2002] 10

11 Coded Aperture Patterned mask in the aperture Changes frequency characteristics of defocus Facilitates blur identification/removal [Levin 2007] [Veeraraghavan 2007] Lens with a mask Captured image Amount of defocus blur ( depth) 11

12 Single-Lens Multi-View Capture Records light rays separately depending on their incident angle Enables light field rendering [Ng 2005] [Veeraraghavan 2007] [Adelson 1992] [Georgeiv 2006] [Liang 2008] 12

13 Matting Automatic matting by multiple cameras [McGuire 2005] 3 cameras with half mirrors Video matting [Joshi 2006] Array of 8 cameras Video matting 13

14 Our Method Features Automatic depth and matte extraction Single hand-held camera Single shot Contributions 1. Improved depth estimation 2. Novel matting algorithm for images captured thru a color-filtered aperture 14

15 Outline Background Related Work Our Method Results Conclusion 15

16 Our Method Color-filtered aperture Depth estimation Matting 16

17 Our Method Color-filtered aperture Depth estimation Matting 17

18 Our Prototype Camera Lens Canon EF 50mm f/1.8 II lens 18

19 Our Prototype Camera Lens Canon EF 50mm f/1.8 II lens Aperture part of the disassembled lens 19

20 Our Prototype Camera Lens Canon EF 50mm f/1.8 II lens Fujifilter SC-58, BPB-53, and BPB-45 20

21 Our Prototype Camera Lens Canon EF 50mm f/1.8 II lens Fujifilter SC-58, BPB-53, and BPB-45 Our prototype lens with color-filters 21

22 Our Prototype Camera Lens Took me just a few hours to fabricate Using a micro-screwdriver and a box cutter Canon EF 50mm f/1.8 II lens Fujifilter SC-58, BPB-53, and BPB-45 Our prototype lens with color-filters 22

23 Captured Image 23

24 Red Plane 24

25 Green Plane 25

26 Blue Plane 26

27 Captured Image Has depth-dependent color-misalignment NOT due to chromatic aberration 27

28 Why Do Colors Get Misaligned? Color filters Background Foreground object Lens Image sensor 28

29 Why Do Colors Get Misaligned? Color filters Background Foreground object Lens Image sensor 29

30 Why Do Colors Get Misaligned? Color filters Background Foreground object Lens Image sensor Color filters Background Lens Image sensor 30

31 Why Do Colors Get Misaligned? Color filters Background Foreground object Lens Image sensor Color filters Background Lens Image sensor 31

32 Our Method Color-filtered aperture Depth estimation Matting 32

33 Depth Estimation Our camera captures 3 views in the RGB planes Stereo reconstruction problem Red plane Green plane Blue plane 33

34 Depth Estimation Our camera captures 3 views in the RGB planes Stereo reconstruction problem However, their intensities donʼt match Contribution 1: improved correspondence measure between the RGB planes Red plane Green plane Blue plane 34

35 Original Image 35

36 Disparity = 1 36

37 Disparity = 2 37

38 Disparity = 3 38

39 Disparity = 4 39

40 Disparity = 5 40

41 Disparity = 6 41

42 When Is The Color Aligned? Disparity 42

43 Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] 43

44 Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] 44

45 Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] 45

46 Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] 1 B R G 46

47 Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] Misalign by 1 pixel 1 B Disparity = 0 Disparity = R G 47

48 Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] Misalign by 1 pixel Disparity = 0 Disparity = 1 B B R G 0 0 R G 48

49 Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] Misalign by 3 pixels Disparity = 0 Disparity = 1 Disparity = 3 B B R G 0 0 R G 49

50 Color-Alignment Measure Local color distribution of natural images tends to form a line [Omer 2004, Levin 2006] Misalign by 3 pixels Disparity = 0 Disparity = 1 Disparity = 3 B B B R G 0 0 R G 0 0 R G 50

51 Color-Alignment Measure L = σ λ 0λ1λ 2 2 r σ 2 g σ 2 b Variances along the principal axes (eigenvalues) Variances along the RGB axes Disparity = 0 Disparity = 1 Disparity = 3 B B B R G 0 0 R G 0 0 R G 51

52 Color-Alignment Measure L = σ λ 0λ1λ 2 2 r σ 2 g σ 2 b Variances along the principal axes (eigenvalues) Variances along the RGB axes Disparity = 0 Disparity = 1 Disparity = 3 B L = BL = 0.11 BL = R G 0 0 R G 0 0 R G 52

53 Depth Estimation Solve for the disparity that makes the color-alignment measure minimum Captured image Pixel-wise estimates (intensity depth) 53

54 Depth Estimation Solve for the disparity that makes the color-alignment measure minimum With smoothness constraints Graph-cut optimization [Boykov 2001] Captured image Pixel-wise estimates (intensity depth) After graph-cuts 54

55 Our Method Color-filtered aperture Depth estimation Matting 55

56 Matting Problem of estimating foreground opacity I( x, y) = α( x, y) F( x, y) + (1 α( x, y)) B( x, y) Input image Matte Foreground color Background color 56

57 Trimap Assigns each pixel to one of 3 labels Strictly foreground (α = 1) Strictly background (α = 0) Unknown (α to be computed) strictly background un k nown Captured image strictly foreground Trimap 57

58 Trimap Assigns each pixel to one of 3 labels Strictly foreground (α = 1) Strictly background (α = 0) Unknown (α to be computed) Generated from the depth map strictly background un k nown strictly foreground Captured image Trimap 58

59 Trimap-Based Matting Captured image Trimap 59

60 Trimap-Based Matting Errors remain where the foreground and background colors are similar Captured image Trimap [Levin 2006] 60

61 Trimap-Based Matting Errors remain where the foreground and background colors are similar Contribution 2: matte error correction using color misalignment cues Captured image Trimap [Levin 2006] 61

62 Basic Idea Estimate foreground and background colors based on the current matte Captured image Current matte 62

63 Basic Idea Estimate foreground and background colors based on the current matte Captured image Estimated foreground color Current matte Estimated background color 63

64 Basic Idea Estimate foreground and background colors based on the current matte Detect inconsistent color misalignments Captured image Estimated foreground color Current matte Estimated background color 64

65 Synthetic Toy Example Synthesized input image Ground truth matte 65

66 Synthetic Toy Example Background Synthesized input image Ground truth matte Camera 66

67 Synthetic Toy Example Background Synthesized input image Ground truth matte Foreground Camera 67

68 Synthetic Toy Example Hard example Similar foreground and background colors Background Foreground Foreground Background 68

69 Synthetic Toy Example Hard example Foreground Background Similar foreground and background colors But solvable Color misalignment cues from ʻxʼ textures Foreground Background 69

70 Trimap-Based Matting Background Foreground Input image Trimap 70

71 Trimap-Based Matting Background Foreground Input image Estimated matte Trimap 71

72 Trimap-Based Matting Background Foreground Input image Estimated matte Trimap Ground truth 72

73 Matting Algorithm Background Foreground Current matte Input image 73

74 Matting Algorithm Background Foreground Current matte Estimated foreground color Input image Estimated background color 74

75 Matting Algorithm Current matte Estimated foreground color Detected color-misalignment Input image Estimated background color 75

76 Matting Algorithm Current matte Estimated foreground color Detected color-misalignment Input image Estimated background color Detected color-alignment 76

77 Matting Algorithm Current matte Estimated foreground color Detected color-misalignment Input image Estimated background color Detected color-alignment 77

78 Matting Algorithm Background Foreground Final matte Input image 78

79 Matting Algorithm Background Foreground Final matte Ground truth Input image 79

80 Outline Background Related Work Our Method Results Conclusion 80

81 Results of Depth & Matte Extraction Captured image Depth Matte 81

82 Comparison: Depth Estimation With the previous color-filter methods Local estimation to show raw performance Captured Captured [Amari 1992] [Chang 2002] Our method 82

83 Comparison: Matting With the trimap-based matting methods The trimaps were generated by our method Trimap Trimap [Levin 2006] [Wang 2007] Our method 83

84 Comparison with Ground Truth Mattes Ground truth Composite image (color-aligned) Composite image (color-misaligned) [Levin 2006] [Wang 2007] Our method 84

85 Image Editing Image composition Color-alignment reconstruction Novel view synthesis Refocusing Video matting 85

86 Example 1: Composition Matte Different Background Matte Captured image Different Background Composite 86

87 Example 2: Color-Alignment Reconst. Cancel the estimated misalignment Foreground color Matte Re-compose Background color Cancel the estimated misalignment 87

88 Reconstructed Image 88

89 Captured Image 89

90 Examples 3 & 4: View/Focus Synthesis 90

91 Example 5: Video Matting 91

92 Outline Background Related Work Our Method Results Conclusion 92

93 Conclusion Automatic depth and matte extraction using a color-filtered aperture Improved depth estimation Novel matting algorithm 93

94 Conclusion Automatic depth and matte extraction using a color-filtered aperture Improved depth estimation Novel matting algorithm Easy-to-use computational photography Put color filters in a camera lens Take a single photo with a hand-held camera 94

95 Conclusion Automatic depth and matte extraction using a color-filtered aperture Improved depth estimation Novel matting algorithm Easy-to-use computational photography Put color filters in a camera lens Take a single photo with a hand-held camera Limitation Entirely red objects cannot be handled 95

96 Thank You!! Any questions? Acknowledgments Takeshi Naemura Yusuke Iguchi Takuya Saito Johanna Wolf Zoltan Szego Paulo Silva Saori Horiuchi 96

97 Other Possible Filter Arrangements Our prototype More lightefficient Even more lightefficient but with lower depth resolution Higher depth resolution Simple horizontal arrangement Arbitrary shapes Combination with coded aperture Multi-aperture 97