Adaptive Coded Aperture Photography

Transcription

1 Adaptive Coded Aperture Photography Oliver Bimber, Haroon Qureshi, Daniel Danch Institute of Johannes Kepler University, Linz Anselm Grundhoefer Disney Research Zurich Max Grosse Bauhaus University Weimar

2 Motivation 2

3 Motivation 2

4 Motivation 2

5 Motivation 2

6 Motivation original JPEG compressed 2

7 Motivation Narrow apertures large depth of field (= high frequencies in out-of-focus regions) in low light throughput (= low signal-to-noise ratio) JPEG compression attenuates high frequencies (in focused and out-of-focus regions) Can we optimize apertures with respect to JPEG compression? frequencies attenuated by JPEG compression do not have to be supported optically by the aperture results in higher light throughput (= higher signal-to-noise ratio or shorter shutter times) 4

8 Related Work Veeraraghava n, et al,, 07 Levin, et al,, 07 Bando, et al,, 08 Zhou, et al,, 09 Liang, et al,, 08 Nagahara, et al,, 10 Grosse, et al,, 08 Grosse, et al,, 10 this paper binary intensity color static dynamic adapted zero crossings Fourier magnitudes noise models Applications: post-exposure refocusing, defocus deblurring, depth reconstruction, matting, light field acquisition, projector-defocus compensation. 5

9 Related Work Veeraraghava n, et al,, 07 Levin, et al,, 07 Bando, et al,, 08 Zhou, et al,, 09 Liang, et al,, 08 Nagahara, et al,, 10 Grosse, et al,, 08 Grosse, et al,, 10 this paper binary intensity color static dynamic adapted zero crossings Fourier magnitudes noise models Applications: post-exposure refocusing, defocus deblurring, depth reconstruction, matting, light field acquisition, projector-defocus compensation, increasing light throughput 6

10 Previous Work pre-computed input image F* binarize (optional) dynamic aperture pattern FT threshold frequencies apply pseudo-inverse FT projected image (Adaptive) Coded Aperture Projection, Grosse, Wetzstein, Grundhoefer, and Bimber, ACM Transaction on Graphics,

11 8

12 set aperture and capture image 8

13 set aperture and capture image JPEG compression and frequency filtering 8

14 set aperture and capture image JPEG compression and frequency filtering compute and set coded aperture 8

15 set aperture and capture image JPEG compression and frequency filtering compute and set code aperture re-capture image 8

16 set aperture and capture image JPEG compression and frequency filtering compute and set code aperture re-capture image 8 transform bokeh and depth of field

17 set aperture and capture image JPEG compression and frequency filtering compute and set code aperture re-capture image final image 8 transform bokeh and depth of field

18 Prototype 9

19 important frequencies masked spectrum original (all frequencies) Institute of Frequency Filtering q=90 =0.2854% q=70 =0.4231% q=50 =0.5024% q=30 =0.6205% 10

20 Aperture Computation Construct a binary frequency mask (m) and compute intensity aperture pattern (a) by minimizing the variance of its Fourier transform for all important frequencies: 11

21 Aperture Computation Construct a binary frequency mask (m) and compute intensity aperture pattern (a) by minimizing the variance of its Fourier transform for all important frequencies: M is the diagonal matrix containing the binary frequency mask values of m, F is the discrete Fourier transform matrix (i.e., the set of orthogonal Fourier basis functions in its columns), a is the unknown vector of the coded aperture pattern, and e is the vector of all ones - this can be solved quickly with the pseudo-inverse: The conjugate-transpose pseudo-inverse matrix F is constant and can be pre-computed! 11

22 Aperture Computation binarize low magnitude high q=70 masked spectrum q=50 masked spectrum MTF of binarized mask MTF of coded intensity mask MTF of binarized mask MTF of coded intensity mask 12

23 Bokeh Transformation 13

24 Bokeh Transformation bokeh transformation 13

25 coded (after bokeh transformation) coded (before bokeh transformation) regular Institute of Bokeh Transformation back-focus close-up front (back-focus) front-focus close-up back (front-focus) 14

26 Bokeh Transformation Capturing an image through an aperture with given PSF can be considered as convolution (multiplication in frequency domain): 15

27 Bokeh Transformation Capturing an image through an aperture with given PSF can be considered as convolution (multiplication in frequency domain): Bokeh transformation can be carried out as follows: 15

28 Bokeh Transformation Capturing an image through an aperture with given PSF can be considered as convolution (multiplication in frequency domain): Bokeh transformation can be carried out as follows: However, the scales (s and s ) are entirely unknown since the scene depth is unknown! 15

29 17x17 15x15 13x13 11x11 9x9 7x7 5x5 3x3 1x1 convolution scale (s ) Bokeh Transformation simulated measured 1x1 3x3 5x5 7x7 9x9 11x11 13x13 15x15 17x17 deconvolution scale (s ) 1x1 3x3 5x5 7x7 9x9 11x11 13x13 15x15 17x17 16

30 convolution deconvolution

31 Bokeh Transformation color and brightness matching 18

32 Bokeh Transformation S S S S S S S S S S S S S S S S S S S S S S S S color and brightness matching 18

33 Bokeh Transformation S S S S S S S S S S S S S S S S S S S S S S S S color and brightness matching 18

34 Bokeh Transformation S S S S S S S S S S S S S S S S S S S S S S S S color and brightness matching 18

35 Bokeh Transformation S S S S S S S S S S S S S S S S S S S S S S S S color and brightness matching 18

36 scale difference deconvolution (s ) convolution (s ) Institute of Bokeh Transformation 19

37 Intensity Masks with PWM 3.2s/6.4s + 1.6s/6.4s + 0.8s/6.4s + 0.4s/6.4s + (1/5)s/6.4s + (1/10)s/6.4s + (1/20)s/6.4s + (1/40)s/6.4s = 20

38 Intensity Masks with PWM 3.2s/6.4s + 1.6s/6.4s + 0.8s/6.4s + 0.4s/6.4s + (1/5)s/6.4s + (1/10)s/6.4s + (1/20)s/6.4s + (1/40)s/6.4s = 20

39 Intensity Masks with PWM 3.2s/6.4s + 1.6s/6.4s + 0.8s/6.4s + 0.4s/6.4s + (1/5)s/6.4s + (1/10)s/6.4s + (1/20)s/6.4s + (1/40)s/6.4s = 3.2s 1.6s 0.8s 0.4s 1/5s 1/10s 1/20s 1/40s 20

40 coded regular Institute of Results uncompressed (original) uncompressed (increased and matched brightness) close-up (regular) close-up (coded) 21

41 coded regular Institute of Results compressed close-up (regular) close-up (coded) 21

42 regular aperture opening coded regular coded compression regular Institute of q=90 q=70 q=50 q= % 10% 27% 22 regular aperture opening (2%, 10%, 27%)

43 lighting conditions coded regular coded focus regular Institute of Results front center back focus (front, center, back) high low medium lighting conditions (high, low, medium) 23

44 Results Note that if we ignored other noise sources, such as dark noise and read noise, and considered shot noise only, then the gain in SNR would be proportional to the square root of the light throughput gain. 24

45 Limitations and Future Work LCA has low light transmittance (only 30% when completely transparent), low contrast (7:1), and is small (limited DOF difference) use larger reflective DMAs or LCoS panels! Coded aperture pattern is scaled manually to roughly match the depth of field while remaining depth-of-field differences are removed by the bokeh transformation automize this scale estimation Explore alternatives downsampling instead of / together with compression (i.e., trade resolution / compression for light through put or shutter time) simple extension of circular apertures for 1/f distributions 25

46 Thank You!