Computational Photography

Transcription

1 Computational Photography

2 Digital photography Photography has been rapidly changing in recent years, with the transition from analog to digital

3 Digital photography Computing can now be involved in every step of the imaging process. This is having a huge impact on the art and practice of photography. We don t just take a picture we sample the light field

4 The light field

5 The light field The light field describes the amount of light (radiance) traveling in every direction through every point in space Possibly also as a function of wavelength and time The plenoptic function One goal of image based rendering is to construct the light field from a set of images, so that new images can be synthesized

6 Computational photography Traditional (analog and digital) cameras have limitations, e.g.: Cannot capture scenes with significant differences between bright and dark areas Limited field of view Can only focus at a single plane Computational photography is an emerging research field that attempts to extend or enhance the capabilities of digital photography by adding computational elements to the imaging process Optics + sensors + illumination + processing + interaction Convergence of computer vision, digital imaging, and graphics

7 Computational photography Typical process: take several images, combine them (perhaps with user interaction), create better image(s) May modify Field of view, depth of field/focus, dynamic range, spatial resolution, wavelength resolution, temporal resolution Aperture, focus, sensor design, exposures, illumination Computational photography can provide Improved dynamic range Variable focus, resolution, and depth of field Aesthetic image framing Content-based image editing Hints about shape, reflectance, and lighting New interactive forms of photography

8 Traditional Photography Sensor array Lens Pixels Image

9 Computational Photography Generalized sensor Generalized Optics Computation Samples Image

10 E.g., a computational lens Varioptic Liquid Lens: Electrowetting Applied voltage modifies the curvature of the liquidliquid interface, leading to a variable focal length lens

11 The Plenoptic Function (a.k.a. the light field) Q: What is the set of all things that we can ever see? A: The Plenoptic Function Let s start with a stationary person and try to parameterize everything that he can see

12 A grayscale image is intensity of light Seen from a single view point At a single time Averaged over the wavelengths of the visible spectrum (can also do P(x,y), but spherical coordinate are nicer) P(q,f)

13 A color image is intensity of light Seen from a single view point At a single time As a function of wavelength P(q,f,l)

14 A color movie is intensity of light Seen from a single view point Over time As a function of wavelength P(q,f,l,t)

15 A holographic movie is intensity of light Seen from ANY viewpoint Over time As a function of wavelength P(q,f,l,t,V X,V Y,V Z )

16 The Plenoptic Function P(q,f,l,t,V X,V Y,V Z ) Can reconstruct every possible view, at every moment, from every position, at every wavelength Contains every photograph, every movie, everything that anyone has ever seen! It completely captures our visual reality.

17 Sampling the Plenoptic Function (top view)

18 Synthesizing novel views

19 Image based rendering We don t need a (3D) model of the object Instead, model the object by storing its light field Stuff

20 Some examples of computational photography Research and products

21 Flash/no-flash image pairs Eisemann and Durand SIGGRAPH04 Petschnigg et al. SIGGRAPH04 No-flash Flash Result

22 Flash/no-flash image pairs Agrawal et al. SIGGRAPH05 Ambient Flash Result Reflection Layer

23 High dynamic range imaging Raskar et al. Ambient Flash

24 HDR imaging: varying exposure

25 Flash Brightness Exposure Time

26 ??? Result

27 Image retargeting Setlur et al. SIGGRAPH04 Rearranging the image content to optimize the information of interest, given the constraints (e.g., reduced aspect ratio)

28 The Lytro camera Link

29 A practical light field camera uv-plane st-plane

30 The prototype camera Adaptive Optics microlens array 125μ square-sided microlenses pixels lenses = pixels per lens

31 Digital refocusing Σ Σ refocusing = summing windows extracted from several microlenses

32 Example of digital refocusing

33 Cinemagraph ( Still photographs in which a minor (and repeated) movement occurs Typically in Animated GIF format A Cinemagraph is an image that contains within itself a living moment that allows a glimpse of time to be experienced and preserved endlessly. 34

34 Coded aperture photography Levin et al. SIGGRAPH 2007 Regular camera with patterned aperture (made from cardboard!) in place of regular aperture From this, can get scene depth and do refocusing Uses deconvolution with the known patterns

35 Output #1: Depth map Single input image: Output #2: All-focused image

36

37

38 Lens and defocus Lens aperture Image of a point light source Lens Camera sensor Point spread function Focal plane

39 Lens and defocus Lens aperture Image of a defocused point light source Object Lens Camera sensor Point spread function Focal plane

43 Idea 2: Coded Aperture Mask (code) in aperture plane - make defocus patterns different from natural images and easier to discriminate Conventional aperture Our coded aperture

44 Solution: lens with occluder Object Lens Camera sensor Point spread function Focal plane

45 Solution: lens with occluder Aperture pattern Image of a defocused point light source Object Lens with coded aperture Camera sensor Point spread function Focal plane

50 Regularizing depth estimation Try deblurring with 10 different aperture scales x arg min f x _ y 2 Convolution error l 2 + i ( x ) Derivatives prior i Keep minimal error scale in each local window + regularization Input Local depth estimation Regularized depth

51 Regularizing depth estimation Regularized depth Local depth estimation Input

52 Input

53 All-focused (deconvolved)

54 Close-up Original image All-focus image

55 Input

56 All-focused (deconvolved)

57 Application: Digital refocusing from a single image

64 Coded aperture: pros and cons Image AND depth at a single shot No loss of image resolution Simple modification to lens Depth is coarse unable to get depth at untextured areas, might need manual corrections. But depth is a pure bonus Loss some light But deconvolution increases depth of field

65 Some projects in computational photography

66 Composition Context Photography w/daniel Vaquero

67 Composition context photography How can the act of framing an image help to produce better images? A wider array of images to choose from? Higher resolution, higher SNR, wider FOV, optimal subject pose, etc.

68 Composition context photography The composition context of a picture: the viewfinder frames, their capture parameters, and inertial sensor data collected while the user is framing the photograph

69

70 The Frankencamera Stanford, Ulm, Nokia RC, UCSB, Disney Research (SIGGRAPH 2010) An open source experimental camera architecture and API for computational photography Fully programmable: permits control and synchronization of the sensor and image processing pipeline at the microsecond time scale, as well as the ability to incorporate and synchronize external hardware like lenses and flashes.

71

72 Generalized Autofocus w/daniel Vaquero

73 Generalized autofocus Choices have to be made in focusing images Large depth-of-field small aperture noisy image Can we provide alternatives? New focusing options? Large depth-of-field with wide aperture? Selective focus?

74 All-in-focus imaging Images focused at different distances (focal stack)

75 What is our work about? The capture process for focal stacks For fusion, we use existing techniques, but we do it on-camera Our scenario, typical in camera phones and low-end consumer cameras Fixed lens aperture Handheld camera Limited memory and processing power

76 Contrast-based passive autofocus Sharp But low SNR

77 Our method: Generalized autofocus Contribution: autofocus for focal stacks Find the minimal set of required images to generate an allin-focus composite The choice of this set depends on the scene Why is minimizing the number of images important? Faster capture Less sensitive to motion Requires less memory and processing power depth

78 Method overview Nokia N900 depth 1. Capture a focal stack of low-resolution images while continuously changing the focal distance ( lens sweep ) 2. Analyze sharpness of low-resolution focal stack 3. Determine the minimal set of required images 4. Recapture the images in the minimal set in high-resolution 5. Perform fusion of the images on-camera Desired: Good user interface to specify what should be in focus

79

80 Standard autofocus

81 All-in-focus result (24 image in stack, 3 used)

82 Stack images (4)

83 All-in-focus result (24 image in stack, 4 used)

84 Generalized Autofocus

85 Ongoing work Improve image fusion User selectable focus Location and range Between x and y In focus here and here and here Provide focusing opportunities not possible with lens-only solutions