Plenoptic Modeling. Ruigang Yang CS 684 CS684 1

Transcription

1 Plenoptic Modeling Ruigang Yang CS 684 CS684 1

2 A Continuum of IBMR methods Geometry-based Image-based Image Image Image Image Image Shape Recovery 3D model Computer Graphics New 2D image New 2D image New 2D image Image Image Image Image Image Additional constraints New 2D image New 2D image New 2D image Stereo Space Carving Façade Lumigraph Light Field More geometric primitives More image samples CS684 2

3 3D Shape Reconstruction Hard CS684 3

4 Image-base Representation Plenoptic Function Plenus (complete) + Optics...describes all of the image information visible from a particular viewing position. From Adelson and Bergen, The Plenoptic Function and Elements of Early Vision CS684 4

5 Plenoptic Function Introduced to CG by McMillan and Bishop in 1995 Captures the light received at any point in space, from any direction, at any time. Function of location, direction of gaze, wavelength, and time. f ( V, V, V, θ, φ, λ, t) x y z CS684 5

6 Image-based Representations 7D Ideal f ( V, V, V, θ, φ, λ, t) x y z 6D 5D 4D 3D 2D Consider only 3 frequencies (RGB) Consider rays through scene at one time instant Consider only lines through scene Consider one dimension fewer directions/positions Consider only directions from viewpoint CS684 6

7 Image-based Representations 7D 6D 5D 4D 3D 2D Ideal Consider only 3 frequencies (RGB) Consider rays through scene at one time instant Consider only lines through scene Consider one dimension fewer directions/positions Consider only directions from viewpoint CS684 7

8 Panoramas 2D Plenoptic Function Single center of projection Only rotations Stitched (QTVR, PhotoVista, etc.) Panorama Tools at QTVR Demo: Mars Image CS684 8

9 Characteristics of Panorama Relatively easy to make 3D Navigation bad Hot spots to click. Can t really tell where to go next. No sense of 3D CS684 9

10 Columbia s Omnicam Web Site CS684 10

11 UNC Camera-cluster CS684 11

12 Image-based Representations 7D Ideal f ( V, V, V, θ, φ, λ, t) x y z 6D 5D 4D 3D 2D Consider only 3 frequencies (RGB) Consider rays through scene at one time instant Consider only lines through scene Consider one dimension fewer directions/positions Consider only positions on a surface CS684 12

13 Texture Mapping Map photograph onto surfaces Geometry + Textures Photographs CS684 13

14 Texture Mapping CS684 14

15 View-Dependent Texture Mapping (VDTM) CS684 15

16 VDTM To simulate small details not captured in the model CS684 16

17 Image-based Representations 7D Ideal f ( V, V, V, θ, φ, λ, t) x y z 6D 5D 4D 3D 2D Consider only 3 frequencies (RGB) Consider rays through scene at one time instant Consider only lines through scene Consider one dimension fewer directions/positions Consider only positions on a surface CS684 17

18 Concentric Mosaics Different parameterization (3D) Make vertical slit (line scan) images at concentric circles CS684 18

19 Store Mosaic As a panorama CS684 19

20 Indexing into Mosaic Find tangents at proper locations. CS684 20

21 Off-Plane Rays Not correct. They manually select a depth. CS684 21

22 Making Mosaic Use standard camera. Tangential or radial. CS684 22

23 Results CS684 23

24 View Interpolation Create novel images by resampling photographs References images sample the Plenoptic function CS684 24

25 View Interpolation Methods Warp nearby reference images to novel viewpoint Blend warped images Key issue: find the correspondence CS684 25

26 Correspondences Lines CS684 26

27 Correspondences Key feature pixels CS684 27

28 View Morphing Mona Lisa Stereo Pair (cross your eyes and fuse them!) Steve Seitz, Chuck Dyer 1996 Show Video CS684 28

29 Matching Find correspondences automatically classic Vision problem Feature tracking Stereo Many others Left Right Disparity Map CS684 29

30 Problems with View-interpolation Visibility changes disocclusion CS684 30

31 Partial Solutions Fill in holes with neighboring pixels Use more photos CS684 31

32 Image-based Representations 7D Ideal f ( V, V, V, θ, φ, λ, t) x y z 6D 5D 4D 3D 2D Consider only 3 frequencies (RGB) Consider rays through scene at one time instant Consider only lines through scene Consider one dimension fewer directions/positions Consider only positions on a surface CS684 32

33 Empty Space 4D Color is OK anywhere along ray as long as in empty space Details in the next lecture CS684 33

34 Image-based Representations 7D Ideal f ( V, V, V, θ, φ, λ, t) x y z 6D 5D 4D 3D 2D Consider only 3 frequencies (RGB) Consider rays through scene at one time instant Consider only lines through scene Consider one dimension fewer directions/positions Consider only positions on a surface CS684 34

35 Plenoptic Modeling (McMillan & Bishop 1995) 5D plenoptic function Cylindrical parameterization A set of panoramas with per pixel depth CS684 35

36 Determine Image Flow Fields Use cylindrical epipolar geometry to relate pixels Same as planar case except epipolar lines become quadratic parametric curves Use angular disparity values to construct image flow vectors CS684 36

37 Results Input Images 2 Cylindrical projections Rendering results CS684 37

38 Next Lecture Light Field Rendering 7D Ideal f ( V, V, V, θ, φ, λ, t) x y z 6D 5D 4D 3D 2D Consider only 3 frequencies (RGB) Consider rays through scene at one time instant Consider only lines through scene Consider one dimension fewer directions/positions Consider only positions on a surface CS684 38