Kinect & 3D Tamara Berg Advanced Mul7media
Recovering 3D from Images How can we automa7cally compute 3D geometry from images? What cues in the image provide 3D informa7on? Slide credit: S. Narasimhan
Visual Cues for 3D Shading Merle Norman Cosme-cs, Los Angeles Slide credit: S. Narasimhan
Visual Cues for 3D Shading Texture The Visual Cliff, by William Vandivert, 1960 Slide credit: S. Narasimhan
Visual Cues for 3D Shading Texture Focus From The Art of Photography, Canon Slide credit: S. Narasimhan
Visual Cues for 3D Shading Texture Focus Mo7on Slide credit: S. Narasimhan
Why do we have two eyes? Cyclope vs. Odysseus Slide credit: S. Narasimhan
Stereo Reconstruc7on The Stereo Problem Shape from two (or more) images Biological mo7va7on Slide credit: S. Narasimhan
Mul7- view stereo Slide credit: S. Seitz & S Lazebnik
What is stereo vision? Generic problem formula7on: given several images of the same object or scene, compute a representa7on of its 3D shape Slide credit: S. Seitz & S Lazebnik
What is stereo vision? Generic problem formula7on: given several images of the same object or scene, compute a representa7on of its 3D shape Images of the same object or scene Arbitrary number of images (from two to thousands) Arbitrary camera posi7ons (camera network or video sequence) Calibra7on may be ini7ally unknown Representa7on of 3D shape Depth maps Meshes Point clouds Patch clouds Volumetric models Layered models Slide credit: S. Seitz & S Lazebnik
Binocular Stereo Slide credit: S. Narasimhan
Binocular Stereo Basic Principle: Triangula7on Gives reconstruc7on as intersec7on of two rays Requires calibra7on point correspondence Slide credit: S. Narasimhan
Stereo Corresondence
Stereo Correspondence Determine Pixel Correspondence Pairs of points that correspond to same scene point Epipolar Constraint Reduces correspondence problem to 1D search along conjugate epipolar lines Java demo: h_p://www.ai.sri.com/~luong/research/meta3dviewer/epipolargeo.html Slide credit: S. Narasimhan
Basic Stereo Algorithm For each epipolar line For each pixel in the leb image compare with every pixel on same epipolar line in right image pick pixel with minimum match cost Improvement: match windows This should look familar... Correla7on, Sum of Squared Difference (SSD), etc. Slide credit: S. Narasimhan
Size of Matching window W = 3 W = 20 Effect of window size Smaller window Good/bad? Larger window Good/bad? Be_er results with adap1ve window T. Kanade and M. Okutomi, A Stereo Matching Algorithm with an Adap1ve Window: Theory and Experiment,, Proc. Interna7onal Conference on Robo7cs and Automa7on, 1991. D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. Interna7onal Journal of Computer Vision, 28(2):155-174, July 1998 Slide credit: S. Narasimhan
Stereo Results Data from University of Tsukuba Scene Ground truth Slide credit: S. Narasimhan
Results with Window Search Window- based matching (best window size) Ground truth Slide credit: S. Narasimhan
From feature matching to dense stereo 1. Extract features 2. Get a sparse set of ini7al matches 3. Itera7vely expand matches to nearby loca7ons 4. Use visibility constraints to filter out false matches 5. Perform surface reconstruc7on Yasutaka Furukawa and Jean Ponce, Accurate, Dense, and Robust Mul-- View Stereopsis, CVPR 2007. Slide credit: S. Seitz & S Lazebnik
Stereo from community photo collec7ons For photos taken from the Internet, we need structure from mo1on techniques to reconstruct both camera posi7ons and 3D points Slide credit: S. Seitz & S Lazebnik
Stereo from community photo collec7ons M. Goesele, N. Snavely, B. Curless, H. Hoppe, S. Seitz, Mul7- View Stereo for Community Photo Collec7ons, ICCV 2007 h_p://grail.cs.washington.edu/projects/mvscpc/ Slide credit: S. Seitz & S Lazebnik
What about the kinect?
Basic Stereo Algorithm What s the hard part? Slide credit: S. Narasimhan
Active stereo with structured light Let s make correspondence easy!! L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisi7on Using Color Structured Light and Mul7- pass Dynamic Programming. 3DPVT 2002 Slide credit: S. Seitz
Active stereo with structured light Project structured light patterns onto the object Simplifies the correspondence problem Allows us to use only one camera Instead of 2 cameras, use camera + projector camera projector L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisi7on Using Color Structured Light and Mul7- pass Dynamic Programming. 3DPVT 2002 Slide credit: S. Seitz
Active stereo with structured light Project structured light patterns onto the object Simplifies the correspondence problem Allows us to use only one camera camera projector L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisi7on Using Color Structured Light and Mul7- pass Dynamic Programming. 3DPVT 2002 Slide credit: S. Seitz
Active stereo with structured light L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisi7on Using Color Structured Light and Mul7- pass Dynamic Programming. 3DPVT 2002 Slide credit: S. Seitz
Kinect in ac7on
Depth Maps
What about people?
Recovering skeletons
How?
Training Data - Capture a large database of mo7on capture (mocap) of human ac7ons. - Retarget mocap to meshes spanning the range of body shapes and sizes - Render depth and body part images
Classify based on depth features
Test Results
Kinect in ac7on
Kool Apps Kinect for recogni7on Kinectbot