Depth from a single camera

Transcription

1 Depth from a single camera Fundamental Matrix Essential Matrix Active Sensing Methods School of Computer Science & Statistics Trinity College Dublin Dublin 2 Ireland 1 1

2 Geometry of two cameras: general case Optical Axes not parallel Consider X, X and t as vectors left hand camera as origin K and K are calibration matrices Left projection of u and right projection of u 2 2

3 Geometry of two cameras: general case X, X and t are coplanar Substitute with u and K expressions defined up to scale i.e. t not defined Change t from vector to matrix form a skew symmetric matrix can be created if t 0 rewriting we get middle part of this eq. can be collapsed into one matrix F F is the Fundamental matrix of two views 3 3

4 The Fundamental Matrix F captures all the information between a pair of cameras if correspondence is solved Properties Bilinear relation between any two views Incorporates K, K, R and t F has rank 2 7 degrees of freedom 2 epipoles 4 4

5 Estimating the fundamental matrix Epipolar Geometry has 7 degrees of freedom epipoles (2+2), map 3 lines from 1st to 2nd (3) 7 points will provide F via a non-linear algorithm but it s unstable 8 non-coplanar points normalized and scaled [Longuet-Higgins81] stable and fast, noise? Over determine with more than 8 points Least Squares Solution, SVD then used to find F 5 5 5

6 Estimation of F Multiple View Geometry, Hartley and Zisserman 2000 find features find set of possible matches RANSAC robust estimation of 7 correspondences for N iterations select random 7 correspondences estimate re-projection error (d) of correspondence compute number inliers consistent with F (i.e. d threshold) if there are 3 real solutions for F the number of inliers is computed for each and and largest is retained Recompute F using all the inliers minimising a cost function 6 6

7 Example from Multiple View Geometry 7 7

8 OpenCV function int cvfindfundamentalmat( const CvMat* points1, const CvMat* points2, CvMat* fundamental_matrix, int method=cv_fm_ransac, double param1=1., double param2=0.99, CvMat* status=null); points1: Array of the first image points of 2xN, Nx2, 3xN or Nx3 size (where N is number of points). Multi-channel 1xN or Nx1 array is also acceptable. The point coordinates should be floating-point (single or double precision) points2 Array of the second image points of the same size and format as points1 fundamental_matrix: The output fundamental matrix or matrices. The size should be 3x3 or 9x3 (7-point method may return up to 3 matrices). method CV_FM_7POINT - for 7-point algorithm. N == 7, CV_FM_8POINT - for 8- point algorithm. N >= 8, CV_FM_RANSAC - for RANSAC algorithm. N >= 8, CV_FM_LMEDS - for LMedS algorithm. N >= 8 param1: The parameter is used for RANSAC or LMedS methods only. It is the maximum distance from point to epipolar line in pixels, beyond which the point is considered an outlier param2: The parameter is used for RANSAC or LMedS methods only. It denotes the desirable level of confidence that the matrix is correct. 8 8

9 The Essential Matrix When camera calibration is known Normalise image measurements simplified version of relations used in F Captures information about the relative motion of a calibrated camera. Rotation and Translation 9 9

10 Ego Motion Estimation One or two calibrated cameras Unknown movement (R & t) Algorithm Find correspondences (u i & u' i ) Normalise data Compute the fundamental matrix Compute the essential matrix Adjust E to deal with numerical inaccuracies Determine R & t from the essential matrix using singular value decomposition 10 10

11 Review of Stereo disparity methods Paper review D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47(1/2/3):7-42, April- June

12 Other approaches to depth extraction Laser Ranging (metris.com) shape from reflected brightness canesta.com, 3dvsystems.com) shape from silhouette (scanbull.com) Stereo in Surgery (brainlab.com 12 12

13 Radiometry / Structured light Images illuminated under a sequence of lighting patterns each pixel is assigned a code based on its grey level under each pattern codes used to find correspondence 13 13

14 Surface Reflectance What viewer sees depends on Light source: strength, position, type Object: local surface properties & orientation Surface Albedo Lambertian Surface Phong Model

15 Shape from shading Gradient Space Every point has a surface normal z(x, y) =surface height n = 1 1+p2 + q 2 p = δz δx q = δz δy [p,q] is the 2D gradient space representation of surface orientation 15 15

16 Bidirectional reflectance distribution function - BRDF Brightness of Surface Patch for a specific surface material, light source and viewer directions 16 16

17 Image Irradiance Equation Recover surface orientation from intensity images E(x, y) =R[p(x, y),q(q, y)] = R( δz δx, δz δy ) 17 17

18 Characteristic Strip Method We know the depth of point [x,y,z] T with known p,q Occluding boundaries Specularities Taking small steps propagate surface normals Not a very stable method Many to one mapping 18 18

19 Global Optimisation Method For every pixel make a guess at orientation Apply two constraints The intensity should be close to predicted by the reflectance map p and q vary smoothly i.e. small Lapacians Apply Lagrange multipliers to minimise the energy over the image, λ is the Lagrange multiplier Energy(x, y) =[f(x, y) R(p, q)] 2 + λ[( 2 p) 2 +( 2 q) 2 ] 19 19

20 Photometric Stereo One fixed viewing direction Change the direction of the illumination E.g. 3 or more images of the Lambertian surface Surface normals can be uniquely determined based on shading variations 20 20

21 Sphere - multiple illuminations 21 21

22 Surface Normals 22 22

23 Surface Rendering 23 23

24 Summary Stereo Camera calibration Epipolar geometry Essential Martrix Fundamental Matrix Correspondence Problem - feature points Shape from Shading Intensity as guide to surface orientation Smoothly changing surfaces 24 24

25 S Barsky and M Petrou, The 4-source photometric stereo technique for three-dimensional surfaces in the presence of highlights and shadows. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-25, pp

26 26 26

27 27 27

28 More Depth from Single image Depth from a single Image: Google Talk Photo Pop Up popup/index.html ImageInterpretation/ 28 28