Structure from motion II

Transcription

1 Structure from motion II Gabriele Bleser Thanks to Marc Pollefeys, David Nister and David Lowe

2 Introduction Previous lecture: structure and motion I Robust fundamental matrix estimation Factorization Today: structure and motion II Structure and motion loop Triangulation Drift reductionmethods Wide baseline matching (SIFT) Next lectures: dense reconstruction Lecture 3D Computer Vision 2

3 Sequential SAM Alternating estimation of camera poses and 3D feature locations (triangulation) from a (continuous) image sequence. e e F matrix estimation and factorization (last lecture) 2D feature location (from image processing) Camera pose Initialize first camera poses How? Lecture 3D Computer Vision 3

4 Sequential SAM Alternating estimation of camera poses and 3D feature locations (triangulation) from a (continuous) image sequence. 3D feature location 2D feature location (from image processing) Camera pose Initialize 3D points (triangulation) Lecture 3D Computer Vision 4

5 Sequential SAM Alternating estimation of camera poses and 3D feature locations (triangulation) from a (continuous) image sequence. 3D feature location 2D feature location (from image processing) Camera pose Estimate next camera pose Lecture 3D Computer Vision 5

6 Sequential SAM Alternating estimation of camera poses and 3D feature locations (triangulation) from a (continuous) image sequence. 3D feature location 2D feature location (from image processing) Camera pose Initialize additional 3D points Lecture 3D Computer Vision 6

7 Sequential SAM Alternating estimation of camera poses and 3D feature locations (triangulation) from a (continuous) image sequence. E.g. recursive filtering 3D feature location 2D feature location (from image processing) Camera pose Refine known 3D points with new camera poses Lecture 3D Computer Vision 7

8 Offline SAM Alternating estimation of camera poses and 3D feature locations (triangulation) from a (continuous) image sequence. 3D feature location Not sequential anymore: mostly in offline SAM, or local bundle adjustment for drift reduction 2D feature location (from image processing) Camera pose Refine known cameras with new 3D points Lecture 3D Computer Vision 8

9 We know Outline how to track features in a continuous image sequence (lecture 6) how to estimate camera poses from 2D/3D correspondences (lectures 2, 3, 5) from 2D/2D correspondences (lectures 4, 7) Now: Triangulation: estimate the 3D point X given at least 2 known cameras Cand C and 2 corresponding feature points xand x (i.e. 2 camera views). X x x C C Lecture 3D Computer Vision 9

10 Parallel cameras (simple case) x Z Then, y = y and depth Z can be computed from disparity: x d Derivation via equal triangles Note: feature displacement (disparity) is inversely proportional to depth as 0, Lecture 3D Computer Vision 10

11 Converging cameras: rectification Can be converted/rectified to parallel camera geometry: project images to plane parallel to baseline 2D homography How could this be done with methods that you know? Lecture 3D Computer Vision epipolar plane 11

12 Converging cameras: general configuration Arbitrary camera poses 3D triangulation required X j x 1j x 3j C 1 x 2j C 2 C 3 Lecture 3D Computer Vision 12

13 Triangulation Given: corresponding measured (i.e. noisy) feature points x and x and camera poses, P and P Problem: in the presence of noise, back projected rays do not intersect Rays are skew in space How could this be solved? and measured points do not lie on corresponding epipolar lines Lecture 3D Computer Vision 13

14 Reminder: direct linear transform Objective Given a sufficient number, n, of measurements determine such that Algorithm (i) Set up linear equation system (m is number of parameters): (ii) If b is zero solve with SVD. (iii) Else solve with pseudo inverse. (iv) Determine X from x. For triangulation: What is the measurement, what the unkown? Which equation is used? How many measurements are needed? Why? Lecture 3D Computer Vision 14

15 Triangulation: algebraic solution Equation for one camera view (camera pose P and feature point x) 4 entries, 3 DOF i th row of P (3x4) matrix (2x4) for one equation 3 equations, only 2 linearly independent drop third row Lecture 3D Computer Vision 15

16 Triangulation: algebraic solution Stack equations for all camera views: Minimizes algebraic error no geometrical interpretation X has 3 DOF (4 parameters due to homogeneous representation) One camera view gives two linearly independent equations Two camera views are sufficient for a minimal solution Solution for X is eigenvector of A T A corresponding to smallest eigenvalue Homogenize solution for X to obtain Euclidean vector Lecture 3D Computer Vision 16

17 Triangulation: vector solution Compute the mid point of the shortest line between the two rays Lecture 3D Computer Vision 17

18 Triangulation: minimize geometric error Estimate 3D point, which exactly satisfies the supplied camera geometry and, so it projects as where and are closest to theactualimagemeasurements. Assumes perfect camera poses! Lecture 3D Computer Vision 18

19 Triangulation: properties The smaller the angle (small baseline, big distance), the bigger the reconstruction uncertainty! X? β X? x x x x C C C C Lecture 3D Computer Vision 19

20 Structure and motion: the big problem Drift (accumulating error) How can we reduce this? Different methods in online and offline case Lecture 3D Computer Vision 20

21 Drift reduction (feature level) Reduce drift in feature tracking/matching Extend feature tracks Reacquire lost features, e.g. Advanced optical flow algorithm (lecture 6) Wide baseline matching (later today) Lecture 3D Computer Vision 21

22 Drift reduction (geometry level) Careful (precise, robust) 3D point initialization How? Triangulate over a history of measurements Enforce a minimal angle θ Use RANSAC to eliminate outliers Accept only well reconstructed points Incorporate measurement uncertainties (error propagation) E.g. simple stochastic model and WLS estimation (lecture 5): All entities modelled as Gaussian random variables Lecture 3D Computer Vision 22

23 Drift reduction (geometry level) Iterative/recursive reconstruction Refine 3D structure over time (e.g. update a 3D point each time the feature is observed) Methods: (Local) bundle adjustment Repeated triangulation Recursive filtering (e.g. extended Kalman filter) Lecture 3D Computer Vision 23

24 Results: online SAM (360 rotation) With drift reduction: error reduced by a factor of 10 Without drift reduction Lecture 3D Computer Vision 24

25 Global bundle adjustment: jointly optimize over all camera poses and 3D points (lecture 7) Offline drift reduction Estimate Residual/reprojection error Parameter vector containing all camera poses and 3D points Lecture 3D Computer Vision 25

26 Offline drift reduction Global bundle adjustment: jointly optimize over all camera poses and 3D points (lecture 7) Requires features to be matched between very different views Simple feature matching/tracking is not enough Wide baseline matching required Lecture 3D Computer Vision 26

27 Wide baseline matching Requirement to cope with larger variations between images Translation, rotation, scaling geometric Foreshortening, other distortions transformations Non diffuse reflections photometric Illumination changes Examples: Could such correspondences be found using the advanced optical flow algorithm (KLT with affine model and multi scale extension)? Lecture 3D Computer Vision 27

28 Reminder: Advanced optical flow algorithm Usually two stage approach: Stage 1: Estimate pure translation from frame to frame (coarse to fine) Stage 2: Estimate the affine transformation and illumination parameters to the initial feature appearance (use previous estimate as initial guess),,,,,,,,,, Lecture 3D Computer Vision 28

29 Lowe s SIFT features SIFT = Scale Invariant Feature Transform Extensively used for wide baseline matching Rotation and scale invariant keypoint (feature) detector and descriptor Robust against other geometric distortion and photometric changes Lecture 3D Computer Vision 29

30 Explanation: 2 nd order gradient filters Laplacian of Gaussians (LoG): gauss filtered sum of 2 nd derivatives in x and y direction Example: Example filter mask p Edges are zero crossing Edge profile 2 nd derivative Lecture 3D Computer Vision 30

31 Explanation: 2nd order gradient filters Approximate LoG with a Difference of Gaussians (DoG): 2 L Gxx x y Gyy x y (,, ) (,, ) (Laplacian) DoG G( x, y, k ) G( x, y, ) (Difference of Gaussians)

32 Explanation: 2 nd order gradient filters Example: DoG Search for zerocrossing to find edges Gauss with σ=4 Gauss with σ=6 Search for local maxima and minima to detect stable blob like image features [Mikolajczyk, 2002] Lecture 3D Computer Vision 32

33 SIFT keypoint detector: position and scale SIFT uses a pyramid of DoGs for scale invariant keypoint detection Algorithm: From the input image, construct a pyramid of DoGs Search for local maxima and minima in both position and scale space Provides candidates for scale invariant keypoints Lecture 3D Computer Vision 33

34 SIFT keypoint detector: orientation Next: assign orientation. Why? Algorithm: Create histogram of local gradient directions computed at selected positions and scales Assign canonical orientation at peak of smoothed histogram Now each keypoint specifies 2D position, scale and orientation 0 2 Lecture 3D Computer Vision 34

35 SIFT keypoint descriptor Compute from keypoint specific image patch: Centered around keypoint (as usual) Oriented and scaled according to keypoint (rotation, scale invariant extension) Based on orientation histograms: Sample image gradients over (16x16) array Create array of orientation histograms (4x4) histogram array, 8 orientations per cell 128 dimensional descriptor (normalize to length 1) Lecture 3D Computer Vision 35

36 SIFT descriptor matching Simple strategy : Compute keypoints for both images Compute pair wise distances between keypoint sets (e.g. Euclidean vector distance) Assign closest keypoints, avoid double assignments (usually many outliers) Use RANSAC for geometry estimation See references for advanced methods! Lecture 3D Computer Vision 36

37 Outlook: next lectures Dense reconstruction overall goal is 3D model generation Example: Spherical input pictures of a church Feature matching (SIFT) Solve correspondence and triangulation problem for each pixel! Sparse reconstruction Dense reconstruction Lecture 3D Computer Vision 37

38 References Hartley R. and Sturm P.: Triangulation, International Conference on Computer Analysis of Images and Patterns, Bleser et al: Real time Vision based Tracking and Reconstruction, Journal of Real Time Image Processing 2 (2007), 2 3, pp , Berlin, Heidelberg, New York : Springer Verlag. D. G. Lowe. Distinctive Image Features from Scale Invariant Keypoints. International Journal of Computer Vision, 60:91 110, November Lecture 3D Computer Vision 38