Uncalibrated Camera calibrated coordinates Uncalibrated Two-View Geometry Linear transformation pixel coordinates l 1 2 Overview Uncalibrated Camera Calibration with a rig Uncalibrated epipolar geometry Ambiguities in image formation Calibrated camera Image plane coordinates Camera extrinsic parameters Perspective projection y c x Stratified reconstruction Autocalibration with partial scene knowledge Uncalibrated camera Pixel coordinates Projection matrix 3 4
Taxonomy on Uncalibrated Reconstruction Calibration with a Rig Use the fact that both 3-D and 2-D coordinates of feature points on a pre-fabricated object (e.g., a cube) are known. is known, back to calibrated case is unknown Calibration with complete scene knowledge (a rig) estimate Uncalibrated reconstruction despite the lack of knowledge of Autocalibration (recover from uncalibrated images) Use partial knowledge Parallel lines, vanishing points, planar motion, constant intrinsic Ambiguities, stratification (multiple views) 5 6 Calibration with a Rig Uncalibrated Epipolar Geometry Given 3-D coordinates on known object Eliminate unknown scales Recover projection matrix Epipolarconstraint Factor the into and using QR decomposition Solve for translation Fundamental matrix Equivalent forms of 7 8
Properties of the Fundamental Matrix Properties of the Fundamental Matrix A nonzero matrix is a fundamental matrix if a singular value decomposition (SVD) with Epipolar lines Epipoles has Image correspondences for some. There is little structure in the matrix 9 except that 10 Two view linear algorithm 8-point algorithm Estimating Fundamental Matrix Find such F that the epipolar error is minimized Solve the LLSE problem: Pixel coordinates Fundamental matrix can be estimated up to scale Solution eigenvector associated with smallest eigenvalue of Denote Compute SVD of F recovered from data Project onto the essential manifold: Rewrite Collect constraints from all points 11 cannot be unambiguously decomposed into pose and calibration 12 3
Calibrated vs. Uncalibrated Space Calibrated vs. Uncalibrated Space Distances and angles are modified by S 13 14 Motion in the distorted space What Does F Tell Us? Calibrated space Uncalibrated space can be inferred from point matches (eight-point algorithm) Cannot extract motion, structure and calibration from one fundamental matrix (two views) Uncalibrated coordinates are related by allows reconstruction up to a projective transformation (as we will see soon) Conjugate of the Euclidean group encodes all the geometric information among two views when no additional information is available 15 16
Decomposing the Fundamental Matrix Decomposition of the fundamental matrix into a skew symmetric matrix and a nonsingular matrix Projective Reconstruction From points, extract, followed by computation of projection matrices and structure Canonical decomposition Decomposition of is not unique Given projection matrices recover structure Unknown parameters - ambiguity Projective ambiguity non-singular 4x4 matrix Corresponding projection matrix 17 Both and are consistent with the epipolar geometry give the same fundamental matrix 18 Projective Reconstruction Euclidean vs Projective reconstruction Given projection matrices recover projective structure This is a linear problem and can be solve using linear least-squares Euclidean reconstruction true metric properties of objects lenghts (distances), angles, parallelism are preserved Unchanged under rigid body transformations => Euclidean Geometry properties of rigid bodies under rigid body transformations, similarity transformation Projective reconstruction projective camera matrices and projective structure Projective reconstruction lengths, angles, parallelism are NOT preserved we get distorted images of objects their distorted 3D counterparts --> 3D projective reconstruction => Projective Geometry Euclidean Structure Projective Structure 19 20
Homogeneous Coordinates (RBM) Homogenous and Projective Coordinates 3-D coordinates are related by: Homogeneous coordinates: Homogenous coordinates in 3D before attach 1 as the last coordinate render the transformation as linear transformation Before 4th coordinat cnot be zero 0 Projective coordinates all points are equivalent up to a scale 2D- projective plane Homogeneous coordinates are related by: 3D- projective space Each point on the plane is represented by a ray in projective space 21 Homogenous and Projective Coordinates 22 Vanishing points Representation of a 3-D line in homogeneous coordinates Ideal points last coordinate is 0 ray parallel to the image plane point at infinity never intersects the image plane When λ -> - vanishing points last coordinate -> 0 Projection of a line - line in the image plane 23 24 6
Ambiguities in the image formation Potential Ambiguities Ambiguities in Image Formation Structure of the (uncalibrated) projection matrix Ambiguity in K (K can be recovered uniquely Cholesky or QR) For any invertible 4 x 4 matrix In the uncalibrated case we cannot distinguish between camera imaging word from camera imaging distorted world Structure of the motion parameters In general, is of the following form Just an arbitrary choice of reference frame In order to preserve the choice of the first reference frame we can restrict some DOF of 25 26 Structure of the Projective Ambiguity Stratified (Euclidean) Reconstruction 1 st frame as reference General ambiguity while preserving choice of first reference frame Choose the projective reference frame then ambiguity is Decomposing the ambiguity into affine and projective one can be further decomposed as Note the different effect of the 4-th homogeneous coordinate 27 28
Affine upgrade Affine upgrade using vanishing points Upgrade projective structure to an affine structure How to compute Maps the points To points with affine coordinates Exploit partial scene knowledge Vanishing points, no skew, known principal point Special motions Pure rotation, pure translation, planar motion, rectilinear motion Constant camera parameters (multi-view) Vanishing points last homogeneous affine coordinate is 0 29 30 Affine Upgrade Euclidean upgrade Need at least three vanishing points We need to estimate remaining affine ambiguity 3 equations, 4 unknowns (-1 scale) Solve for Alternatives: Set up structure and update the projective In the case of special motions (e.g. pure rotation) no projective ambiguity cannot do projective reconstruction Estimate directly (special case of rotating camera follows) Multi-view case estimate projective and affine ambiguity together Use additional constraints of the scene structure (next) Autocalibration (Kruppa equations) 31 32
Euclidean upgrade using vanishing points Geometric Stratification Vanishing points intersections of the parallel lines Vanishing points of three orthogonal directions Orthogonal directions inner product is zero Provide directly constraints on matrix S has 5 degrees of freedom, 3 vanishing points 3 constraints (need additional assumption about K) Assume zero skew and aspect ratio = 1 33 34 Applications of projective geometry Rotation Only - Calibrated Case Calibrated Two views related by rotation only Mapping to a reference view rotation can be estimated Mapping to a cylindrical surface Vermeer s Music Lesson 35 36
Rotation Only - Uncalibrated Case Overview of the methods Calibrated Two views related by rotation only Conjugate rotation can be estimated Given C, how much does it tell us about K (or S)? Given three rotations around linearly independent axes S, K can be estimated using linear techniques Applications image mosaics 37 38 Summary Direct Autocalibration Methods The fundamental matrix satisfies the Kruppa s equations If the fundamental matrix is known up to scale Under special motions, Kruppa s equations become linear. Solution to Kruppa s equations is sensitive to noises. 39 40
Direct Stratification from Multiple Views Summary of (Auto)calibration Methods From the recovered projective projection matrix we obtain the absolute quadric contraints Partial knowledge in (e.g. zero skew, square pixel) renders the above constraints linear and easier to solve. The projection matrices can be recovered from the multiple-view rank method to be introduced later. 41 42 Projective transformations in 2D Images of planes (+ rectification) And what remains invariant Projective transformation 8-DOF (collinearity, cross ratios) There is one-to-one mapping between two images of a plane or between image plane and world plane Affine transformation 6-DOF (parallelism, ration of areas, length ratios) 2D projective transformation H homography (3x3 matrix) Estimation of homography from point correspondences 1. eliminate unknown depth Similarity transformation 4-DOF (angles, length ratios) Rigid Body Motion 3-DOF (angles, lengths, areas) 2. get two independent constraints per point (9-1) unknowns 3. need at least 4 points to estimate H 4. H is can be estimated up to a scale factor 43 44
Using H Solving for homographies p p Image based rectification (given some points in 3D world) compute H which would map them into a square Use H to rectify the entire image In calibrated case inter-image homography 45 46 Solving for homographies LLS solution to solving a system of homogeneous equations Solution eigenvector associated with the smallest eigenvalue of A T A 47