Globally Optimal Inlier Set Maximization With Unknown Rotation and Focal Length

Transcription

1 Globally Optimal Inlier Set Maximization With Unknown Rotation and Focal Length Jean-Charles Bazin 1, Yongduek Seo, Richard Hartley 3, Marc Polleeys 1 1 Department o Computer Science, ETH Zurich, Switzerland Department o Media Technology, Sogang University, South Korea 3 Australian National University and NICTA, Canberra, Australia Abstract. Identiying inliers and outliers among data is a undamental problem or model estimation. This paper considers models composed o rotation and ocal length, which typically occurs in the context o panoramic imaging. An eicient approach consists in computing the underlying model such that the number o inliers is maximized. The most popular tool or inlier set maximization must be RANSAC and its numerous variants. While they can provide interesting results, they are not guaranteed to return the globally optimal solution, i.e. the model leading to the highest number o inliers. We propose a novel globally optimal approach based on branch-and-bound. It computes the rotation and the ocal length maximizing the number o inlier correspondences and considers the reprojection error in the image space. Our approach has been successully applied on synthesized data and real images. Keywords: Consensus set maximization, branch-and-bound, inlier detection, RANSAC. 1 Introduction Distinguishing inliers and outliers among data is a undamental problem and constitutes a necessary step or model estimation, notably in computer vision. An eicient approach to identiy inliers and outliers consists in estimating the underlying model in such a way that the number o inliers is maximized. The most popular technique must be RANSAC [8] and has been applied or numerous computer vision tasks ranging rom 3D reconstruction to object recognition. Despite its popularity and that interesting results can be obtained, RANSAC is not guaranteed to maximize the number o inliers in a globally optimal way. This paper is dedicated to rotational homography with unknown ocal length, i.e. models composed o rotation and ocal length, which typically occurs in the context o panoramic imaging [5]. We propose a globally optimal approach that computes the camera rotation and the ocal length so that the maximum number o inlier correspondences between two images is guaranteed to be obtained. Previous work investigated how to maximize the number o inliers. First, several variants o RANSAC have been proposed, or example MLESAC, LO- RANSAC and preemptive-ransac [, 17, 19, 7]. While they generally perorm

2 Jean-Charles Bazin, Yongduek Seo, Richard Hartley, Marc Polleeys better than the original RANSAC in terms o the number o identiied inliers, they are not guaranteed to obtain the optimal result. In contrast to random sampling, Li [13] applied an optimal branch-and-bound technique in combination with convex and concave envelops [15]. However this approach is limited to distance deinitions and constraints that are strictly linear with respect to the sought model. Kahl et al. [1] proposed an optimal method also based on branchand-bound in combination with L 1 -norm to partially reduce the sensitivity to outliers. A post-validation step was proposed by Olsson et al. [18]. While this approach is useul to veriy the optimality o a potential solution, it does not provide a mechanism to explicitly compute the optimal solution. Bazin et al. [] proposed an approach to maximize the number o inliers under a pure rotational model. In contrast to their work, (i) we do not assume that the ocal length is known in advance, (ii) we compute the ocal length, in addition to the rotation, (iii) instead o the angular error, we consider the meaningul Euclidian distance in the image space in pixels [], which requires deriving the reprojection bounds in the image, and (iv) we introduce a rotation parametrization that permits to reduce the correlation between the ocal length and the rotation parameters. Yang et al. [3] recently proposed a globally optimal Iterative Closest Point (ICP) algorithm or rigid registration (rotation + translation) o two 3D point sets and can be applied to maximize the number o inlier correspondences between these two sets. This method is dedicated to 3D point sets (e.g. registration error in 3D space) and thus cannot be straightorwardly generalized or unknown ocal length and used or our application in the image space. Formulation Let us note x i = (x i, y i ) and x i = (x i, y i ) the ith input pair o D eature points in correspondence (e.g. obtained by SIFT [1]), respectively in the irst and second images, with i = 1... N and where N is the number o matches. The two images are taken with a camera located at a ixed position and turning with any 3D rotation R. It is assumed that the camera is intrinsically calibrated (e.g. camera center is known), except or the ocal length which is unknown and same or the two images. In the ollowing, all the measurements (x i, x i ) are centered, i.e. the camera center s coordinates (C x, C y ) are subtracted to the points coordinates in pre-processing, and thus the intrinsic calibration matrix is reduced to K = diag(,, 1) []. In the absence o noise and outliers, any two measurements (x i, x i ) in correspondence veriy: x i = Tx i = K R K 1 x i and d R, (x i, x i) = 0 i (1) where d R, (x, x ) = K RK 1 x x, that is the Euclidian distance in pixels between the measurement in the right image x and the measurement o the let image x projected into the right image by the transormation T that depends on the camera rotation R and the ocal length. In the ollowing, d R, (x, x ) is called the reprojection error.

3 Inlier Set Maximization With Unknown Rotation and Focal Length 3 Due to noise and outliers, these relations might not be veriied or all the input matches. Following the residual tolerance method [8], we deine a match (x i, x i ) as an inlier i the reprojection error is than a residual tolerance δ, i.e. d R, (x i, x i ) δ. Otherwise the match is considered an outlier. Let S represent the set o input matches: S = {(x i, x i ), i = 1... N}. The set S is partitioned into an inlier-set S I S containing the inlier matches and an outlier-set S O S with S O = S S I. The cardinality o S I corresponds to the number o inlier matches. Maximizing the number o inliers with unknown rotation and unknown ocal length can now be ormulated as: max S I,R, card(s I ) (a) s.t. d R, (x i, x i) δ, i S I S (b) R SO(3) Solving System in a globally optimal way is a challenging task mainly due to the non-linearity, the non-convexity and the rotation constraint. System can also be considered as a typical chicken-and-egg problem: i the inliers are known, then the underlying model (rotation and ocal length) can be computed [6], and reciprocally, i the model is known, the inliers can be retrieved (simple check o the inlier constraint at Eq. (b)). Unortunately neither the inliers nor the model is known apriori. A method would be to test all the possible combinations o inliers/outliers. While the number o combinations is inite ( N ), it is generally untractable in practice. Another naive method would be to test all the possible models but the model search space has an ininite cardinality and thus is untractable. In practice, a popular method to solve System is to apply RANSAC in combination with a -point algorithm [], but as explained above, RANSAC is not guaranteed to return the globally optimal solution. 3 Proposed approach This section presents the proposed approach to solve System in a globally optimal way. We start by introducing a particular parametrization o the transormation T, then study the projection bounds when the parameters o the transormation lie in a given range, and inally explain how to use these bounds in the ramework o branch-and-bound. (c) 3.1 Parametrization Rotation can be parameterized in several ways such as Euler angles, quaternion and axis-angle. In our work, we propose to use a parametrization o the orm R = R z (θ) R r R z (φ) (3) where R z (θ) is a rotation about the z-axis (optical axis) by an angle θ and R r is a rotation about the y-axis to be explained presently.

4 Jean-Charles Bazin, Yongduek Seo, Richard Hartley, Marc Polleeys One key advantage o this parametrization is the way it interacts with K in the deinition (1) o T. One may easily veriy that K commutes with a rotation R z, i.e. K R z = R z K. This important observation provides a simple orm or T: T = K R K 1 = K R z (θ)r r R z (φ) K 1 = R z (θ) K R r K 1 R z (φ). () We deine R r as the rotation about the y-axis with the property that K R r K 1 takes origin (0, 0, 1) to the point (r, 0, 1). The mapping K R r K 1 is o the orm cos(α) 0 sin(α) 1 0 tan(α) K R y (α)k 1 = = 0 1/ cos(α) 0 sin(α) / 0 cos(α) tan(α) / 0 1 (5) up to (irrelevant) scale. In order to take (0, 0, 1) to (r, 0, 1), it ollows that tan(α) = r/, so cos(α) = / + r. For easier notation, let us consider the reciprocal ocal length g = 1/ instead o the ocal length. Ater some mathematical manipulations, we inally obtain the mapping U(g, r) = K R r K 1 = 1 0 r r g 0. (6) rg 0 1 Finally, the complete transormation T gets simpliied to: 1 0 r T = R z (θ) r g 0 R z (φ). (7) rg 0 1 Interpretation The two rotations R z (φ) and R z (θ) carry out rotations about the image origin. One may think o these as being rotations in the let and right hand images respectively. Thus, the way a point is transormed by T is as ollows: 1. The point in the let image is rotated around the origin by an angle φ by R z (φ).. Then it is mapped into the right image by U(g, r) (see (6)), parameterized by g and r (i.e. and α). 3. Finally, it is rotated around the origin o the right image by an angle θ by R z (θ). An interesting observation is that in order to test a correspondence between points x i (in the let image) and x i (in the right image), one may do a prerotation o x i by φ and x i by θ. In this operation, the circular inlier neighbourhoods (o radius δ) remain circular and the radius is unchanged. All this is possible thanks to the rotation parametrization o (3) that permits R z to commute with K, and in turn, to separate R z and. This reduces the correlation o and the rotation parameters, which is desirable in the reduction o the error bound, and also simpliies the derivations o the bounds in pixel error (more details in the next sections).

5 Inlier Set Maximization With Unknown Rotation and Focal Length 5 Additional properties The mapping U(g, r) applied to a point x = (x, y, 1) provides a point x given by x = (x, y, 1) x = (x + r, y 1 + r g, 1 rxg ). (8) Ater dehomogenizing, this becomes ( ) (x, y) (x, y x + r ) = 1 rxg, y 1 + r g 1 rxg. (9) An important note is that x depends only on x, and not on y. In particular, this mapping takes a (vertical) line o the orm x = c to a new line x = (c + r)/(1 rcg ). This will be used or a quick-and-easy way o checking whether a match is a potential inlier given a certain transormation. Another important property is the trajectory o the point (x, y) mapped into the right image as r varies. From (9), one can show that this trajectory is a hyperbola with the equation (1 + g x 0) (y ) (g y 0) (x ) = y 0 () where (x 0, y 0 ) represents the initial point when r = 0. The initial point (x 0, y 0 ) is the same as (x, y), but written with the subscript so as to indicate that this point is constant here and (x, y ) are the variables. 3. Intervals o r and g This section investigates where a point x = (x, y) is mapped to under a transormation U(g, r), when the parameters r and g lie in a range. Let us suppose that r min r r max and g min g g max. With these intervals, the mapped point U(g, r)x must lie inside some bounded region in the right image and in the ollowing, we compute the bounds on its x and y coordinates. Bounds on y. Based on the above observations and derivations, U(g, r)x must lie between the two hyperbolas deined by () or g = g min and g = g max. This gives a very convenient way to determine whether the mapping o x into the right image passes close to (i.e. up to δ) its putative corresponding point x or any value o r. The hyperbolas are illustrated in Figure 1. This igure shows that or values o g close to zero (very large ocal lengths), the point moves almost horizontally, whereas or larger values o g (shorter ocal lengths) the trajectory becomes more curved. Bounds on x. As shown previously, where a point x = (x, y) maps to does not depend on y, only on x (see (9)). As y varies in the point (x, y), the mapped point (x, y ) varies along a vertical line as well. Thus, U maps vertical lines to vertical lines. Taking derivatives o the line x = (x + r)/(1 rxg ) with respect to r and g gives x r = 1 + g x x grx(r + x) ( 1 + g and = rx) g ( 1 + g rx). (11)

6 6 Jean-Charles Bazin, Yongduek Seo, Richard Hartley, Marc Polleeys y g= g=0.0 g=0.0 g= g=0.00 g= x Fig. 1. Hyperbolic trajectories o a point transormed by U(g, r) with dierent values o g, given the initial point (x 0, y 0) = (5, ) and or any values o r. The derivative with respect to r is always positive. Thereore, the minimum value x min o x will be achieved at r = r min and the maximum x max at r = r max. The sign o the derivative with respect to g corresponds to the sign o grx(r+ x), which depends on the values o x and r (note that g = 1/ 0 since ocal length is always positive). Let us study the minimum o x. As explained right above, it is achieved with r = r min. Let note s = sign(r min x(r min +x)). Thus the sign o the derivative is the same as s: i s 0 (rec. s 0) then the derivative is positive (rec. negative), then the minimum o x is obtained at g = g min (rec. g = g max ). Concretely, the minimum value o x is achieved at one o the two parameter values (r, g) = (r min, g min ) or (r min, g max ) depending on the sign o r min x(r min + x). A similar derivation can be ollowed or the maximum o x : the value at which x takes its maximum value is at either (r, g) = (r max, g min ) or (r max, g max ) depending on the sign o r max x(r max + x). Summarizing this, the minimum and maximum o x are achieved at two o the our corners o the rectangle deined by the range o the parameters r and g. Thus, we see that we can bound the range o the point x = (x, y ) to lie between two vertical lines x = x min and x = x max and to lie between the two hyperbolas corresponding to the values o g min and g max. This gives a simple test to see whether the point x can transorm to the target point x within a suitable radius (i.e. up to δ). Concretely, the test can be conducted as ollows: 1. In terms o the initial point x = (x 0, y 0 ) compute the bounds x min and x max and test i x lies between them.. Test i x lies between the two hyperbolas deined or g min and g max and with the initial point (x 0, y 0 ). I either o these two tests ails, then it is sure that x is not within range. Otherwise, x might be within range: (x, x ) is a potential inlier match but this has to be conirmed with urther investigations, as detailed in the next section. 3.3 Branch-and-bound To eiciently deal with bounds, we ollow the branch and bound algorithm (noted BnB in the ollowing). BnB is a general ramework or global optimization [11] that recently gained popularity in the ield o computer vision [1,, 9]. Given

7 Inlier Set Maximization With Unknown Rotation and Focal Length 7 a search space o the model to estimate, BnB iteratively subdivides this search space into smaller subspaces, identiies and removes the subspaces that do not contain the optimal solution via a easibility test and reines the remaining subspaces, i.e. the subspaces that can potentially contain the optimal solution. Our search space is composed o the rotation space SO(3) parameterized by the angles (θ, φ, α) (see Section 3.1) and the ocal length range. We call cube a delimited part o that space. We now ask the question: how to estimate the and bounds o the number o inliers that can be obtained by any models in a given cube? Lower bound. Let us start with the bound. The center o a given cube C corresponds to a speciic point in the search space and thus corresponds to known rotation angles (θ, φ, α) and ocal length, which in turn provides R and K. Then we can simply count the number o matches (x i, x i ) veriying the inlier constraint (b). One may note that this number does not necessarily correspond to the lowest number o inliers that could be obtained in the cube C, but provides a practical bound o the maximum number o inliers that can be obtained in the cube C. Upper bound. Computing the bound is more challenging. We note [θ l, θ u ], [φ l, φ u ], [α l, α u ] and [ l, u ] the deinition ranges o these parameters in a given cube C. The rotation o a point x i (in the let image) by R z (φ) with φ [φ l, φ u ] deines a circular arc centered at the origin and o length φ u φ l. The x and y bounds o this arc can be easily obtained analytically and we note them ([x i ], [y i ]) = ([x i,l, x i,u ], [y i,l, y i,u ]) where l and u stand or and values. Similarly, let us note ([x i ], [y i ]) = ([x i,l, x i,u ], [y i,l, y i,u ]) the bounds o the point x i (in the right image) rotated by R z(θ) with θ [θ l, θ u ]. In Section 3., we investigated how the ranges o r and g (i.e. α and ) inluence the mapping U and we derived the x and y bounds o the point x mapped by U. This approach can be easily generalized to an interval o point ([x i ], [y i ]) instead o a given point x. The extended x and y bounds o the interval ([x i ], [y i ]) mapped by U can be computed in a similar way, analytically or by simple interval arithmetics [16]. This provides the interval o x i mapped by U in the right image (obtained by the deinition ranges o φ, α, ) and we compare this interval to the interval o ([x i ], [y i ]) (obtained by the deinition range o θ). I the intervals intersect (up to δ), then the match (x i, x i ) is a potential inlier under a model contained in C. I they do not intersect, then it is deinitively an outlier or the cube C. We conduct this procedure or all the input matches and count the number o potential inliers, which in turn provides an bound o the number o inliers or C. Search strategy. We explained above how to compute the and bounds o the number o inliers or a given cube. We now discuss how to iteratively discard the non-easible cubes and conduct the search strategy. First o all, the search starts with a cube list L that is initialized with one cube covering the entire search space (see above). During the subdivision o a

8 8 Jean-Charles Bazin, Yongduek Seo, Richard Hartley, Marc Polleeys cube C j, each dimension o this cube is split into two intervals o equal length, which provides, in total, = 16 disjoint smaller cubes whose side length is hal o the side length o the original cube C j. The cube C j is removed rom L and replaced by its subdivided cubes. Let l be the highest bound obtained so ar or the number o inliers computed beorehand by any existing methods. Let us note l j and u j the and bounds o the number o inliers o a cube C j. I u j < l then it is sure that the optimal model (i.e. the model leading to the highest number o inliers) is not contained in C j because C j does not contain any models that can lead to at least or more than l inliers. Thus the cube C j is considered non-easible and can be removed rom the search space. On the contrary (i.e. i u j l ) then the cube C j is subdivided or urther investigation. In case l j > l then l is updated by l j. This procedure is applied iteratively or all the cubes contained in the cube list L. We deine the maximum bound u as the highest bound among the easible cubes currently present in the list L. Along the BnB iterations, the non-easible cubes are discarded (the search space reduces), the size o the cubes decreases (by subdivision), the gap between the and bounds l j and u j computed or each cube C j diminishes, and l and u converge. The search stops when the list L contains at least one cube Cj whose bound l j equals the maximum bound u because it means that the model at the center o Cj (i.e. used to compute l j) provides the maximum number o inliers u. Finally the BnB procedure simply returns the model (R, ) associated to the center o Cj and this model leads to the maximum number o inliers u that can be obtained inside the search space. Results This section presents some experimental results obtained or synthesized data and real images. Implementation details, additional algorithmic explanations and supplementary results are available on the authors website. Our branch-andbound approach has been implemented in C++ and ran on a computer equipped with an Intel Core i7 CPU.8GHz (a single core is used) and 1GB RAM. We run BnB on the whole rotation search space SO(3) and a conservative realistic range or the ocal length [0, 500], unless otherwise stated. Our approach takes between a ew seconds and a ew minutes depending mainly on the number o points and the search space size. We compare our approach to the conventional RANSAC and the optimized LO-RANSAC [7], both being reerred to as RANSACs. They both embed the state-o-the-art minimal solution approach o Brown et al. [], that we reer to as the -point algorithm. We consider LO-RANSAC in addition to the conventional RANSAC, because it is known to perorm very well in practice. Among the versions o LO-RANSAC, we apply the inner RANSAC with iteration because it has been shown that it provides the best results [7]. The number o RANSAC iterations is automatically computed with the true outlier ratio (i not available, e.g. or real data, set to %), a guaranteed accuracy o 99% and a minimal sampling o points or the -point algorithm []. Since dierent runs o RANSACs might

9 Inlier Set Maximization With Unknown Rotation and Focal Length 9 percentage o experiments distribution o the nb o inliers RANSAC LO RANSAC Nb o detected inliers Upper and bounds Fig.. Comparison between RANSACs and the proposed approach. Let: distribution o the number o inliers obtained by RANSAC and LO-RANSAC. Their best run leads to 88 and 89 inliers respectively. Right: convergence o the BnB bounds to 90 inliers. percentage o experiments execution time (in sec) percentage o experiments execution time (in sec) Fig. 3. Distribution o the execution time o our approach or N = 30 (let) and N = 300 data points (right) with 70% o outliers. lead to dierent results (because o the random data sampling), we repeat each experiment over 00 runs with the same input data and parameters. Finally, the algorithms are compared with respect to the number o inliers detected..1 Synthesized data We randomly generate a set o N = 300 correspondences o D points between the let and right images. To reproduce realistic settings, we corrupt the x and y coordinates o the points by a Gaussian noise (std=0.5pixels) and create a percentage p = 70% o outliers with δ = pixel. Figure compares the number o inliers obtained by the proposed approach and RANSACs. Our BnB approach obtains 90 inliers, which corresponds to the number o synthesized inliers. On the contrary, RANSAC spans between 61 and 88 inliers and never obtains the number o synthesized inliers, that is 90. As expected, LO-RANSAC perorms better than RANSAC: results span between 71 and 89 inliers. While it does not obtain the number o synthesized inliers neither, the distribution is clearly improved: RANSAC obtained more than 8 inliers in 6% o the 00 runs, whereas it occurred in 7% o the runs or LO-RANSAC. One might wonder why neither RANSAC nor LO-RANSAC managed to obtain the true number o inliers. This actually happened in about % o our experiments or RANSAC and 65% or LO-RANSAC. This can be explained by two reasons. First, the perormance o RANSACs depends on the data noise:

10 Jean-Charles Bazin, Yongduek Seo, Richard Hartley, Marc Polleeys or completeness, we conducted several experiments with noise-ree data, but still corrupted by outliers, and the true number o inliers was obtained in a percentage o experiments similar to the selected guaranteed accuracy o 99%. Second, RANSAC can hypothesize only models that are directly supported by the selected points. The concrete consequence is that RANSAC cannot return the optimal number o inliers i the associated model cannot be hypothesized by the minimal points. Thanks to its local optimization steps, LO-RANSAC perorms better than RANSAC, but still relies on the data points support to hypothesize models, and thereore this limitation is removed only partially. For a complete comparison o RANSACs and our BnB approach, we conducted more than 0,000 experiments with various data amounts (N = 0 points), proportions o outliers (p = 0% 90%) and ocal length ranges ([500, 700] to [, 5000]). In all the experiments, the BnB bounds always converged to the true number o inliers, and the number o inliers obtained by our approach was always higher than or equal to the number obtained by RANSAC and LO-RANSAC. We conducted 00 dierent experiments with newly randomly generated data and measured the execution time o our BnB approach or each experiment. The distribution o the execution time or N = 30 and N = 300 data points (70% o outliers) is illustrated in Figure 3. As expected, the execution time increases with the number o points since there are more data to process, and the igure shows that the approach is scalable with the data amount. RANSACs deinitely run aster than the BnB approach but we do not aim to compete with RANSACs in terms o speed: our key goal is to obtain the globally optimal solution.. Real data We now present results obtained by our approach on real images. We perorm the intrinsic calibration o the camera by Bouguet s toolbox [3], and the camera center is then applied to the point measurements to center their coordinates. Putative correspondences between two images are obtained by extracting and matching SIFT eatures [1]. The number o RANSAC iterations is computed with a very conservative outlier ratio o % and we run RANSACs 00 times or each experiment. In the case the ocal length and its limits are totally unknown, we use a conservative ocal length range [0, 500]. This ocal length range covers most o the practical cases and can be enlarged i needed. Optionally, a rough ocal length estimation and/or its limits can be obtained via EXIF tags like in [1, ] or known approximately rom the camera device. A irst representative result with a small overlap is shown in Figure. The images are acquired with a Sony NEX-3 camera. A rough estimate o the ocal length is available rom the EXIF tag and we use a range o ±0 around this value (we will show results without any EXIF tag inormation in Figure 5). 163 putative correspondences are obtained by SIFT. The number o inliers obtained by RANSAC and LO-RANSAC span between 75 and, and between 77 and respectively, as shown in Figure -(a). Their maximum number o inliers, that is, is obtained during only 9% o the 00 runs by RANSAC, and 19% by

11 Inlier Set Maximization With Unknown Rotation and Focal Length RANSAC LO RANSAC Upper and bounds percentage o experiments distribution o the nb o inliers Nb o detected inliers (a) x (b) (c) (d) (e) Fig.. (a) Distribution o the number o inliers obtained by RANSAC and LORANSAC. (b) Convergence o the BnB bounds to inliers. (c) Inlier (green) and outlier (red) matches detected by our approach. (d) Highlighted overlap o the input images on the resulting panoramic view. (e) The inal panoramic view x x x Upper and bounds 0 Upper and bounds Upper and bounds Upper and bounds x (a) (b) (c) (d) Fig. 5. Convergence o the BnB bounds to inliers with a ocal length range o ±300 (a), ±500 (b), ±00 (c) around a rough ocal length estimation, and with a very conservative ocal length range o [0, 500] (d).

12 1 Jean-Charles Bazin, Yongduek Seo, Richard Hartley, Marc Polleeys LO-RANSAC. The BnB bounds converge to inliers, as shown in Figure -(b). The inlier and outlier matches identiied by our BnB approach are shown in Figure -(c), and the resulting panoramic view in Figure -(d,e). For completeness, Figure 5 illustrates the evolution o the BnB bounds with dierent ocal length ranges or the image pair o Figure. First it shows that the bounds always converged. Also, a ocal length range o ±0 took about 5. iterations (see Figure -(b)) and a range o ±00 took about 16 iterations (see Figure 5-(c)), that is about.96 times more iterations while the search space is 5 times larger. This shows that the method is scalable with the ocal length range. Figure 5-(d) shows the convergence o the BnB bounds with a very conservative ocal length range, which shows that our method can be applied or practical cases when the ocal length and its limits are totally unknown. RANSAC LO RANSAC Upper and bounds percentage o experiments distribution o the nb o inliers Nb o detected inliers 5 (a) x (b) (c) (d) (e) Fig. 6. Same legend as in Figure. The best run o RANSAC and LO-RANSAC leads to inliers and our BnB bounds converge to 8 inliers.

13 Inlier Set Maximization With Unknown Rotation and Focal Length 5 RANSAC LO RANSAC Upper and bounds percentage o experiments distribution o the nb o inliers Nb o detected inliers (a) x (b) (c) (d) (e) Fig. 7. Same legend as in Figure. The best run o RANSAC and LO-RANSAC leads to 69 inliers and our BnB bounds converge to 70 inliers. An additional representative result with a large overlap is shown in Figure putative correspondences are obtained by SIFT. The number o inliers obtained by RANSAC and LO-RANSAC span between 9 and, and between 38 and respectively. Their maximum number o inliers, that is, is obtained during only 3%, by RANSAC, and 6%, by LO-RANSAC, o the 00 runs. In contrast, our BnB bounds converge to 8 inliers, which is additional inliers than the best result o RANSAC and LO-RANSAC among the 00 runs. For this image pair, the bound increases very quickly to 3 inliers, and

14 1 Jean-Charles Bazin, Yongduek Seo, Richard Hartley, Marc Polleeys the gap between the and the bounds reduces slowly. This is because the reprojection errors o some correspondences were close to the inlier threshold and thus the size o the cubes (i.e. the uncertainty o the model) needs to be suiciently small to be able to decide whether such correspondence is inlier or outlier, and this cube size is continuously reduced along the BnB iterations. We also apply our approach on images acquired by a smartphone. Figure 7 shows a representative result with a conservative ocal length range: the RANSAC distribution, the convergence o the BnB bounds, the inlier/outlier correspondences obtained by our approach and the resulting panoramic view. Some additional results or dierent scenes and cameras are available on the authors website. Similarly to the experiments with synthesized data, the number o inliers obtained by our approach on all our experiments with real images was always higher than or equal to the number o inliers obtained by RANSAC and LO-RANSAC, which conirms the validity o our approach. 5 Conclusion This paper aced the problem o inlier set maximization in the image space with unknown rotation and ocal length. The most popular approach to solve this problem is RANSAC but it has several limitations especially the lack o optimality. We proposed a new approach based on branch-and-bound that maximizes the inlier set in a globally optimal way: it returns the rotation and the ocal length leading to the highest number o inlier correspondences in the image space. The validity o the approach has been conirmed by experiments on synthesized data and real images. An interesting direction or uture work would be to estimate, in addition to the ocal length, some extra intrinsic parameters, such as the camera center. We also plan to investigate the generalization o our approach o inlier set maximization in the image space beyond pure-rotation motion and or other models. Especially, in the context o undamental matrix estimation, the goal would be to compute the rotation and translation o the camera, as well as the ocal length, in such a way that the number o inlier correspondences in the image is maximized, up to an inlier threshold in pixels (e.g. with respect to the distance to the epipolar line). Acknowledgments This research, which has been partially carried out at BeingThere Centre, was supported by the Singapore National Research Foundation under its International Research Singapore Funding Initiative and administered by the IDM Programme Oice. The work was also supported by the Sogang University Research Grant o 109. NICTA is unded by the Australian Government through the ARC. Yongduek Seo is the corresponding author.

15 Reerences Inlier Set Maximization With Unknown Rotation and Focal Length Agarwal, S., Snavely, N., Simon, I., Seitz, S.M., Szeliski, R.: Building Rome in a day. In: ICCV (09). Bazin, J.C., Seo, Y., Polleeys, M.: Globally optimal consensus set maximization through rotation search. In: ACCV (1) 3. Bouguet, J.Y.: Camera calibration toolbox or Matlab. Brown, M., Hartley, R., Nister, D.: Minimal solutions or panoramic stitching. In: CVPR (07) 5. Brown, M., Lowe, D.: Automatic panoramic image stitching using invariant eatures. IJCV (07) 6. Choi, K., Lee, S., Seo, Y.: A branch-and-bound algorithm or globally optimal camera pose and ocal length. Image and Vision Computing () 7. Chum, O., Matas, J.: Optimal randomized RANSAC. TPAMI (08) 8. Fischler, M.A., Bolles, R.C.: Random sample consensus: A paradigm or model itting with applications to image analysis and automated cartography. In: Communications o the ACM (1981) 9. Hartley, R., Kahl, F.: Global optimization through rotation space search. IJCV (09). Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, second edn. (0) 11. Horst, R., Tuy, H.: Global optimization: deterministic approaches. Springer Verlag (06) 1. Kahl, F., Agarwal, S., Chandraker, M., Kriegman, D., Belongie, S.: Practical global optimization or multiview geometry. IJCV (08) 13. Li, H.: Consensus set maximization with guaranteed global optimality or robust geometry estimation. In: ICCV (09) 1. Lowe, D.: Distinctive image eatures rom scale-invariant keypoints. IJCV (03) 15. McCormick, G.: Computability o global solutions to actorable nonconvex programs: part I - convex underestimating problems. Mathematical Programming (1976) 16. Moor, R.: Interval Analysis. Prentice-Hall (1966) 17. Nistér, D.: Preemptive RANSAC or live structure and motion estimation. In: ICCV (03) 18. Olsson, C., Enqvist, O., Kahl, F.: A polynomial-time bound or matching and registration with outliers. In: CVPR (08) 19. Raguram, R., Frahm, J.M., Polleeys, M.: A comparative analysis o RANSAC techniques leading to adaptive real-time random sample consensus. In: ECCV (08). Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: Exploring photo collections in 3D. In: SIGGRAPH (06) 1. Sun, M., Telaprolu, M., Lee, H., Savarese, S.: An eicient branch-and-bound algorithm or optimal human pose estimation. In: CVPR (1). Torr, P., Zisserman, A.: MLESAC: A new robust estimator with application to estimating image geometry. Computer Vision and Image Understanding (00) 3. Yang, J., Li, H., Jia, Y.: Go-ICP: solving 3D registration eiciently and globally optimally. In: ICCV (13). Yu, C., Seo, Y., Lee, S.W.: Global optimization or estimating a BRDF with multiple specular lobes. In: CVPR ()