CAMERA AUTOCALIBRATION AND HOROPTER CURVES

CAMERA AUTOCALIBRATION AND HOROPTER CURVES JOSÉ IGNACIO RONDA, ANTONIO VALDÉS, AND FERNANDO JAUREGUIZAR * Grupo de Tratamieto de Imágees ** Dep. de Geometría y Topología Uiversidad Politécica de Madrid Uiversidad Complutese de Madrid E-28040 Madrid, Spai E-28040 Madrid, Spai {jir,j}@gti.ssr.upm.es Atoio Valdes@mat.ucm.es Abstract. We describe a ew algorithm or the obtaimet o the aie ad Euclidea calibratio o a camera uder geeral motio. The algorithm exploit the relatioships o the horopter curves associated to each pair o cameras with the plae at iiity ad the absolute coic. Usig these properties we deie cost uctios whose miimizatio by meas o geeral purpose techiques provides the required calibratio. The experimets show the good covergece properties, computatioal eiciecy ad robust perormace o the ew techiques. Keywords: camera autocalibratio, horopter, projective geometry, oliear optimizatio. 1. Itroductio The semial paper [2] was the irst to show the possibility o calibratig a camera rom a set o views, avoidig the use o ay kid o calibratig object. Their method was based i the so-called Kruppa equatios, which permit to locate the plae at iiity ad the absolute coic i it, so recoverig the Euclidea structure o space. Additioal objects ad cocepts came later to provide alterative tools ad methods to solve the sel-calibratio problem. I [18] the absolute quadric was itroduced. This is a sigular imagiary quadric i the dual space P 3 which ecodes simultaeously the positio o the plae at iiity ad the absolute coic. I [] the homographies betwee images iduced by the plae at iiity are employed to obtai the uimodular costrait rom which the plae at iiity ca be oud. The key observatio o this work is that or the plae at iiity the associated homography is cojugated to a rotatio ad thereore its eigevalues are all o them o equal modulus. Other approaches to sel-calibratio ca be oud i [4] ad [7]. Aother atural geometric object associated to a pair o idetically calibrated cameras is the horopter curve, deied as the set o poits i space which have the same image coordiates i both cameras, ad geerically is a twisted cubic. They were itroduced i moder computer visio by Maybak i [9], appearig as oe o the irreducible compoets o the quartic curve obtaied as itersectio o two ambiguous suraces. Some o the iterestig properties o horopter curves ca be oud i [9], [10], [7]. Up to our kowledge, horopter curves have ot bee used so ar i their ull potetial or the sel-calibratio problem, with the remarkable exceptios o the works o Armstrog et al. [1] ad o Schaalitzky [15]. I the irst oe it is show how horopters ca be used or the recovery o the plae at iiity i the special case o plaar camera motios. I the secod the properties o horopter curves are employed to obtai ew polyomial equatios or the obtaimet o the plae at iiity. All the previously metioed methods, as well as the preset work, deal with the recovery o arbitrary but costat itrisic parameters o the camera. Research has also bee coducted elsewhere to cover the case o varyig itrisic parameters with some restrictios (see e.g., [8] ad the reereces therei). The mai idea behid the algorithms is to take advatage o the highly sigular coiguratio adopted by the horopter curves i relatio with the plae at iiity ad the absolute coic, which is preserved by homographies o the space. The, startig rom a projective calibratio, this allows or the deiitio o cost uctios o the set o plaes o the space which vaish or the plae at iiity idepedetly o the adopted projective coordiates. The miimizatio is carried out by meas o a Leveberg-Marquardt algorithm. Experimets show the good covergece ad stability properties o the techique, eve i the presece o oise, alog with its computatioally eiciecy. Besides, the approach does ot require the precise iitializatio which is usually imperative. The paper is orgaized as ollows: Sectio 2 reviews the projective camera model ad ormalizes the selcalibratio problem. Sectio 3 provides a sel-cotaied presetatio o the relevat properties o horopter curves. Date: December 1, 2005. This work has bee partly supported by the Comisió Itermiisterial de Ciecia y Tecología (CICYT) o the Spaish Govermet. 1

2 JOSÉ IGNACIO RONDA, ANTONIO VALDÉS, AND FERNANDO JAUREGUIZAR Some o these properties are ew ad or the others we have provided ew proos toward a more cosistet ad geometric-orieted presetatio. Sectio 3 describes the algorithms ad, ially, Sectio 4 provides the experimetal results. The mai cotributios o this work, appart rom the autocalibratio algorithms, cosist i relatig horopter curves with the uimodular costrait o [], which is doe i Theorems 3.1.3 ad 3.2. Theorems 3.5 ad 3.6 also preset ew results cocerig the geometry o horopter curves. 2. Mathematical model o camera sel-calibratio Our model or a real camera will be the traditioal pihole camera [3], which is deied by its optical ceter C ad the image plae π. The image plae is edowed with a aie coordiate system (O;u,v), where O π is a arbitrary origi o coordiates ad vectors u ad v orm a basis o the plae, ot ecessarily orthoormal. I practice this coordiate system is that give by the pixel structure o the camera. The capture o the 3D-scee is the mathematically modeled as the correspodece o each space poit with its projectio oto the plae π with ceter C. For coveiece, we cosider the Euclidea space coordiate system {C;x,y,z} where x = u/ u, y is deied so that {x,y} is a orthoormal basis o π with the same orietatio o {u,v}, ad z completes them to a orthoormal basis ad poits toward the camera rom C. Usig homogeeous coordiates attached to the previously deied reereces, say Q c = (x c, y c, z c, t c ) T ad q = (u, v, w) T or space ad image poits, respectively, the equatios o the projectio are q = λk(i 0)Q c, where I is the 3 3 idetity matrix, λ is a projective proportioality costat, ad K is the matrix o itrisic parameters give by cot u 0 u u K = 0 v 0. v si 0 0 1 I this matrix is the ocal legth, i.e., the Euclidea distace rom the optical ceter to the image plae, is the agle rom u to v, ad (u 0, v 0 ) T are the o-homogeeous image coordiates o the, i.e., the orthogoal projectio o C oto π [3]. More geerally, i we wat to use ad arbitrary Euclidea space coordiate system istead o the oe associated to the camera, the projectio equatios become q = λk (R t) Q e where Q e = (x, y, z, t) T are the space poit homogeeous coordiates i the ew reerece ad R ad t are respectively the rotatio matrix ad traslatio vector givig the motio rom the secod to the irst space systems. Eve more geerally, we ca use a arbitrary projective 3D coordiate system istead o a Euclidea oe, with coordiates Q = (X, Y, Z, T) T related to the Euclidea coordiates by a 4 4 regular matrix G so that Q e = µ GQ, where µ is a projective proportioality costat. I this case the equatio o the projectio is give by q = νpq where P = K (R t)g ad we have abbreviated ν = λµ. The kowledge o the projectio matrix P reerred to the metioed 3D projective coordiate system ad the 2D aie coordiate system o the camera is called a projective calibratio o the camera. I the case o multiple cameras, a projective calibratio o the set is the kowledge o their projectio matrices with respect to a commo projective space reerece ad their respective image plae reereces. A importat issue is to obtai a projective calibratio respetig the covex hull o the poits i the image (see [6], [11]). The aie calibratio o the set o cameras cosist i the obtaimet o the projectio matrices with respect to ay aie space reerece. We recall that a aie reerece is just a projective reerece {X 0,X 1,X 2,X 3,E} such that X 1,X 2 ad X 3 belog to the plae at iiity (see [16]). Thereore the aie calibratio ca be achieved by determiig the coordiates o the plae at iiity. Aalogously, a Euclidea calibratio is a aie calibratio or which the aie reerece is Euclidea, i.e., such that the absolute coic has equatios X 2 + Y 2 + Z 2 = T = 0.

CAMERA AUTOCALIBRATION AND HOROPTER CURVES 3 This correspods to the usual otio that the basis o the associated Cartesia reerece is orthoormal (up to a scale actor). 3. Horopter curves 3.1. Deiitio ad basic properties. Let us cosider two cameras idetical i every respect but i their space positio. This meas that each elemet o the secod camera ca be obtaied rom the correspodig elemet o the irst oe by applyig to it a commo Euclidea motio. The set o 3D poits whose projectios o both cameras have idetical coordiates is called the horopter associated to the pair o cameras. I this paper we will oly develop those properties o the horopter curves which are related to the recovery o itrisic ad extrisic parameters. The iterested reader may cosult [9], [10], [7] or related iormatio o the subject. Let us compute the equatio o the horopter associated with a couple o cameras with projectio matrices P i = λ i K(R i t i )G, i = 1, 2. A poit Q projects oto each camera o poits o the same homogeeous projective coordiates i ad oly i (1) P 1 Q = P 2 Q, or some costat C { } = P 1 uderstood as a projective parameter, i the sese o [16]. Note that the particular cases = 0 or = correspod to the camera ceters, which always belog to the horopter. Equatio (1) meas that Q belogs to the ull-space o the matrix P 1 P 2. To compute this kerel, let us deote by a T i, bt i ad c T i the rows o the matrix P i. A poit Q is i this kerel i ad oly i it satisies the system o equatios (a T 1 at 2 )Q = 0 (b T 1 bt 2 )Q = 0 (c T 1 c T 2 )Q = 0. This meas that Q lies simultaeously i the plaes o coordiates a 1 a 2, b 1 b 2 ad c 1 c 2. Let us deote by α = (α 1, α 2, α 3, α 4 ) T the coeiciets o a geeric plae. The plaes o the star through Q (we recall see [16] that this is the set o plaes ad lies cotaiig Q) are give by the equatio det(α, a 1 a 2, b 1 b 2, c 1 c 2 ) = 0. The coordiates o Q are thereore the coeiciets o (α 1, α 2, α 3, α 4 ) i this equatio. From this we see that the homogeeous coordiates o Q are give by our polyomials o degree three i. Geerically, these polyomials are idepedet ad the horopter is thereore a twisted cubic. For more iormatio o twisted cubics, see [16]. We will deote the parametric equatio o the horopter by h = h(). 3.2. The horopter ad the absolute coic. Beig the horopter a cubic curve, it geerically meets each plae i three poits (maybe o complex coordiates). Let us deote the plae at iiity by π. A poit Q π o Euclidea coordiates Q e G 1 Q = (x, y, z, 0) T belogs to the horopter i ad oly i (P 1 P 2 )Q K(λ 1 R 1 λ 2 R 2 λ 1 t 1 λ 2 t 2 )Q e = 0. Settig ˆQ e = (x, y, z) T ad α = λ 2 /λ 1, this is equivalet to (R 1 α R 2 ) ˆQ e = 0 (R T 2 R 1 α I) ˆQ e = 0. Observe that R = R T 2 R 1 is the rotatio part o the motio carryig the irst camera to the secod oe. The last equatio meas that ˆQ e is a eigevector o R with eigevalue α. Beig R a rotatio matrix, its eigevalues are 1, e iϕ ad e iϕ where ϕ is the rotatio agle. The eigevector associated with the eigevalue α = 1 must be real ad correspods to the directio o the rotatio axis. Cosiderig a eigevector ˆQ e o eigevalue e iϕ we have that ˆQ et ˆQe = ˆQ et R T R ˆQ e = (R ˆQ e ) T (R ˆQ e ) = e iϕ ˆQeT e iϕ ˆQe = e 2iϕ ˆQeT ˆQe. So, as log as e 2iϕ 1, i.e, ϕ 0, π (as we will assume rom ow o) we obtai that ˆQ et ˆQe = 0, i.e, its coordiates satisy the equatio x 2 + y 2 + z 2 = 0. This meas that the complex eigevectors are poits o the absolute coic (see [16], [3] or iormatio o the absolute coic ad how it ecodes the Euclidea or, more precisely, coormal structure o the space). Deotig by ˆQ e 0 the real eigevector ad by ˆQ e 1, ˆQe 2 the complex cojugate eigevectors o R, a similar computatio shows that ˆQ et 0 ˆQ e 1 = ˆQ et 0 ˆQ e 2 = 0,

4 JOSÉ IGNACIO RONDA, ANTONIO VALDÉS, AND FERNANDO JAUREGUIZAR provided that ϕ 0. This meas that the polar lie o the poit o coordiates ˆQ e 0 with respect to the absolute coic is the lie through the poits o coordiates ˆQ e 1 ad ˆQ e 2. So we have the ollowig result: Theorem 3.1. (1) The horopter curve attached to a pair o cameras related with a motio o agle dieret rom 0 ad π itersects the plae at iiity at three poits, oe o which is real ad the other two are complex cojugate ad lie o the absolute coic. (2) The pole o the lie determied by the complex poits is the real oe ad the real poit represets the directio o the screw axis o the motio. 1 (3) The horopter reaches the plae at iiity at parameters with ratios 1 : e iϕ : e iϕ, ϕ beig the rotatio agle o the motio. I particular, the three parameters have the same modulus. Remark 3.1. Sice all the cameras share the same calibratio matrix, the retial coordiates provide us with atural isomorphisms amog them, relatig poits with the same coordiates i dieret retial plaes. Thereore we ca idetiy these plaes with a virtual retial plae Π edowed with coordiates (u, v, w) compatible with the isomorphisms ad project oto it the itersectio poits ad the absolute coic usig ay o the projectio matrices (or eve all o them). A relevat eature o this costructio is that, as is well kow (see [7]), the absolute coic projects oto the same coic o Π idepedetly o the projectio matrix used. The polarity relatios are preserved by the homographies iduced by these projectios. Let us cosider a set o projectively calibrated cameras P l, l = 1,..., with idetical itrisic parameters. For each pair o cameras (i, j), i < j, we deote by h ij = h ij () the correspodig horopter, by Q ij k, k = 0, 1, 2 its itersectio poits with π ad by r ij kl the projectio rij kl = P lq ij k. O course, rij ki = P iq ij k = P jq ij k = rij kj, but the other projectios will be dieret, i geeral (see igure 1). I A is the deiite symmetric matrix o the projected absolute coic i Π we have (2) ( r ij 0l ) T Ar ij kl = 0 ad ( r ij kl ) T Ar ij kl = 0, where k = 1, 2, 1 i < j, ad l = 1,...,. Note that although the projected poits r ij kl, l i, j, are homographically related to the rij ki = rij kj, the cosideratio o all these poits is ot redudat i the cotext o the search or the plae at iiity. Ideed, i the cadidate plae is the plae at iiity, this homography leaves ivariat the projected absolute coic, but i this is ot the case, ivariace o the coic does ot hold true, ad thereore these additioal costraits are ot redudat. These equatios provide us with a method to recover the projectio o the absolute coic rom the kowledge o the coordiates o the plae at iiity. They are i the heart o our algorithms, as will be explaied later. Π Q 0 Q 0 13 Q 13 2 Q 2 Q 1 Q 23 1 Q 13 1 Q 23 2 Q 23 0 r 2 1 22 r r 1 3 r 0 3 r 0 1 r 1 1 r 0 2 r 1 2 r 2 3 camera 1 camera 2 camera 3 Figure 1. Diagram o relatioships betwee the itersectios o the horopters with the plae at iiity, the absolute coic, ad their projectios. 1 This is just a projective versio o the act that the real lie determied by the complex poits is the directio o the pecil o plaes orthogoal to the screw axis.

CAMERA AUTOCALIBRATION AND HOROPTER CURVES 5 3.3. Horopters ad the uimodular costrait. Twisted cubics ca be geerated by meas o a homography relatig two stars i space. For the coveiece o the reader, we recall here this Steier-type costructio (see [16]). The star o a poit has a structure o projective plae ad so we ca cosider homographies betwee stars. Such homographies ca be geerated usig a homography o the ambiet space H : P 3 P 3 as ollows: For each pair o poits C 1 ad C 2 = H(C 1 ) we ca deie a homography Star(C 1 ) Star(C 2 ) just by sedig the lie l Star(C 1 ) (resp. the plae π) to the lie H(l) Star(C 2 ) (resp. the plae H(π)). For simplicity, we will deote the homography betwee these stars agai by H. Note that ay homography betwee stars is iduced by a homography o the ambiet space: to see this it is eough to realize that a homography betwee stars is equivalet to a homography betwee two plaes o the dual space P 3, which has a iiity o extesios to the whole space. I geeral, a lie o the irst star does ot itersect its image. However, the set o lies which do itersect its image produce a twisted cubic, i.e., give a homography H : Star(C 1 ) Star(C 2 ) the set {P P 3 : P = l H(l), l Star(C 1 )}. is a twisted cubic. Ad, coversely, every twisted cubic ca be geerated i this way. The motio betwee the cameras is a homography o the space which iduces the correspodig homography betwee the stars based o the optical ceters o the cameras. The associated twisted cubic is the horopter previously deied. From this, it ollows immediately that the horopter depeds oly o the relative motio betwee the cameras ad the positio o the irst optical ceter. So let us cosider a homography H : Star(C 1 ) Star(C 2 ). For each plae τ ot cotaiig either o the poits C 1 ad C 2, there is a atural homography H τ : τ τ iduced by H, deied as ollows: Give a poit Q τ, we deie H τ (Q) = H(C 1 Q) τ. Geerically, a homography o a projective plae has three ixed poits. By its very deiitio, the three ixed poits o H τ are the three itersectio poits o τ with the twisted cubic associated to H. Theorem 3.2. (Uimodular costrait) Let H : Star(C 1 ) Star(C 2 ) be a homography ad τ a plae ot cotaiig ay o the ceters o the stars. Let us cosider the associated homography H τ : τ τ, which we suppose to be geeric, i the sese that it has exactly three dieret ixed poits, say, Q 0, Q 1 ad Q 2. Let h = h() be the twisted cubic attached to H, ad let us deote by i the parameters o the poits Q i or i = 0, 1, 2, ad by µ j the parameters o C j, j = 1, 2. The the eigevalues λ 0, λ 1 ad λ 2 o H τ have ratios give by λ 0 λ 2 = {µ 1, µ 2, 0, 2 }, λ 1 λ 2 = {µ 1, µ 2, 1, 2 }. I the particular case that the ceters are reached at parameters 0 ad, we have λ 0 λ 2 = {0,, 0, 2 } = 0 2, λ 1 λ 2 = {0,, 1, 2 } = 1 2. Proo. Let us cosider the projective reerece {X 0,X 1,X 2,X 3,E} o P 3, where X 0 = C 1,X 1 = Q 0,X 2 = Q 1,X 3 = Q 2,E = C 2. I terms o this reerece, the plae τ = X 1 X 2 X 3 has equatio X = 0. The cubic that reaches the poits at the required parameters has the orm h() = ( µ2 µ 1 µ 1, µ 2 0 0, µ 2 1 1, µ 2 2 2 as ca be easily checked. To determie the homography H τ we eed to kow the image o a ourth poit. Note that the twisted cubic ca be reparametrizated by meas o a homographic chage o coordiate = a + b c, ad bc 0 + d without aectig the cross ratios o the parameters. So we ca assume, reparametrizatig the curve i ecessary, that oe o the parameters λ i, µ j coicides with. The we ca cosider the poit Q = h( ) = (µ 2 µ 1, µ 2 0, µ 2 1, µ 2 2 ). Sice H τ (C 1 Q τ) = C 2 Q τ, the poit C 1 Q τ = (0, µ 2 0, µ 2 1, µ 2 2 ) maps to the poit o coordiates (0, µ 1 0, µ 1 1, µ 1 2 ). Usig ow (Y, Z, T) as coordiates or the plae τ, the homography H τ maps (1, 0, 0) (1, 0, 0) (0, 1, 0) (0, 1, 0) (0, 0, 1) (0, 0, 1) ). (µ 2 0, µ 2 1, µ 2 2 ) (µ 1 0, µ 1 1, µ 1 2 )

C 2 C1 6 JOSÉ IGNACIO RONDA, ANTONIO VALDÉS, AND FERNANDO JAUREGUIZAR ad so it is give by a matrix o the orm (up to multiples) µ 1 0 µ 2 0 0 0 µ H τ 0 1 1 µ 2 1 0 0 0 µ 1 2 µ 2 2 so λ 0 : λ 1 : λ 2 = µ1 0 µ 2 0 : µ1 1 µ 2 1 : µ1 2 µ 2 2 rom which the result ollows. Sice we have see that i the particular case o a horopter attached to a pair o cameras with idetical itrisic parameters the optical ceters are reached by the horopter at = 0,, we obtai that the eigevalues λ 0, λ 1 ad λ 2 o the iter-image homography iduced by the plae at iiity are i the same ratio that the parameters 0, 1 ad 2 at which the horopter reaches the plae at iiity, i.e., λ 0 : λ 1 : λ 2 = 0 : 1 : 2. This equality, together with Theorem 3.1.3 implies the uimodular costrait itroduced by Polleeys et al. []. Compare with [15]. Next Theorem shows how the parameters o these poits are related with the eigevalues o H τ. Note that this result, together with Theorem 3.1.3 implies the uimodular costrait 3.4. Coes ad horopters. There is a 2-parameter amily o quadrics cotaiig a give twisted cubic (see [16]), ad this relates the horopter with the ambiguity pheomeo (see the iterestig book o Maybak [10] or more iormatio o this subject). For each poit P o the twisted cubic, the set o rays joiig P with all the other poits o the cubic is a coe, icludig the taget lie at P as a limit case. To see this, it is eough to check that the curve obtaied itersectig this surace with a plae ot icludig P is a coic. Accordig to Bezout s Theorem it is eough to see that a geeric lie o the plae itersects the curve i just two poits. To id them is equivalet to idig the itersectios, dieret rom P, o the plae determied by the lie ad P with the twisted cubic. These itersectio are, i geeral, three poits, the vertex P ad two other oes. This meas that the curve is a coic ad thereore that the ruled surace is a coe (c. [16]). I the particular case o a horopter curve, we ca cosider the coe with vertex the real poit at iiity o the horopter, Q 0, which is a cylider. To see what type o coic is the base o this cylider, let us cosider a plae orthogoal to the axis o the cylider, i.e., a plae itersectig the plae at iiity i the polar lie o Q 0 with respect the absolute coic. By Theorem 3.1, this plae cotais the complex cojugate poits o the horopter at iiity, Q 1 ad Q 2, the cyclic poits at iiity o the plae. Thereore the coic is a circle (see igure 2). We have proved the ollowig Theorem (this result is well kow ad requetly metioed i the literature, e.g., [14]). Theorem 3.3. The horopter curve is cotaied withi a circular cylider ormed by all the lies touchig the horopter with the same directio that the screw axis o the motio., Figure 2. A horopter i its cylider showig the screw axis o the motio.

CAMERA AUTOCALIBRATION AND HOROPTER CURVES 7 For the sake o completeess, we iclude the ollowig property. Let us costruct a coe cotaiig the horopter takig as vertex oe o the optical ceters. The curve obtaied itersectig the coe with the correspodig retial plae is a coic. It is ot diicult to id the equatio o this coic: I the poits q ad q are the images i the irst ad secod cameras, respectively, o the same poit i space, the their coordiates satisy the relatio q T Fq = 0, where F is the udametal matrix o the pair. I q ad q are the images o a poit o the horopter h(), the q ad q have the same coordiates. We abuse otatio ad write q = q. Thereore q T Fq = 0 I we decompose F = F S + F A as a sum o its symmetric part F S = (F + F T )/2 ad its atisymmetric part F A = (F F T )/2, we have that q T Fq = q T F S q = 0, i.e., q lies over a coic o matrix F S. Note that although F is a sigular matrix, its atisymmetric part is ot sigular, i geeral. So we have proved the ollowig Theorem, which ca also be oud i [7]). Theorem 3.4. The projectio o the horopter o each retia o the camera pair is a coic with matrix the symmetric part o the udametal matrix o the pair. 3.5. Other properties o the horopters. Give a plae π Star(C 1 ) let us cosider its image π = H(π) Star(C 2 ), which we suppose to be dieret rom π. Let us cosider the lie l = π π, which we suppose dieret rom C 1 C 2. The H iduces a homography o l, sedig each P l to the poit P = H(C 1 P) l. Ay homography o a lie has either two dieret ixed poits, or oe double ixed poit. Note that the ixed poits belog to the associated twisted cubic, so l is a chord o this cubic. I we have a double ixed poit o l the l is taget to the twisted cubic at that poit. I the particular case o the horopter curve, let l be the screw axis o the motio ad π the plae determied by C 1 ad l. Beig the screw axis ivariat uder the motio, the image o π is the plae π determied by C 2 ad l. Hece the screw axis l = π π is a chord o the horopter. But, sice the motio acts o l as a traslatio (assumig that the motio is ot a pure rotatio), the oly ixed poit o the homography iduced o l is the poit at iiity, which is double. So we have the ollowig Theorem: Theorem 3.5. I the motio betwee the cameras is ot a pure traslatio, the the taget to the horopter at its real poit at iiity is the screw axis o the motio. Proo. We have already cosidered the case o a geeric motio. I the motio is a pure rotatio, the horopter actorices i a lie (the screw axis) ad a circle, ad so the Theorem holds true. Let us ow study the eect o a coordiate chage o the horopter curve. So we have ew projective coordiates Q = ( X, Ȳ, Z, T) T related with the ormer oes by a o-sigular matrix A as ollows: Q = GQ. The we will have ew camera matrices P i = ν i P i G 1, ad the horopter h = h( ) is deied by 0 = ( P 1 P 2 ) h( ) = (ν 1 P 1 ν 2 P2 )G 1 h( ), thereore ) ν2 h( ) = Gh(. ν 1 We see rom this that or a give coordiate system ad a order o the cameras all the possible parameterizatios o the correspodig horopter are related by a coordiate chage o the orm = k, k 0. I ay o these parameterizatios the optical ceter o the irst camera is reached at = 0 ad that o the secod camera is reached at =. I there is o privileged order o cameras, we must also cosider chages o parameter o the orm = k/, as it ollows rom equatio [1]. So we have the ollowig Theorem: Theorem 3.6. Uder a chage o projective coordiates Q = GQ, the correspodig horopter h = h( ) is related to h = h() by h( ) = Gh ( λ ) ( ) or h( ) = Gh λ accordig to whether we iterchage or ot the order o the cameras ad where λ is a costat which depeds o the choice o the represetatives o the matrices o the cameras. The parameters 0 ad always correspod to the optical ceters o the cameras. 4. Camera sel-calibratio usig horopter curves I this sectio we propose two dieret algorithms based o the properties o the horopters itroduced above. The irst algorithm cosists i a stratiied calibratio which irst provides the plae at iiity ad the the absolute coic. Its irst phase (aie calibratio) ca be see as a implemetatio o the uimodular costrait algorithm o [] i terms o horopter curves. The secod phase (Euclidea calibratio) provides the absolute coic by

8 JOSÉ IGNACIO RONDA, ANTONIO VALDÉS, AND FERNANDO JAUREGUIZAR estimatig liearly the best-it coic with respect to the itersectios o the horopters with the plae at iiity (relatios give i Theorem 3.1). The secod oe gives directly the Euclidea structure o space. While the irst algorithm is computatioally simpler, the secod oe is more robust i the presece o oise, as we will see i the ext sectio. Both algorithms use geeral purpose optimizatio techiques (a Leveberg-Marquardt algorithm as i [13]). 4.1. Algorithm 1: Horopters ad the uimodular costrait. This algorithm operates i two phases. I the irst oe the plae at iiity is obtaied by meas o a optimizatio process i which the target uctio is based o the uimodular costrait. The secod phase provides the projected absolute coic i a direct ashio usig the relatioships give i Remark 3.1. The cost uctio employed or the aie calibratio assigs to a cadidate plae o coordiates u = (u 0, u 1, u 2, u 3 ) the quatity 2 ( ij k 2, c 1 (u) = 1) i,j=1 i<j k,l=1 k l where ij 0, ij 1, ij 2 are the parameters o the poits o itersectio o the plae u with the horopter associated to the camera pair (i, j). This cost uctio has bee desiged to be ivariat uder permutatios ad scalig o the parameters. Note that with a adequate parameterizatio o the space o plaes, this leads to a three-dimesioal search. I practice, this will be achieved by selectig oe o the u i to be oe. It would also be possible to carry out just a two-dimesioal search usig as parameters the modulus ad the phase o the ij k or a give horopter hij. However, umerical istability has bee observed, which discourages this approach. Besides, three-dimesioal search turs out to perorm quite eicietly, as will be explaied i the ollowig sectio. Note that the computatioal cost o oe evaluatio o c 1 is essetially that o idig the roots o a third degree polyomial equatio. For the startig poit o the optimizatio process, a irst guess o the plae at iiity is obtaied by meas o a liear algorithm which provides a exact solutio i the cameras are orthogoal, i.e., i K is a diagoal matrix (see [17]). The good covergece properties o the algorithm allow to employ a startig poit obtaied by this method eve i the camera is very ar rom veriyig this coditio, as will be see i the sectio o results. Oce the plae at iiity has bee oud, dieret liear methods ca be employed to calculate the projectio o the absolute coic (see e.g. []). The properties o the horopters provide ew liear methods or this task. A irst approach would be to estimate the absolute coic as the best-it coic i the plae at iiity with respect to the icidece ad polarity relatios give by Theorem 3.1. This would require the deiitio o a suitable projective reerece i this plae. To avoid this, we take advatage o the costrutio i Remark 3.1 by projectig the itersectio poits oto the virtual plae Π ad calculatig the best-it coic with respect to equatios (2). The obtaied coic is the projectio o the absolute coic, whose matrix is give by A = (KK T ) 1, so that the itrisic parameter matrix ca be obtaied rom it by meas o Cholesky actorizatio. 4.2. Algorithm 2: Horopters ad absolute coic. This algorithm is a alterative to the previous oe i which both aie ad Euclidea calibratio are perormed at the same time by meas o a optimizatio process. The ew target uctio measures to what extet a cadidate plae veriies equatios (2). For the sake o clarity we deie this uctio i two steps. First, we associate to a give plae o coordiates u = (u 0, u 1, u 2, u 3 ) T ad a symmetric matrix A the umber (u, A) give by (u, A) = { ( ) T r ij 2 2 ( ) T 0l Ar ij kl + r ij } 2 kl Ar ij kl i<j l=1 with otatios as i Remark 3.1 ad takig ormalized represetatives o the r ij kl. This is a quadratic orm i the coeiciets o A, so with respect to these coeiciets stadard miimizatio ca be applied: We write ij l (u, A) = a T Ma where M is a 6 6 matrix whose etries deped o the r ij kl ad at = (a 1,..., a 6 ) where A = a 1 a 2 a 3 a 2 a 4 a 5. a 3 a 5 a 6 The we deie the cost o the plae u as k=1 c(u) = mi (u, A) a =1

CAMERA AUTOCALIBRATION AND HOROPTER CURVES 9 where a stads or the Euclidea orm o the vector. The c(u) is give by the miimum eigevalue o M ad the vector a is the correspodig eigevector. I u is the true plae at iiity, this eigevalue must be zero, sice the best-it coic meets exactly all the costraits. Although there are more sophisticated methods to id the best-it coic, we will see i the ollowig sectio that this liear techique suices to provide satisactory results. The risk that the optimizatio process leads to a spurious miimum motivates the itroductio o additioal terms i the cost uctio. As the matrix o the projected absolute coic is deiite, pealty terms have bee icluded i the target uctio to icrease the cost o a cadidate plae leadig to a o-deiite matrix. The ially selected target uctio has the orm c 2 (u) = c(u)g 1 (A u ) + αg 2 (A u ) where A u is the ormalized symmetric matrix that miimizes (u, A) or the plae u, g 1 (A) is the pealty or closeess to sigularity, g 2 (A) is that or o-deiiteess ad α is a weightig actor. We deie g 1 (A) = 1 + 1/λ mi p g 2 (A) = sig(λ i) sig(λ max) where λ mi (resp. λ max ) is the eigevalue o A o miimum (resp. maximum) absolute value, p is a suitable power ad (λ i ) is the sequece o eigevalues o A. I our experimets we have take α = 10, p = 6. The iitializatio o the algorithm is the same as that o the previous oe, ad it is also robust with respect to the startig poit. However, as we ext see, it is more robust with respect to oise. λ i 5. Experimetal results To evaluate the perormace o the techiques a sceario has bee simulated i which a set o three cameras capture a scee cosistig i a set o 100 radomly positioed 3D poits. The Euclidea coordiates o these poits are obtaied rom a uiorm distributio with support a cetered cube o side two uits. The three cameras are located at radom with pricipal axes passig close to the ceter o the poit distributio. More speciically, the irst camera has optical ceter C 1 = t 1 = (0, 0, d) T ad projects oto plae z = h, so that its projectio matrix i stadard orm ca be writte as P 1 = K(I t 1 ). The other cameras are radomly rotated ad traslated versios o the irst oe, with projectio matrices P i = K(R i t i ), where the R i are radom rotatio matrices ragig over the whole rotatio group, ad t i = R i t 1 + t i, with the t i radom vectors with idepedet compoets uiormly distributed withi [ s, s]. This arragemet iteds to model a set o cameras approximately poitig toward the ceter o coordiates, ad located all o them at a similar distace o this ceter. We project the 3D poits ad perturb the resultig aie coordiates with oise o idepedet compoets uiormly distributed withi [, ]. The values o the itrisic parameters have bee take so that the projectios lie roughly withi the rage that would be typical or values measured i pixels o images obtaied with a video camera. Projective calibratio is irst perormed by camera pairs usig the algorithm described i [5]. The estimatio o the udametal matrix is computed by meas o the elemetary eight-poit algorithm [7]. The ive o the observed 3D poits are selected at radom to establish a commo projective basis. We parameterize the space o plaes just by settig oe o the compoets o the plae coordiates u = (u 0, u 1, u 2, u 3 ) T to be oe. I practice, a very slight improvemet is observed i the our possible choices are employed ad the result o miimum cost is selected, which is what we do i our experimets. Although the geometry o the algorithms is idepedet o scalig, its umerical procedures are ot: For example, the extractio o eigevalues is scalig-depedet. I practice, scalig o the data has proved to be very sigiicat i the perormace o the algorithms. I our case, a homothety (u, v) (λu, λv) is applied to the o-homogeeous aie observed oisy coordiates, ad its eects are corrected i the output o the algorithm. A rule o thumb or the value o λ is to take it as the iverse o the maximum observed coordiate. For a give selectio o itrisic parameters ad oise level, a set o 300 dieret experimets is cosidered, each with a dieret set o camera positios ad 3D poit coordiates. With high oise levels the algorithms ail occasioally to provide a deiite matrix or the estimated absolute coic. This limitatio is easily overcome by reprocessig the data with a dieret choice or the projective basis. As each optimizatio takes a ew secods or the secod algorithm (o a Itel Petium 4 machie ruig uder Liux at 1500 MHz), which is the most costly, this redudacy is perectly aordable. I our experimets each processig is actually perormed with dieret basis util three dieret successes are obtaied (with a maximum o te attempts), ad the miimum cost solutio is selected.

10 JOSÉ IGNACIO RONDA, ANTONIO VALDÉS, AND FERNANDO JAUREGUIZAR Figure 3 show the average relative error curves or the irst (igure 3.a) ad secod (igure 3.b) algorithms together with the correspodig curves ad or two itermediate algorithms resultig rom two dieret combiatios o the cost uctios β 1 c 1 + β 2 c 2 (igure 3.c ad 3.d). We cosider the case o three cameras with parameters idicated i the captio. Figure 4 shows aalogous results or a dieret set o itrisic parameters. These results clearly show that the icrease i computatioal cost o algorithm 2 is justiied by a improvemet i perormace. Besides, it is see that a combiatio o both cost uctios does ot ecessarily imply a additioal gai. Nacho: tiees que meter ms iguras Reereces [1] M. Armstrog, A. Zisserma, R. Hartley, Sel-calibratio rom image triplets,, I Europea Coerece o Computer Visio, pages 3 16. 1996 [2] S. Maybak ad O. Faugeras, A theory o sel calibratio o a movig camera. I Iteratioal Joural o Computer Visio, 8 (2), pages 3 141. 1992 [3] O. Faugeras, Three-Dimesioal Computer Visio: A Geometric Viewpoit, MIT Press, Cambridge, Massachusetts. 1993 [4] A. Fusiello, Ucalibrated Euclidea recostructio: a review. I Image ad Video Computig, 18, pages 555-563, 2000. [5] R.I. Hartley, Projective recostructio ad ivariats rom multiple images, I IEEE Tras. o PAMI, vol. 16, o. 10, Oct. 1994. [6] R. I. Hartley, Chirality, Iteratioal Joural o Computer Visio, 26 (1), pp. 41-46, 1998. [7] R.I. Hartley ad A. Zisserma, Multiple View Geometry i Computer Visio. Cambridge Uiversity Press, Cambridge, UK, 2000. [8] R.I. Hartley, E. Hayma, L. de Agapito ad I. Reid, Camera calibratio ad the search or iiity. I Proceedigs o the Iteratioal Coerece o Computer Visio, 1999. [9] S.J. Maybak, The Projective Geometry o Ambiguous Suraces. I Philosophical Trasactios: Physical Scieces ad Egieerig, 332 (1623), pages 1 47, 1990. [10] S.J. Maybak, Recostructio rom projectios, Spriger-Verlag, Berli, 1993. [11] D. Nister, Calibratio with robust use o cheirality by quasi-aie recostructio o the set o camera projectio cetres, I Proceedigs 8th IEEE Iteratioal Coerece o Computer Visio. ICCV 2001, 2001, pp. 116-23, vol.2. [] M. Polleeys, L. Va Gool, Stratiied sel-calibratio with the modulus costrait, I IEEE Tras. o Patter Aalysis ad Machie Itelligece, 21 (6), pages 707-724. 1999. [13] W.H. Press et al., Numerical Recipes i C++: The Art o Scietiic Computig, 2d. ed.. Cambridge Uiversity Press, Cambridge, MA. 2002. [14] L. Qua ad Z. La, Liear N-Poit Camera Pose Determiatio. I IEEE Trasactios o Patter Aalysis ad Machie Itelligece, vol. 21, o. 8, pages 774 780, 1999. [15] F. Schaalitzky, Direct Solutio o Modulus Costraits. I Proc. Idia Coerece o Computer Visio, Graphics ad Image Processig, Bagalore. 2000. [16] J.G. Semple ad G.T. Keeboe, Algebraic Projective Geometry, Oxord Classic Texts i Physical Scieces, Oxord Uiversity Press, Oxord, 1998. [17] Y. Seo, A. Heyde, Auto-calibratio rom the orthogoality costraits, I Iteratioal Coerece o Computer Visio, Barceloa, Spai, 2000. [18] W. Triggs, Auto-calibratio ad the absolute quadric. I Proc. IEEE Coerece o Computer Visio ad Patter Recogitio, pages 609 614. 1997.

CAMERA AUTOCALIBRATION AND HOROPTER CURVES 11 2 2 1 1 1 1 (a) (b) 2 2 1 1 1 1 (c) (d) Figure 3. Average results or / u = 250, v / u =, = 0.8π/2, u 0 = 80, v 0 = 80, d = 1.6, s = 0.1, ad oise amplitud rom 0 to 3 pixels. I (c) β 1 = 1, β 2 = 10 2. I (d) β 1 = 1, β 2 = 10 5. 2 2 1 1 1 1 (a) (b) 2 2 1 1 1 1 (c) (d) Figure 4. Average results or / u = 1000, v / u = 1, = π/2, u 0 = 250, v 0 = 250, other parameters as i previous igure.