Matchng Images wth Dfferent Resolutons Yves Dufournaud, Cordela Schmd, Radu Horaud To cte ths verson: Yves Dufournaud, Cordela Schmd, Radu Horaud. Matchng Images wth Dfferent Resolutons. Internatonal Conference on Computer son & Pattern Recognton (CPR 00), Jun 000, Hlton Head Island, Unted States. IEEE Computer Socety,, pp., 000, <http://eeexplore.eee.org/xpls/abs all.jsp?arnumber=>. <0.09/CPR.000.>. <nra-009> HAL Id: nra-009 https://hal.nra.fr/nra-009 Submtted on 0 Dec 00 HAL s a mult-dscplnary open access archve for the depost and dssemnaton of scentfc research documents, whether they are publshed or not. The documents may come from teachng and research nsttutons n France or abroad, or from publc or prvate research centers. L archve ouverte plurdscplnare HAL, est destnée au dépôt et à la dffuson de documents scentfques de nveau recherche, publés ou non, émanant des établssements d ensegnement et de recherche franças ou étrangers, des laboratores publcs ou prvés.
Matchng Images wth Dfferent Resolutons Yves Dufournaud Cordela Schmd Radu Horaud INRIA RHÔNE-ALPES & GRAIR-CNRS av. de l Europe 0 Montbonnot, France Yves.Dufournaud@nralpes.fr Abstract In ths paper we address the problem of matchng two mages wth two dfferent resolutons: a hgh-resoluton mage and a low-resoluton one. On the premse that changes n resoluton act as a smoothng equvalent to changes n scale, a scale-space representaton of the hgh-resoluton mage s produced. Hence the one-to-one classcal mage matchng paradgm becomes one-to-many because the lowresoluton mage s compared wth all the scale-space representatons of the hgh-resoluton one. Key to the success of such a process s the proper representaton of the features to be matched n scale-space. We show how to extract nterest ponts at varable scales and we devse a method allowng the comparson of two mages at two dfferent resolutons. The method comprses the use of photometrc- and rotatonnvarant descrptors, a geometrc model mappng the hghresoluton mage onto a low-resoluton mage regon, and an mage matchng strategy based on the robust estmaton of ths geometrc model. Extensve experments show that our matchng method can be used for scale changes up to a factor. Introducton The problem of matchng two mages has been an actve topc of research n computer vson for the last two decades. The vast majorty of exstng methods consder two vews of the same scene where the vewponts dffer by small offsets n poston, orentaton and vewng parameters such as focal length. Under such condtons, the mages assocated wth the two vews have comparatve reso- Y. Dufournaud acknowledges support from Aerospatale. Fgure. An example of matchng a lowresoluton mage wth a hgh-resoluton one. lutons and hence they encapsulate scene features at approxmatvely the same scale. In ths paper we address a somehow dfferent problem that has only lttle been addressed n the past. We consder for matchng two mages wth very dfferent resolutons. More precsely, f we denote by the approxmate dstance from an observed scene object to a vewpont and by the focal length assocated wth the vewng parameters, the mage resoluton may be defned as or more generally as a functon of. Therefore we are nterested n developng a matchng technque whch takes as nput a low resoluton mage, mage #, and a hgh resoluton mage, mage #, such that ther assocated resolutons and satsfy the constrant. In practce t wll be shown that, usng the approach advocated below, t s possble to match two mages such that. As an example we consder the mage par n Fgure. Both mages were taken wth a camera placed at klometers (.9 mles) away from the top of the mountan. For the frst mage (left) we used a focal length equal to mm whle for the second one (rght) we used a focal length equal to mm. Notce that the hgh-resoluton mage corresponds to a small regon of the low-resoluton one and t
s qute dffcult to fnd the exact poston and sze of ths regon. Clearly, a scene object and/or texture may appear at dfferent szes and postons n the two mages. Therefore, the search space assocated wth the featureto-feature matchng of two such mages s larger and more complex than the one assocated wth the classcal stereo matchng paradgm. The classcal approach to mage matchng extracts nterestng pont-features from each mage, matches them based on cross-correlaton, computes the eppolar geometry through the robust estmaton of the fundamental matrx, and establshes many other matches once ths matrx s known. For a number of reasons, ths method cannot be appled anymore:. Pont-feature extracton and matchng are resoluton dependent processes.. The hgh-resoluton mage corresponds to a small regon of the low-resoluton one and hence the latter contans many features whch do not have a match n the former.. It may be dffcult to estmate the eppolar geometry because there s not enough depth assocated wth the hgh resoluton mage. The soluton suggested n ths paper conssts of consderng a scale-space representaton of the hgh-resoluton mage and of matchng the low-resoluton mage aganst the scale-space descrpton of the hgh-resoluton one. A scalespace representaton may be obtaned by smoothng an mage wth Gaussan kernels of ncreasng standard devatons. Therefore, the hgh-resoluton mage wll be descrbed by a dscrete set of mages at varous scales. On the premse that decreasng the resoluton can be modeled as a smoothng equvalent to a scale change, the one-to-one mage matchng problem at hand becomes a one-to-many mage matchng problem. In ths paper we descrbe such a matchng method. Key to ts success are the followng features: The scale-space representaton of nterest ponts together wth ther assocated descrptors. A geometrc model descrbng the mappng from the hgh-resoluton mage to the low-resoluton one. An mage-matchng strategy whch combnes pont-topont assgnments wth a robust estmaton of the geometrc mappng. Several authors addressed the problem of matchng two mages gathered from two very dfferent ponts of vew [,, 9] but they dd not consder a change n resoluton. The use of scale-space n conjuncton wth stereo matchng has been restrcted to herarchcal matchng: correspondences obtaned at low resoluton constran the search space at hgher resolutons [,, 0]. Scale-space propertes are thoroughly studed n [] and the same author attempted to characterze the best scale at whch an mage feature should be represented []. A smlar dea s presented n [] to detect stable ponts n scale space. Our work s closely related wth [] whch attempts to match two mages of the same object gathered wth two dfferent zoom settngs. Pont-to-pont correspondences are characterzed n scale space by correlaton traces. The method s able to recover the scale factor for whch two mage ponts are the most smlar but t cannot deal wth camera rotatons. Image descrptors that are nvarant wth respect to local affne greyvalue changes, mage rotatons, and mage translatons were studed theoretcal n [9] and an effcent mplementaton was proposed n []. These descrptors are based on convolutons wth Gaussan kernels and ther dervatves. They are therefore consstent wth scale-space representatons. They are best appled to nterest ponts and a recent study showed that the Harrs corner detector s the most relable one []. However, they are not scalenvarant and, n spte of good theoretcal models for such nvarants [, ], t s more judcous from a practcal pont of vew to compute local descrptors at varous scales n a dscrete scale-space []. Paper organzaton. The remander of ths paper s organzed as follows. Secton brefly outlnes the geometrc model assocated wth the mage par. Secton suggests a framework for adaptng the detecton of nterest ponts to scale changes. Secton descrbes the hgh-resoluton to low-resoluton matchng and secton presents results. Geometrc modelng One of the key observatons enablng the matchng of two mages at two dfferent resolutons s that the hghresoluton mage corresponds to a small regon of the lowresoluton one. Hence, one reasonable assumpton s to consder that the mappng between the hgh resoluton mage and the correspondng low-resoluton regon s a plane projectve transformaton,.e., the scene correspondng to ths regon s planar. Such a homography may well be represented by a homogeneous full rank matrx H. Let be a pont n the frst mage (low resoluton) and be a pont n the second mage (hgh resoluton). One can characterze a regon n the low-resoluton mage such that the
GF >. = > : $ GF <; > $ $ > m $ r m ponts wthn ths regon verfy: H () Smlarly, ponts outsde ths regon, say do not verfy ths equaton. In the general case t s qute tedous to fnd a parameterzaton of H. Moreover, mage descrptors whch are nvarant to such a general plane-to-plane projectve transformaton are dffcult to compute and therefore t s dffcult to properly select potental canddate ponts satsfyng eq. (). We can further smplfy the geometrc model and consder a restrcted class of homographes, namely a rotaton about the optcal axs, a translaton, and a scale factor:! "#%$'&)(+*-,.'$/*0!, $'*0, $'&)(*, 9 9 : () Notce that the projectve equalty n eq. () s replaced by an equalty and two pont-to-pont correspondences are suffcent to lnearly estmate such a smlarty transformaton. In practce t wll be useful to replace the -vectors and used above by -vectors = and =?> such that: A@BDCE: IH =:J and K L@BMCE > Wth ths notaton, eq. () becomes =?>Q NH =O> :PJ R=SRUT where R s the rotaton matrx and T s the translaton assocated wth the mage transformaton. Ideally, one would lke to characterze mage ponts by descrptors nvarant to mage rotaton, translaton and scale. Unfortunately, as already outlned, scale-nvarant mage descrptors are hard to compute n practce. Therefore, the matchng strategy wll buld a dscrete scale space on top of the hgh-resoluton mage thus by-passng the scale-nvarance problem. The mage matchng problem at hand becomes the problem of () extractng sets of ponts from the two mages, =W YX[Z\Z[Z)X=?]_^ and =?> X\Z[Z\Z[X=`>ab^, () properly characterzng these ponts such that pont-to-pont correspondences are allowed, and () determnng the largest set of such correspondences compatble wth a homography between the hgh-resoluton mage and a low-resoluton regon. Scale-space nterest pont detecton In order to match two mages one has to defne a measure of smlarty. One possble defnton s correlaton. In our case, ths can be wrtten as: c dfe+gh $ R =. d ^^. d ^kj where l s $ a wndow around =. Therefore, one must fnd a scale factor and a rotaton matrx R for whch the expresson above s mnmzed. The search space assocated wth such a technque s very large and the assocated non-lnear mnmzaton procedure has problems. Alternatvely, one may use nterest ponts whch are detected by a rotaton-nvarant operator and characterze these ponts by rotaton-nvarant descrptors. Such an nterest pont detector was proposed n []. More precsely, consder an mage pont = and the assocated mage greyvalue =W^. Interest ponts are detected by:. Compute the mage dervatves n the C and E drectons,, and \m. These computatons are carred out by convoluton wth the dfferental of a Gaussan kernel of standard devaton n.. Form the auto-correlaton matrx. Ths matrx C =oxpnqxr ns^ averages dervatves n a wndow around a ns^ s used for weghtng : pont =. A Gaussan t C =oxnqxr ns^o ut ns^vxw r =oxpny^ =oxny^ m =oxny^ [m =oxpny^{z (). = s an nterest pont f the matrx C has two sgnfcant egenvalues, that s f the determnant and trace of ths matrx verfy: -}[~ C^.U trace C^f () where s a fxed threshold and a parameter. Notce that the nterest pont detector defned above s rotaton-nvarant ths s due to the symmetry of matrx C. However, $ IT IS NOT nvarant to a change n the mage sze or mage resoluton. Wthout loss of generalty we can therefore omt the mage-plane rotaton at nterest ponts detected by the operator descrbed above. Under the assumpton that the greyvalues are properly normalzed, the smlarty condton that must be satsfed s > =O>^o =W^ where, as before, s the hgh-resoluton mage and > s the low resoluton one. Snce the rotaton s omtted we have =O>s =ƒr T. Takng the dervatves of the above expresson $ wth respect to the mage coordnates and E, we obtan and C $. m Therefore, the relatonshp between the nterest pont detector appled to the hgh-resoluton mage and the nterest pont detector appled to the low-resoluton mage s: C> = > X nqx ny^? r : $ C =oxnqxr ns^
: t m : m : We consder now the scale-space assocated wth the hgh resoluton mage. The scale-space s obtaned by convolvng the ntal mage wth a Gaussan kernel who s standard devaton s ncreasng monotoncally, say n wth. At some scale n ths space the hgh resoluton mage s gven by: =oxp ny^o =W^ v t =oxˆ [ny^ At ths scale, the mage s frst order dervatves wrte: =oxp ny^? =W^v t =oxˆ [ny^ =oxp ny^? =W^v t =oxˆ [ny^ m Therefore, one can detect nterest ponts at any scale by smply replacng n wth n n eqs. () and (). If the task conssts of matchng the hgh-resoluton mage wth the low-resoluton one >, t s crucal to select the scale of at whch ths matchng has $ to be performed. The scale $ must absorb the sze rato, therefore one may wrte. The nterest pont detector at scale s defned by: C =oxp nqxˆ r nš^o r ns^ v w =oxp ny^ =oxp ny^ m =oxˆ ny^ m =oxp ny^ z () In order to llustrate the results obtaned wth ths scalespace nterest pont detector, we appled t to the hghresoluton mage of Fgure (rght) at scales,.e.,, and. Fgure shows these results where n and nb uœ r. s= s= Robust mage matchng s= The scale-space extracton and representaton of nterest ponts that are rotaton-nvarant wll enable us to devse the one-to-many mage matchng technque descrbed below. The man dea s to compare the low-resoluton mage at one scale wth the hgh-resoluton mage at many scales. Hence, the scale at whch ths matchng process provdes the best results, provdes the correct one-to-one assgnments between nterest ponts. Because the matchng s supported by the robust estmaton of a homography between the two mages, the estmated parameters wll provde among others the resoluton rato between the two mages. Wthout loss of generalty, we assume that the hghresoluton mage, mage #, s represented at dfferent scales n, n,..., n wth n. At each scale, nterest ponts are extracted usng eq. (). Furthermore, a number of dfferental nvarants are extracted at each scale as well. These descrptors are photometrc-, mage rotaton-, and mage translaton-nvarant. Lkewse, nterest ponts and ther descrptors are computed and assocated wth the low-resoluton mage, mage #, at only one scale, n. Fgure. Interest ponts detected at scales. s=
We then consder one-by-one the scale-space representatons of mage # and attempt to fnd whch one of these mages best matches a regon n mage #. Snce there s a strong relatonshp between scale and resoluton, one may assume that the scale of the best match corresponds to the resoluton rato between mages # and #. s= At each scale one-to-one correspondences are determned by drect comparson of the descrptors assocated wth the nterest ponts. In practce there are such descrptors lmted to thrd-order dervatves of the ntensty sgnal. These descrptors are nvarant to mage rotaton as well as local affne changes of llumnaton. Two such -vector descrptors are compared usng the Mahalanobs dstance. Ths dstance requres the covarance matrx assocated wth each descrptor. Ths matrx encapsulates sgnal nose, varatons n photometry, naccuracy of nterest pont locaton, and so forth. s estmated statstcally over a large set of mage samples. In order to solve as many ambgutes as possble, each one-to-one assgnment thus establshed s checked for local coherence. Namely, for each one of the two ponts n an assgnment we consder ther neghbors and check whether the two groups of ponts n the two neghborhoods are mutually compatble. Ths local compatblty check based on local geometrc dstrbuton has a cost [] but t s worth the effort because t allows to elmnate spurous matches. The pont matchng process just descrbed s appled at scales. Next, we have to evaluate the qualty of mage-tomage matchngs based on these pont matches n order to select the scale assocated wth the best match. We therefore estmate a mappng between the two mages as defned by eq. () and use robust statstcs [, ]. Once an approxmate scale has been selected usng the strategy just descrbed, a robust estmator takes as nput the potental one-to-one pont assgnments, computes the best homography between the two mages, and splts the pont assgnments nto two sets: () nlers,.e. ponts lyng n the small regon correspondng to the homography mappng of the hgh resoluton mage onto the low resoluton one and () outlers,.e. ponts that are ether outsde ths regon or msmatched ponts nsde the regon. Commonly used robust estmators nclude M-estmators, least-medan-squares (LMedS), and RANdom SAmple Consensus (RANSAC). In our case, the number of outlers may be qute large. Ths occurs n partcular when the two mages have very dfferent resolutons and hence only 0% or less of the low-resoluton mage corresponds to the hgh resoluton one. Therefore, we ruled out M-estmators because they tolerate only a few outlers. Among the two remanng technques, we preferred RANSAC because t allows the user to defne n advance the number of potental outlers through the selecton of a threshold. Hence, ths 0 9 0 9 0 9 0 9 0 9 0 9 9 9 0 0 s= 9 0 9 0 9 0 9 s= 0 s= Fgure. Pont-to-pont assgnments obtaned at four scales. threshold can be chosen as a functon of the scale factor. Detals concernng threshold selecton can be found n []. Experments The matchng strategy just descrbed was appled and tested over a large number of mage pars where the resoluton factor between the two mages vared from to. Here we present three examples. The fnal result of applyng the matchng to the par of Fgure s shown n Fgure. Let us explan n detal how ths type of result s obtaned for another example, e.g. Table and Fgures and. Interest ponts are frst extracted from the low-resoluton mage at one scale ( ) and from the hgh-resoluton mage at dfferent scales ( to ). Therefore, eght mage matchngs are performed. The result of pont-to-pont matchng s shown on Fgure at four dfferent scales:,,, and. Obvously scale and scale have the best matches assocated wth them and scale s a better canddate. Therefore, :
90 9 0 Fgure. The hgh-resoluton mage s mapped onto the low-resoluton one usng the homography consstent wth pont-to-pont assgnments. Resoluton factor No. of ponts No. of matches Predcted Computed Intal guess Inlers Outlers (%). 9 - - 0. 9 %. 90 % 0 % 0 % 9 %. - - 0. - - Table. Ths table shows, at each scale, the computed resoluton factor, the number of ponts n the hgh-resoluton mage, the number of potental matches, the fnal number of matches, and the percentage of outlers. Notce that scales and yeld very smlar results. Conclusons In ths paper we presented a new method for matchng two mages wth two dfferent resolutons. We showed that t s enough to represent the hgh-resoluton mage n scalespace and we descrbed a one-to-many robust mage matchng strategy. Key to the success of ths method s the scalespace representaton of nterest ponts. In spte of a huge number of publcatons n the magematchng doman, t seems to us that none of the exstng methods s able to deal wth large changes n resoluton. Here we have been able to match mages wth a resoluton factor of. In practce the mages shown n ths paper were gathered by varyng the focal length usng the zoom-lens of a dgtal camcorder. The advent of dgtal photography opens new felds of applcatons and we beleve that our matchng technque wll allow the smultaneous explotaton of multple vewponts and varable resoluton. t would have been suffcent to run the robust matchng algorthm at scale only. In practce we run the latter algorthm at all the scales and dsplayed the results n Table. Thus we can verfy that the best match s, ndeed, obtaned at _. Out of ponts detected at ths scale, of them have a potental assgnment n the low-resoluton mage and of them are fnally selected by the robust matchng technque. The latter rejected 0% of the matches. Fnally the homography thus obtaned was appled to the hgh resoluton mage and ths mage s reproduced on top of the lowresoluton one (cf. Fgure ). A thrd example s dsplayed on Fgure. References [] G. Csurka, D. Demrdjan, and R. Horaud. Fndng the collneaton between two projectve reconstructons. CIU, ():0, 999. [] M. Fschler and R. Bolles. Random sample consensus: A paradgm for model fttng wth applcatons to mage analyss and automated cartography. Graphcs and Image Processng, (): 9, 9. [] N. Georgs, M. Petrou, and J. Kttler. On the correspondence problem for wde angular separaton of non-coplanar ponts. IC, :, 99.
9 0 9 0 Fgure. The fnal result obtaned for the second mage par at scale. All of the matches are correct. [] F. Glazer, G. Reynolds, and P. Anandan. Scene matchng by herarchcal correlaton. In CPR, 9. [] B. B. Hansen and B. S. Morse. Multscale mage regstraton usng scale trace correlaton. In CPR, 999. [] C. Harrs and M. Stephens. A combned corner and edge detector. In Alvey son Conference, 9. [] R. Horaud and T. Skordas. Stereo matchng through feature groupng and maxmal clques. PAMI,(): 0, 99. [] J. Koendernk. The structure of mages. Bologcal Cybernetcs, 0: 9, 9. [9] J. Koendernk and A. van Doorn. Representaton of local geometry n the vsual system. Bologcal Cybernetcs, :, 9. [0] M. S. Lew and T. S. Huang. Optmal mult-scale matchng. In CPR, 999. [] T. Lndeberg. Scale-Space Theory n Computer son. Kluwer Academc Publshers, 99. [] T. Lndeberg. Feature detecton wth automatc scale selecton. IJC, 0():9, 99. [] D. G. Lowe. Object recognton from local scale-nvarant features. In ICC, 999. [] P. Meer, D. Mntz, A. Rosenfeld, and D. Km. Robust regresson methods for computer vson: a revew. IJC, ():9 0, 99. [] P. Prtchett and A. Zsserman. Wde baselne stereo matchng. In ICC, 99. [] L. H. Quam. Herarchcal warp stereo. In Readng n computer son, pages 0. Morgan Kaufman, 9. [] C. Schmd and R. Mohr. Local grayvalue nvarants for mage retreval. PAMI, 9():0, 99. [] C. Schmd, R. Mohr, and C. Bauckhage. Comparng and evaluatng nterest ponts. In ICC, 99. [9] T. Tuytelaars, L.. Gool, L. D haene, and R. Koch. Matchng of affnely nvarant regons for vsual servong. In ICRA, 999. 9 0 0 9 Fgure. An other example wth