ComprehensiveColourImageNormalization GrahamD.Finlayson1,BerntSchiele2,andJamesL.Crowley3 1TheColour&ImagingInstitute TheUniversityofDerby 2MITMediaLab UnitedKingdom 3INRIARh^onesAlpes 38330Montbonnot CambridgeMA France USA colourbutareplacedatdierentpositionsthencorrespondingrgbpixels ducestwodierentcolourimages.ifthetwoilluminantsarethesame Abstract.Thesamesceneviewedundertwodierentilluminantsin- arerelatedbysimplescalefactors.incontrastifthelightinggeometry colourchannels(e.g.alltheredpixelvaluesorallthegreenpixels)that areascalingapart.itiswellknownthattheimagedependenciesdueto lightinggeometryandilluminantcolourcanberespectivelyremovedby normalizingthemagnitudeofthergbpixeltriplets(e.g.bycalculating isheldxedbutthecolourofthelightchangesthenitistheindividual thernormalizationsucestoaccountforchangesinboththelighting chromaticities)andbynormalizingthelengthsofeachcolourchannel (byrunningthe`grey-world'colourconstancyalgorithm).however,nei- normalizethergbpixels(toremovedependenceonlightinggeometry) geometryandilluminantcolour. andthennormalizether,gandbcolourchannels(toremovedependence colour.ourapproachisdisarminglysimple.wetakethecolourimageand whichremovesimagedependencyonlightinggeometryandillumination Inthispaperwepresentanewcomprehensiveimagenormalization thisprocessuntilwereachastablestate;thatisreachapositionwhere onilluminantcolour).wethenrepeatthisprocess,normalizergbpixels thenr,gandbcolourchannels,andthenrepeatagain.indeedwerepeat procedurealwaysconvergestothesameanswer.moreover,convergence eachnormalizationisidempotent.cruciallythisiterativenormalization thatappearintheliterature:swain'sdatabase,thesimonfraserdatabase, SangWokLee'sdatabase.Inallcases,forrecognitionbycolourdistributioncomparison,thecomprehensivenormalizationimprovesrecognition reportedintheliterature).alsorecognitionforthecompositedatabase rates(theresultsarenearperfectandinallcasesimproveonresults weconsideredtheobjectrecognitionproblemforthreeimagedatabases isveryrapid,typicallytakingjust4or5iterations. Toillustratethevalueofour\comprehensivenormalization"procedure (comprisingalmost100objects)isalsonearperfect.

1Introduction Thelightreachingoureyeisafunctionofsurfacereectance,illuminantcolour andlightinggeometry.yet,thecoloursthatweperceivedependalmostexclusivelyonsurfacereectance;thedependenciesduetolightinggeometryand illuminantcolourareremovedthroughsomesortofimagenormalizationprocedure.asanexample,thewhitepageofabooklookswhitewhetherviewed underblueskyorunderarticiallightandremainswhite,independentofthe positionofthelightsource.whileanalogousnormalizationsexistincomputer visionfordiscountinglightinggeometryorilluminantcolourtheredoesnotexistanormalizationwhichcandoboth,togetheratthesametime.yet,sucha comprehensivenormalizationisclearlyneededsincebothlightinggeometryandilluminantcolourcanbeexpectedtochangefromimagetoimage.a comprehensivenormalizationisdevelopedinthispaper. Imagenormalizationresearchincomputervisiongenerallyproceedsintwo stagesandwewilladoptthesamestrategyhere.first,thephysicsofimage formationarecharacterizedandthedependencyduetoagivenphysicalvariableismadeexplicit.inasecondstage,methodsforremovingthisdependency (thatis,cancelingdependentvariables)aredeveloped.asanexampleofthis kindofreasoning,itiswellknownthat,assumingalinearcameraresponse,if lightintensityisscaledbyafactorsthentheimagescalesbythesamefactor: eachcaptured(r;g;b)pixelbecomes(sr;sg;sb).relativetothissimplephysical modelitiseasytoderiveanormalizationprocedurewhichisindependentofthe intensityoftheviewingilluminant: r r+g+b; g r+g+b; b r+g+b (1) Thenormalizedimagecoloursaresometimesrepresentedusingonlythechromaticitiesr r+g+bandg r+g+b(sinceb r+g+b=1?r?g r+g+b). Thenormalizationshownin(1)iswellused,andwellaccepted,inthecomputervisionliterature(e.g.[SW95,CB97,MMK95,FDB91,Hea89])anddoesan admirablejobofrenderingimagecoloursindependentofthepoweroftheviewingilluminant.asweshallseelater,lightinggeometryingeneral(thisincludes thenotionsoflightsourcedirectionandlightsourcepower)aectsonlythe magnitudeofacapturedrgbandsothenormalizationshownin(1)performs wellindiversecircumstances. Dependencyduetoilluminationcolourisalsoverysimpletomodel(subject tocertaincaveatswhichareexploredlater).if(r1;g1;b1)and(r2;g2;b2)denote cameraresponsescorrespondingtotwoscenepointsviewedunderonecolour oflightthen(r1;g1;b1)and(r2;g2;b2)denotetheresponsesinducedby thesamepointsviewedunderadierentcolouroflight[wb86](thered,green andbluecolourchannelsscalebythefactors,and).clearly,itiseasyto derivealgebraicexpressionswhere,andcancel: (2r1 r1+r2;2g1 g1+g2;2b1 b1+b2);(2r2 r1+r2;2g2 g1+g2;2b2 b1+b2) (2)

denominatortermbecomesthesumofallpixelsandnumeratorsarescaledby thatis,to`grey'.forthisreason,equation(2)issometimescalled`grey-world' normalization[hun95,gjt88,buc80]. N.Noticethatafternormalization,themeanimagecolourmapsto(1,1,1); Thetwopixelcasesummarizedin(2)naturallyextendstoN-pixels:the itisusefultostepthroughaworkedexample.let(s1r1;s1g2;s1b1)and dencyonbothlightinggeometryandilluminantcolour.toseethatthisisso, (s2r2;s2g2;s2b2)denoteimagecolourscorrespondingtotwoscenepoints where(s1;s2)and(;;)arescalarsthatmodellightinggeometryandilluminantcolourrespectively.underlightinggeometrynormalization,equation(1), Unfortunately,neithernormalization(1)or(2)sucestoremovedepen- thepixelsbecome:( andunderilluminantcolournormalization,equation(2): ( r1+g1+b1; r2+g2+b2; r1+g1+b1; r2+g2+b2; r1+g1+b1); r2+g2+b2) (3) Inbothcasesonlysomeofthedependentvariablescancel.Thisisunsatisfactory (2s1r1 s1r1+s2r2; s1g1+s2g2; 2s1g1 s1b1+s2b2);(2s2r2 2s1b1 s1r1+s2r2; s1g1+s2g2; 2s2g2 s1b1+s2b2) 2s2b2 sincebothlightinggeometryandilluminantcolourwillchangefromimageto image. (4) solvedusingnormalizedcolourdistributionmanifolds.intheirmethodimages ismodelledexplicitly.theyshowthatthelightinggeometrynormalizeddistributionofcoloursinasceneviewedunderallilluminantcoloursoccupiesacontinuousmanifoldindistributionspace.inlaterworkbyberwickandlee[bl98] arenormalizedforlightinggeometryandthevariationduetoilluminantcolour LinanLee[LL97]proposedthatthisproblem(cancelationfailure)couldbe computationaloverhead.ahighdimensionalmanifoldis,atthebestoftimes, themanifold`onthey'.unfortunatelyboththesesolutionsincurasubstantial `matched'byashiftinilluminantcolour;thismatchingeectivelyreconstructs malizedimagecolourdistributionsaredenedtobethesameiftheycanbe thismanifoldisrepresentedimplicitly.hereapairoflightinggeometrynor- agivenmanifold).similarlythedistributionmatchingsolution,whichoperates unwieldyandimpliescostlyindexing(i.e.todiscoverifadistributionbelongsto atthesametime.ourapproachissimplicityitself.wetakeaninputimageand malizeawayvariationduetoilluminantcolourandlightinggeometrytogether byexhaustivedistributioncorrelation,isveryexpensive. normalizeforlightinggeometryusingequation(1).wethennormalizeforilluminantcolourusingequation(2).wetheniterateonthistheme,successively Inthispaperwedevelopanewcomprehensivenormalizationwhichcannor- stepisidempotent. normalizingawaylightinggeometryandlightcolouruntileachnormalization

verges.second,thattheconvergentimageisunique:thesamesceneviewed underanylightinggeometryandunderanyilluminantcolourhasthesamecomprehensivelynormalizedimage.wealsofoundthatconvergenceisveryrapid, typicallytakingjust4or5iterations. Weprovetwoveryimportantresults.First,thatthisprocessalwaysconeratedsyntheticimagesofayellow/greywedgeviewedunderwhite,blueand normalforgrey),90o(halfwaybetweenbothsurfaces)and135o(closetothenormalofyellow).theimagecaptureconditionsareillustratedatthetopoffigure1 Toillustratethepowerofthecomprehensivenormalizationprocedurewegen- orangecolouredlightswhichwereplacedatanglesof45o(closetothesurface bottomoftheguretogetherwithcorrespondingnormalizedimages.lighting ingpositionbutnotilluminantcolour.conversely,illuminantcolournormalized geometrynormalization(equation(1))sucestoremovevariationduetolight- images(equation2)areindependentoflightcolourbutdependonlightingposition.onlythecomprehensivenormalizationsucestoremovevariationdueto lightinggeometryandilluminantcolour. (abluelightat80oisshown).the9generatedsyntheticimagesareshownatthe neath,arealmostthesame.thisexperimentsisrepeatedonasecondimage imagesappear.aftercomprehensivenormalizationtheimages,shownunder- apairoflightinggeometriesandilluminantcolours.noticehowdierentthe showninfigure2.thetoptwoimagesareofthesameobjectviewedunder Examplesofthecomprehensivenormalizationactingonrealimagesare pairwithsimilarresults1. distributions(orinthecaseofourexperimentsbythedistributionofcolours e.g.[so95],[nb93]and[ll97])whereobjectsarerepresentedbyimagecolour nitionparadigmsuggestedbyswainandballard[sb91](whichiswidelyused severalobjectrecognitionexperimentsusingrealimages.weadoptedtherecog- Asayetmorerigoroustestofcomprehensivenormalizationwecarriedout comparison:querydistributionsarecomparedtoobjectdistributionsstoredin adatabaseandtheclosestmatchidentiesthequery. incomprehensivelynormalizedimages).recognitionproceedsbydistribution jee(13database,26queries),brewsterandlee(8objectsand9queries)anda compositeset87objectsand67queries,comprehensivenormalizationfacilitated almostperfectrecognition.forthecompositedatabaseallbut6oftheobjects arecorrectlyidentiedandthosethatarenotareidentiedinsecondplace.this FortheimagedatabasesofSwain(66databaseobjects,31queries),Chatter- performanceisquitestartlinginitsownright(itisalargedatabasecompiledby avarietyofresearchgroups).moreover,recognitionperformancesurpasses,by appliedindividually. far,thatsupportedbythelightinggeometryorilluminantcolournormalizations normalizationispresentedinsection3togetherwithproofsofuniquenessand thenormalizationsshownaboveinequations(1)and(2).thecomprehensive 1AllfourinputimagesshowninFigure2weretakenbyBerwickandLee[BL98] Insection2ofthispaperwediscusscolourimageformationandderive

Fig.1.Ayellow/greywedge,showntopofgure,isimagedunder3lightinggeometries and3lightcolours.theresulting9imagescomprisethe33gridofimagesshown izationremovestheeectsofbothilluminantcolourandlightinggeometry(thesingle normalizationsuces(top3imagesinthelastcolumn).thecomprehensivenormal- suces(rst3imagesinthelastrow).forxedlightinggeometry,illuminantcolour imageshownbottomright) topleftabove.whenilluminantcolourisheldxed,lightinggeometrynormalization

hensivelynormalized(bottompairofimages).noticehoweectivelycomprehensive Fig.2.Apeanutcontainerisimagedundertwodierentlightinggeometriesandilluminantcolours(topofgure).Aftercomprehensivenormalizationtheimagesappear thesame(2ndpairofimages).apairof`split-pea'images(thirdrow)arecompre- normalizationremovesdependenceilluminantcolourandlightinggeometry

convergence.insection4theobjectrecognitionexperimentsaredescribedand resultspresented.thepapernisheswithsomeconclusionsinsection5. 2ColourImageFormation Thelightreectedfromasurfacedependsonthespectralpropertiesofthesurfacereectanceandoftheilluminationincidentonthesurface.Inthecaseof Lambertiansurfaces(theonlyonesweconsiderhere),thislightissimplythe productofthespectralpowerdistributionofthelightsourcewiththepercent sensorresponse: spectralreectanceofthesurface.assumingasinglepointsourcelight,illumination,surfacereectionandsensorfunction,combinetogetherinforminga isthe3-vectorofresponsefunctions(red-,greenandblue-sensitive),eisthe whereiswavelength,pisa3-vectorofsensorresponses(rgbpixelvalue),f p^x;e=ex:nxz!sx()e()f()d (5) illuminationstrikingsurfacereectancesxatlocationx.integrationisoverthe visiblespectrum!.here,andthroughoutthispaper,underscoringdenotesvector ofexmodelsthepoweroftheincidentlightatx.notethatthisimpliesthat surfacenormalatxandexisinthedirectionofthelightsource.thelength onto^xonthesensorarray.theprecisepowerofthereectedlightisgoverned bythedot-producttermex:nx.here,nxistheunitvectorcorrespondingtothe quantities.thelightreectedatx,isproportionaltoe()sx()andisprojected R!Sx()E()F()allowsustosimplify(5): thefunctione()isactuallyconstantacrossthescene.substitutingqx;efor lightinggeometry(butdoeschangewithilluminantcolour).equation(6),which Itisnowunderstoodthatqx;Eisthatpartofascenethatdoesnotvarywith p^x;e=qx;eex:nx dealsonlywithpoint-sourcelightsiseasilygeneralizedtomorecomplexlighting (6) thethecameraresponseisequalto: lightswithlightingdirectionvectorsequaltoex;i(i=1;2;;m).inthiscase geometries.supposethelightincidentatxisacombinationofmpointsource Ofcourse,allthelightingvectorscanbecombinedintoasingleeectivedirection vector(andthistakesusbacktoequation(6)): p^x;e=qx;emxi=1ex;i:nx (7) ex=mxi=1ex;i)p^x;e=qx;eex:nx (7a)

sucestomodelextendedlightsourcessuchasuorescentlights[pet93]. equalsthesumoftheresponsestoeachindividuallight.simplethough(7)is,it Sincewenowunderstandthedependencybetweencameraresponseandlightinggeometry,itisascalarrelationshipdependentonex:nx,itisastraightforward mattertonormalizeitaway: Equation(7)conveystheintuitiveideathatthecameraresponsetomlight Whenp^x;E=(r;g;b)thenthenormalizationreturns:(r P3i=1p^x;E i = ex:nxp3i=1qx;e qx;eex:nxi = P3i=1qx;E Itisusefultoviewthedynamicsofthisnormalizationintermsofacomplete r+g+b;g i r+g+b;b r+g+b). (8) image.letusplacethenimagergbpixelsinrowsofann3imagematrix I.Itisclearthat(8)scalestherowsofItosumtoone.ThefunctionR() row-normalizesanimagematrixaccordingto(8): Here,andhenceforth,adoublesubscripti;jindexestheijthelementofamatrix. R(I)i;j= P3k=1Ii;k Ii;j Letusnowconsidertheeectofilluminantcolouronthergbsrecordedby (8a) mattersstillfurthercamerasensorsaredeltafunctions:f()=(?i) (i=1;2;3).undere()thecameraresponseisequalto: acamera.hereweholdlightinggeometry,thevectorsex,xed.tosimplify andundere1(): p^x;e i =ex:nxz!sx()e()(?i)d=ex:nxsx(i)e(i) (9) Combining(9)and(10)togetherweseethat: p^x;e1 i =ex:nxz!sx()e1()(?i)d=ex:nxsx(i)e1(i) (10) recordedineachcolourchannelscalebyamultiplicativefactor(onefactorper Equation(11)informsusthat,asthecolourofthelightchanges,thevalues p^x;e1 i =E1(i) E(i)p^x;E scalars).itisasimplemattertoremovedependenceonilluminationcolour: eachofthered,greenandbluecolourchannels)thenunderachangeinlight colourchannel).ifr,gandbdenotethenvaluesrecordedinanimage(for colourthecapturedimagebecomesr,gandb(where,andare

PNi=1Gi=N=3G PNi=1Ri=N=3R N=3G N=3R PNi=1Bi=N=3B N=3B (12) columntosumton=3.thisn=3tallyisfarfromarbitrary,butratherensures sameasthetotalimagesumcalculatedpostrownormalization.thus,inprinciple,animagecanbeinbothrow-andcolumn-normalform(andthegoal IntermsoftheN3imagematrixI,thenormalizationactstoscaleeach thatthetotalsumofallpixelspost-columnnormalizationisnwhichisthe ofcomprehensivenormalization,discussedinthenextsection,isfeasible).the functionc()columnnormalizesiaccordingto(12): izationpresentedin(12)deltafunctionsensitivitieswereselectedforourcamera. Itisprudenttoremindthereaderthatinordertoarriveatthesimplenormal- C(I)i;j=N=3Ii;j PNk=1Ik;j (12a) Whilesuchaselectionisnotgenerallyapplicable,studies[FF96,FDF94b,FDF94a] haveshownthatmostcamerasensorsbehave,orcanbemadetobehave,like deltafunctions. 3TheComprehensiveNormalization Thecomprehensivenormalizationprocedureisdenedbelow: 2.Ii+1=C(R(Ii)) 1.I0=I 3.Ii+1=Ii Initialization Iterationstep Thecomprehensiveprocedureiterativelyperformsrowandcolumnnormalizationsuntiltheterminationconditionismet.Inpracticetheprocesswillterminate whenanormalizationstepinducesachangelessthanacriterionamount. vergenceanduniqueness.theprocedureissaidtoconverge,ifforallimages xedscene)thenuniquenessfollows. terminationisguaranteed.iftheconvergentimageisalwaysthesame(forany Obviouslythisiterativeprocedureisusefulifandonlyifwecanshowcon- Terminationcondition (13) thediscussioninsection2,weknowthatviewingthesamesceneunderadierent lightinggeometryandilluminantcolourusingthetoolsofmatrixalgebra.from Asasteptowardsprovinguniquenessitisusefultoexaminetheeectsof

lightinggeometryresultsinanimagewherepixels,thatisrowsofi,arescaled. ThisideacanbeexpressedasanNNdiagonalmatrixDrpremultiplyingI: colourchannels;thatis,ascalingofthecolumnsofi.thismaybewrittenas Ipostmultipliedbya33matrixDc: Similarly,achangeinilluminantcolourresultsinascalingofindividual DrI (14) matrices:ianddridc. underapairoflightinggeometriesandilluminantcoloursinducestheimage Equations(14)and(15)takentogetherinformusthatthesamesceneviewed IDc (15) normalizedcounterpartarerelated: scalestherowsandthenthecolumnsofibypre-andpost-multiplyingwith canbecascadedtogetherandsowendthatanimageanditscomprehensively theappropriatediagonalmatrix.theoperationsofsuccessivenormalizations Bydenition,eachiterationofthecomprehensivenormalizationprocedure wherecomprehensive()isafunctionimplementingtheiterativeprocedureshown in(13)andthesymbolconveystheideaofasequenceofdiagonalmatrices. comprehensive(i)=dridc (16) Proof.LetusassumethatC16=C2.By(16),C1=Dr1IDc1andC2=Dr2DrIDcDc2 andc2=comprehensive(dridc)thenc1=c2(proofofuniqueness). forsomediagonalmatricesdr1,dr2,dc1anddc2.itfollowsthat: Theorem1.Assumingtheiterativeprocedureconverges,ifC1=comprehensive(I) Clearly,foranyDaandDbsatisfying(17)sodokDaand1kDbso,withoutloss C16=C2,DaandDbarenotequaltoidentity(orscaled)identitymatrices. whereda=dr2dr[dr1]?1anddb=[dc1]?1dcdc2.bytheassumptionthat C2=DaC1Db (17) ofgeneralityweassumethatdbi;i>1.wealsoassumethatdb1;1>db2;2>db3;3 ofc1).sincec2iscomprehensivelynormalizedwecanexpressthecomponents (sinceifthisisnotthecaseitcanbemadetruebyinterchangingthecolumns ofdaintermsofdbandc1.inparticulartheithdiagonaltermofdais,and mustbe,thereciprocalofthesumoftheithrowofc1db: Fromwhichitfollowsthat: Dai;i= Db1;1C1i;1+Db2;2C1i;2+Db3;3C1i;3 1 (18) C2i;1= Db1;1C1i;1+Db2;2C1i;2+Db3;3C1i;3 (19)

SincehaveassumedthatDb1;1>Db2;2>Db3;3,itfollowsthat whichimpliesthat Db1;1C1i;1+Db1;1C1i;2+Db1;1C1i;3< Db1;1C1i;1+Db2;2C1i;2+Db3;3C1i;3 (20) lessthanthecorrespondingelementininc2.however,thiscannotbethecase Equation(21)informsusthateveryelementintherstcolumnofC1isstrictly C1i;1+C1i;2+C1i;3< Db1;1C1i;1+Db2;2C1i;2+Db3;3C1i;3C1i;1<C2i;1 sincebothc1andc2arecomprehensivelynormalizedwhichimpliesthatthesum oftheirrespectedrstcolumnsmustbethesame(andthiscannotbethecase iftheinequalityin(21)holds). Proof.OurprooffollowsdirectlyfromSinkhorn'sthoerem[Sin64]whichwe Wehaveacontradictionandso,C1=C2anduniquenessisproven. Theorem2.Thecomprehensivenormalization,(13),procedurealwaysconverges. ut thattheprocesswheretherowsofbareiterativelyscaledtosumton=3and invokehereasalemma. thenthecolumnsarescaledtosumton=3(inananalogousprocessto(13))is Lemma.LetBdenoteanarbitrarynnallpositivematrix.Sinkhornshowed guaranteedtoconverge2. letmatrixbbea3n3nwherethen3imagematrixiiscopiedntimes inthehorizontaland3timesintheverticaldirection: First,notethatimages,underanyimagingconditions,areallpositive.Now, B=24III sumtonandthecolumnsofbdcsumton(noten=3n=3).fromtheblock SupposethatDrandDcarediagonalmatricessuchthattherowsofDrB 3 5 (22) structureofb,itfollowsthatdri;i=dri+kn;i+kn(i=1;2;;n)(k=2;3). SimilarlybecausecolumnssumtoN,Dci;i=Dci+kN;i+kN(i=1;2;3)andk= canwritedrbandbdcas: (2;;N).SettingDa=Dri;i(i=1;2;;N)andDb=Dci;i(i=1;2;3),we DrB=24DaIDaIDaI DaIDaIDaI 3 5;BDc=24IDbIDbIDb35 3NN=N=3.ThatiseachIinBisnormalizedaccordingtothefunctionsR() NEachrowinDaIsumsto3 3NN=1andeachcolumninIDbsumsto (23) 2Infactwecouldchooseanypositivenumberhere;n=3willworkwellforourpurposes.

andc()andsoaftersucientiterations,sinkhorn'siterativeprocessconverges to: Sinkhorn(B)=24comprehensive(I)comprehensive(I)comprehensive(I) Clearly,Sinkhorn'stheoremimpliestheconvergenceofthecomprehensivenor- (24) 3 5 malizationprocedureandourproofiscomplete. veryrapidily:4or5iterationsgenerallysuces. Experimentally,wefoundthatthecomprehensivenormalizationconverges ut 4ObjectRecognitionExperiments WecarriedoutimageindexingexperimentsfortheSwainandBallard[SB91], imagescombined.swainandballard'simagesetcomprises66databaseand31 SimonFraser[Cha95,GFF95],BerwickandLee[BL98]imagesetsandasetofall queryimages.allimagesaretakenunderaxedcolourlightsourceandthere areonlysmallchangesinlightinggeometry.becausethereare,eectively,no confoundingfactorsinswain'simages,weexpectgoodindexingperformance forlightinggeometry,illuminantcolourandcomprehensivenormalizations.the queryimages.queryimagescontainthesameobjectsbutviewedunderlarge SimonFraserdatasetcomprisesasmalldatabaseof13objectimagesand26 setthereare8objectimagesand9queryimages.againqueriesimagesare changesinlightinggeometryandilluminantcolour.inleeandberwick'simage Thecompositesetcomprises87databaseimagesand67queries. capturedunderdierentconditions(viewinggeometryandlightcolourchange). ages,databaseandquery,weseparatelycarriedoutlightinggeometry,illuminant colourandcomprehensivenormalizations.atasecondstagecolourhistograms, structed.thisinvolveshistogrammingonlythe(g;b)tuplesinthenormalized representingthecolourdistributions,ofthevariouslynormalizedimagesarecon- Totesttheecacyofeachnormalizationweproceedasfollows.Forallim- images.thepixelvaluerisdiscardedbecauser+g+b=1afterlightinggeometryandcomprehensivenormalizations,andsoisadependentvariable.after (G;B)colourspace(whichhavevaluesbetween0and1)denethebinsforthe colourhistograms.ifhiandqdenotesthehistogramsfortheithdatabaseand illuminantcolournormalizationonaverager+g+b=1.a1616partionof queryimagesthenthesimilarityoftheithimagetothequeryimageisdened wherejj:jj1denotesthel1(orcity-blockdistance)betweenthecolourdistributions.thisdistanceisequaltothesumofabsolutedierencesofcorresponding jjhi?qjj1 as: (25)

spondstosmalldistances. histogrambins.reassuringly,ifhi=qthenjjhi?qjj1=0;closenesscorre- shouldcontainthesameobjectasthequeryimage).tables1,2and3summarizeindexingperformanceforallthreenormalizationsoperatingonallfourdattionsinthedatabase.thesedistancesaresortedintoascendingorderandthe Foreachquerycolourdistribution,wecalculatethedistancetoalldistribu- rankofthecorrectanswerisrecorded(ideallytheimagerankedinrstplace sets.twoperformancemeasuresareshown:the%ofqueriesthatwerecorrectly matched(%in1stplace)andtherankoftheworstcasematch. ImageSet Swain's SimonFraser LeeandBerwick33.33 %correctworstranking 42.313thoutof13 96.7 2ndoutof66 Table1.Indexingperformanceoflightinggeometrynormalization Composite 58.286thoutof87 ImageSet Swain's SimonFraser LeeandBerwick67.7 %correctworstranking 80.8 87.1 6thoutof13 5thoutof66 Table2.Indexingperformanceofilluminantnormalization Composite 79.116thoutof87 4thoutof9 ImageSet Swain's SimonFraser %correctworstranking 80.62ndoutof66 Table3.Indexingperformanceofcomprehensivenormalization LeeandBerwick Composite 93.12ndoutof87 100 1stoutof13 1stoutof9 suggestthatlightinggeometrynormalizationworksbestandthecomprehensivenormalizationworst.thisis,infact,notthecase:allthreenormalizationinggeometrynormalizationsplacethecorrectanswersinthetoptworanks workverywell.noticethatonlythecomprehensivenormalizationandlight- AcursorylookatthematchingperformanceforSwain'simagesappearsto

andthisisanadmirablelevelofperformancegivensuchalargedatabase. vastlysuperior,100%recognitionissupportedcomparedwith42.3%and80.8% FortheSimonFraserDatabasethecomprehensivenormalizationprocedureis noothercolourdistributioncomparisonmethodhascomeclosetodelivering are13thand6threspectively.thisisquiteunacceptablegiventheverysmall forlightinggeometryandilluminantcolournormalizations.thelatternormalizationsalsoperformverypoorlyintermsoftheworstcaserankingswhich size(just13objects)ofthesimonfraserdatabase.itisworthnotingthat 100%recognitiononthisdataset[FCF96](thesemethodsinclude,colour-angular indexing[fcf96],ane-invariantsofcolourhistograms[hs94]andcolourconstantcolourindexing[ff95]).thesamerecognitionstoryisrepeatedforthnitionandtheothernormalizationperformverypoorly. BerwickandLeedatabase.Comprehensivenormalizationsupports100%recog- thesewerecompiledbythreedierentresearchgroups.alsothemeansofrecognitionisasimplecolourdistributioncomparisonwhichisboundtofailwhen placematchesthatarerecordedhavecolourdistributionswhicharesimilarto performanceisquitestartling.thedatabaseislargecomprising87objectsand normalizationandtheworstcasematchisinsecondplace.suchrecognition teresting.over93%(of67)queriesarecorrectlyidentiedusingcomprehensive Perhapstherecognitionresultsforthecompositedatasetarethemostin- images,orobjects,havethesamemixtureofcolours.indeed,mostofthe,2nd theoverallbestmatch.forexampleanimageof`campbell'schickensoup'is confusedwithanimageof`campbell'sspecialchickensoup'.bothimagesare colournormalizationperformsbetter,arecognitionrateof79%butagainthe andtheworstcaserankingisanincrediblypoor86th(outof87).illuminant predominantlyredandwhite(asweexpectwithcampbell'ssoup). worstcasematchisunacceptable:16thplacedoutof87. usedindividually,performverypoorly.theformersucceedsjust58%ofthetime Incomparisonthelightinggeometryandilluminantcolournormalizations, Thecoloursrecordedinanimagedependonboththelightinggeometryandthe 5Conclusion colouroftheilluminant.unlesstheseconfoundingfactorscanbediscounted, colourscannotbemeaningfullycomparedacrossimages(andsoobjectrecognitionbycolourdistributioncomparisoncannotwork).inthispaperwedeveloped anewcomprehensivenormalizationprocedurewhichcanremovedependency duetolightinggeometryandilluminantcolourandsocanfacilitatecross-image togetheranditeratively.weprovethatthisiterativeprocessalwaysconverges whichdiscounteitherlightinggeometryorilluminantcolourandapplythem toauniquecomprehensivelynormalizedimage. colourcomparison. Ourapproachissimplicityitself.Wesimplyinvokenormalizationprocedures inasettheobjectrecognitionexperiments.forfourimagedatabases:swain's Thepowerofourcomprehensivenormalizationprocedurewasillustrated

post-comprehensivenormalizationwasfoundtobenearperfect.importantly, database,thesimonfraserdatabase,sangwoklee'sdatabase,andacomposite set(withalmost100objects),recognitionbycolourdistributioncomparison, References performancegreatlysurpassedthatachievableusingthelightinggeometryor illuminantcolournormalizationsindividually. [BL98]D.BerwickandS.W.Lee.Achromaticityspaceforspecularity-,illumination [CB97]J.L.CrowleyandF.Berard.Multi-modaltrackingoffacesforvideocommunications.InCVPR97,pages640{645,1997. [Buc80]G.Buchsbaum.Aspatialprocessormodelforobjectcolourperception. JournaloftheFranklinInstitute,310:1{26,1980. pagesession2.3,1998. color-andilluminationpose-invariant3-dobjectrecognition.iniccv98, [Cha95]S.S.Chatterjee.Colorinvariantobjectandtexturerecognition,1995.MSc [FCF96]G.D.Finlayson,S.S.Chatterjee,andB.V.Funt.Colorangularindexing.In thesis,simonfraseruniversity,schoolofcomputingscience. TheFourthEuropeanConferenceonComputerVision(VolII),pages16{27. [FDB91]B.V.Funt,M.S.Drew,andM.Brockington.Recoveringshadingincolor [FDF94a]G.D.Finlayson,M.S.Drew,andB.V.Funt.Colorconstancy:Generalized EuropeanVisionSociety,1996. [FDF94b]G.D.Finlayson,M.S.Drew,andB.V.Funt.Spectralsharpening:Sensortransformationsforimprovedcolorconstancy.J.Opt.Soc.Am.A, 11(5):1553{1563,May1994. images,1991.submittedforpublication. [FF95]B.V.FuntandG.D.Finlayson.Colorconstantcolorindexing.IEEEtransactionsonPatternanalysisandMachineIntelligence,1995. diagonaltransformssuce.j.opt.soc.am.a,11:3011{3020,1994. [FF96]G.D.FinlaysonandB.V.Funt.Coecientchannels:Derivationandrelationshiptoothertheoreticalstudies.COLORresearchandapplication, [GFF95]S.S.ChatterjeeG.D.FinlaysonandB.V.Funt.Colorangleinvariantsfor 21(2):87{96,1996. [GJT88]R.Gershon,A.D.Jepson,andJ.K.Tsotsos.From[r;g;b]tosurfacere- objectrecognition.in3rdis&tandsidcolorimagingconference,pages 44{47.1995. [HS94]G.HealeyandD.Slater.\Globalcolorconstancy:recognitionofobjectsby [Hea89]G.Healey.Usingcolorforgeometry-insensitivesegmentation.J.Opt.Soc. Am.A,6:920{937,1989. ectance:computingcolorconstantdescriptorsinimages.ininternational useofilluminationinvariantpropertiesofcolordistributions".journalof JointConferenceonArticialIntelligence,pages755{758,1987. [LL97] [Hun95]R.W.G.Hunt.TheReproductionofColor.FountainPress,5thedition,1995. S.LinandS.W.Lee.Usingchromaticitydistributionsandeigenspaceanalysisforpose-,illumination-andspecularity-invariantrecognitionof3dobject. theopticalsocietyofamerica,a,11(11):3003{3010,november1994. [MMK95]J.Matas,R.Marik,andJ.Kittler.Onrepresentationandmatchingof oncomputervision,pages726{732.ieeecomputersociety,june1995. multi-colouredobjects.inproceedingsofthefthinternationalconference InCVPR97,pages426{431,1997.

[NB93]W.NiblackandR.Barber.TheQBICproject:Queryingimagesbycontent [Pet93]A.P.Petrov.Onobtainingshapefromcolorshading.COLORresearchand [SB91] usingcolor,textureandshape.instorageandretrievalforimageandvideo DatabasesI,volume1908ofSPIEProceedingsSeries.1993. [Sin64]R.Sinkhorn.Arelationshipbetweenarbitrarypositivematricesanddoubly M.J.SwainandD.H..Ballard.Colorindexing.InternationalJournalof application,18(4):236{240,1993. [SO95]M.A.StrickerandM.Orengo.Similarityofcolorimages.InStorageandRetrievalforImageandVideoDatabasesIII,volume2420ofSPIEProceedings stochasticmatrices.annalsofmathematicalstatistics,35:876{879,1964. ComputerVision,7(11):11{32,1991. [WB86]J.A.WortheyandM.H.Brill.HeuristicanalysisofvonKriescolorconstancy. [SW95]B.SchieleandA.Waibel.Gazetrackingbasedonface-color.InInternational Series,pages381{392.Feb.1995. WorkshoponAutomaticFace-andGesture-Recognition,June1995. JournalofTheOpticalSocietyofAmericaA,3(10):1708{1712,1986.