SCHOOLOFCOMPUTERSTUDIES RESEARCHREPORTSERIES UniversityofLeeds Report95.4 AcquisitionsandApplications Generic3-DShapeModel: DivisionofArticialIntelligence XShen&DCHogg by February1995
sequencesandrepresentedbythecontrolpointsofab-splinesurface.the genericmodelisderivedbyprincipalcomponentanalysisonthealignedtrainingshapes.usingtheacquiredgenericmodels,3-dshaperecovery,trackinvidualshapesinthetrainingsetareextractedautomaticallyfromtheimage fromatrainingsetofshapesdepictedinacorpusofimagesequences.indi- Thepaperdescribesamethodforgeneratingagenericdeformablemodel Abstract 1Introduction thedomainofvehiclesappearingintracscenes. andobjectidenticationareimplementedwithinoneprocedure.experimental 3-Dshapemodelsareextensivelyusedfortrackingandrecognisingknownobjects resultsarepresentedbothforgenerationandapplicationofthemodelwithin shapesofobjectsfrom2-dimages.themodelobtainedinthisworkisspecicto (e.g.[1,2,3]).intheworkreportedin[4,5],amethodisproposedforrecovering3-d ofobjectsisoftenmoreusefulinmanypracticalapplicationswheretheobjectsofthe sameclassarenotidenticalinshape(e.g.vehicles,people).wewishtoproducesuch amodelbygeneralisationfromthespecicshapesofobjectsinatrainingset. characterisinganyshapeinstanceinthemodelledclass.physically-basedmodels[6,7] eachpresentedobject.however,agenericshapemodelbeingabletorepresentaclass achievethisbyintroducingdynamicaldeformationstoaccommodatevariationsin shape.theyhavebeensuccessfullyusedintheapplicationssuchasshaperecovery andtracking[8,5,9].asignicantprobleminutilisingthesemodelsisthatthe scopeofthegenericmodelaccommodatesallshapeswithinthesolutionspaceofthe Anecessarypropertyofsuchagenericmodelisthatitshouldbecapableof dynamicalsimulation,butisnotrestrictedtoaspecictargetclassofobjects.as clutter,andtherefore,mayfailtodetectandtrackatargetshapecorrectly. aresult,theassociatedmethodsareoftensensitivetoimagenoiseandbackground Bajcsy[10]usesuperquadricsasagenericshapemodel.Arestrictedsolutionspace accommodatesseveralshapevariationsincludingtaperingandbending. canbeobtainedbyspecifyingtherangeofadmissibleshapeparameters.thismodel toasubspaceofthetotalcongurationspace.ideally,thissubspaceshouldbelearned Onesolutionistoaddvariableparameterstorigidgeometricmodels.Solinaand fromexamples.thisistheapproachadoptedbycootesetal.[11]whopresenta Analternativeapproachistoconstrainthedeformationofaexibleshapemodel 1
describingmodelsas2-dsplinecurves,and(2)acquiringtrainingshapeinstances automaticallyfromimagesandthereforeautomatingtheentireprocedureofmodel methodtoproducea2-dgenericshapemodel(\pointdistributionmodel")representedbyasetoflandmarks.baumbergandhogg[12]extendtheirmethodby(1) construction. modeliscomposedofameanshapeandaorthonormalbasisextractedbyanalysing recognition.forexperimentalpurposes,theobjectsweareinterestedinarevehicles, althoughthemethodcanalsobeappliedtootherclassesofobject.thegeneric shapemodel,representingagivenclassofobjects,directlyfromsequencesof2- Dimages,anddemonstratesitsapplicationsto3-Dshaperecovery,trackingand Inthispaper,wepresentamethodforautomaticallyproducingageneric3-D eigenvectorsdescribingthemostsignicantvariancesintheshape.theseeigenvectors thenformaorthonormalbasisdeningthespaceofadmissibleshapes.theproduced thevariancesofshapesinasetoftrainingshapesobtainedusingthemethodproposed modelisexibleinthevariancesofshape,andisalsorestrictedtoaspecicclass in[5].eachtrainingshapeisrstrepresentedbyashapevectorwhichconsistsof ofobjects.applicationsin3-dshaperecovery,trackingandrecognitioncanthenbe scalingandtranslation.principlecomponentanalysisisusedtoobtainasetofunit thesetofcontrolpointsofab-splinesurface.theshapevectorsarealignedby 2AcquisitionofGenericModel whichinturnareusedtoidentifytheobject. easilyachievedwithinoneprocedurebyttingthemodeltotheobjectineachimage inordertoestimatetheposeparametersofthemodel,andtheshapeparameters 2.13-Dshaperecovery In[5],amethodforrecoveringthe3-Dshapesofobjectsfrom2-Dimagesequences groundplane.withthismethod,instancesofvariousshapesinagenericclassof isproposed,assumingtheobjectisrigidandmirrorsymmetrical,andmovesona shapescanbeobtained.figure1showssomeresultsfromthiswork.therecovered shapemodelisaclosedsurfacewithtwopoleswhichisrepresenteddiscretelyby PQsamplepointsr(p;q)onthesurface r(p;q)=264u(p;q) V(p;q) W(p;q)375;r(p;0)=r(p;Q) r(p 1;0)=r(P 1;q);p=0;1;:::;P 1 r(0;0)=r(0;q) 2 q=0;1;:::;q (1)
Figure1:Recoveryof3-Dvehicleshapesfromimages.Eachtargetobjectisshown inaframefromthecorrespondingsequence.belowthisframeisshowntheobtained shapemodel. Toconciselyrepresenttheshapesinchosentrainingset,ashapevectorisextracted 2.2Extractingashapevector r(p;0)andr(p;hq2i),p=0;1;:::;p 1. Themodelismirrorsymmetricalaboutaverticalplanethroughthesamplepoints 0i<Mand0j<N,whichdenesabicubicB-splinesurfaceS(u;v)= qq.thesplines(u;v)isexpressedas[13] fromeachshapeinstance.theshapevectorconsistsofasetofcontrolpoints,xij, fsx(u;v);sy(u;v);sz(u;v)gtapproximatingthemodelr(p;q),0p<pand0 where,bi(u)andbj(v)arethe4thorderbasisfunctionsdenedonthefollowing knotvectorsrespectively: S(u;v)=M 1 Xi=0N 1 u=[0;0;0;0;1;2;;m 3;M 3;M 3;M 3] Xj=0XijBi(u)Bj(v) (2) weletx00=x0jandxm 1;0=XM 1;j,forj=0;1;:::;N 1,toensuretheclosed takenasbn k(n+v)toensurethesurfaceisclosedalongthecoordinatesv.also, surfacehastwopoles. Forthesituation0v<3,thebasisfunctionsBN k(v),1k<3 [v],aresimply v=[0;1;2;;n 1;N;N+1;N+2] 3
approximationleadstothefollowingequations: Theparametervaluesupqandvpqarecomputedasfollowing: Minimisingthesquarederrorbetweenthegeneratedsurfacer(p;q)andthespline upq=8><>:0 r(p;q)=m 1 (M 3)Ppi=1jr(i;q) r(i 1;q)j Xi=0N 1 PP 1 i=0jr(i;q) r(i 1;q)jforp>0 Xj=0XijBi(upq)Bj(vpq) forp=0 (4) (3) Writingequation(3)inmatrixform,wehave vpq=8><>:0 (N 1)Pqj=1jr(p;j) r(p;j 1)j PQ 1 j=0jr(p;j) r(p;j 1)jforq>0 =B forq=0 (6) (5) ofthefollowingforms: where,andarek3andl3(k=(p 2)Q+2,andl=(M 2)N+2)matrices, =264U00U10U11U1;Q 1UP 2;Q 1UP 1;0 =264X00X10X11X1;N 1XM 2;N 1XM 1;0 W00W10W11W1;Q 1WP 2;Q 1WP 1;0375T V00V10V11V1;Q 1VP 2;Q 1VP 1;0 Z00Z10Z11Z1;N 1ZM 2;N 1ZM 1;0375T Y00Y10Y11Y1;N 1YM 2;N 1YM 1;0 (8) (7) wherem=hj 1 Bisaklmatrix, Bij(upq;vpq)=8><>:B0(upq) Bm(upq)Bn(vpq)otherwise BM 1(upq) forj=0 tion(6).however,thecomputationofmatrixbanditsinverseisexpensive.barsky Foreachtrainingshape,thecontrolpointsXijcanbecalculatedbysolvingequa- Ni+1,andn=(j 1)modN. forj=l 1 (9) andgreenberg[14]proposeamethodtosolvetheequationwithoutcomputingtheinverseofb,byinvestigatingthepropertyofthematrixb.here,weuseanalternative waybasedonthatproposedbybaumbergandhogg[12]for2-dcase. 4
spondtothexeduniformlyspacedparametervalues: Foreachtrainingshape,wecalculatePQnewdatapointsr0(p;q)whichcorre- Thedatar0(p;q)canbecomputedbyiterativelyapplyinglinearinterpolationsalong pcoordinatesfollowedbylinearinterpolationsalongqcoordinatesusing u0pq(m 3)p U0(i;q) U(i 1;q) u0i;q ui 1;q=U(i;q) U(i 1;q) v0pq(n 1)q Q 1 P 1 natesareanalogous.inourexperiments,theaboveiterativeinterpolationsalways withu0(0;q)=u(0;q)andu0(p;0)=u(p;0).interpolationsforvandwcoordi- U0(p;j) U(p;j 1) v0pj vp;j 1 =U(p;j) U(p;j 1) uiq ui 1;q;ui 1;q<u0iquiq;0<i<M succeededinproducingtherequirednewdatapointsr0(p;q),althoughaformalproof oftheconvergenceneedstobedeveloped. upj up;j 1;vp;j 1<v0pjvpj;0<j<N controlpointsofthesplineconsistsoftheshapevectorg, forallofthetrainingshapes,andonlyneedstobecomputedonce.thecomputed Usingthesenewdatapoints,thematrixB(andtherefore,itsinverse)isxed Fortheconvenienceofthefollowingdiscussion,thecoordinatetriplesintheshape vectorareindexedas(xi;yi;zi),i=0,1,:::,k 1. 2.3Aligningshapevectors G=hX00Y00Z00X10:::ZM 2;N 1XM 1;0YM 1;0ZM 1;0iT(10) scalingandtranslation.foragivenshape,thistransformationcanbeestimated Havingproducedasetofshapes,thenextstepistoaligneachshapewithareference shape.inourimplementation,thersttrainingshapeisusedasthereferenceshape. Duetothemirrorsymmetryoftheshape,alignmentofeachshapeinvolvesonly byscalingandtranslatingtheshapetobealigned,g,i.e. usingalinearleastsquaresapproximation. Let^Gbethevectorofthereferenceshape,andG0betheshapevectorobtained 264X0i Y0 Z0i375=264Xi+tx i 5Zi+ty375 Yi+ty (11)
where,isthescaleparameter,andtx,ty,andtzarethetranslationparameters.to andtzwhichminimisethefollowingerror: aligntheshapevectorgwiththereferenceshapevector^g,wechoosethe,tx,ty, Usingthestandardmethod,weobtainthefollowingequationsfromwhichthesolution isderived: 264a1a2a3a4 a2a500 Error=(^G G0)T(^G G0) (12) where, a400a5 a30a50 3 75264tx ty tz 3 75=264b1 a1=k 1 Xi=0(X2i+Y2 i+z2i);a2=k 1 Xi=0Xi b2 b3 b4 3 75 (13) b1=k 1 a3=k 1 Xi=0(^XiXi+^YiYi+^ZiZi);b2=K 1 Xi=0Yi;a4=K 1 Xi=0Zi;a5=K 2.4Principlecomponentsanalysis b3=k 1 Xi=0^Yi;b4=K 1 Xi=0^Zi Xi=0^Xi asfollows: componentanalysis[15].nisthetotalnumberofshapesinthetrainingset. Thecontrolpointsintheshapevectoraretreatedasthelandmarkpointsinthe \pointdistributionmodel"(cootes,etal[11]).forthealignedshapesinthetraining set,thedeviationsfromthemeanshapeg=1npni=1giareanalysedusingprinciple A3K3KcovariancematrixCisthencalculatedusing ForeachalignedshapevectorG,itsdeviation,dG,fromthemeaniscomputed where,e()istheexpectedvalue. C=E(dGdGT) dg=g G (14) shownthattheeigenvectorcorrespondingtothelargesteigenvaluedescribesthemost thencorrespondtothemodesofvariationsofthetrainingshapes.ithasalsobeen Theuniteigenvectors,ei(eTiei=1,i=1;2;:::;3K)ofthecovariancematrix (15) 6
thesmostsignicantmodesofvariations.sisdeterminedusingfollowingcriterion: signicantmodeofvariation,andthattheproportionofthetotalvarianceexplained byeacheigenvectorisequaltothecorrespondingeigenvalue[15].toproducea concise,yeteectivegenericshapemodel,wechooseseigenvectorscorrespondingto (e.g.95%)tothetotalvariation.thegenericshapemodelisthenexpressedasfollows: where,iisthei'theigenvalueofc,ii+1,andsisapre-selectedproportion G=G+ew P3K Psi=1i>s (16) w2ws]tisavectorofweights(calledshapeparameters)foreacheigenvector. shapes.shapemodelsoftheobjectsinthesameclassasthetrainingonescanthen where,e=[e1e2es]isthematrixconsistingoftherstseigenvectors;w=[w1 Theseigenvectorsthenformaorthonormalbasisforthespaceofadmissible (17) areacceptable. begeneratedbyvaryingtheparameterswiwithinsuitablelimits.sincethevariance ofwioverthetrainingsetisi,wechoosewiusingthefollowingcriterion: 2.5Experimentalresults whichmeansthatshapesdistributedwithinthreestandarddeviationsofthemean 3qiwi3qi (18) variationsareshowninfig.2.theresultsshowthattherstmodecharacterisesthe variationfromavantoacar,andthesecondmodecharacterisesthevariationfrom accountedforover95%ofthevarianceofthetrainingshapes.someofthemodesof Wehaveappliedtheabovemethodtoasetof15vehicleswhoseimagesequences arecapturedbymonitoringaparkingspace.threeoftheseobjectsareshownin Fig.1.Eachshapevectorconsistsof162controlpoints.Wefoundtherst6modes ahatchbacktoasaloon.analysisonothermodesisanalogous. 3Applications Usingagenericshapemodel,wecanachieve3-Dshaperecovery,trackingandrecognitionwithinoneprocedurewhichcanbeimplementedinthefollowingway.Beginning withthemeanmodelfortherstimageinthesequence,themodelisrsttransformedtoastatewhichismostconsistentwiththeobjectin3-d.theshapeofthe 7
Figure2:Eectsofvarying(a)therstshapeparameter,(b)thesecondshape (b) thebestapproximationtotheobjectshapetodate,andisfedtothesameoperations thedistancebetweentheobjectproleandtheproleofthemodeldenedbythese objectisthenestimatedbyndingshapeparameterswiwhichminimiseameasureof parameters.themodelcharacterisedbytheseestimatedshapeparametersisthen parameter,and(c)thefourthshapeparameterby2:0standarddeviations. appliedtothenextimage.shapeparametersestimatedattheendofthesequenceare usedtodenetherecoveredshapeoftheobject.recognitionoftheobjectisachieved bycomparingtheseshapeparameterswiththoseofthemodelsinthedatabase. 3.1Estimatingthetransformation Sincetherigidobjectisassumedtomoveonagroundplane,thetransformationfrom themodeltotheobjectischaracterisedbythescale,orientation,andtranslation (px;py;pz)(pzisxed)inthefollowingtransformationmatrix: Theorientationoftheobjectisagainestimatedbyback-projectingtheimagemotiontrajectorytothegroundplane[5].Othertransformationparametersarethen sincos0py estimatedbyminimisingameasureofdistancebetweentheobjectproleandthe 0 001 0pz 3 75 264cos sin0px projecttothemodelprole.deneameritfunctionoftheunknowntransformation impliedmodelproleforthatorientation. LetXj,j=1;2;:::;NbetheNpointsintheshapevectorwhichapproximately 8
prole. parameters(;px;py)asfollows: Figure3:Determinationofcorrespondencepointsbetweenmodelproleandobject x j x j x x standardeuclideannorm.ifthereismorethanoneintersectionpoint,thefarthest where,x0jistheintersectionwiththeobjectproleoftherayfromx0,theprojection ofthemodelcenter,passingthroughxj,theprojectionofxj(fig.3)1.kkisthe 1(;px;py)=NXj=1kx0j xjk c 0 (19) object region profile areoutsidetheobjectregion,thenxc,thecentroidoftheobjectregionissubstituted onefromx0isused.inthecasewhenboththeprojectionofthemodelcenterandxj forx0. 3.2Estimatingshapeparameters optimisationmethod[16]isthenusedtoestimate(;px;py).theinitialestimatefor theparametersaretakenasthevaluesforthepreviousimage. perspectiveprojectionbetweenthexjandxjmeans Theaimistondtheparameters(;px;py)whichminimisethefunction 1isnon-linear.Anon-linear 1.The wi,similarlydenedasfollows: Oncethetransformationisdetermined,theobjectshapeisestimatedbynding Estimationofthewiisachieved,againbyminimisingameritfunctionofunknowns theindependentshapeparameterswiwhichdenethebestttotheobjectshape. Inordertoensurethattheestimatedshapeisplausible,theshapeparametersare constrainedtoliewithinahyper-ellipsoid[17].foreachimage,theshapeparametersareupdatedwiththeaboveestimate,andthengloballyconstrainedusingthe 1Thecalibrationmatrixwhichallowstheprojectionfromtheworldcoordinatestoimage(pixel) 2(w1;w2;:::;ws)=NXj=1kx0j xjk (20) coordinatesisassumedtobeknown(see[5]fordetails) 9
followingscheme: Thisglobalconstraintalsopreventstheshaperecoveryfrombeingsensitivetoerrors d2=sxi=1w2i intheextractionoftheobjectprole. w0i=8<:(dmax=d)wid>dmax wi i otherwise (21) 3.3Recognition (22) Theestimatedshapeparameterscanbeusedtoformameasureforobjectrecognition orclassication.pentlandandsclaro[7]usecorrelationoftwovectorsdescribing changesintheshape,andoftenhavelargerabsolutevaluesthanotherparameters. parametertothedescriptionoftheobjectshapeisnotequal.shapeparameters twoobjectsasameasureforcomparison.inourcase,thecontributionofeachshape Simplyusingcorrelationmayignoretheeectoftheparameterscorrespondingto correspondingtothemostsignicantmodesofvariationdescribethemostcommon thelesssignicantmodesofvariationwhich,onthecontrary,arenormallycrucial indiscriminatingthedierentobjects.wethereforeusethefollowingmeasureto comparetwoobjectswithshapeparametersw1andw2: = spsi=1w1i spsi=1w1i i2+spsi=1w2i i w2i theisintheinterval[0;1].fortwosimilarshapes,theapproachesto0. Theeigenvaluesinthemeasureareusedtonormalisetheeectofeachshapeparameter.Thismeasureisproportionaltothedistancebetweentwovectors.Thevalueof i2 (23) recoveredobjectshapesfrom6sequencesrespectively.theobjectsshownin(a),(b) 3.4Experimentalresults Weappliedtheabovemethodtoseveralimagesequenceswheretheobjectsmayor and(c)arealsousedinthetrainingset.therecoveredshapefortheobject(f)is Therecoveredshapeisthemostsimilarshapetothisobjectamongtheshapesinthe trainingset. maynotbewithinthetrainingset(butbelongtothesameclass).fig.4showsthe notcorrect.thisisbecausethetrainingsetdoesnotincludethiskindofobject. 10
(a) (b) (c) otherpairsofobjectsisanalogous.amongallpairsofobjects,(b)and(d)arethe Figure4:Recoveredobjectshapesusinggenericshapemodel. mostsimilar. (e)thanothers,andobject(b)ismoresimilartoobject(d)thanothers.analysison Theresultsaregivenintable1.Itcanbeseenthatobject(a)ismoresimilartoobject Therecoveredshapesarecomparedeachotherbycomputingthemeasure(23). (d) (f) 4Conclusion Wehaveproposedamethodforproducingageneric3-Dshapemodelbystatistically analysingasetoftrainingshapesrecoveredautomaticallyfrom2-dimagesequences. 11
b0.740.000.680.450.890.54 d0.770.450.580.000.820.61 a0.000.740.820.770.510.88 c0.820.680.000.580.680.61 e0.510.890.680.820.000.97 abcdef f0.880.540.610.610.970.00 constraint,themodelcanprovideasensiblesolutionspacewhichwebelievesupports robustmodel-basedapplications.applicationsofutilisingthismodelfor3-dshape ThemodelisrepresentedbyasetofcontrolpointsofaB-splinesurface,andconsists ofameanshapeandaorthonormalbasisdeningtheshapespace.withaglobal Table1:Resultsofrecognition haveshownthatthemethodgivesveryencouragingresults. recovery,trackingandrecognitionhavebeendemonstrated.experimentalresults References [3]D.G.Lowe.RobustModel-basedMotionTrackingThroughtheIntegrationof [2]R.T.ChinandC.R.Dyer.Model-basedRecognitioninRobotVision.ACM [1]P.J.BeslandR.C.Jain.Three-dimensionalObjectRecognition.ACMComputingSurveys,17(1):75{145,March1985. ComputingSurveys,18(1):67{108,March1986. SearchandEstimation.InternationalJournalofComputerVision,8(2):113{122, [5]X.ShenandD.Hogg.3-DShapeRecoveryUsingADeformableModel.Image [4]X.ShenandD.Hogg.ShapeModelsfromImageSequences.InJan-OlofEklundh, 1994. 1992. [6]D.Terzopoulos,J.Platt,A.Barr,andK.Fleischer.ElasticallyDeformable andvisioncomputingjournal,1995.toappear. editor,computervision{eccv'94,volumei,pages225{230.springer-verlag, Models.ACMComputerGraphics,21(4):205{214,July1987. 12
[8]D.Terzopoulos,A.Witkin,andM.Kass.ConstraintsonDeformableModels: [7]A.PentlandandS.Sclaro.Closed-formSolutionforPhysicallyBasedShape ModellingandRecognition.IEEETransactionsonPatternAnalysisandMachineIntelligence,PAMI-13(7):715{729,July1991. [10]F.SolinaandR.Bajcsy.RecoveryofParametricModelsfromRangeImages: [9]D.TerzopoulosandD.Metaxas.TrackingNonrigid3DObjects.InA.Blake anda.yuille,editors,activevision,pages75{89.mitpress,1992. Recovering3DShapeandNonrigidMotion.ArticialIntelligence,36(1):91{123, TheCaseforSuperquadricswithGlobalDeformations.IEEETransactionson 1988. [11]T.F.Cootes,C.J.Taylor,D.H.Cooper,andJ.Graham.TrainingModels 1992.Springer-Verlag. BritishMachineVisionConference1992,pages9{18,Leeds,UK,September ofshapefromsetsofexamples.indavidhoggandrogerboyle,editors, PatternAnalysisandMachineIntelligence,PAMI-12(2):131{147,February1990. [13]R.H.Bartels,J.C.Beatty,andB.A.Barsky.AnIntroductiontoSplinesfor [12]A.BaumbergandD.Hogg.LearningFlexibleModelsfromImageSequences.In 308.Springer-Verlag,1994. Jan-OlofEklundh,editor,ComputerVision{ECCV'94,volumeI,pages299{ [15]I.T.Jollie.PrincipleComponentsAnalysis.Springer-Verlag,1986. [14]B.A.BarskyandD.P.Greenberg.DeterminingaSetofB-SplineControl VerticestoGenerateanInterpolatingSurface.ComputerGraphicsandImage Processing,14:203{226,1980. useincomputergraphicsandgeometricmodelling.morgankaufmann,1987. [17]T.F.CootesandC.J.Taylor.ActiveShapeModels{`SmartSnakes'.InDavid [16]L.E.Scales.IntroductiontoNon-linearOptimization.Macmillan,1985. 266{275,Leeds,UK,September1992.Springer-Verlag. HoggandRogerBoyle,editors,BritishMachineVisionConference1992,pages 13