Mean Shift: A Robust Approach Toward Feature Space Analysis

Transcription

1 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 4, NO. 5, MAY Mea Sft: A Robust Approac Toward Feature Space Aalyss Dor Comacu, Member, IEEE, ad Peter Meer, Seor Member, IEEE AbstractÐA geeral oparametrc tecque s proposed for te aalyss of a complex multmodal feature space ad to deleate arbtrarly saped clusters t. Te basc computatoal module of te tecque s a old patter recogto procedure, te mea sft. We prove for dscrete data te covergece of a recursve mea sft procedure to te earest statoary pot of te uderlyg desty fucto ad, tus, ts utlty detectgte modes of te desty. Te relato of te mea sft procedure to te Nadaraya- Watso estmator from kerel regresso ad te robust M-estmators of locato s also establsed. Algortms for two low-level vso tasks, dscotuty preservg smootg ad mage segmetato, are descrbed as applcatos. I tese algortms, te oly user set parameter s te resoluto of te aalyss ad eter gray level or color mages are accepted as put. Extesve expermetal results llustrate ter excellet performace. Idex TermsÐMea sft, clusterg, mage segmetato, mage smootg, feature space, low-level vso. æ 1 INTRODUCTION LOW-LEVEL computer vso tasks are msleadgly dffcult. Icorrect results ca be easly obtaed sce te employed tecques ofte rely upo te user correctly guessg te values for te tug parameters. To mprove performace,te executo of low-level tasks sould be task drve,.e.,supported by depedet g-level formato. Ts approac,owever,requres tat,frst,te lowlevel stage provdes a relable eoug represetato of te put ad tat te feature extracto process be cotrolled oly by very few tug parameters correspodg to tutve measures te put doma. Feature space-based aalyss of mages s a paradgm wc ca aceve te above-stated goals. A feature space s a mappg of te put obtaed troug te processg of te data small subsets at a tme. For eac subset,a parametrc represetato of te feature of terest s obtaed ad te result s mapped to a pot te multdmesoal space of te parameter. After te etre put s processed,sgfcat features correspod to deser regos te feature space,.e.,to clusters,ad te goal of te aalyss s te deleato of tese clusters. Te ature of te feature space s applcato depedet. Te subsets employed te mappg ca rage from dvdual pxels,as te color space represetato of a mage,to a set of quas-radomly cose data pots,as te probablstc Houg trasform. Bot te advatage ad te dsadvatage of te feature space paradgm arse from te global ature of te derved represetato of te put. O oe ad,all te evdece for te presece of a. D. Comacu s wt te Imagg ad Vsualzato Departmet, Semes Corporate Researc, 755 College Road East, Prceto, NJ E-mal: comac@scr.semes.com.. P. Meer s wt te Electrcal ad Computer Egeerg Departmet, Rutgers Uversty, 94 Brett Road, Pscataway, NJ E-mal: meer@cap.rutgers.edu. Mauscrpt receved 17 Ja. 001; revsed 16 July 001; accepted 1 Nov Recommeded for acceptace by V. Solo. For formato o obtag reprts of ts artcle, please sed e-mal to: tpam@computer.org, ad referece IEEECS Log Number sgfcat feature s pooled togeter,provdg excellet tolerace to a ose level wc may reder local decsos urelable. O te oter ad,features wt lesser support te feature space may ot be detected spte of beg salet for te task to be executed. Ts dsadvatage, owever,ca be largely avoded by eter augmetg te feature space wt addtoal spatal) parameters from te put doma or by robust postprocessg of te put doma guded by te results of te feature space aalyss. Aalyss of te feature space s applcato depedet. Wle tere are a pletora of publsed clusterg tecques,most of tem are ot adequate to aalyze feature spaces derved from real data. Metods wc rely upo a pror kowledge of te umber of clusters preset cludg tose wc use optmzato of a global crtero to fd ts umber),as well as metods wc mplctly assume te same sape most ofte ellptcal) for all te clusters te space,are ot able to adle te complexty of a real feature space. For a recet survey of suc metods,see [9,Secto 8]. I Fg. 1,a typcal example s sow. Te color mage Fg. 1a s mapped to te tree-dmesoal L*u*v* color space to be dscussed Secto 4). Tere s a cotuous trasto betwee te clusters arsg from te domat colors ad a decomposto of te space to ellptcal tles wll troduce severe artfacts. Eforcg a Gaussa mxture model over suc data s doomed to fal,e.g.,[49], ad eve te use of a robust approac wt cotamated Gaussa destes [67] caot be satsfactory for suc complex cases. Note also tat te mxture models requre te umber of clusters as a parameter,wc rases ts ow calleges. For example,te metod descrbed [45] proposes several dfferet ways to determe ts umber. Arbtrarly structured feature spaces ca be aalyzed oly by oparametrc metods sce tese metods do ot ave embedded assumptos. Numerous oparametrc clusterg metods were descrbed te lterature ad tey ca be classfed to two large classes: erarccal clusterg ad desty estmato. Herarccal clusterg tecques eter aggregate or dvde te data based o /0/$17.00 ß 00 IEEE

2 604 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 4, NO. 5, MAY 00 Fg. 1. Example of a feature space. a) A color mage. b) Correspodg L*u*v* color space wt 110; 400 data pots. some proxmty measure. See [8,Secto 3.] for a survey of erarccal clusterg metods. Te erarccal metods ted to be computatoally expesve ad te defto of a meagful stoppg crtero for te fuso or dvso) of te data s ot stragtforward. Te ratoale bed te desty estmato-based oparametrc clusterg approac s tat te feature space ca be regarded as te emprcal probablty desty fucto p.d.f.) of te represeted parameter. Dese regos te feature space tus correspod to local maxma of te p.d.f., tat s,to te modes of te ukow desty. Oce te locato of a mode s determed,te cluster assocated wt t s deleated based o te local structure of te feature space [5],[60],[63]. Our approac to mode detecto ad clusterg s based o te mea sft procedure,proposed 1975 by Fukuaga ad Hostetler [1] ad largely forgotte utl Ceg's paper [7] rekdled terest t. I spte of ts excellet qualtes,te mea sft procedure does ot seem to be kow statstcal lterature. Wle te book [54,Secto 6..] dscusses [1],te advatages of employg a mea sft type procedure desty estmato were oly recetly redscovered [8]. As wll be prove te sequel,a computatoal module based o te mea sft procedure s a extremely versatle tool for feature space aalyss ad ca provde relable solutos for may vso tasks. I Secto,te mea sft procedure s defed ad ts propertes are aalyzed. I Secto 3,te procedure s used as te computatoal module for robust feature space aalyss ad mplemetatoal ssues are dscussed. I Secto 4,te feature space aalyss tecque s appled to two low-level vso tasks: dscotuty preservg flterg ad mage segmetato. Bot algortms ca ave as put eter gray level or color mages ad te oly parameter to be tued by te user s te resoluto of te aalyss. Te applcablty of te mea sft procedure s ot restrcted to te preseted examples. I Secto 5,oter applcatos are metoed ad te procedure s put to a more geeral cotext. THE MEAN SHIFT PROCEDURE Kerel desty estmato kow as te Parze wdow tecque patter recogto lterature [17,Secto 4.3]) s te most popular desty estmato metod. Gve data pots x, ˆ 1;...; te d-dmesoal space R d,te multvarate kerel desty estmator wt kerel K x ad a symmetrc postve defte d d badwdt matrx H, computed te pot x s gve by X ^f x ˆ1 K H x x ; 1 ˆ1 were K H x ˆjH j 1= K H 1= x : Te d-varate kerel K x s a bouded fucto wt compact support satsfyg [6,p. 95] Z K x dx ˆ 1 lm R d kxk!1 kxkd K x ˆ0 Z Z 3 xk x dx ˆ 0 xx > K x dx ˆ c K I; R d R d were c K s a costat. Te multvarate kerel ca be geerated from a symmetrc uvarate kerel K 1 x two dfferet ways K P x ˆYd ˆ1 K 1 x K S x ˆa k;d K 1 kxk ; 4 were K P x s obtaed from te product of te uvarate kerels ad K S x from rotatg K 1 x R d,.e.,k S x s radally symmetrc. Te costat a 1 k;d ˆ RR K d 1 kxk dx assures tat K S x tegrates to oe,toug ts codto ca be relaxed our cotext. Eter type of multvarate kerel obeys 3),but,for our purposes,te radally symmetrc kerels are ofte more sutable. We are terested oly a specal class of radally symmetrc kerels satsfyg K x ˆc k;d k kxk ; wc case t suffces to defe te fucto k x called te profle of te kerel,oly for x 0. Te ormalzato costat c k;d,wc makes K x tegrate to oe,s assumed strctly postve. Usg a fully parameterzed H creases te complexty of te estmato [6,p. 106] ad, practce,te badwdt matrx H s cose eter as dagoal H ˆ dag 1 ;...; d Š, 5

3 COMANICIU AND MEER: MEAN SHIFT: A ROBUST APPROACH TOWARD FEATURE SPACE ANALYSIS 605 or proportoal to te detty matrx H ˆ I. Te clear advatage of te latter case s tat oly oe badwdt parameter >0 must be provded; owever,as ca be see from ),te te valdty of a Eucldea metrc for te feature space sould be cofrmed frst. Employg oly oe badwdt parameter,te kerel desty estmator 1) becomes te well-kow expresso ^f x ˆ 1 d X ˆ1 K x x : 6 Te qualty of a kerel desty estmator s measured by te mea of te square error betwee te desty ad ts estmate,tegrated over te doma of defto. I practce, owever,oly a asymptotc approxmato of ts measure deoted as AMISE) ca be computed. Uder te asymptotcs,te umber of data pots!1,wle te badwdt!0 at a rate slower ta 1. For bot types of multvarate kerels,te AMISE measure s mmzed by te Epaeckov kerel [51,p. 139],[6,p. 104] avg te profle k E x ˆ 1 x 0 x x>1; wc yelds te radally symmetrc kerel K E x ˆ 1 c 1 d d 1 kxk kxk oterwse; were c d s te volume of te ut d-dmesoal spere. Note tat te Epaeckov profle s ot dfferetable at te boudary. Te profle k N x ˆexp 1 x x 0 9 yelds te multvarate ormal kerel K N x ˆ d= exp 1 kxk 10 for bot types of composto 4). Te ormal kerel s ofte symmetrcally trucated to ave a kerel wt fte support. Wle tese two kerels wll suffce for most applcatos we are terested,all te results preseted below are vald for arbtrary kerels wt te codtos to be stated. Employg te profle otato,te desty estmator 6) ca be rewrtte as ^f ;K x ˆck;d d X ˆ1 x x k : 11 Te frst step te aalyss of a feature space wt te uderlyg desty f x s to fd te modes of ts desty. Te modes are located amog te zeros of te gradet rf x ˆ0 ad te mea sft procedure s a elegat way to locate tese zeros wtout estmatg te desty..1 Desty Gradet Estmato Te desty gradet estmator s obtaed as te gradet of te desty estmator by explotg te learty of 11) ^rf ;K x r^f ;K x ˆ c k;d d X ˆ1 x x k 0 x x : 1 We defe te fucto g x ˆ k 0 x ; 13 assumg tat te dervatve of te kerel profle k exsts for all x 0; 1,except for a fte set of pots. Now,usg g x for profle,te kerel G x s defed as G x ˆc g;d g kxk ; 14 were c g;d s te correspodg ormalzato costat. Te kerel K x was called te sadow of G x [7] a slgtly dfferet cotext. Note tat te Epaeckov kerel s te sadow of te uform kerel,.e.,te d-dmesoal ut spere,wle te ormal kerel ad ts sadow ave te same expresso. Itroducg g x to 1) yelds, ^rf ;K x ˆ c X k;d x x d x x g " ˆ1 ˆ c X k;d x x # P d g ˆ1 xg 4 P ˆ1 ˆ1 g x x x x 3 x5; 15 were P ˆ1 g x x s assumed to be a postve umber. Ts codto s easy to satsfy for all te profles met practce. Bot terms of te product 15) ave specal sgfcace. From 11),te frst term s proportoal to te desty estmate at x computed wt te kerel G ^f ;G x ˆcg;d d X ˆ1 x x g : 16 Te secod term s te mea sft P ˆ1 x x x g m ;G x ˆ P ˆ1 g x x x; 17.e.,te dfferece betwee te wegted mea,usg te kerel G for wegts,ad x,te ceter of te kerel wdow). From 16) ad 17), 15) becomes yeldg ^rf ;K x ˆ ^f ;G x c k;d c g;d m ;G x ; 18 m ;G x ˆ1 c ^rf ;K x ^f ;G x : 19 Te expresso 19) sows tat,at locato x,te mea sft vector computed wt kerel G s proportoal to te ormalzed desty gradet estmate obtaed wt kerel K. Te ormalzato s by te desty estmate x computed wt te kerel G. Te mea sft vector tus always pots toward te drecto of maxmum crease te desty. Ts s a more geeral formulato of te property frst remarked by Fukuaga ad Hostetler [0,p. 535],[1],ad dscussed [7]. Te relato captured 19) s tutve,te local mea s sfted toward te rego wc te majorty of te

4 606 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 4, NO. 5, MAY 00 pots resde. Sce te mea sft vector s alged wt te local gradet estmate,t ca defe a pat leadg to a statoary pot of te estmated desty. Te modes of te desty are suc statoary pots. Te mea sft procedure, obtaed by successve. computato of te mea sft vector m ;G x,. traslato of te kerel wdow) G x by m ;G x, s guarateed to coverge at a earby pot were te estmate 11) as zero gradet,as wll be sow te ext secto. Te presece of te ormalzato by te desty estmate s a desrable feature. Te regos of low-desty values are of o terest for te feature space aalyss ad, suc regos,te mea sft steps are large. Smlarly,ear local maxma te steps are small ad te aalyss more refed. Te mea sft procedure tus s a adaptve gradet ascet metod.. Suffcet Codto for Covergece Deote by fy j g jˆ1;... te sequece of successve locatos of te kerel G,were,from 17), P ˆ1 x x x g y j 1 ˆ P ˆ1 g x x j ˆ 1; ;... 0 s te wegted mea at y j computed wt kerel G ad y 1 s te ceter of te tal posto of te kerel. Te correspodg sequece of desty estmates computed wt kerel K, f ^f ;K j g jˆ1;...,s gve by ^f ;K j ˆ ^f ;K y j j ˆ 1;...: 1 As stated by te followg teorem,a kerel K tat obeys some mld codtos suffces for te covergece of te sequeces fy j g jˆ1;... ad f ^f ;K j g jˆ1;.... Teorem 1. If te kerel K as a covex ad mootocally decreasg profle, te sequeces y j ad jˆ1;... f ^f ;K j g jˆ1;... coverge ad f ^f ;K j g jˆ1;... s mootocally creasg. Te proof s gve te Appedx. Te teorem geeralzes te result derved dfferetly [13],were K was te Epaeckov kerel ad G te uform kerel. Te teorem remas vald we eac data pot x s assocated wt a oegatve wegt w. A example of ocovergece we te kerel K s ot covex s sow [10,p. 16]. Te covergece property of te mea sft was also dscussed [7,Secto v]. Note,owever,tat almost all te dscusso tere s cocered wt te ªblurrgº process wc te put s recursvely modfed after eac mea sft step.) Te covergece of te procedure as defed ts paperwasattrbuted [7]to tegradetascetatureof 19). However,as sow [4,Secto 1.],movg te drecto of te local gradet guaratees covergece oly for ftesmal steps. Te step sze of a gradet-based algortm s crucal for te overall performace. If te step sze s too large,te algortm wll dverge,wle f te step sze s too small,te rate of covergece may be very slow. A umber of costly procedures ave bee developed for step sze selecto [4,p. 4]. Te guarateed covergece as sow by Teorem 1) s due to te adaptve magtude of te mea sft vector,wc also elmates te eed for addtoal procedures to cose te adequate step szes. Ts s a major advatage over te tradtoal gradet-based metods. For dscrete data,te umber of steps to covergece depeds o te employed kerel. We G s te uform kerel,covergece s aceved a fte umber of steps sce te umber of locatos geeratg dstct mea values s fte. However,we te kerel G mposes a wegtg o te data pots accordg to te dstace from ts ceter),te mea sft procedure s ftely coverget. Te practcal way to stop te teratos s to set a lower boud for te magtude of te mea sft vector..3 Mea Sft-Based Mode Detecto Let us deote by y c ad ^f ;K c ˆ ^f ;K y c te covergece pots of te sequeces fy j g jˆ1;... ad f ^f ;K j g jˆ1;..., respectvely. Te mplcatos of Teorem 1 are te followg. Frst,te magtude of te mea sft vector coverges to zero. Ideed,from 17) ad 0) te jt mea sft vector s m ;G y j ˆy j 1 y j ad,at te lmt,m ;G y c ˆy c y c ˆ 0. I oter words,te gradet of te desty estmate 11) computed at y c s zero r ^f ;K y c ˆ0; 3 due to 19). Hece, y c s a statoary pot of ^f ;K. Secod, sce f ^f ;K j g jˆ1;... s mootocally creasg,te mea sft teratos satsfy te codtos requred by te Capture Teorem [4,p. 45],wc states tat te trajectores of suc gradet metods are attracted by local maxma f tey are uque wt a small egborood) statoary pots. Tat s,oce y j gets suffcetly close to a mode of ^f ;K,t coverges to t. Te set of all locatos tat coverge to te same mode defes te bas of attracto of tat mode. Te teoretcal observatos from above suggest a practcal algortm for mode detecto:. Ru te mea sft procedure to fd te statoary pots of ^f ;K,. Prue tese pots by retag oly te local maxma. Te local maxma pots are defed,accordg to te Capture Teorem,as uque statoary pots wt some small ope spere. Ts property ca be tested by perturbg eac statoary pot by a radom vector of small orm ad lettg te mea sft procedure coverge aga. Sould te pot of covergece be ucaged up to a tolerace),te pot s a local maxmum..4 Smoot Trajectory Property Te mea sft procedure employg a ormal kerel as a terestg property. Its pat toward te mode follows a smoot trajectory,te agle betwee two cosecutve mea sft vectors beg always less ta 90 degrees. Usg te ormal kerel 10),te jt mea sft vector s gve by P ˆ1 x exp x x m ;N y j ˆy j 1 y j ˆ P ˆ1 exp x x y : 4 j Te followg teorem olds true for all j ˆ 1; ;..., accordg to te proof gve te Appedx.

5 COMANICIU AND MEER: MEAN SHIFT: A ROBUST APPROACH TOWARD FEATURE SPACE ANALYSIS 607 Teorem. Te cose of te agle betwee two cosecutve mea sft vectors s strctly postve we a ormal kerel s employed,.e., m ;N y j > m ;N y j 1 km ;N y j kkm ;N y j 1 k > 0: 5 As a cosequece of Teorem,te ormal kerel appears to be te optmal oe for te mea sft procedure. Te smoot trajectory of te mea sft procedure s cotrast wt te stadard steepest ascet metod [4,p. 1] local gradet evaluato followed by le maxmzato) wose covergece rate o surfaces wt deep arrow valleys s slow due to ts zgzaggg trajectory. I practce,te covergece of te mea sft procedure based o te ormal kerel requres large umber of steps, as was dscussed at te ed of Secto.. Terefore, most of our expermets,we ave used te uform kerel, for wc te covergece s fte,ad ot te ormal kerel. Note,owever,tat te qualty of te results almost always mproves we te ormal kerel s employed..5 Relato to Kerel Regresso Importat sgt ca be gaed we 19) s obtaed approacg te problem dfferetly. Cosderg te uvarate case suffces for ts purpose. Kerel regresso s a oparametrc metod to estmate complex treds from osy data. See [6,capter 5] for a troducto to te topc,[4] for a more -dept treatmet. Let measured data pots be X ;Z ad assume tat te values X are te outcomes of a radom varable x wt probablty desty fucto f x, x ˆ X ;ˆ 1;...;, wle te relato betwee Z ad X s Z ˆ m X ˆ 1;...;; 6 were m x s called te regresso fucto ad s a depedetly dstrbuted,zero-mea error,e Šˆ0. A atural way to estmate te regresso fucto s by locally fttg a degree p polyomal to te data. For a wdow cetered at x,te polyomal coeffcets te ca be obtaed by wegted least squares,te wegts beg computed from a symmetrc fucto g x. Te sze of te wdow s cotrolled by te parameter, g x ˆ 1 g x=. Te smplest case s tat of fttg a costat to te data te wdow,.e.,p ˆ 0.It ca be sow,[4,secto 3.1],[6,Secto 5.],tat te estmated costat s te value of te Nadaraya- Watso estmator, P ˆ1 ^m x; ˆ g x X Z P ˆ1 g x X ; 7 troduced te statstcal lterature 35 years ago. Te asymptotc codtoal bas of te estmator as te expresso [4,p. 109],[6,p. 15], E ^m x; m x j X 1 ;...;X Š m00 x f x m 0 x f 0 x gš; f x 8 were gš ˆR u g u du. Defg m x ˆx reduces te Nadaraya-Watso estmator to 0) te uvarate case), wle 8) becomes E ^x x jx 1 ;...;X Š f 0 x f x gš ; 9 wc s smlar to 19). Te mea sft procedure tus explots to ts advatage te eret bas of te zero-order kerel regresso. Te coecto to te kerel regresso lterature opes may terestg ssues,owever,most of tese are more of a teoretcal ta practcal mportace..6 Relato to Locato M-Estmators Te M-estmators are a famly of robust tecques wc ca adle data te presece of severe cotamatos,.e., outlers. See [6],[3] for troductory surveys. I our cotext oly,te problem of locato estmato as to be cosdered. Gve te data x ;ˆ 1;...;; ad te scale,wll defe ^^,te locato estmator as ^^ ˆ argm X J ˆargm ˆ1 x! ; 30 were, u s a symmetrc,oegatve valued fucto, wt a uque mmum at te org ad odecreasg for u 0. Te estmator s obtaed from te ormal equatos 0 1 ^^ r J ^^ ˆ x ^^ x w@ A ˆ 0; 31 were w u ˆd u du : Terefore,te teratos to fd te locato M-estmate are based o P ˆ1 x w ^^ x ^^ ˆ P ; 3 ˆ1 w ^^ x wc s detcal to 0) we w u g u. Takg to accout 13),te mmzato 30) becomes X ^^ x! ˆ argmax k ; 33 ˆ1 wc ca also be terpreted as ^^ ˆ argmax ^f ;K j x 1 ;...; x : 34 Tat s,te locato estmator s te mode of te desty estmated wt te kerel K from te avalable data. Note tat te covexty of te k x profle,te suffcet codto for te covergece of te mea sft procedure Secto.) s accordace wt te requremets to be satsfed by te objectve fucto u. Te relato betwee locato M-estmators ad kerel desty estmato s ot well-vestgated te statstcal lterature,oly [9] dscusses t te cotext of a edge preservg smootg tecque.

6 608 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 4, NO. 5, MAY 00 3 ROBUST ANALYSIS OF FEATURE SPACES Multmodalty ad arbtrarly saped clusters are te defg propertes of a real feature space. Te qualty of te mea sft procedure to move toward te mode peak) of te ll o wc t was tated makes t te deal computatoal module to aalyze suc spaces. To detect all te sgfcat modes,te basc algortm gve Secto.3 sould be ru multple tmes evolvg prcple parallel) wt talzatos tat cover te etre feature space. Before te aalyss s performed,two mportat ad somewat related) ssues sould be addressed: te metrc of te feature space ad te sape of te kerel. Te mappg from te put doma to a feature space ofte assocates a o-eucldea metrc to te space. Te problem of color represetato wll be dscussed Secto 4,but te employed parameterzato as to be carefully examed eve a smple case lke te Houg space of les,e.g., [48],[61]. Te presece of a Maalaobs metrc ca be accommodated by a adequate coce of te badwdt matrx ). I practce,owever,t s preferable to ave assured tat te metrc of te feature space s Eucldea ad,tus,te badwdt matrx s cotrolled by a sgle parameter, H ˆ I. To be able to use te same kerel sze for all te mea sft procedures te feature space,te ecessary codto s tat local desty varatos ear a sgfcat mode are ot as large as te etre support of a sgfcat mode somewere else. Te startg pots of te mea sft procedures sould be cose to ave te etre feature space except te very sparse regos) tessellated by te kerels wdows). Regular tessellatos are ot requred. As te wdows evolve toward te modes,almost all te data pots are vsted ad,tus,all te formato captured te feature space s exploted. Note tat te covergece to a gve mode may yeld slgtly dfferet locatos due to te tresold tat termates te teratos. Smlarly,o flat plateaus,te value of te gradet s close to zero ad te mea sft procedure could stop. Tese artfacts are easy to elmate troug postprocessg. Mode caddates at a dstace less ta te kerel badwdt are fused,te oe correspodg to te gest desty beg cose. Te global structure of te feature space ca be cofrmed by measurg te sgfcace of te valleys defed alog a cut troug te desty te drecto determed by two modes. Te deleato of te clusters s a atural outcome of te mode seekg process. After covergece,te bas of attracto of a mode,.e.,te data pots vsted by all te mea sft procedures covergg to tat mode,automatcally deleates a cluster of arbtrary sape. Close to te boudares,were a data pot could ave bee vsted by several dvergg procedures,majorty logc ca be employed. It s mportat to otce tat, computer vso, most ofte we are ot dealg wt a abstract clusterg problem. Te put doma almost always provdes a depedet test for te valdty of local decsos te feature space. Tat s,wle t s less lkely tat oe ca recover from a severe clusterg error,allocato of a few ucerta data pots ca be relably supported by put doma formato. Te multmodal feature space aalyss tecque was dscussed detal [1]. It was sow expermetally, tat for a sytetc,bmodal ormal dstrbuto,te tecque aceves a classfcato error smlar to te optmal Bayesa classfer. Te beavor of ts feature space aalyss tecque s llustrated Fg.. A twodmesoal data set of 110; 400 pots Fg. a) s decomposed to seve clusters represeted wt dfferet colors Fg. b. A umber of 159 mea sft procedures wt uform kerel were employed. Ter trajectores are sow Fg. c,overlapped over te desty estmate computed wt te Epaeckov kerel. Te prug of te mode caddates produced seve peaks. Observe tat some of te trajectores are prematurely stopped by local plateaus. 3.1 Badwdt Selecto Te fluece of te badwdt parameter was assessed emprcally [1] troug a smple mage segmetato task. I a more rgorous approac,owever,four dfferet tecques for badwdt selecto ca be cosdered.. Te frst oe as a statstcal motvato. Te optmal badwdt assocated wt te kerel desty estmator 6) s defed as te badwdt tat aceves te best compromse betwee te bas ad varace of te estmator,over all x R d,.e.,mmzes AMISE. I te multvarate case,te resultg badwdt formula [54,p. 85],[6,p. 99] s of lttle practcal use,sce t depeds o te Laplaca of te ukow desty beg estmated,ad ts performace s ot well uderstood [6,p. 108]. For te uvarate case,a relable metod for badwdt selecto s te plug- rule [53],wc was prove to be superor to leastsquares cross-valdato ad based cross-valdato [4],[55,p. 46]. Its oly assumpto s te smootess of te uderlyg desty.. Te secod badwdt selecto tecque s related to te stablty of te decomposto. Te badwdt s take as te ceter of te largest operatg rage over wc te same umber of clusters are obtaed for te gve data [0,p. 541].. For te trd tecque,te best badwdt maxmzes a objectve fucto tat expresses te qualty of te decomposto.e.,te dex of cluster valdty). Te objectve fucto typcally compares te ter- versus tra-cluster varablty [30],[8] or evaluates te solato ad coectvty of te deleated clusters [43].. Fally,sce most of te cases te decomposto s task depedet,top-dow formato provded by te user or by a upper-level module ca be used to cotrol te kerel badwdt. We preset [15],a detaled aalyss of te badwdt selecto problem. To solve te dffcultes geerated by te arrow peaks ad te tals of te uderlyg desty,two locally adaptve solutos are proposed. Oe s oparametrc,beg based o a ewly defed adaptve mea sft procedure,wc explots te plug- rule ad te sample pot desty estmator. Te oter s semparametrc, mposg a local structure o te data to extract relable scale formato. We sow tat te local badwdt sould maxmze te magtude of te ormalzed mea sft vector. Te adaptato of te badwdt provdes superor results we compared to te fxed badwdt procedure. For more detals,see [15].

7 COMANICIU AND MEER: MEAN SHIFT: A ROBUST APPROACH TOWARD FEATURE SPACE ANALYSIS 609 Fg.. Example of a D feature space aalyss. a) Two-dmesoal data set of 110; 400 pots represetgte frst two compoets of te L*u*v* space sow Fg. 1b. b) Decomposto obtaed by rug 159 mea sft procedures wt dfferet talzatos. c) Trajectores of te mea sft procedures draw over te Epaeckov desty estmate computed for te same data set. Te peaks retaed for te fal classfcato are marked wt red dots. 3. Implemetato Issues A effcet computato of te mea sft procedure frst requres te resamplg of te put data wt a regular grd. Ts s a stadard tecque te cotext of desty estmato wc leads to a bed estmator [6,Appedx D]. Te procedure s smlar to defg a stogram were lear terpolato s used to compute te wegts assocated wt te grd pots. Furter reducto te computato tme s aceved by employg algortms for multdmesoal rage searcg [5,p. 373] used to fd te data pots fallg te egborood of a gve kerel. For te effcet Eucldea dstace computato,we used te mproved absolute error equalty crtero,derved [39]. 4 APPLICATIONS Te feature space aalyss tecque troduced te prevous secto s applcato depedet ad,tus,ca be used to develop vso algortms for a wde varety of tasks. Two somewat related applcatos are dscussed te sequel: dscotuty preservg smootg ad mage segmetato. Te versatlty of te feature space aalyss eables te desg of algortms wc te user cotrols performace troug a sgle parameter,te resoluto of te aalyss.e.,badwdt of te kerel). Sce te cotrol parameter as clear pyscal meag,te ew algortms ca be easly tegrated to systems performg more complex tasks. Furtermore,bot gray level ad color mages are processed wt te same algortm, te former case,te feature space cotag two degeerate dmesos tat ave o effect o te mea sft procedure. Before proceedg to develop te ew algortms,te ssue of te employed color space as to be settled. To obta a meagful segmetato, perceved color dffereces sould correspod to Eucldea dstaces te color space cose to represet te features pxels). A Eucldea metrc,owever,s ot guarateed for a color space [65, Sectos 6.5.,8.4]. Te spaces L*u*v* ad L*a*b* were especally desged to best approxmate perceptually uform color spaces. I bot cases, L,te lgtess relatve brgtess) coordate,s defed te same way,te two spaces dffer oly troug te cromatcty coordates. Te depedece of all tree coordates o te tradtoal RGB color values s olear. See [46,Secto 3.5] for a readly accessble source for te coverso formulae. Te metrc of perceptually uform color spaces s dscussed

8 610 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 4, NO. 5, MAY 00 te cotext of feature represetato for mage segmetato [16]. I practce,tere s o clear advatage betwee usg L*u*v* or L*a*b*; te proposed algortms,we employed L*u*v* motvated by a lear mappg property [65,p.166]. Our frst mage segmetato algortm was a stragtforward applcato of te feature space aalyss tecque to a L*u*v* represetato of te color mage [11]. Te modularty of te segmetato algortm eabled ts tegrato by oter groups to a large varety of applcatos lke mage retreval [1],face trackg [6],object-based vdeo codg for MPEG-4 [],sape detecto ad recogto[33],ad texture aalyss [47],to meto oly a few. However,sce te feature space aalyss ca be appled ucaged to moderately ger dmesoal spaces see Secto 5),we subsequetly also corporated te spatal coordates of a pxel to ts feature space represetato. Ts jot doma represetato s employed te two algortms descrbed ere. A mage s typcally represeted as a two-dmesoal lattce of p-dmesoal vectors pxels),were p ˆ 1 te gray-level case,tree for color mages,ad p>3 te multspectral case. Te space of te lattce s kow as te spatal doma,wle te gray level,color,or spectral formato s represeted te rage doma. For bot domas,eucldea metrc s assumed. We te locato ad rage vectors are cocateated te jot spatal-rage doma of dmeso d ˆ p,ter dfferet ature as to be compesated by proper ormalzato. Tus,te multvarate kerel s defed as te product of two radally symmetrc kerels ad te Eucldea metrc allows a sgle badwdt parameter for eac doma K s; r x ˆ C x s! k x r! k ; 35 s p r were x s s te spatal part, x r s te rage part of a feature vector, k x te commo profle used bot two domas, s ad r te employed kerel badwdts,ad C te correspodg ormalzato costat. I practce,a Epaeckov or a trucated) ormal kerel always provdes satsfactory performace,so te user oly as to set te badwdt parameter ˆ s ; r,wc,by cotrollg te sze of te kerel,determes te resoluto of te mode detecto. 4.1 Dscotuty Preservg Smootg Smootg troug replacg te pxel te ceter of a wdow by te wegted) average of te pxels te wdow dscrmately blurs te mage,removg ot oly te ose but also salet formato. Dscotuty preservg smootg tecques,o te oter ad, adaptvely reduce te amout of smootg ear abrupt cages te local structure,.e.,edges. Tere are a large varety of approaces to aceve ts goal,from adaptve Weer flterg [31],to mplemetg sotropc [50] ad asotropc [44] local dffuso processes, a topc wc recetly receved reewed terest [19],[37], [56]. Te dffuso-based tecques,owever,do ot ave a stragtforward stoppg crtero ad,after a suffcetly large umber of teratos,te processed mage collapses to a flat surface. Te coecto betwee asotropc dffuso ad M-estmators s aalyzed [5]. s r A recetly proposed oteratve dscotuty preservg smootg tecque s te blateral flterg [59]. Te relato betwee blateral flterg ad dffuso-based tecques was aalyzed [3]. Te blateral flters also work te jot spatal-rage doma. Te data s depedetly wegted te two domas ad te ceter pxel s computed as te wegted average of te wdow. Te fudametal dfferece betwee te blateral flterg ad te mea sft-based smootg algortm s te use of local formato Mea Sft Flterg Let x ad z ;ˆ 1;...;,be te d-dmesoal put ad fltered mage pxels te jot spatal-rage doma. For eac pxel, 1. Italze j ˆ 1 ad y ;1 ˆ x.. Compute y ;j 1 accordg to 0) utl covergece, y ˆ y ;c. 3. Assg z ˆ x s ; yr ;c. Te superscrpts s ad r deote te spatal ad rage compoets of a vector,respectvely. Te assgmet specfes tat te fltered data at te spatal locato x s wll ave te rage compoet of te pot of covergece y r ;c. Te kerel wdow) te mea sft procedure moves te drecto of te maxmum crease te jot desty gradet,wle te blateral flterg uses a fxed,statc wdow. I te mage smooted by mea sft flterg, formato beyod te dvdual wdows s also take to accout. A mportat coecto betwee flterg te jot doma ad robust M-estmato sould be metoed. Te mproved performace of te geeralzed M-estmators GM or bouded-fluece estmators) s due to te presece of a secod wegt fucto wc offsets te fluece of leverage pots,.e.,outlers te put doma [3,Secto 8E]. A smlar at least sprt) twofold wegtg s employed te blateral ad mea sft-based fltergs,wc s te ma reaso for ter excellet smootg performace. Mea sft flterg wt uform kerel avg s ; r ˆ 8; 4 as bee appled to te ofte used gray-level camerama mage Fg. 3a),te result beg sow Fg. 3b. Te regos cotag te grass feld ave bee almost completely smooted,wle detals suc as te trpod ad te buldgs te backgroud were preserved. Te processg requred fractos of a secod o a stadard PC 600 Mz Petum III) usg a optmzed C++ mplemetato of te algortm. O te average, 3:06 teratos were ecessary utl te fltered value of a pxel was defed,.e.,ts mea sft procedure coverged. To better vsualze te flterg process,te 400 wdow marked Fg. 3a s represeted tree dmesos Fg. 4a. Note tat te data was reflected over te orzotal axs of te wdow for a more formatve dsplay. I Fg. 4b,te mea sft pats assocated wt every oter pxel bot coordates) from te plateau ad te le are sow. Note tat covergece pots black dots) are stuated te ceter of te plateau,away from te dscotutes deleatg t. Smlarly,te mea sft trajectores o te le rema o t. As a result,te fltered data Fg. 4c) sows clea quasomogeeous regos. Te pyscal terpretato of te mea sft-based flterg s easy to see by examg Fg. 4a,wc, fact, dsplays te tree dmesos of te jot doma of a

9 COMANICIU AND MEER: MEAN SHIFT: A ROBUST APPROACH TOWARD FEATURE SPACE ANALYSIS 611 Fg. 3. Camerama mage. a) Orgal. b) Mea sft fltered s ; r ˆ 8; 4. gray-level mage. Take a pxel o te le. Te uform kerel defes a parallelepped cetered o ts pxel ad te computato of te mea sft vector takes to accout oly tose pxels wc ave bot ter spatal coordates ad gray-level values sde te parallelepped. Tus,f te parallelepped s ot too large,oly pxels o te le are averaged ad te ew locato of te wdow s guarateed to rema o t. A secod flterg example s sow Fg. 5. Te 5151 color mage baboo was processed wt mea sft flters employg ormal kerels defed usg varous spatal ad rage resolutos, s ; r ˆ 8 3; Wle te texture of te fur as bee removed,te detals of te eyes ad te wskers remaed crsp up to a certa resoluto). Oe ca see tat te spatal badwdt as a dstct effect o te output we compared to te rage color) badwdt. Oly features wt large spatal support arerepreseted te fltered mage we s creases. O te oter ad,oly features wt g color cotrast survve we r s large. Smlar beavor was also reported for te blateral flter [59,Fg. 3]. 4. Image Segmetato Image segmetato,decomposto of a gray level or color mage to omogeeous tles,s arguably te most mportat low-level vso task. Homogeety s usually defed as smlarty pxel values,.e.,a pecewse costat model s eforced over te mage. From te dversty of mage segmetato metods proposed te lterature,we wll meto oly some wose basc processg reles o te jot doma. I eac case,a vector feld s defed over te samplg lattce of te mage. Fg. 4. Vsualzato of mea sft-based flterg ad segmetato for gray-level data. a) Iput. b) Mea sft pats for te pxels o te plateau ad o te le. Te black dots are te pots of covergece. c) Flterg result s ; r ˆ 8; 4. d) Segmetato result.

10 61 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 4, NO. 5, MAY 00 Fg. 5. Baboo mage. Orgal ad fltered. Te attracto force feld defed [57] s computed at eac pxel as a vector sum of parwse afftes betwee te curret pxel ad all oter pxels,wt smlarty measured bot spatal ad rage domas. Te rego boudares are te detfed as loc were te force vectors dverge. It s terestg to ote tat,for a gve pxel,te magtude ad oretato of te force feld are smlar to tose of te jot doma mea sft vector computed at tat pxel ad projected to te spatal doma. However, cotrast to [57],te mea sft procedure moves te drecto of ts vector,away from te boudares. Te edge flow [34] s obtaed at eac locato for a gve set of drectos as te magtude of te gradet of a smooted mage. Te boudares are detected at mage locatos wc ecouter two opposte drectos of flow. Te quatzato of te edge flow drecto,owever,may troduce artfacts. Recall tat te drecto of te mea sft s dctated solely by te data. Te mea sft procedure-based mage segmetato s a stragtforward exteso of te dscotuty preservg smootg algortm. Eac pxel s assocated wt a sgfcat mode of te jot doma desty located ts egborood,after earby modes were prued as te geerc feature space aalyss tecque Secto 3) Mea Sft Segmetato Let x ad z ;ˆ 1;...;,be te d-dmesoal put ad fltered mage pxels te jot spatal-rage doma ad L te label of te t pxel te segmeted mage. 1. Ru te mea sft flterg procedure for te mage ad store all te formato about te d-dmesoal covergece pot z,.e.,z ˆ y ;c.. Deleate te jot doma te clusters C p pˆ1...m by groupg togeter all z wc are closer ta s te spatal doma ad r te rage doma,.e.,cocateate te bass of attracto of te correspodg covergece pots. 3. For eac ˆ 1;...;,assg L ˆfp j z C p g. 4. Optoal: Elmate spatal regos cotag less ta M pxels. Te cluster deleato step ca be refed accordg to a pror formato ad,tus,pyscs-based segmetato algortms,e.g.,[],[35],ca be corporated. Sce ts process s performed o rego adjacecy graps,erarccal tecques lke [36] ca provde sgfcat speed-up. Te effect of te cluster deleato step s sow Fg. 4d. Note te fuso to larger omogeeous regos of te

11 COMANICIU AND MEER: MEAN SHIFT: A ROBUST APPROACH TOWARD FEATURE SPACE ANALYSIS 613 Fg. 6. MIT mage. a) Orgal. b) Segmeted s ; r ;M ˆ 8; 7; 0. c) Rego boudares. Fg. 7. Room mage. a) Orgal. b) Rego boudares deleated wt s ; r ;M ˆ 8; 5; 0, draw over te put. result of flterg sow Fg. 4c. Te segmetato step does ot add a sgfcat overead to te flterg process. Te rego represetato used by te mea sft segmetato s smlar to te blob represetato employed [64]. However,wle te blob as a parametrc descrpto multvarate Gaussas bot spatal ad color doma),te partto geerated by te mea sft s caracterzed by a oparametrc model. A mage rego s defed by all te pxels assocated wt te same mode te jot doma. I [43],a oparametrc clusterg metod s descrbed wc,after kerel desty estmato wt a small badwdt,te clusters are deleated troug cocateato of te detected modes' egboroods. Te mergg process s based o two tutve measures capturg te varatos te local desty. Beg a erarccal clusterg tecque, te metod s computatoally expesve; t takes several mutes MATLAB to aalyze a,000 pxel subsample of te feature space. Te metod s ot recommeded to be used te jot doma sce te measures employed te mergg process become effectve. Comparg te results for arbtrarly saped sytetc data [43,Fg. 6] wt a smlarly callegg example processed wt te mea sft metod [1,Fg. 1] sows tat te use of a erarccal approac ca be successfully avoded te oparametrc clusterg paradgm. All te segmetato expermets were performed usg uform kerels. Te mprovemet due to jot space aalyss ca be see Fg. 6 were te graylevel mage MIT was processed wt s ; r ;M ˆ 8; 7; 0. A umber of 5 omogeeous regos were detfed fractos of a secod,most of tem deleatg sematcally meagful regos lke walls,sky,steps,scrpto o te buldg,etc. Compare te results wt te segmetato obtaed by oe-dmesoal clusterg of te gray-level values [11,Fg. 4] or by usg a Gbbs radom feldsbased approac [40,Fg. 7]. Te jot doma segmetato of te color room mage preseted Fg. 7 s also satsfactory. Compare ts result wt te segmetato preseted [38,Fgs. 3e ad 5c] obtaed by recursve tresoldg. I bot tese examples, oe ca otce tat regos wc a small gradet of llumato exsts lke te sky te MIT or te carpet te room mage) were deleated as a sgle rego. Tus,te jot doma mea sft-based segmetato succeeds overcomg te eret lmtatos of metods based oly o gray-level or color clusterg wc typcally oversegmet small gradet regos. Te segmetato wt s ; r ;M ˆ 16; 7; 40 of te color mage lake s sow Fg. 8. Compare ts result wt tat of te multscale approac [57,Fg. 11]. Fally,oe ca compare te cotours of te color mage s ; r ;M ˆ 16; 19; 40 ad preseted Fg. 9 wt tose from [66,Fg. 15],obtaed troug a complex global optmzato,ad from [41,Fg. 4a],obtaed wt geodesc actve cotours. Te segmetato s ot very sestve to te coce of te resoluto parameters s ad r. Note tat all mages used te same s ˆ 8,correspodg to a

12 614 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 4, NO. 5, MAY 00 Fg. 8. Lake mage. a) Orgal. b) Segmeted wt s ; r ;M ˆ 16; 7; 40. Fg. 9. Had mage. a) Orgal. b) Rego boudares deleated wt s ; r ;M ˆ 16; 19; 40 draw over te put spatal wdow,wle all mages used s ˆ 16 correspodg to a wdow. Te rage parameter r ad te smallest sgfcat feature sze M cotrol te umber of regos te segmeted mage. Te more a mage devates from te assumed pecewse costat model,larger values ave to be used for r ad M to dscard te effect of small local varatos te feature space. For example,te eavly textured backgroud te ad mage s compesated by usg r ˆ 19 ad M ˆ 40,values wc are muc larger ta tose used for te room mage r ˆ 5;M ˆ 0 sce te latter better obeys te model. As wt ay low-level vso algortm,te qualty of te segmetato output ca be assessed oly te cotext of te wole vso task ad,tus,te resoluto parameters sould be cose accordg to tat crtero. A mportat advatage of mea sft-based segmetato s ts modularty wc makes te cotrol of segmetato output very smple. Oter segmetato examples wc te orgal mage as te rego boudares superposed are sow Fg. 10 ad wc te orgal ad labeled mages are compared Fg. 11. As a potetal applcato of te segmetato,we retur to te camerama mage. Fg. 1a sows te recostructed mage after te regos correspodg to te sky ad grass were maually replaced wt wte. Te mea sft segmetato as bee appled wt s ; r ;M ˆ 8; 4; 10. Observe te preservato of te detals wc suggests tat te algortm ca also be used for mage edtg,as sow Fg. 1b. Te code for te dscotuty preservg smootg ad mage segmetato algortms tegrated to a sgle system wt grapcal terface s avalable at ttp:// 5 DISCUSSION Te mea sft-based feature space aalyss tecque troduced ts paper s a geeral tool wc s ot restrcted to te two applcatos dscussed ere. Sce te qualty of te output s cotrolled oly by te kerel badwdt,.e.,te resoluto of te aalyss,te tecque sould be also easly tegrable to complex vso systems were te cotrol s relqused to a closed loop process. Addtoal sgts o te badwdt selecto ca be obtaed by testg te stablty of te mea sft drecto across te dfferet badwdts,as vestgated [57] te case of te force feld. Te oparametrc toolbox developed ts paper s sutable for a large varety of computer vso tasks were parametrc models are less adequate,for example,modelg te backgroud vsual survellace [18]. Te complete soluto toward autoomous mage segmetato s to combe a badwdt selecto tecque lke te oes dscussed Secto 3.1) wt top-dow taskrelated g-level formato. I ts case,eac mea sft process s assocated wt a kerel best suted to te local structure of te jot doma. Several terestg teoretcal

13 COMANICIU AND MEER: MEAN SHIFT: A ROBUST APPROACH TOWARD FEATURE SPACE ANALYSIS 615 Fg. 10. Ladscape mages. All te rego boudares were deleated wt s ; r ;M ˆ 8; 7; 100 ad are draw over te orgal mage. ssues ave to be addressed,toug,before te beefts of suc a data drve approac ca be fully exploted. We are curretly vestgatg tese ssues. Te ablty of te mea sft procedure to be attracted by te modes local maxma) of a uderlyg desty fucto, ca be exploted a optmzato framework. Ceg [7] already dscusses a smple example. However,by troducg adequate objectve fuctos,te optmzato problem ca acqure pyscal meag te cotext of a computer vso task. For example, [14],by defg te dstace betwee te dstrbutos of te model ad a caddate of te target,orgd objects were tracked a mage sequece uder severe dstortos. Te dstace was defed at every pxel te rego of terest of te ew frame ad te mea sft procedure was used to fd te mode of ts measure earest to te prevous locato of te target. Te above-metoed trackg algortm ca be regarded as a example of computer vso tecques wc are based o stu optmzato. Uder ts paradgm,te soluto s obtaed by usg te put doma to defe te optmzato problem. Te stu optmzato s a very powerful metod. I [3] ad [58],eac put data pot was assocated wt a local feld votg kerel) to produce a more dese structure from were te sougt formato salet features,te yperplae represetg te fudametal matrx) ca be relably extracted. Te mea sft procedure s ot computatoally expesve. Careful C++ mplemetato of te trackg algortm allowed real tme 30 frames/secod) processg of te vdeo stream. Wle t s ot clear f te segmetato algortm descrbed tspapercabemadesofast,gve tequaltyof te rego boudares t provdes,t ca be used to support edge detecto wtout sgfcat overead tme. Kerel desty estmato, partcular,ad oparametrc tecques, geeral,do ot scale well wt te dmeso of te space. Ts s mostly due to te empty space peomeo[0,p. 70],[54,p. 93] by wc most of te mass a g-dmesoal space s cocetrated a small rego of te space. Tus,weever te feature space as more ta say) sx dmesos,te aalyss sould be approaced carefully. Employg projecto pursut, wc te desty s aalyzed alog lower dmesoal cuts,e.g.,[7],s a possblty. To coclude,te mea sft procedure s a valuable computatoal module wose versatlty ca make t a mportat compoet of ay computer vso toolbox. APPENDIX Proof of Teorem 1. If te kerel K as a covex ad mootocally decreasg profle, te sequeces fy j g jˆ1;... ad f ^f ;K j g jˆ1;... coverge, ad f ^f ;K j g jˆ1;... s mootocally creasg. Sce s fte,te sequece ^f ;K 1) s bouded, terefore,t s suffcet to sow tat ^f;k s strctly mootoc creasg,.e.,f y j 6ˆ y j 1,te ^f ;K j < ^f ;K j 1 ; for j ˆ 1;... Wtout loss of geeralty,t ca be assumed tat y j ˆ 0 ad,tus,from 16) ad 1) ^f ;K j 1 ^f ;K j ˆ c k;d X y j 1 x k d ˆ1 k x Te covexty of te profle k x mples tat k x k x 1 k 0 x 1 x x 1 : A:1 A: for all x 1 ;x 0; 1, x 1 6ˆ x,ad sce g x ˆ k 0 x, A.) becomes k x k x 1 g x 1 x 1 x : A:3

14 616 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 4, NO. 5, MAY 00 Fg. 11. Some oter segmetato examples wt s ; r ;M ˆ 8; 7; 0. Left: orgal. Rgt: segmeted. Now,usg A.1) ad A.3),we obta ^f ;K j 1 ^f ;K j c X k;d x d g kx k ky j 1 x k ˆ1 ˆ ck;d X x d g y > j 1 x ky j 1 k " ˆ1 ˆ ck;d X y > x d j 1 x g ky j 1 k X x # g ˆ1 ˆ1 A:4 ad,recallg 0),yelds ^f ;K j 1 ^f ;K j c k;d d ky j 1k X ˆ1 x g : A:5 Te profle k x beg mootocally decreasg for all x 0,te sum P ˆ1 g x s strctly postve. Tus,as log as y j 1 6ˆ y j ˆ 0,te rgt term of A.5) s strctly postve,.e., ^f ;K j 1 > ^f ;K j. Cosequetly,te sequece f ^f ;K j g jˆ1;... s coverget. To prove te covergece of te sequece fy j g jˆ1;..., A.5) s rewrtte for a arbtrary kerel locato y j 6ˆ 0. After some algebra,we ave

15 COMANICIU AND MEER: MEAN SHIFT: A ROBUST APPROACH TOWARD FEATURE SPACE ANALYSIS 617 Fg. 1. Camerama mage. a) Segmetato wt s ; r ;M ˆ 8; 4; 10 ad recostructo after te elmato of regos represetg sky ad grass. b) Supervsed texture serto. ^f ;K j 1 ^f ;K j c k;d d ky j 1 y j k X y j x g : ˆ1 A:6 Now,summg te two terms of A.6) for dces j; j 1...j m 1,t results tat ^f ;K j m ^f ;K j c k;d d ky j m y j m 1 k X y j m 1 x g... ˆ1 c k;d ky d j 1 y j k X y j x g ˆ1 c k;d ky d j m y j m 1 k... ky j 1 y j k M c k;d ky d j m y j k M; A:7 were M represets te mmum always strctly postve) of te sum P ˆ1 g y j x for all fy j g jˆ1;.... Sce f ^f ;K j g jˆ1;... s coverget,t s also a Caucy sequece. Ts property cojucto wt A.7) mples tat fy j g jˆ1;... s a Caucy sequece,ece,t s coverget te Eucldea space. tu Proof of Teorem. Te cose of te agle betwee two cosecutve mea sft vectors s strctly postve we a ormal kerel s employed. We ca assume,wtout loss of geeralty tat y j ˆ 0 ad y j 1 6ˆ y j 6ˆ 0 sce,oterwse,covergece as already bee aceved. Terefore,te mea sft vector m ;N 0 s P ˆ1 x exp x m ;N 0 ˆy j 1 ˆ x exp : B:1 P ˆ1 We wll sow frst tat,we te wegts are gve by a ormal kerel cetered at y j 1,te wegted sum of te projectos of y j 1 x oto yj 1 s strctly egatve,.e., X ky j 1 k y > j 1 x exp y j 1 x < 0: B: ˆ1 Te space R d ca be decomposed to te followg tree domas: D 1 ˆ x R d y > j 1 x 1 ky j 1k D ˆ x R d 1 ky j 1k < y > j 1 x ky j 1k B:3 o D 3 ˆ x R d ky j 1 k < y > j 1 x ad after some smple mapulatos from B.1),we ca derve te equalty X ky j 1 k y > j 1 x exp x x D ˆ X y > j 1 x ky j 1 k exp x B:4 : x D 1[D 3 I addto,for x D,we ave ky j 1 k y > j 1 x 0, wc mples ky j 1 x k ˆky j 1 k kx k y > j 1 x kx k ky j 1 k B:5 from were X ky j 1 k y > j 1 x exp y j 1 x y j 1 exp X ky j 1 k y > j 1 x exp x : x D B:6 x D Now,troducg B.4) B.6),we ave X ky j 1 k y > j 1 x exp y j 1 x x D y j 1 exp X y > j 1 x ky j 1 k exp x x D 1[D 3 B:7

16 618 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 4, NO. 5, MAY 00 ad,by addg to bot sdes of B.7),te quatty X ky j 1 k y > j 1 x exp y j 1 x ; x D 1 [D 3 after some algebra,t results tat X ky j 1 k y > j 1 x exp y j 1 x ˆ1 y j 1 exp X ky j 1 k y > j 1 x exp x x D 1 [D 3 exp ky j 1 k y > j 1 x 1 : Te rgt sde of B.8) s egatve because ky j 1 k y > j 1 x B:8 ad te last product term as opposte sgs bot te D 1 ad D 3 domas. Terefore,te left sde of B.8) s also egatve,wc proves B.). We ca use ow B.) to wrte P ky j 1 k < y > ˆ1 x exp y j 1 x j 1 P ˆ1 exp y j 1 x ˆ y > j 1yj ; B:9 from were y > j 1 y j y j 1 ky j 1 kky j y j 1 k > 0 or by takg to accout 4) ACKNOWLEDGMENTS m ;N y j > m ;N y j 1 km ;N y j kkm ;N y j 1 k > 0: B:10 Te support of te US Natoal Scece Foudato uder grats IRI ad IRI s gratefully ackowledged. Prelmary versos for some parts of te materal were preseted [13] ad [14]. Te autors would lke to tak Jo Ket from te Uversty of Leeds ad Davd Tyler of Rutgers for dscussos about te relato betwee te mea sft procedure ad M-estmators. REFERENCES [1] G. Aggarwal,S. Gosal,ad P. Dubey,ªEffcet Query Modfcato for Image Retreval,º Proc. 000 IEEE Cof. Computer Vso ad Patter Recogto, vol. II,pp , Jue 000. [] R. Bajcsy,S.W. Lee,ad A. Leoards,ªDetecto of Dffuse ad Specular Iterface Reflectos ad Iter-Reflectos by Color Image Segmetato,º It'l J. Computer Vso, vol. 17,pp. 41-7,1996. [3] D. Baras,ªBlateral Flterg ad Asotropc Dffuso: Towards a Ufed Vewpot,º IEEE Tras. Patter Aalyss ad Patter Aalyss, to appear. tu [4] D.P. Bertsekas, Nolear Programmg. Atea Scetfc,1995. [5] M.J. Black,G. Sapro,D.H. Marmot,ad D. Heeger,ªRobust Asotropc Dffuso,º IEEE Tras. Image Processg, vol. 7,pp ,1998. [6] G.R. Bradsk,ªComputer Vso Face Trackg as a Compoet of a Perceptual User Iterface,º Proc. IEEE Worksop Applcatos of Computer Vso, pp ,Oct [7] Y. Ceg,ªMea Sft,Mode Seekg,ad Clusterg,º IEEE Tras. Patter Aalyss ad Mace Itellgece, vol. 17,o. 8, pp ,Aug [8] E. Co ad P. Hall,ªData Sarpeg as a Prelude to Desty Estmato,º Bometrka, vol. 86,pp ,1999. [9] C.K. Cu,I.K. Glad,F. Godtlebse,ad J.S. Maro,ªEdge- Preservg Smooters for Image Processg,º J. Am. Statstcal Assoc., vol. 93,pp ,1998. [10] D. Comacu,ªNoparametrc Robust Metods for Computer Vso,º PD tess, Dept. of Electrcal ad Computer Eg., Rutgers Uv.,1999. Avalable at ttp:// rul/researc/teses.tml. [11] D. Comacu ad P. Meer,ªRobust Aalyss of Feature Spaces: Color Image Segmetato,º Proc IEEE Cof. Computer Vso ad Patter Recogto, pp ,Jue [1] D. Comacu ad P. Meer,ªDstrbuto Free Decomposto of Multvarate Data,º Patter Aalyss ad Applcatos, vol.,pp. - 30,1999. [13] D. Comacu ad P. Meer,ªMea Sft Aalyss ad Applcatos,º Proc. Sevet It'l Cof. Computer Vso, pp ,Sept [14] D. Comacu,V. Rames,ad P. Meer,ªReal-Tme Trackg of No-Rgd Objects Usg Mea Sft,º Proc. 000 IEEE Cof. Computer Vso ad Patter Recogto, vol. II,pp ,Jue 000. [15] D. Comacu,V. Rames,ad P. Meer,ªTe Varable Badwdt Mea Sft ad Data-Drve Scale Selecto,º Proc Egt It'l Cof. Computer Vso, vol. I,pp ,July 001. [16] C. Coolly,ªTe Relatosp betwee Colour Metrcs ad te Appearace of Tree-Dmesoal Coloured Objects,º Color Researc ad Applcatos, vol. 1,pp ,1996. [17] R.O. Duda ad P.E. Hart, Patter Classfcato ad Scee Aalyss. Wley,1973. [18] A. Elgammal,D. Harwood,ad L. Davs,ªNo-Parametrc Model for Backgroud Subtracto,º Proc. Sxt Europea Cof. Computer Vso, vol. II,pp ,Jue 000. [19] B. Fscl ad E.L. Scwartz,ªAdaptve Nolocal Flterg: A Fast Alteratve to Asotropc Dffuso for Image Eacemet,º IEEE Tras. Patter Aalyss ad Mace Itellgece, vol. 1,o. 1, pp. 4-48,Ja [0] K. Fukuaga, Itroducto to Statstcal Patter Recogto, secod ed. Academc Press,1990. [1] K. Fukuaga ad L.D. Hostetler,ªTe Estmato of te Gradet of a Desty Fucto,wt Applcatos Patter Recogto,º IEEE Tras. Iformato Teory, vol. 1,pp. 3-40,1975. [] J. Guo,J. Km,ad C. Kuo,ªFast ad Accurate Movg Object Extracto Tecque for MPEG-4 Object Based Vdeo Codg,º Proc. SPIE Vsual Comm. ad Image Processg, vol. 3653,pp ,1999. [3] G. Guy ad G. Medo,ªIferece of Surfaces,3D Curves,ad Juctos from Sparse,Nosy,3D Data,º IEEE Tras. Patter Aalyss ad Mace Itellgece, vol. 19,pp ,1997. [4] W. HaÈrdle, Appled Noparameterc Regresso. Cambrdge Uv. Press,1991. [5] M. Herb,N. Boet,ad P. Vautrot,ªA Clusterg Metod Based o te Estmato of te Probablty Desty Fucto ad o te Skeleto by Ifluece Zoes,º Patter Recogto Letters, vol. 17,pp ,1996. [6] P.J. Huber, Robust Statstcal Procedures, secod ed. SIAM,1996. [7] J.N. Hwag ad S.R. Lay,ad A. Lppma,ªNoparametrc Multvarate Desty Estmato: A Comparatve Study,º IEEE Tras. Sgal Processg, vol. 4,pp ,1994. [8] A.K. Ja ad R.C. Dubes, Algortms for Clusterg Data. Pretce Hall,1988. [9] A.K. Ja,R.P.W. Du,ad J. Mao,ªStatstcal Patter Recogto: A Revew,º IEEE Tras. Patter Aalyss ad Mace Itellgece, vol.,o. 1,pp. 4-37,Ja [30] L. Kauffma ad P. Rousseeuw, Fdg Groups Data: A Itroducto to Cluster Aalyss. J. Wley & Sos,1990.

17 COMANICIU AND MEER: MEAN SHIFT: A ROBUST APPROACH TOWARD FEATURE SPACE ANALYSIS 619 [31] D.T. Kua,A.A. Sawcuk,T.C. Strad,ad P. Cavel,ªAdaptve Nose Smootg Flter for Images wt Sgal Depedet Nose,º IEEE Tras. Patter Aalyss ad Mace Itellgece, vol. 7,o., pp ,Mar [3] G. L,ªRobust Regresso,º Explorg Data Tables, Treds, ad Sapes, D.C. Hoagl,F. Mosteller,ad J.W. Tukey,eds.,pp Wley, [33] L. Lu ad S. Sclaroff,ªDeformable Sape Detecto ad Descrpto va Model-Based Rego Groupg,º Proc IEEE Cof. Computer Vso ad Patter Recogto, vol. II,pp. 1-7, Jue [34] W.Y. Ma ad B.S. Majuat,ªEdge flow: A Framework of Boudary Detecto ad Image Segmetato,º IEEE Tras. Image Processg, vol. 9,pp ,000. [35] B.A. Maxwell ad S.S. Safer,ªSegmetato ad Iterpretato of Multcolored Objects wt Hglgts,º Computer Vso ad Image Uderstadg, vol. 77,pp. 1-4,000. [36] A. Motavert,P. Meer,ad A. Rosefeld,ªHerarccal Image Aalyss Usg Irregular Tessellato,º IEEE Tras. Patter Aalyss ad Mace Itellgece, vol. 13,o. 4,pp ,Apr [37] J. Motel ad A. Begdad,ªA New Iterpretato ad Improvemet of Nolear Asotropc Dffuso for Image Eacemet,º IEEE Tras. Patter Aalyss ad Mace Itellgece, vol. 1,o. 9,pp ,Sept [38] Y. Ota,T. Kaade,ad T. Saka,ªColor Iformato for Rego Segmetato,º Computer Grapcs ad Image Processg, vol. 13, pp. -41,1980. [39] J. Pa,F. McIes,ad M. Jack,ªFast Clusterg Algortms for Vector Quatzato,º Patter Recogto, vol. 9,pp ,1996. [40] T.N. Pappas,ªA Adaptve Clusterg Algortm for Image Segmetato,º IEEE Tras. Sgal Processg, vol. 40,pp , 199. [41] N. Paragos ad R. Derce,ªGeodesc Actve Cotours for Supervsed Texture Segmetato,º Proc IEEE Cof. Computer Vso ad Patter Recogto, vol. II,pp. 4-47,Jue [4] B. Park ad J. Marro,ªComparso of Data-Drve Badwdt Selectors,º J. Am. Statstcal Assoc., vol. 85,pp. 66-7,1990. [43] E.J. Pauwels ad G. Frederx,ªFdg Salet Regos Images,º Computer Vso ad Image Uderstadg, vol. 75,pp ,1999. [44] P. Peroa ad J. Malk,ªScale-Space ad Edge Detecto Usg Asotropc Dffuso,º IEEE Tras. Patter Aalyss ad Mace Itellgece, vol. 1,o. 7,pp ,July [45] K. Popat ad R.W. Pcard,ªCluster-Based Probablty Model ad Its Applcato to Image ad Texture Processg,º IEEE Tras. Image Processg, vol. 6,pp ,1997. [46] W.K. Pratt, Dgtal Image Processg, secod ed. Wley,1991. [47] D. Rdder,J. Kttler,O. Lemmers,ad R. Du,ªTe Adaptve Subspace Map for Texture Segmetato,º Proc. 000 It'l Cof. Patter Recogto, pp. 16-0,Sept [48] T. Rsse,ªHoug Trasform for Le Recogto: Complexty of Evdece Accumulato ad Cluster Detecto,º Computer Vso Grapcs ad Image Processg, vol. 46,pp ,1989. [49] S.J. Roberts,ªParametrc ad No-Parametrc Usupervsed Cluster Aalyss,º Patter Recogto, vol. 30,pp. 61-7,1997. [50] P. Sat-Marc ad J.S. Ce,G. Medo,ªAdaptve Smootg: A Geeral Tool for Early Vso,º IEEE Tras. Patter Aalyss ad Mace Itellgece, vol. 13,o. 6,pp ,Jue [51] D.W. Scott, Multvarate Desty Estmato. Wley,199. [5] R. Sedgewck, Algortms C++. Addso-Wesley,199. [53] S. Seater,M. Joes,ªA Relable Data-Based Badwdt Selecto Metod for Kerel Desty Estmato,º J. Royal Statstcs Soc. B, vol. 53,pp ,1991. [54] B. W. Slverma, Desty Estmato for Statstcs ad Data Aalyss. Capma ad Hall,1986. [55] J. Smooff, Smootg Metods Statstcs. Sprger-Verlag,1996. [56] Specal Issue o Partal Dfferetal Equatos ad Geometry- Drve Dffuso Image Processg ad Aalyss, IEEE Tras. Image Processg, vol. 7,Mar [57] M. Tabb ad N. Auja,ªMultscale Image Segmetato by Itegrated Edge ad Rego Detecto,º IEEE Tras. Image Processg, vol. 6,pp ,1997. [58] C.K. Tag,G. Medo,ad M.S. Lee,ªEppolar Geometry Estmato by Tesor Votg 8D,º Proc. Sevet It'l Cof. Computer Vso, vol. I,pp ,Sept [59] C. Tomas ad R. Maduc,ªBlateral Flterg for Gray ad Color Images,º Proc. Sxt It'l Cof. Computer Vso, pp , Ja [60] A. Touza ad J.G. Postare,ªClusterg by Mode Boudary Detecto,º Patter Recogto Letters, vol. 9,pp. 1-1,1989. [61] T. Tuytelaars,L. Va Gool,M. Proesmas,ad T. Moos,ªTe Cascaded Houg Trasform as a Ad Aeral Image Iterpretato,º Proc. Sxt It'l Cof. Computer Vso, pp. 67-7,Ja [6] M.P. Wad ad M. Joes, Kerel Smootg. Capma ad Hall, [63] R. Wlso ad M. Spa,ªA New Approac to Clusterg,º Patter Recogto, vol. 3,pp ,1990. [64] C. Wre,A. Azarbayeja,T. Darrell,ad A. Petlad,ªPfder: Real-Tme Trackg of te Huma Body,º IEEE Tras. Patter Aalyss ad Mace Itellgece, vol. 19,o. 7,pp ,July [65] G. Wyszeck ad W.S. Stles, Color Scece: Cocepts ad Metods, Quattatve Data ad Formulae. secod ed. Wley,198. [66] S.C. Zu ad A. Yulle,ªRego Competto: Ufyg Sakes, Rego Growg,ad Bayes/MDL for Multbad Image Segmetato,º IEEE Tras. Patter Aalyss ad Mace Itellgece, vol. 18,o. 9,pp ,Sept [67] X. Zuag,Y. Huag,K. Palaappa,ad Y. Zao,ªGaussa Mxture Desty Modelg Decomposto,ad Applcatos,º IEEE Tras. Image Processg, vol. 5,pp ,1996. Dor Comacu receved te Dpl. Eg. ad PD degrees electrocs from te Polytecc Uversty of Bucarest 1988 ad 1995 ad te PD degree electrcal egeerg from Rutgers Uversty From 1988 to 1990, e was wt ICE Felx Computers Bucarest. I 1991, e joed te Departmet of Electrocs ad Telecommucatos at te Polytecc Uversty of Bucarest ad e eld researc appotmets Germay ad Frace. From 1996 to 1999, e was wt te Ceter for Advaced Iformato Processgassocated wt Rutgers Uversty. Sce 1999, e as bee a member of te teccal staff at Semes Corporate Researc Prceto, New Jersey. Hs researc terests clude robust metods for autoomous computer vso, oparametrc aalyss, real-tme vso systems, vdeo survellace, cotet-based access to vsual data, ad data compresso. He as coautored umerous papers, coferece papers, ad book capters te area of vsual formato processg. He receved te Best Paper Award at te IEEE Coferece Computer Vso ad Patter Recogto 000. He s a member of te IEEE. Peter Meer receved te Dpl. Eg. degree from te Bucarest Polytecc Isttute, Bucarest, Romaa, 1971, ad te DSc degree from te Teco, Israel Isttute of Tecology, Hafa, Israel, 1986, bot electrcal egeerg. From 1971 to 1979, e was wt te Computer Researc Isttute, Cluj, Romaa, workgo researc ad developmet of dgtal ardware. From 1986 to 1990, e was a assstat researc scetst at te Ceter for Automato Researc, Uversty of Marylad at College Park. I 1991, e joed te Departmet of Electrcal ad Computer Egeerg, Rutgers Uversty, Pscataway, New Jersey, were e s curretly a assocate professor. He as eld vstgappotmets Japa, Korea, Swede, Israel ad Frace ad was o te orgazg commttees of umerous Iteratoal worksops ad cofereces. He s a assocate edtor of te IEEE Trasacto o Patter Aalyss ad Mace Itellgece, a member of te edtoral board of Patter Recogto, ad e was a guest edtor of Computer Vso ad Image Uderstadg for te Aprl 000 specal ssue o ªRobust Statstcal Tecques Image Uderstadg.º He s coautor of a award wg paper Patter Recogto 1989, te best studet paper 1999, ad te best paper te 000 IEEE Coferece Computer Vso ad Patter Recogto. Hs researc terest s applcato of moder statstcal metods to mage uderstadg problems. He s a seor member of te IEEE.. For more formato o ts or ay oter computg topc, please vst our Dgtal Lbrary at ttp://computer.org/publcatos/dlb.