Nonparametric Density Estimation on A Graph: Learning Framework, Fast Approximation and Application in Image Segmentation

Transcription

1 Nonparametrc Densty Estmaton on A Graph: Learnng Framewor, Fast Approxmaton and Applcaton n Image Segmentaton Zhdng Yu Oscar C. Au Ketan Tang Chunjng Xu Dept. of Electronc and Computer Engneerng Hong Kong Unversty of Scence and Technology {zdyu, eeau, tt}@ust.h Shenzhen Inst. of Advanced Technology Chnese Academy of Scences cj.xu@sat.ac.cn Abstract We present a novel framewor for tree-structure embedded densty estmaton and ts fast approxmaton for mode seeng. The proposed method could fnd dverse applcatons n computer vson and feature space analyss. Gven any undrected, connected and weghted graph, the densty functon s defned as a jont representaton of the feature space and the dstance doman on the graph s spannng tree. Snce the dstance doman of a tree s a constraned one, mode seeng can not be drectly acheved by tradtonal mean shft n both doman. we address ths problem by ntroducng node shftng wth force competton and ts fast approxmaton. Our wor s closely related to the prevous lterature of nonparametrc methods. One shall see, however, that the new formulaton of ths problem can lead to many advantages and new characterstcs n ts applcaton, as wll be llustrated later n ths paper. 1. Introducton Nonparametrc densty estmaton provdes a versatle tool for feature space analyss such as clusterng and local maxma detecton. The ratonale behnd, as ponted out by Comancu et al., s that feature space can be regarded as the emprcal probablty densty functon pdf) of the represented parameter. Fndng local estmated densty maxma or mode seeng) results n the computatonal module of mean shftv [1], an old pattern recognton technque. The robust nature of mean shft leads to wde applcatons n low level computer vson, ncludng edge preserved smoothng, mage segmentaton and object tracng. Recent wors tres to mprove ts performance by ntroducng asymmetrc based ernels n specfc tass, or sees to reduce ts complexty wth fast algorthms. Ths wor has been supported n part by the Research Grants Councl RGC) of the Hong Kong Specal Admnstratve Regon, Chna. GRF ) Fgure 1. Example of data clusterng usng the proposed mode seeng algorthm wth h 1 = 180 and =40. We nvestgate the problem of tree-structure embedded densty estmaton, provdng a novel angle loong nto ths problem. Our method ntroduces metrcs learned from a spannng tree nto mode seeng. In partcular, we adopt mnmum spannng tree MST) to learn compact structures n the feature space or on a connected graph. On one hand, the ncluson of MST helps to fnd manfold structures for feature space analyss and data clusterng. On the other hand, the graph-based attrbute wors compatbly wth regonal level mage operatons n computer vson. A wde range of computer vson problems n prncple requres regonal support, where relaton between mage regons are typcally depcted wth a weghted graph and graph-based methods have consequently become a powerful tool. Such characterstc offers several ntutonally reasonable advantages. Frst, regon-wse operaton allows one to nvestgate and desgn more versatle and powerful features, as a regon often contans much more nformaton than a sngle pxel. Second, adoptng regon as basc processng unt can largely allevate the computatonal burden. In the paper, we only llustrate the applcatons of our method n data clusterng and regon-based mage segmentaton, due to the lmt of page length. Fgure 1 shows one example of data clusterng usng our proposed method. The potental applcaton of ths algorthm, however, s consderable, as mode seeng has dverse applcatons. Ths paper s organzed as follows: In Secton, we brefly ntroduce the bacground and closely related wors. 01

2 Readers already famlar wth nonparametrc densty estmaton and mean shft may jump to Secton 3, where we descrbe the proposed method and dscuss ts mportant propertes. Some expermental results regardng clusterng and applcaton of our method n mage segmentaton are llustrated n Secton 4, showng that the method s an effectve one. Fnally, conclusons are made n the last Secton.. Bacground and related wors Gven a set of ndependent and dentcally dstrbuted data ponts, nonparametrc densty estmaton sees to approxmate ts pdf. Instead of representng the pdf by a sngle parametrc model or a mxture model, the method fnds a small number of nearest or most smlar) tranng nstances and nterpolate from them. To obtan smooth pdf estmaton, gaussan ernel s commonly utlzed as the ernel densty estmator, also nown as Parzen wndow. The paradgm of densty estmaton and clusterng ncludes a famly of mode seeng algorthms wth Parzen densty estmaton. More recently, several wors have explored the mprovement of tradtonal mean shft algorthm. In [], the author ntroduced asymmetrc ernel to mean shft object tracng. The scale and orentaton of the ernel s automatcally and adaptvely selected, dependng on the observatons at each teraton. In [3], A new mode seeng algorthm called the medod shft was proposed. The purpose of medod shft s to extend mode seeng to general metrc spaces. The method, however, requres huge computatonal load and tends to result n over-fragmentaton. It essentally becomes a fnte pont searchng problem and s qute dfferent from our method n terms of both purpose and algorthmc process. In [4], the authors proposed the quc shft algorthm whch s consderably faster than mean shft and medod shft. Ther emphass tends to concentrate on algorthm acceleraton whle preservng ts performance. The GPU mplementaton of quc shft was dscussed n [5] to further speed up the algorthm from the hardware perspectve. There has also been other wors tryng to mprove the effcency of mode seeng [8]. Consderng the nearest neghbor property of MST, our method to some extent are related to prevous wors that generalze mean shft to non-lnear manfolds [9], or ntroduce nonlnear ernelzed or manfold metrcs [3, 4]. Our method can acheve some smlar goals but the dea remans very dfferent. We also notce there exst a great many wors concernng MST based graph segmentatons [10]. Even though our method have also utlzed MST, we generally thn t belongs to the famly of mode seeng methods where the algorthm characterstcs are qute dfferent from many graph based segmentaton methods. Hence these methods may not fall wthn the scope of comparson n ths paper. In fact our wor presents a general framewor of embeddng tree structures nto the mode seeng process. Therefore t s straght forward for one to plug n many other trees and brng n addtonal algorthm characterstcs. 3. Graph-based densty estmaton We propose to perform densty estmaton on a jont doman represented by the node feature space and the dstance space defned by the mnmum spannng tree of that graph. There are several advantages operatng on an MST-based structure. Frst, tree-based structure helps to unquely defne dstances for any node par, as a tree does not have crcles. Of course, one could drectly defne the parwse node dstances n the Eucldean space, resultng n the tradtonal mean shft. But ths bascally dscards the structural nformaton preserved by a graph. In applcatons such as mage segmentaton, spatal nformaton preserved by a graph can be very mportant. Second, an MST s the connected graph structure where all nodes are connected wth least edges numbers and weghts. In other words, an MST can be regarded as a compact structure that preserves mportant nformaton about the cluster structure n a feature space. Although the ntroducton of a tree structure n practce could possbly be problematc - as t faces the rs of large tree structure varaton nduced by nose ponts, especally for those mportant tree roots - one shall see, the proposed method wors pretty well and robustly n real mage segmentaton tests. In addton, such formulaton helps to mprove mode seeng performancesfor many manfoldshaped clusters. There are several exstng methods extractng an MST. In ths paper, we adopt the Krusal s Algorthm to obtan the MST structure from the graph. We then defne the densty functon and descrbe ts mode seeng process n the followng part of ths secton Proposed densty estmator Gven N samples represented by the set V = {v = 1,...,N,v R d } and the undrected weghted graph G =V, E), the mnmum spannng tree S =V, E S ) s a connected graph of G wth E S E, E S = N 1. For any node par, j) where j, there exsts a unque path E j such that E j E S, and j s connected by E j and deletng any element of the set results n the dsconnecton of and j. In addton, we defne E j to be,f = j. Property 3.1 For any gven node par, j), the set of connectng edges E j s unque. The above attrbute comes drectly from the tree structure. The proof s smple: f there s more than one E j then there exsts at least one crcle, whch contradcts wth the proposton. The unque dstance defnton on an MST facltates the defnton of densty for a gven locaton. We propose to use a jont representaton of the MST dstance space or MST space for short) and the feature space 0

3 to defne the densty estmator. Consder the smplest case where the MST space ernel center s located exactly at a tree node v j, then the densty estmator can be wrtten as follows: dvj, v ) ) v v fv) =c 0, 1) 1 where dv j, v )= v 1,v ) E j v 1 v s the cumulatve weght of edges that connects the two nodes, v s the feature space ernel center, h 1 and are the bandwdth parameters controllng the wndow sze and c 0 s a constant term determned by the sample sze and bandwdth. x) s the profle of a normal ernel: x) =exp 1 x). To defne a densty estmator for any locaton on the MST space, we have to frst defne the branch of an MST node. Here by sayng any locaton we actually allow the MST space ernel center to be located on an MST edge between neghborng nodes. In other words, the ernel can shft on the constraned space defned by MST. Suppose v negh s a neghborng node of v, we have the followng defnton: Defnton 3.1 The branch of a gven tree node v wth respect to ts connected edge v, v negh ) s a set of nodes and edges B = V B, E B ), such that V B = {v j j, v, v negh ) E j }, E B = {v, v j ) j, v, v negh ) E j }. The branch of a node s an nduced subgraph rooted at v, and descendng from ts referenced connected edge. There exst at least one correspondng MST edge - denoted as e ref - where the MST space ernel center s located on. If the center s located exactly on a tree node, then one may choose any edge connectng ths node to one of ts neghborng nodes as e ref. Suppose that the two nodes connected by e ref are respectvely v ref1 and v ref, and that the dstances from the ernel center to v ref1 and v ref are respectvely x 1 and x x 1 +x = dv ref1 v ref )= v ref1 v ref ), then the densty estmator defned wth respect to v ref1 can be wrtten as: ˆf eref,vref1 v,x 1 )= dvref1, v ) x 1 ) c 0,v V ref1 c 0,v / V ref1 1 1 ) v v + dvref1, v )+x 1 ) ) v v. 3) where V ref1 s the set of branch nodes wth respect to v ref1 and e ref. Smlarly, we can defne the densty estmator wth respect to v ref : ˆf eref,vref v,x )= dvref, v ) x ) c 0,v V ref c 0,v / V ref 1 1 ) v v + dvref, v )+x ) ) v v. where V ref s defned n a smlar way. Assocated wth the above densty estmator are some good propertes that facltates the mode seeng process: Property 3. ˆf eref,vref1 = ˆf eref,vref, e ref E The above equalty holds n the sense that V ref1 V ref = V and V ref1 V ref =, whch ndcates {v v V ref1 } = {v v / V ref }. In addton, snce dv ref1, v ) x 1 = dv ref1, v ref )+dv ref, v ) x 1 = dv ref, v )+x when v V ref1, we obtan the followng equalty:,v V ref1 =,v / V ref 1 dvref1, v ) x 1 ) ) v v 1 4) dvref, v )+x ) ) v v. The equalty relaton between the second term of 3) and the frst term of 4) can be proved smlarly. Property 3. states that the estmated densty does not depend on the choce of reference pont. Property 3.3 If e ref1 and e ref are two edges that connects the same node v ref, ˆferef1,vref v, 0) = ˆf eref,vref v, 0), v ref V. Property 3.3 states that the estmated densty does not depend on the choce of reference edge when the MST space ernel s located on a tree node. Here we consder the specal stuaton where the MST space ernel s shftng from one edge to another. When the ernel s located on v ref,the densty estmator degenerates to 1), as x =0. The same condton also holds when we defne the densty estmator wth respect to any other edge connectng to v ref,whch ndcates the above property. Property 3.4 The ernel defned on the MST dstance space s contnuous and s pecewse dfferentable. 03

4 Accordng to the defnton of densty estmator, one s easy to verfy the pecewse contnuty and dfferentablty gven the MST space ernel s located on the same edge. Together wth Property 3.3, we can obtan Property 3.4. The above property also nfers the contnuty and pecewse dfferentablty of the densty estmator snce t s a lnear combnaton of contnuous and pecewse dfferentable ernels. 3.. Mode seeng wth force competton We see the mode by maxmzng the densty estmator wth respect to v and x smultaneously. The step s to pecewsely estmate the densty gradent, whch s smlar to mean shft. Tang the dervatve of the densty estmator wth respect to v, one get the estmated densty gradent: where K jont, s the product of the feature space ernel and the negatve dervatve of the MST space ernel profle: dv ref,v ) x) ) h v v K jont, = h 1 f v V ref dv ref,v )+x) ) h v v 1 otherwse Equaton 7) can be further rewrtten as: ˆf eref,vref v,x) = [ x c 0 ][ K jont, K jont, dv ref, v ) 1 8),v V ref ) K jont, dv ref, v ) / ] K jont, x,v / V ref ˆf eref,vref v,x) The last term of 8) results n the dsplacement of the MST space ernel, whch s the so called force competton. Force v = c 0 v v competton can also be regarded as a specal case of unvarate mean shft wth v v v)k g h ref representng the orgn. One = c [ 0 v v ] [ K g K g could magne t as a tug of war where data ponts weghted v v ] by K jont are tuggng along each sde of v ref. The shftng step sze, however, should be chosen carefully snce h v h K g v v v ˆf eref,vref s only pecewse dfferentable. Suppose we use 5) the ms to denote the last term of 8), the dsplacement of the MST space ernel s defned as: where gx) = x), K s the MST space ernel functon: { dvref, v K = ) x) / 1) f v V ref1 dv ref, v )+x) / 1 ) otherwse The second term n 5) s the well nown mean shft vector for the feature space ernel center v: mv) = K g v v h v K g v v v. 6) [1] has already developed a sound theoretcal bass for mean shft algorthm concernng ts physcal meanng, convergence analyss and relaton to other feature space analyss methods. Here we wll not extend the dscusson. Now consder the second varable. Tang the dervatve of ˆf eref,vref v,x) wth respect to x, wehave: ˆf eref,vref v,x) = x c 0 dv ref, v ) x)k jont, 1,v V ref + c 0 dv ref, v ) x)k jont,, 1,v / V ref 7) mx) =max x, mn e ref x, ms)) 9) The above term generantees that the MST space ernel s always shfted along the same reference edge. Here we see to provde more ntuton by dscussng some propertes of the densty gradent estmaton: Property 3.5 The estmaton of densty gradent does not depend on the choce of reference node v ref. Snce the densty estmator s pecewse dfferentable on the edge, accordng to Property 3. we can verfy the above property. The estmated densty gradent, however, does depend on the choce of reference edge when the MST space ernel reaches a tree node wth more than two connectng edges. Dfference n the choce of the reference edge results n the followng nequalty: V vref,eref1 V vref,eref V, where V vref,eref1 s the branch node set wth respect to node v ref and ts connectng edge e ref, and smlar for V vref,eref. Such nequalty leads to the sudden jump of estmated densty gradent at some tree nodes. Theorem 3.1 Gven any node v ref where the MST space ernel s located and there are more than two connectng edges, the number of reference edge e ref wth postve MST space ernel dsplacement s no more than 1. 04

5 Proof: Wthout loss of generalty, suppose the MST space ernel s located on node v ref wth three connectng edges e ref1, e ref and e ref3,andd eref1 >D eref >D eref3, where D eref s defned as follows: D eref = dvref, v ) ),v V vref,eref 1 v v dv ref, v ). The force competton term ms vref,eref equals to the estmated densty gradent wth respect to v ref and e ref tmes a postve scalar: ms vref,eref1 = c ˆf eref,vref v,x) x x=0 = D ref1 D ref D ref3. Smlarly, we have ms vref,eref = D ref D ref1 D ref3 and ms vref,eref3 = D ref3 D ref1 D ref. Snce D eref1 >D eref >D eref3 and D eref > 0, ms vref,eref and ms vref,eref3 can not possbly be larger than 0. The only postve ms vref,eref comes when D ref1 >D ref + D ref3 and the above proof can be easly extended to nodes wth multple edges. Thus we have proved the above Theorem Algorthmc descrpton Theorem 3.1 states that when the MST space ernel s located on any tree node, ether ths node s a local maxma, or there s only one edge to whch shftng the ernel results n the ncrease of the densty. The conveyed ntuton here s mportant: each tme the MST space ernel s shftng from one edge to another, one does not face the problem of multple selectable paths snce there s at most one edge that ncreases the estmated densty. Such property leads to the bass of our mplemented algorthm and ts fast approxmaton method. The mode seeng algorthm s a step sze controlled gradent ascent: 1. For each data pont v, =1,,..., N, ntalze the ts feature space ernel poston as the data pont tself. Select v as v ref and ntalze the MST space ernel on the reference node.. Compute the MST space ernel shft wth the followng rules: If the MST space s exactly located on any tree node, calculate m j x) x=0 wth respect to all ts connectng edges e j. If There exsts one postve m j, select the correspondng edge e j as the reference edge e ref. mx) =m j as the MST space ernel shft. Else mx) =0. Else calculate mx) wth respect to v ref and e ref. 3. Calculate the step control factor α: If mx) =0, α =1. Else α = mx) / ms. 4. Compute the feature space ernel shft and scale t wth α: m v) =αmv). 5. Smultaneously shft the MST space ernel and the feature space ernel wth respect to the ernel shfts calculated n Step and Step 4. The MST space ernel s shfted wth the followng rule: If the MST space ernel s exactly located on a node If mx) = e ref, shft the MST space ernel to the neghborng node connected by e ref and select the neghborng node as the new reference node. Elsef mx) =0, the MST space ernel stays on the current node. Else update the ernel poston on the edge: x = mx). Elsef the MST space ernel s located on an edge If mx) == x, shft the MST space ernel to the reference node. Elsef mx) =e ref x, shft the MST space ernel to the neghborng node connected by e ref and select the neghborng node as the new reference node. Else update the ernel poston on the edge: x = mx)+x. 6. Repeat Step to Step 5 untl convergence Fast approxmaton Due to the pecewse dfferentablty and step control, the above algorthm gves the best mode seeng performance but requres more teratons before convergence. In addton, the algorthm contans numerous f-then-else condtons, whch s not frendly to hardware mplementaton. Here we also propose a fast approxmaton to the orgnal algorthm by teratvely shftng the MST space ernel and the feature space ernel. The method s straght forward: 1. For each data pont, ntalze the MST space ernel and the feature space ernel.. Shft the feature space ernel accordng to 6). 05

6 3. If there exst neghborng nodes that ncrease the estmated densty, shft the MST space ernel to the nearest one. Otherwse, stop shftng. 4. Repeat Step and 3 untl convergence. In all of the followng experments, we only mplement the above fast algorthm. 4. Expermental results We show three sets of experments usng our proposed algorthm. The frst set of experments demonstrates the performance of the method n the tas of data clusterng. Fgure a) shows a character shaped dstrbuton contanng 934 data ponts and ts clusterng result. The bandwdth parameters h 1 and were respectvely set to 150 and 40 for ths experment. Fgure b) shows the mxture of 4 gaussan dstrbutons wth a total of 1500 data ponts. Here we set h 1 to 700 and to 150. From the two experments one could observe that the method wors reasonably well for both arbtrarly shaped and regularly shaped cluster of data. The real challenge comes when we want to cluster the spral-le data dstrbuton wth hghly nonlnear cluster separaton boundares. The example of spral-le data gven n [3] was reproduced wth the Matlab code ndly avalable at new medod.htm. In ths experment h 1 and are respectvely set to 150 and 300. Note that we have acheved the clusterng performance that approxmates the one gven n [3] wthout usng any non-eucldean metrc, whle mean shft or Eucldean medod shft usually wll fal on such tas Fgure 3. Clusterng wth spral-le cluster of data usng the proposed method The second set of experments address the problem dscontnuty preserved smoothng wth superpxelzed mages. As dscussed n prevous secton, regon-wse operaton sgnfcantly reduces the requred computaton power, thus greatly accelerates the mage smoothng and segmentaton process. The ntroducton of MST space ernel wors n compatble wth the regon adjacency graph and n addton, further mproves the smoothng and segmentaton performance. Fgure 4 shows the mages and ther smoothng results usng dfferent methods n the RGB color space. The mages are frst superpxelzed usng normalzed cut[6, 7]. The correspondng Matlab code s ndly provded at mor/research/superpxels/. We set the number of coarse superpxels N sp to 00, the number of fne superpxels N sp to 400 and the number of egenvectors N ev to 40. Each superpxel s then represented by the mean RGB value and the whole mage s mapped to an undrected, weghted regon adjacency graph where edges corresponds to the eght-connectvtes of two regons and edge weghts are defned as the Eucldean dstances between the regon means. We extract the mnmum spannng tree from the regon adjacency graph usng Krusal s Algorthm and perform mode seeng usng our proposed method. Here we fxed h 1 as 30 and as 50 for all the test mages. The obtaned results are llustrated n the second column of fgure 4. To demonstrate the mprovement of algorthm performance by ntroducng the MST space ernel, we compare the results wth medod shft smoothng where each super pxel s represented by the 5D jont representaton of the RGB mean and spatal coordnate mean. The dstance matrx s obtaned by calculatng the Eucldean dstances between each par of super pxels and the parameter Sgma s set to 000. We also compare our results wth quc shft whch s a fast mode seeng algorthm. We run the quc shft algorthm wth the VLFeat Matlab pacage whch s publcly avalable at The parameters rato, ernelsze and maxdst are respectvely set to 0.3, 1 and 30. The results llustrated n fgure 4 ndcates the advantage of usng our proposed method for mage smoothng. We llustrate the potental applcaton of mage segmentaton usng our method n the last set of experments. Note that the segmentaton performance depends largely on the defned feature. Wth superpxelzed mages, the defnton of mage feature becomes much more versatle than pxel based methods. Such framewor allows one to mprove the segmentaton performance by defnng the feature n a sophstcated way, usng textons, texture detectors or other regon statstcs. For smplcty we only adopt regon color hstogram n ths paper. Each regon s represented by a 4-D concatenated hstogram wth each RGB channel returnng a hstogram of 8 bns. We then use prncpal component analyss PCA) to perform dmensonalty reducton on the obtaned hstograms. The percentage of preserved varance for PCA s set to 0.9, a typcal rule of thumb value for PCA. For most of the mages, the reduced dmenson after performng PCA often les n between 4-8, whch s much smaller than the orgnal dmenson number. By runnng PCA we reduces the computatonal complexty and effec- 06

7 a) Fgure. Data clusterng usng the proposed method. a) Clusterng wth lnearly separable data. b) Clusterng wth mxture of gaussans b) Fgure 4. Dscontnuty preserved smoothng wth superpxelzed mages: The frst column contans the orgnal mages. The second column corresponds to the smoothng results usng the proposed method. The second column contans the smoothng results usng medod shft. The last column are the results obtaned by quc shft. tvely avods from sufferng the curse of dmensonalty. The segmentaton results are shown n fgure 5. One could observe that the proposed method s effectve and produces reasonably good segmentatons. 5. Concluson In ths paper, by ntroducng the MST space ernel, we have proposed a novel mode seeng method that can mprove mode seeng performance on manfold-structured data and can wor compatbly wth regon-wse mage pro- 07

8 Fgure 5. Image segmentaton experments wth regon hstogram cesng operatons. We acheved good algorthm performance n clusterng data wth hghly nonlnear separaton boundares wthout usng any manfold dstance or some other non Eucldean metrcs, whch s of consderable challenge. The advantage of usng the proposed method for mage smoothng and segmentaton s also supported by our experments. References [1] D. Comancu and P. Meer. Mean shft: A robust approach toward feature space analyss. IEEE Trans. Pattern Anal. Mach. Intell., 45): , 00. [] A. Ylmaz, Object tracng by Asymmetrc ernel mean shft wth automatc scale and orentaton selecton. In CVPR, 007. [3] Y. A. Sheh, E. A. Khan and T. Kanade. Modeseeng by Medodshfts. In ICCV, 007. [5] A. Vedald and S. Soatto. Really quc shft: Image segmentaton on a GPU. In Worshop on Computer Vson usng GPUs, held wth ECCV, 010. [6] J. Sh and J. Mal. Normalzed cuts and mage segmentaton. IEEE Trans. Pattern Anal. Mach. Intell., 8): , 000. [7] X. Ren and J. Mal. NLearnng a classfcaton model for segmentaton. In ICCV, 003. [8] K. Zhang, J. T. Kwo and M. Tang. Accelerated convergence usng dynamc mean shft. In ECCV, 006. [9] R. Subbarao and P. Meer. Nonlnear mean shft for clusterng over analytc manfolds. In CVPR, 006. [10] O. J. Morrs, M.de J. Lee, and A.G. Constantndes. Graph theory for mage analyss: An approach based on the shortest spannng tree, In IEE Proc. F., Communcatons. Radar & Sgnal Processng, 133:146-15, [4] A. Vedald and S. Soatto. Quc shft and ernel methods for mode seeng. In ECCV,