Cluster Validity Measurement Techniques

Transcription

1 Cluster Validity Measuremet Techiques Ferec Kovács, Csaba Legáy, Attila Babos Departmet of Automatio ad Applied Iformatics Budapest Uiversity of Techology ad Ecoomics Goldma György tér 3, H- Budapest, Hugary {ferec.kovacs, csaba.legay, Abstract: Clusterig is a usupervised process i the data miig ad patter recogitio ad most of the clusterig algorithms are very sesitive to their iput parameters. Therefore it is very importat to evaluate the result of the clusterig algorithms. It is difficult to defie whe a clusterig result is acceptable, thus several clusterig validity techiques ad idices have bee developed. I this paper the most commoly used validity idices are itroduced ad compared to each other. Keywords: data miig, clusterig algorithms, cluster validity, validity idices Itroductio Oe of the best kow problem i the data miig is the clusterig. Clusterig is the task of categorisig obects havig several attributes ito differet classes such that the obects belogig to the same class are similar, ad those that are broke dow ito differet classes are ot. Clusterig is the subect of active research i several fields such as statistics, patter recogitio, machie learig ad data miig. A wide variety of clusterig algorithms have bee proposed for differet applicatios []. Clusterig is mostly usupervised process thus the evaluatio of the clusterig algorithms is very importat. I the clusterig process there are o predefied classes therefore it is difficult to fid a appropriate metric for measurig if the foud cluster cofiguratio is acceptable or ot. Several clusterig validity approaches have bee developed [2] [3]. The mai disadvatage of these validity idices is that they caot measure the arbitrary shaped clusters as they usually choose a represetative poit from each cluster ad they calculate distace of the represetative poits ad calculate some other parameter based o these poits (for example: variace).

2 The rest of the paper is orgaized as follows. Geeral properties of clusterig algorithms ad cluster validity techiques are itroduced i Sectio 2. The detailed ivestigatio of the most commoly used cluster validity idices is give i Sectio 3. The experimetal results ad compariso of the idices are outlied i Sectio 4. 2 Related Work The clusterig problem is to partitio a data set ito groups (clusters) so that the data elemets withi a cluster are more similar to each other tha data elemets i differet clusters [4]. There are differet types of clusterig algorithms ad they ca be classified ito the followig groups []: Partitioal Clusterig: These algorithms decompose directly data set ito a set of disoit clusters. They attempt to determie a iteger umber of partitios that optimise a certai criterio fuctio. This optimisatio is a iterative procedure. Hierarchical Clusterig: These algorithms create clusters recursively. They merge smaller cluster ito larger oes or split larger clusters ito smaller oes. Desity-based Clusterig: The key poit of these algorithms is to create clusters based o desity fuctios. The mai advatage of these algorithms is to create arbitrary shaped clusters. Grid-based Clusterig: These types of algorithms are maily proposed for spatial data miig. They quatise the search space ito fiite umber of cells. The result of a clusterig algorithm ca be very differet from each other o the same data set as the other iput parameters of a algorithm ca extremely modify the behaviour ad executio of the algorithm. The aim of the cluster validity is to fid the partitioig that best fits the uderlyig data. Usually 2D data sets are used for evaluatig clusterig algorithms as the reader easily ca verify the result. But i case of high dimesioal data the visualisatio ad visual validatio is ot a trivial tasks therefore some formal methods are eeded.

3 The process of evaluatig the results of a clusterig algorithm is called cluster validity assessmet. Two measuremet criteria have bee proposed for evaluatig ad selectig a optimal clusterig scheme [5]: Compactess: The member of each cluster should be as close to each other as possible. A commo measure of compactess is the variace. Separatio: The clusters themselves should be widely separated. There are three commo approaches measurig the distace betwee two differet clusters: distace betwee the closest member of the clusters, distace betwee the most distat members ad distace betwee the cetres of the clusters. There are three differet techiques for evaluatig the result of the clusterig algorithms [6]: Exteral Criteria Iteral Criteria Relative Criteria Both iteral ad exteral criteria are based o statistical methods ad they have high computatio demad. The exteral validity methods evaluate the clusterig based o some user specific ituitio. The iteral criteria are based o some metrics which are based o data set ad the clusterig schema. The mai disadvatage of these two methods is its computatioal complexity. The basis of the relative criteria is the compariso of the differet clusterig schema. Oe or more clusterig algorithms are executed multiple times with differet iput parameters o same data set. The aim of the relative criteria is to choose the best clusterig schema from the differet results. The basis of the compariso is the validity idex. Several validity idices have bee developed ad itroduced [7] [8] [9] [0] [] [2]. Most widely used validity idices are itroduced i the followig sectio. 3 Validity Idices I this sectio several validity idices are itroduced. These idices are used for measurig goodess of a clusterig result comparig to other oes which were created by other clusterig algorithms, or by the same algorithms but usig differet parameter values. These idices are usually suitable for measurig crisp clusterig. Crisp clusterig meas havig o overlappig partitios. Table describes the used otatio i validity idices.

4 Notatio c d d( x, y) X X i vi ci ci Meaig Number of clusters Number of dimesio Distace betwee two data elemet Expected value i the th dimesio T X X, where X T is a colum vector Number of elemet i i th cluster th dimesio Number of elemet i th dimesio i the whole data set Cetre poit of the i th cluster i th cluster Number of elemet i the i th cluster Table Notatio i validity idices 3. Du ad Du like Idices These cluster validity idices have bee itroduced i paper [7]. The idex defiitio is give by Equatio. D d( ci, c ) = mi mi, where i=... c = i+... c max ( diam( ck )) k=... c ( i, ) = mi { (, )} ad ( i) = max { (, )} d c c d x y diam c d x y x ci, y c xy, ci If a data set cotais well-separated clusters, the distaces amog the clusters are usually large ad the diameters of the clusters are expected to be small [3]. Therefore larger value meas better cluster cofiguratio. The mai disadvatages of the Du idex are the followig: the calculatio of the idex is time cosumig ad this idex is very sesitive to oise (as the maximum cluster diameter ca be large i a oisy eviromet). Several Du-like idices have bee proposed [6] [3]. These idices use differet defiitio for cluster distace ad cluster diameter. ()

5 3.2 Davies Bouldi Idex The Davies Bouldi idex [8] is based o similarity measure of clusters (R i ) whose bases are the dispersio measure of a cluster (s i ) ad the cluster dissimilarity measure (d i ). The similarity measure of clusters (R i ) ca be defied freely but it has to satisfy the followig coditios [8]: Ri 0 R = R i i if si = 0 ad s = 0 the R i = 0 if s > s k ad di = d ik the R i > R ik if s =s ad d < d the R > R k i ik i ik Usually R i is defied i the followig way: si + s Ri = di di = d vi v si = d x vi c (, ), (, ) i x ci The the Davies Bouldi idex is defied as c DB =, where c R = max R, i =... Ri i= ( ) i i c =... c, i The Davies Boludi idex measures the average of similarity betwee each cluster ad its most similar oe. As the clusters have to be compact ad separated the lower Davies Bouldi idex meas better cluster cofiguratio. 3.3 RMSSDT ad RS Validity Idices Usually hierarchical clusterig algorithms use these idices but they ca be used for evaluatig the results of ay clusterig algorithm. The RMSSTD (root mea square stadard deviatio) idex [9] is the variace of the clusters, formally defied o Equatio 4, thus it measures the homogeeity of the clusters. As the aim of the clusterig process to idetify homogeous groups the lower RMSSTD value meas better clusterig. (2) (3)

6 RMSSTD = i 2 ( x x ) ( ) i=... c k =... d k = i=... c i =... d The motivatio RS (R Squared) idex [9], described o Equatio 5, idex is to measure the dissimilarity of clusters. Formally it measures the degree of homogeeity degree betwee groups. The values of RS rage from 0 to where 0 meas there are o differece amog the clusters ad idicates that there are sigificat differece amog the clusters. SSt SS RS = SS w, where t d i 2 2 t = ( k ), w = i=... c ( k ) = k= =... d k= SS x x SS x x 3.4 SD Validity Idex The bases of SD validity idex [2] are the average scatterig of clusters ad total separatio of clusters. The scatterig is calculated by variace of the clusters ad variace of the dataset, thus it ca measure the homogeeity ad compactess of the clusters. The variace of the dataset ad variace of a cluster are defied i Equatio 6. Variace of the dataset: Variace of a cluster: 2 2 p p p p p p σx = ( xk x ) σv = ( ) i xk vi k= ci k= σ x σ vi σ ( x) = M σ ( vi ) = M d d σ x σ vi (6) The average scatterig for clusters is defied as Scatt = σ c σ c i= ( vi ) ( x) The total separatio of clusters is based o the distace of cluster cetre poits thus it ca measures the separatio of clusters. Its defiitio is give by Equatio 8. (4) (5) (7)

7 ( v v i ) c c c ( i ) v v k= =, max i, =... Dis = v v i mi i, =... c i (8) The SD idex ca be defied based o Equatio 7 ad 8 as follows SD = α Scatt + Dis (9) where α is a weightig factor that is equal to Dis parameter i case of maximum umber of clusters. Lower SD idex meas better cluster cofiguratio as i this case the clusters are compacts ad separated. 3.5 S_Dbw Validity Idex This validity idex has bee proposed i []. Similarly to SD idex its defiitio is based o cluster compactess ad separatio but it also takes ito cosideratio the desity of the clusters. Formally the S_Dbw idex measures the itra-cluster variace ad the iter-cluster variace. The itra cluster variace measures the average scatterig of clusters ad it is described by Equatio 7. The iter cluster desity is defied as follows c c desity( ui ) Des _ bw = c( c ) i= =, max { desity ( v i), desity ( v ) } i (0) where u i is the middle poit of the lie segmet that is defied by the v i ad v clusters cetres. The desity fuctio aroud a poit is defied as follows: it couts the umber of poits i a hyper-sphere whose radius is equal to the average stadard deviatio of clusters. The average stadard deviatio of clusters is defied as c stdev = σ c i= ( v ) i The S_Dbw idex is defied i the followig way: S _ Dbw = Scatt + Des _ bw The defiitio of S_Dbw idicates that both criteria of good clusterig are properly combied ad it eables reliable evaluatio of clusterig results. Lower idex value idicates better clusterig schema. () (2)

8 4 Experimetal Results The clusterig algorithms ad validity idices were evaluated sythetically geerated data set. These data were geerated by our data set geerator. The validity idices were evaluated usig the followig datasets: Well separated clusters: the cluster elemets were geerated aroud the cluster cetres poits usig ormal distributio. Rig shaped clusters: Two cluster, which cotais each other. Arbitrary shaped clusters: some arbitrary shaped clusters close to each other. The used data sets are depicted o Figure. Figure The used data set i experimetal evaluatio Figure 2 shows a compariso of the validity idices o the first data set. The used clusterig algorithm is k-meas algorithm ad i the first case it foud the right clusterig schema but i the secod case it geerates wrog cluster cofiguratio. I this case it easy to idetify that the validity idices ca compare, i appropriate way, the result of the clusterig algorithm.

9 00 0 0, 0,0 k-meas right clusterig schema k-meas w rog clusterig schema Du Davies-Bouldi SD S_Dbw Figure 2 Validity idices o the first dataset Figure 3 shows the validity idices based clusterig of the secod data set. Two clusterig results are compared: a right clusterig result (usig DB-Sca algorithm) ad a oe (usig k-meas algorithm). A result a little bit surprisig as the Du ad S_Dbw idex ca idetify the right clusterig result but the other idices offer wrog decisio. Du D a v ie s - B o u ld i SD S_Dbw , 0,0 DBSca right clusterig cofiguratio k-m ea w rog clusterig cofiguratio Figure 3 Validity idices o the secod data set Two clusterig results, which are based o third data set, are depicted i Figure 4. The compariso of the validity idices are give i Figure 5. It is possible to realise that oly the Du idex ca idetify the right clusterig schema. The mai disadvatage of the curret validity idices is that they caot idetify the right clusterig schema uless the clusters are well separated.

10 Figure 4 Clusterig results o third data set Du Davies-Bouldi SD S_Dbw 0 0, Right clusterig result Wrog clusterig result Figure 5 Clusterig results o third data set Coclusios I this paper several cluster validity idices have bee summarised. These validity idices have bee evaluated with several differet iput dataset ad we tried to compare the efficiecy of these validity idices. The result of this compariso is that these idices ca idetify oly the well separated hyper sphere shaped clusters. As these idices measure the variace of the clusters aroud some represetative poits but some clusters, especially the arbitrary shaped clusters, do ot have represetative cetre poit. Thus it is importat to defie ovel validity idices which ca measure arbitrary shaped clusters.

11 Ackowledgmet This work has bee supported by the Mobile Iovatio Ceter, Hugary. Refereces [] A. K. Jai, M. N. Murty ad P. J. Fly: Data clusterig: a review, ACM Computig Surveys, Vol. 3, No. 3, pp , 999 [2] M. Halkidi, Y. Batistakis ad M. Vazirgiais: Cluster validity methods: part I, SIGMOD Rec., Vol. 3, No. 2, pp , 2002 [3] M. Halkidi, Y. Batistakis ad M. Vazirgiais: Cluster validity methods: part II, SIGMOD Rec., Vol. 3, No. 3, pp. 9-27, 2002 [4] S. Guha, R. Rastogi ad K. Shim: CURE: a efficiet clusterig algorithm for large databases, Proc. of ACM SIGMOD Iteratioal Coferece o Maagemet of Data, pp , 998 [5] M. J. A. Berry ad G. Lioff: Data Miig Techiques for Marketig, Sales ad Customer Support, Joh Wiley & Sos, Ic., 996 [6] S. Theodoridis ad K. Koutroubas: Patter Recogitio, Academic Press, 999 [7] J. C. Du: Well Separated Clusters ad Optimal Fuzzy Partitios, Joural of Cyberetica, Vol. 4, pp , 974 [8] D. L. Davies ad D. W. Bouldi: Cluster Separatio Measure, IEEE Trasactios o Patter Aalysis ad Machie Itelligece, Vol., No. 2, pp , 979 [9] Subhash Sharma: Applied multivariate techiques, Joh Wiley & Sos, Ic., 996 [0] M. Halkidi, Y. Batistakis ad M. Vazirgiais: O Clusterig Validatio Techiques, Joural of Itelliget Iformatio Systems, Vol. 7, No. 2-3, pp , 200 [] M. Halkidi ad M. Vazirgiais: Clusterig Validity Assessmet: Fidig the Optimal Partitioig of a Data Set, Proc. of ICDM 200, pp , 200 [2] M. Halkidi ad M. Vazirgiais ad Y. Batistakis: Quality Scheme Assessmet i the Clusterig Process, Proc. of the 4 th Europea Coferece o Priciples of Data Miig ad Kowledge Discovery, pp , 2000 [3] N. R. Pal ad J. Biswas: Cluster Validatio usig graph theoretic cocepts, Patter Recogitio, Vol. 30, No. 4, 997