Transcription

1 320 The Internatonal Arab Journal of Informaton Technology, Vol. 5, No. 3, July 2008 Comparsons Between Data Clusterng Algorthms Osama Abu Abbas Computer Scence Department, Yarmouk Unversty, Jordan Abstract: Clusterng s a dvson of data nto groups of smlar objects. Each group, called a cluster, conssts of objects that are smlar between themselves and dssmlar compared to objects of other groups. Ths paper s ntended to study and compare dfferent data clusterng algorthms. The algorthms under nvestgaton are: k-means algorthm, herarchcal clusterng algorthm, self-organzng maps algorthm, and expectaton maxmzaton clusterng algorthm. All these algorthms are compared accordng to the followng factors: sze of dataset,, type of dataset and type of software used. Some conclusons that are extracted belong to the performance, qualty, and accuracy of the clusterng algorthms. Keywords: Clusterng, k-means algorthm, herarchcal clusterng algorthm, self-organzng maps algorthm, expectaton maxmzaton clusterng algorthm. Receved January 18, 2007 ; accepted May 2, Introducton Clusterng s a dvson of data nto groups of smlar objects. Each group, called cluster, conssts of objects that are smlar amongst themselves and dssmlar compared to objects of other groups. Representng data by fewer necessarly loses certan fne detals, but acheves smplfcaton. It represents many data objects by few, and hence, t models data by ts [3]. Cluster analyss s the organzaton of a collecton of patterns (usually represented as a vector of measurements, or a pont n a multdmensonal space) nto based on smlarty. Patterns wthn a vald cluster are more smlar to each other than they are to a pattern belongng to a dfferent cluster. It s mportant to understand the dfference between clusterng (unsupervsed classfcaton) and dscrmnate analyss (supervsed classfcaton). In supervsed classfcaton, we are provded wth a collecton of labeled (preclassfed) patterns; the problem s to label a newly encountered, yet unlabeled, pattern. Typcally, the gven labeled (tranng) patterns are used to learn the descrptons of classes whch n turn are used to label a new pattern. In the case of clusterng, the problem s to group a gven collecton of unlabeled patterns nto meanngful. In a sense, labels are assocated wth also, but these category labels are data drven; that s, they are obtaned solely from the data [5, 7, 8, 13]. Some researchers mproved some data clusterng algorthms, others mplemented new ones, and some others studed and compared dfferent data clusterng algorthms. Followng are some of the prevous studes that consdered the effect of dfferent factors on the performance of some data clusterng algorthms and compared the results. However, these studes dffer from my analyss n the algorthms and the factors: [1] appled several ndces to evaluate the performance of clusterng algorthms, ncludng herarchcal clusterng, k-means, PAM and SOM. The ndces were homogenety and separaton scores, slhouette wdth, redundant score (based on redundant genes), and WADP (testng the robustness of clusterng results after small perturbaton). [9] descrbed the mplementaton of an out-of-core technque for the data analyss of very large data sets wth the sequental and parallel verson of the clusterng algorthm AutoClass. They dscussed the out-of-core technque and showed performance results n terms of executon tme and speed up. [8] employed an agglomeratve algorthm to construct a dendrogram and used a smple dstnctness heurstc to extract a partton of the data. They studed the performance of Smlarty- Based Agglomeratve Clusterng (SBAC) algorthm on real and artfcally generated data sets. They demonstrated the effectveness of ths algorthm n unsupervsed dscovery tasks. They llustrated the superor performance of ths approach by makng comparsons wth other clusterng schemes. [10] showed how to perform some typcal mnng tasks usng conceptual graphs as formal but meanngful representatons of texts. Ther methods nvolved qualtatve and quanttatve comparsons of conceptual graphs, conceptual clusterng, buldng a conceptual herarchy, and applcaton of data mnng technques to ths herarchy n order to detect nterestng assocatons and devatons. Ther experments showed that, despte wdespread

2 Comparsons Between Data Clusterng Algorthms 321 dsbelef, detaled meanngful mnng wth conceptual graphs s computatonally affordable. [12] compared two graph-colorng programs: one exact and another based on heurstcs whch can gve, however, provably exact results on some types of graphs. They proved that the exact graph colorng s not necessary for hgh-qualty functonal decomposers. Comparson of ther expermental results wth competng decomposers shows that for nearly all benchmarks ther solutons are the best and tme s usually not too hgh. Jan and Dubes [1988] and Dubes [1993] used a relatve test to compare two structures and to measure ther relatve mert. They also dscussed n detal the ndces that are used for ths comparson. In ths paper dfferent data clusterng algorthms that have not been consdered before are compared accordng to dfferent factors that haven't been studed yet. 2. How Algorthms are Implemented? An extensve web search s done to fnd some data clusterng algorthms mplementaton to test on. After selecton, I ended up wth two of them: Software: It s publc doman software made avalable from MIT Lncoln Laboratory [6]. It s located at the followng ste: ( TreeVew Software: TreeVew are programs that provde a computatonal and graphcal envronment for analyzng data from dfferent datasets [2]. It s located at the followng ste: ( The reasons behnd choosng these two software are: They are the most popular software for mplementng dfferent data clusterng algorthms. They are very powerful n mplementng data clusterng algorthms. They mplement the four data clusterng algorthms that are chosen n ths paper. They have deal dataset as a part of them whch can be used for testng and mplementng the algorthms Data Sample The dataset that s used to test the clusterng algorthms and compare among them s obtaned from the ste: ( or from another ste, whch s, ( Ths s a good dataset to test tme seres clusterng algorthms because eucldean dstance wll not be able to acheve perfect accuracy. In partcular the followng pars of classes wll often be confused (normal/ cyclc) (decreasng trend/ downward shft) and (ncreasng trend/ upward shft). Ths dataset s stored n an ASCII fle, 600 rows, 60 columns, wth a sngle chart per lne. The classes are organzed as follows: Normal Cyclc Increasng trend Decreasng trend Upward shft Downward shft However, ths format s not totally sutable for the two packages used to compare the clusterng algorthms ( Software and TreeVew Software). So some reasonable changes are done n the dataset format to be acceptable for the packages. The dataset formats of the two software are revewed by referrng to the manuals of the software and the changes that are done to the dataset formats do not affect the dataset tself at all. Also, a part of ths data set (200 rows and 20 columns) s taken as an nput fle for the clusterng algorthms to study the sze of the datasets (huge and small datasets) on these algorthms. Fnally, the clusterng algorthms are tested usng the dataset stored n the two packages themselves to study the effect of dfferent datasets on the algorthms Whch Algorthms are Compared? Four dfferent clusterng algorthms are chosen to nvestgate, study, and compare them. The algorthms that are chosen are: k-means algorthm, herarchcal clusterng algorthm, Self-Organzaton Map (SOM) algorthm and Expectaton Maxmzaton (EM) clusterng algorthm. The general reasons for selectng these four algorthms are: Popularty. Flexblty. Applcablty. Handlng hgh dmensonalty. However, detaled reasons behnd selectng every algorthm are lsted n the context. In ths secton, for every algorthm some dea s gven about t; how t works and the reasons for choosng t K-means Algorthm K-means s a well-known parttonng method. Objects are classfed as belongng to one of k groups, k chosen a pror. Cluster membershp s determned by calculatng the centrod for each group (the multdmensonal verson of the mean) and assgnng each object to the group wth the closest centrod. Ths approach mnmzes the overall wthn-cluster dsperson by teratve reallocaton of cluster members.

3 322 The Internatonal Arab Journal of Informaton Technology, Vol. 5, No. 3, July 2008 In a general sense, a k-parttonng algorthm takes as nput a set S of objects and an nteger k, and outputs a partton of S nto subsets S 1,...S 2, S k. It uses the sum of squares as the optmzaton crteron. Let x r be the r th element of S, S be the elements n S, and d(x r, x s) be the dstance between x r and x s. The sum of squares crteron s defned by the cost functon: c( S S S = ( d( xr,xs 2 (1) r= 1x= 1 ) )) In partcular, k-means works by calculatng the centrod of each cluster S, denoted x, and optmzng the cost functon: c( S S = ( d( x,xr 2 (2) r= 1 ) )) The goal of the algorthm s to mnmze the total cost: c( S ) +... c( S ) (3) + Here, the pseudo code of the k-means algorthm s to explan how t works: A. Choose K as the. B. Intalze the codebook vectors of the K (randomly, for nstance) C. For every new sample vector: C.1. Compute the dstance between the new vector and every cluster's codebook vector. C.2. Recompute the closest codebook vector wth the new vector, usng a learnng rate that decreases n tme. The reason behnd choosng the k-means algorthm to study s ts popularty for the followng reasons: Its tme complexty s O(nkl), where n s the number of patterns, k s the, and l s the teratons taken by the algorthm to converge. Its space complexty s O(k+n). It requres addtonal space to store the data matrx. It s order-ndependent; for a gven ntal seed set of cluster centers, t generates the same partton of the data rrespectve of the order n whch the patterns are presented to the algorthm Herarchcal Clusterng Algorthm Parttonng algorthms are based on specfyng an ntal groups, and teratvely reallocatng objects among groups to convergence. In contrast, herarchcal algorthms combne or dvde exstng groups, creatng a herarchcal structure that reflects the order n whch groups are merged or dvded. In an agglomeratve method, whch bulds the herarchy by mergng, the objects ntally belong to a lst of sngleton sets S 1,..., S 2, S n. Then a cost functon s used to fnd the par of sets {S, S j } from the lst that s the k cheapest to merge. Once merged, S and S j are removed from the lst of sets and replaced wth S S j. Ths process terates untl all objects are n a sngle group. Dfferent varants of agglomeratve herarchcal clusterng algorthms may use dfferent cost functons. Complete lnkage, average lnkage, and sngle lnkage methods use maxmum, average, and mnmum dstances between the members of two, respectvely. Followng s the pseudo code of the herarchcal clusterng algorthm to explan how t works: Compute the proxmty matrx contanng the dstance between each par of patterns. Treat each pattern as a cluster. Fnd the most smlar par of usng the proxmty matrx. Merge these two nto one cluster. Update the proxmty matrx to reflect ths merge operaton. If all patterns are n one cluster, stop. Otherwse, go to step 2. The advantages of the herarchcal clusterng algorthms are the reason ths algorthm was chosen for dscusson. These advantages nclude: Embedded flexblty regardng a level of granularty. Ease of handlng of any forms of smlarty or dstance. Consequently applcablty to any attrbutes types. Herarchcal clusterng algorthms are more versatle Self-Organzaton Map Algorthm Inspred by neural networks n the bran, Self- Organzaton Map (SOM) uses a competton and cooperaton mechansm to acheve unsupervsed learnng. In the classcal SOM, a set of nodes s arranged n a geometrc pattern, typcally 2- dmensonal lattce. Each node s assocated wth a weght vector wth the same dmenson as the nput space. The purpose of SOM s to fnd a good mappng from the hgh dmensonal nput space to the 2-D representaton of the nodes. One way to use SOM for clusterng s to regard the objects n the nput space represented by the same node as grouped nto a cluster. Durng the tranng, each object n the nput s presented to the map and the best matchng node s dentfed. Formally, when nput and weght vectors are normalzed, for nput sample x(t) the wnner ndex c (best match) s dentfed by the condton: for all, x( t ) m ( t ) x( t ) m ( t ) (4) c where t s the tme step n the sequental tranng, m s the weght vector of the th node. After that, weght vectors of nodes around the best-matchng node c= c(x) are updated as:

4 Comparsons Between Data Clusterng Algorthms 323 m ( t + 1) = m ( t ) + ah ( x( t ) m ( t )) (5) c( x ), where α s the learnng rate and h c(x), s the neghborhood functon, a decreasng functon of the dstance between the th and c th nodes on the map grd. To make the map converge quckly, the learnng rate and neghborhood radus are often decreasng functons of t. After the learnng process fnshes, each object s assgned to ts closest node. There are varants of SOM to the above classcal scheme. Followng s the pseudo code of the SOM algorthm to explan how t works: A. Choose the dmenson and sze of the map. B. For every new sample vector: B.1. Compute the dstance between the new vector and every cluster's codebook vector. B.2. Recompute all codebook vectors wth the new vector, usng both a dstance radus on the map and learnng rate that decrease n tme. The followng advantages of SOM are behnd choosng ths algorthm for studyng: Whle the vorono regons of the map unts are convex, the combnaton of several map unts allows the constructon of non-convex. Dfferent knds of dstance measures and jonng crtera can be utlzed to form the bg. It has been successfully used for vector quantzaton and speech recognton. The SOM generates a sub-optmal partton f the ntal weghts are not chosen properly The Expectaton Maxmzaton Clusterng Algorthm Expectaton Maxmzaton (EM) s a well-establshed clusterng algorthm n the statstcs communty. EM s a dstance-based algorthm that assumes the data set can be modeled as a lnear combnaton of multvarate normal dstrbutons and the algorthm fnds the dstrbuton parameters that maxmze a model qualty measure, called log lkelhood. EM s chosen to cluster data for the followng reasons among others: It has a strong statstcal bass. It s lnear n database sze. It s robust to nosy data. It can accept the desred as nput. It can handle hgh dmensonalty. It converges fast gven a good ntalzaton. 3. How Algorthms are Compared? The four clusterng algorthms are compared accordng to the followng factors: The sze of the dataset. Number of the. Type of dataset. Type of software. For each factor, four tests are made, one for each algorthm. For example, accordng to the sze of data, each of the four algorthms: k-means, Herarchcal Clusterng, SOM, and EM s executed twce; frst by tryng a huge dataset and then by tryng a small dataset. Table 1 explans how the four algorthms are compared. The total tmes the algorthms have been executed s 32. For each 8-runs group, the results of the executons are studed and compared. The conclusons are wrtten down. Ths step s repeated for all the factors. Table 1. The factors accordng to whch the algorthms are compared. k- means HC SOM EM Sze of Number of Clusters Type of Ideal Ideal Ideal Ideal Type of Software TreeVew TreeVew TreeVew TreeVew Accordng to the, k (see Table 2), except for herarchcal clusterng, all clusterng algorthms compared here requre settng k n advance (for SOM, k s the nodes n the lattce). Here, the performance of dfferent algorthms for dfferent k s s compared n order to test the performances that are related to k. To smplfy the stuaton and to make the comparsons easer, k s chosen equal to 8, 16, 32, and 64, and the lattces for SOM are the square of them. To compare herarchcal clusterng wth other algorthms, the herarchcal tree s cut at two dfferent levels to obtan correspondng numbers of (8, 16, 32 and 64). As a result, as the value of k becomes greater the performance of SOM algorthm becomes lower. However, the performance of k-means and EM

5 324 The Internatonal Arab Journal of Informaton Technology, Vol. 5, No. 3, July 2008 algorthms become better than herarchcal clusterng algorthm. Table 2. The relatonshp between and the performance of the algorthms. Number Of Performance Clusters (K) SOM k-means EM HCA Accordng to the accuracy (see Table 3), SOM shows more accuracy n classfyng most the objects to ther than other algorthms. But as the number of k becomes greater the accuracy of herarchcal clusterng becomes better untl t reaches the accuracy of SOM algorthm. k-means and EM algorthms have less qualty (accuracy) than the others. However, all the algorthms have some ambguty n some nosy data to be clustered. Table 3. The relatonshp between and the qualty of the algorthms. Number Of Qualty Clusters (K) SOM K-Means EM HCA Accordng to the sze of dataset (see Table 4), a huge dataset s used consstng of 600 rows and 60 columns and a small dataset usng 200 rows and 20 columns. The small dataset s extracted as a subset of the huge dataset. The qualty of EM and k-means algorthms becomes very good when usng a huge dataset. The other two algorthms herarchcal clusterng and SOM algorthms show good results when usng a small dataset. As a concluson, parttonng algorthms (lke k-means and EM) are used for huge dataset whle herarchcal clusterng algorthms are used for small dataset. Table 4. The affect of the data sze on the algorthms. K=32 Data Sze SOM K-Means EM HCA Accordng to the type of dataset (see Table 5), a random dataset s used whch s extracted from the nternet and used for dfferent jobs. On the other hand, an deal dataset s used whch s part of the software ( and TreeVew). It s deal because t s desgned to be sutable for testng and tranng the software tself and havng less nosy data whch leads to ambguty. As a result, herarchcal clusterng and SOM algorthms gve better results than k-means and EM algorthms when usng random dataset and the vce versa. Ths ndcates that k-means and EM algorthms are very senstve for nose n the dataset. Ths nose makes t dffcult for the algorthm to nclude an object n a certan cluster. Ths wll affect the results of the algorthm. However, herarchcal clusterng algorthm s more senstve for nosy dataset than SOM algorthm. Table 5. The affect of the data type on the algorthms. K=32 Data Type SOM K-Means EM HCA Ideal Accordng to the type of software, two packages are used to compare between the algorthms: (UNIX envronment) and cluster and TreeVew (WINDOWS envronment). However, runnng the clusterng algorthms usng any one of them gves almost the same results even when changng any of the other three factors (dataset sze, number and dataset type). Ths, I beleve, s because most software use the same procedures and deas n any algorthm mplemented by them. 4. Conclusons After analyzng the results of testng the clusterng algorthms and runnng them under dfferent factors and stuaton, the followng conclusons are obtaned: As the, k becomes greater; the performance of SOM algorthm becomes lower. The performance of k-means and EM algorthms s better than herarchcal clusterng algorthm. SOM algorthm shows more accuracy n classfyng most the objects nto ther sutable than other algorthms. As the value of k becomes greater, the accuracy of herarchcal clusterng becomes better untl t reaches the accuracy of SOM algorthm. k-means and EM algorthms have less qualty (accuracy) than the others. All the algorthms have some ambguty n some (nosy) data when clustered. The qualty of EM and k-means algorthms become very good when usng huge dataset. Herarchcal clusterng and SOM algorthms show good results when usng small dataset. As a general concluson, parttonng algorthms (lke k-means and EM) are recommended for huge dataset whle herarchcal clusterng algorthms are recommended for small dataset. Herarchcal clusterng and SOM algorthms gve better results compared to k-means and EM algorthms when usng random dataset and the vce versa. k-means and EM algorthms are very senstve for nose n dataset. Ths nose makes t dffcult for the algorthm to cluster an object nto ts sutable cluster. Ths wll affect the results of the algorthm.

6 Comparsons Between Data Clusterng Algorthms 325 Herarchcal clusterng algorthm s more senstve for nosy dataset than SOM algorthm. Runnng the clusterng algorthms usng any software gves almost the same results even when changng any of the factors because most software use the same procedures and deas n any algorthm mplemented by them. 5. Future Work Ths paper was ntended to compare between some data clusterng algorthms. Through my extensve search, I was unable to fnd any study that attempts to compare between the four clusterng algorthms under nvestgaton. As a future work, comparsons between these four algorthms (or may other algorthms) can be attempted accordng to dfferent factors other than those consdered n ths paper. One mportant factor s normalzaton. Comparng between the results of algorthms usng normalzed data or non-normalzed data wll gve dfferent results. Of course normalzaton wll affect the performance of the algorthm and the qualty of the results. Another approach may consder usng data clusterng algorthms n applcatons such as object and character recognton or nformaton retreval whch s concerned wth automatc storage and retreval of documents. References [1] Chen G., Jaradat S., Banerjee N., Tanaka T., Ko M., and Zhang M., Evaluaton and Comparson of Clusterng Algorthms n Analyzng ES Cell Gene Expresson Data, Statstca Snca, vol. 12, pp , [2] Esen M., Tree Vew Manual, Stanford Unversty, [3] Han J. and Kamber M., Data Mnng: Concepts and Technques, Morgan Kaufmann Publshers, [4] Jan A., Murty M., and Flynn P., Data Clusterng: A Revew, ACM Computng Surveys, vol. 31, no. 3, [5] Keogh E., Chakrabart K., Pazzan M., and Mehrotra S., Dmensonalty Reducton for Fast Smlarty Search n Tme Seres Databases, Knowledge and Informaton Systems, vol. 3, pp , [6] Kukolch L. and Lppmann R., User s Gude, MIT Lncoln Laboratory, [7] Lepere R. and Trystram D., A New Clusterng Algorthm for Communcaton Delays, n Proceedngs of 16 th IEEE-ACM Annual Internatonal Parallel and Dstrbuted Processng Symposum (IPDPS 02), Fort Lauderdale, USA, [8] L C. and Bswas G., Unsupervsed Learnng wth Mxed Numerc and Nomnal Data, IEEE Transactons on Knowledge and Data Engneerng, vol. 14, no. 4, pp , [9] Mascar E., Pzzut C., and Ramondo G., Usng an Out-of-Core Technque for Clusterng Data Sets, n Proceedngs of 12 th Internatonal Workshop on Database and Expert Systems Applcatons (DEXA), Munch, Germany, pp , [10] Montes-Y-G Mez M., Gelbukh A., and L Pez- L Pez A., Text Mnng at Detal Level Usng Conceptual Graphs, Lecture Notes n Computer Scence, vol. 2393, pp , [11] Ordonez C. and Cereghn P., SQLEM: Fast Clusterng n SQL usng the EM Algorthm, n Proceedngs of the 2000 ACM SIGMOD Internatonal Conference on Management of Data, Dallas, Unted States, pp , [12] Perkowsk M., Malv R., Grygel S., Burns M., and Mshchenko A., Graph Colorng Algorthms for Fast Evaluaton Of Curts Decompostons, n Proceedngs of the 36 th Desgn Automaton Conference(DAC), ACM, Lousana, pp , [13] Rabov A., Lu Z., Wolf L., Yu S. and Zhang L., Clusterng Algorthms for Content-Based Publcaton-Subscrpton Systems, n Proceedngs of the 22 nd Internatonal Conference on Dstrbuted Computng Systems (ICDCS 02), USA, pp. 133, [14] Zha H., He X., Dng C., Smon H., and Gu M., Bpartte Graph Parttonng and Data Clusterng, n Proceedngs of the 10 th Internatonal Conference on Informaton and Knowledge Management, ACM Press, pp , Osama Abu Abbas s an nstructor n the department of Computer Scence at Yarmouk Unversty. He got the BSc n Computer Scence from Yarmouk Unversty, Jordan n He then got the Master degree n Computer Scence and Informaton from Yarmouk Unversty, Jordan n Hs teachng nterest focus on algorthms desgn and analyss, data structure, compler constructon, and artfcal ntellgent programmng.