Cluster Analysis of Data Points using Partitioning and Probabilistic Model-based Algorithms

Transcription

1 Internatonal Journal of Appled Informaton Systems (IJAIS) ISSN : Foundaton of Computer Scence FCS, New York, USA Volume 7 No.7, August Cluster Analyss of Data Ponts usng Parttonng and Probablstc Model-based Algorthms Ajboye Adeleke R. Faculty of Computer Systems & Software Engneerng Unverst Malaysa Pahang Isah-Kebbe Hauwau Faculty of Computer Systems & Software Engneerng Unverst Malaysa Pahang Oladele Tnuke O. Dept. of Computer Scence Unversty of Ilorn Ilorn, Ngera. ABSTRACT Explorng the dataset features through the applcaton of clusterng algorthms s a vable means by whch the conceptual descrpton of such data can be revealed for better understandng, groupng and decson makng. Some clusterng algorthms, especally those that are parttonedbased, clusters any data presented to them even f smlar features do not present. Ths study explores the performance accuraces of parttonng-based algorthms and probablstc model-based algorthm. Experments were conducted usng k- means, k-medods and EM-algorthm. The study mplements each algorthm usng RapdMner Software and the results generated was valdated for correctness n accordance to the concept of external crtera method. The clusters formed revealed the capablty and drawbacks of each algorthm on the data ponts. General Terms Algorthm, Clusterng, Data mnng Keywords Clusterng, Algorthm, K-means, EM-clusterng, K-medods 1. INTRODUCTION Clusterng s an mportant exploratory technque commonly used n descrptve data mnng to unvel some hdden features embedded n the dataset. Clusterng s descrbed n [1] as an automated search for group of related observatons n a data set. The concept nvolves the dvson of data nto groups, also known as clusters. Although, t s well establshed that the smlarty of objects s used for clusterng, the defntons of smlarty and the method employed to obtan smlarty are vared [2]. The object of the same cluster shares some features, and from a machne learnng perspectve, clusters correspond to hdden patterns [3]. The clusterng algorthm automatcally supples clusters found n data wth a conceptual descrpton and accordng to [4], a good conceptual descrpton can be used for better understandng and better decson. K-means and k-medods are the most wdely used clusterng algorthms for selectng group of objects from data sets [5]. Tradtonally, clusterng technques are broadly dvded nto herarchcal and parttonng [3], the noton used n both technques to cluster data defers. Clusterng may also be densty or grd-based, whle herarchcal algorthm does ts groupng as crystals grows, parttonng algorthms learn clusters drectly. The herarchcal clusterng s further subdvded nto agglomeratve and dvsve. Clusterng s an unsupervsed learnng technque and unlke classfcaton and regresson, whch analyse class-labelled data sets, clusterng analyses data objects wthout consultng class labels [6]. It can therefore be used to generate class labels for a group of data. Many smlartes exst between data mnng and machne learnng, but whle machne learnng research often focuses on the accuracy of the model, data mnng research n addton to accuracy places strong emphass on the effcency and scalablty of mnng methods for large data sets; and ways to handle complex types of data and to explore new, alternatve methods [6]. Ths study focuses on comparng the performance accuraces of parttonng-based clusterng algorthms and probablstc model-based clusterng of the dataset beng explored. The paper s organzed as follows: In the next secton, some related works reported n the lterature on the mplementaton of these parttonng algorthms and probablstc model-based clusterng s dscussed. In secton 3, the basc concepts of each algorthm s brefly dscussed; whle the processes nvolved n expermentng wth the data s reported n secton 4. The expermental results s represented and dscussed n secton 5 and the study s concluded n secton RELATED WORK A parttonng-based algorthm such as k-means has been wdely reported n the lterature for the clusterng of data. Specfcally, the algorthm s well known for clusterng of data such as mages [7], vdeo object segmentaton [8], document clusterng [9] etc. However, one of the drawbacks of the algorthm s ts challenges of groupng categorcal varables; k-means can only cluster numerc values. The algorthm s senstve to outlers because such objects are far away from the majorty of the data, and thus, when assgned to a cluster, they can dramatcally dstort the mean value of the cluster [6]. To overcome ths challenge, algorthm to address the shortcomngs s proposed n [8]. Although, several exstng algorthms can handle both numerc and categorcal data, Huang[8], opned that most of them are not effcent when dealng wth large datasets. In a study proposed n [10], k-means s used to generate class labels; the algorthm was combned wth lnear dscrmnant analyss approach to adaptvely select the most dscrmnatng subspace. Study n [11] proposed k-medods to dentfy sets of smlar rules n order to better understand the pattern of the data. The algorthm s wdely proposed n several other studes [12],[5],[13] and[11]. The algorthm was modfed n [5] to get a faster clusterng and to overcome some of ts lmtatons 21

2 Internatonal Journal of Appled Informaton Systems (IJAIS) ISSN : Foundaton of Computer Scence FCS, New York, USA Volume 7 No.7, August such as the problem of fndng natural clusters, the dependency of output on the order of nput data, etc. Due to too much tme consumed whle k-medods s dealng wth very large dataset, the algorthm was scaled n [13] to enhance ts performance. In order to mprove the qualty of solutons and speed, EM algorthm was enhanced n a study proposed n [14] to cluster large data sets havng hgh dmenson. The algorthm s also proposed n [15] for clusterng of spatal data. In segmentaton of mages proposed n [16], the EM algorthm estmates the parameters of the model whch provde segmentaton of the mage nto regons, the resultng output produces a descrpton of each regon s colour and texture characterstcs. 3. PARTITIONING AND PROBABILISTIC MODEL-BASED ALGORITHMS Descrptve algorthms descrbe features n the dataset based on certan notons. Typcal algorthms n ths category nclude: parttonng-based, densty-based, grd-based, herarchcal etc. Generally, parttonng-based algorthm fnds the parttons or the resultng clusters that mnmze ether ntra-cluster dstances or nter-cluster dstances. Gven a set of n objects, a parttonng method constructs k parttons of the data, where each partton represents a cluster and k n; and most parttonng methods are dstance-based [6]. 3.1 k-means Suppose a data set, D, contans n objects n Eucldean space, parttonng methods dstrbute the objects n D nto k clusters, C 1, C k, C D and C C j = for (1, j k). K- means method can only be appled when the mean of a set of objects s defned [6]. A centrod-based parttonng technque uses the centrod of a cluster, C to represent that cluster. The centrod can be defned n varous ways, such as by the mean or method of the objects assgned to the cluster. The C dfference between an object p and c, the representatve of the cluster, s measured by dst(p, c ), where dst(x, y) s the Eucldean dstance between two ponts x and y. The sum of squared error that measures the qualty of cluster between all objects n C and centrod c, can be defned as: E = k n 2 dst ( p, c ) (1) 1 p C where E s the sum of squared error for all objects n the data set; p s the pont n space representng a gven object; and c s the centrod of cluster C. The tme complexty of the k- means algorthm s O(nkt), where n s the total number of objects, k s the number of clusters, and t s the number of teratons [6]. 3.2 EM-algorthm In many applcatons, probablstc model-based clusterng has been shown to be effectve, ths learnng algorthm starts wth an ntal set of parameters and terates untl the clusterng cannot be mproved. Generally, the EM- algorthm may not converge to the optmal soluton [6]. However, many heurstcs have been explored to ths stuaton, most especally by runnng EM process multple tmes usng dfferent random ntal values. The EM algorthm has two steps: an expectaton step and a maxmzaton step; the ntal expectaton step guesses what the parameters are usng pseudo-random numbers. In the maxmzaton step, the mean and varance s used to re-estmate the parameters contnually untl they converge to a local maxmum [17]. Expectaton step assgns objects accordng to the parameters of probablstc clusters, whle maxmzaton step fnds the new clusterng or expected lkelhood n probablstc model-based clusterng [6]. 3.3 k-medods Ths algorthm s an extenson of k-means paradgm; t clusters categorcal data as t uses a smple matchng dssmlarty measure for categorcal objects [18]. Instead of takng the mean value of the objects n a cluster as a reference pont, the actual objects can be pcked to represent the clusters, usng one representatve object per cluster. Each remanng object s assgned to the cluster of whch the representatve object s the most smlar. The parttonng method s then performed based on the prncple of mnmzng the sum of dssmlartes between each object p and ts correspondng representatve object. Ths s the bass for the k-medods method, whch groups n objects nto k clusters by mnmzng the absolute error crteron[6], whch can be defned as: k E = dst ( p, o ) (2) 1 p c where E s the sum of the absolute error for all objects p n the dataset, and o s the representatve object of c. 4. EPERIMENTATIONS Ths study experments on excerpts of the dataset retreved from an open repostory of the World Bank [21]. The data reflect the cross-country nformaton for Sector Investment and Captal n the year Experment n the proposed study s carred out n the RapdMner Software envronment to descrbe the features n the data that form the bass for the groupng. The data has three attrbutes: GDP per captal, urban populaton and surface area. The target groupng s Income (low, medum, hgh). The three partton algorthms used n ths study are represented n Table 1. Whle k-means measures the Eucldean dstance of the data ponts, k-medods measures the mxed Eucldean dstance of the data ponts n order to handle strng values and wth EM-algorthm, data ponts are randomly assgned to the parameter. In order to compare the cluster results to the label (ncome group) n the orgnal data, the dataset was clustered nto three. Table 1 shows the confguraton of the parameters n each algorthm. 22

3 Internatonal Journal of Appled Informaton Systems (IJAIS) ISSN : Foundaton of Computer Scence FCS, New York, USA Volume 7 No.7, August Fg. 1 Expermental setup Table 1: Parameter settngs for the algorthms Algorthms Number of clusters Max. runs Dataset Normalzaton Numercal measure Max optmzaton K-means 3 10 z-transform Eucldean dstance EM-Clusterng 3 10 z-transform Assgn values randomly K-medods 3 10 z-transform Mxed Eucldean The dataset s normalzed n order to express the attrbutes n smaller unts and accordng to [6], normalzed data gve attrbute greater effect. Normalzed and standardzed nvolves transformng the data to fall wthn small or common range such as [-1, 1] or [0.0, 1.0]. Z-score normalzaton s a data standardzaton method that normalzed attrbute values based on the mean and standard devaton of the attrbutes values. A value, x of A s normalzed to x 1 by computng: 1 A A (3) A varaton of z-score normalzaton replaces the standard devaton n (3) by the mean absolute devaton of A. Thus, z-score normalzaton usng the mean absolute devaton s: 1 A S A (4) where A and A are the mean and standard devaton, respectvely, of attrbute A. The mean absolute devaton, S A, s more robust to outlers than the standard devaton, [6]. A 23

4 Internatonal Journal of Appled Informaton Systems (IJAIS) ISSN : Foundaton of Computer Scence FCS, New York, USA Volume 7 No.7, August The clusterng of data usng k-means, EM-algorthm and K-medods conforms to fgures 1, 2 and 3 respectvely: The k-means algorthm 1. Place k ponts nto the space S 2. Assgn each object to the cluster that has the closest centrod 3. Re-compute the postons of the k centrod 4. Repeat steps 2 and 3 untl the centrods reman constant. The EM clusterng algorthm 1. Intalze to 0 and choose arbtrarly 2. (E-step): Compute Q( ) 3. (M-step): Choose +1 to maxmze Q( ) 4. If!= +1, then set to +1 and return to Step 2 The k-medods algorthm 1. Select the ntal medods 2. Determne the new medod of each cluster to update medods 3. Assgn each object to the nearest medod 4. Compute sum of dstance from all objects to ther medods 5. Repeat step 2 untl the sum remans constant. where s an unknown hdden varable. Fg. 2: k-means Fg. 3: EM clusterng Fg. 4: K-medods Input: k: the number of clusters; S: a data set contanng n objects Output: A set of k clusters 5. EPERIMENTAL RESULTS AND DISCUSSION Ths study experment wth two parttonng-based algorthms and a probablstc model-based algorthm, the clusters formed are shown n Fgures 5, 6, and 7. The algorthms operate under smlar parameter settngs as represented n Table 1. For the purpose of valdaton of the formed clusters, the results were exported to excel fle as shown n the expermental setup n Fgure 1 for further computatons. The scatter plots of k- means and k-medods looks much alke and comparng ther results to the class-label of the orgnal data beng analysed, each has 61% and 62% accuracy respectvely as dsplayed n Table 2. The EM-algorthm shows a dfferent result entrely and the comparsons show that t s 51.8% accurate. In general, the three algorthms are very fast, whle k-means reman the fastest among them. Fg. 6: Scatter plot usng k-medods algorthm Table 2: Performance accuracy of each algorthm on the dataset Algorthm Accuracy k-means 61% k-medods 62% EM-clusterng 51.8% Fg. 5: Scatter plot usng k-means algorthm 24

5 Internatonal Journal of Appled Informaton Systems (IJAIS) ISSN : Foundaton of Computer Scence FCS, New York, USA Volume 7 No.7, August Fg. 7: Scatter plot usng EM algorthm Gven a set, the true representatve of the set [19], s a representatve set of clusterng Centrods C 1,..., C k f C. As reported n [20], the class of an object cannot be predcted by a clusterng algorthm, but t may be estmated by examnng the clusterng result for the class-label data. In order to determne the accuracy of the clusters formed, the class-label of the orgnal data set s mapped to each cluster and the percentage accuracy s determned based on (5). S = Sample correctly mapped to the class-label n each cluster S T =Total number of sample data n each cluster k Accuracy = n 1 where k = 3 S ST x 100 (5) 6. CONCLUSION In ths paper, the performance of some parttonng-based clusterng algorthms and probablstc model-based algorthm, namely: k-means, k-medods and EM-algorthm on structured data are explored wth a vew to revealng how accurate each algorthm could perform n groupng the dataset. The expermental results are compared to the classlabel of the orgnal data and the performance accuracy of each algorthm dsplayed n table 2 shows that, k-means and k-medods are more effcent than EM-algorthm n the clusterng of data ponts. Whle k-means appear to be the fastest among the three algorthms and s known for ts excellent performance on large data, the algorthm however requres that, the value of k be contnuously vared to get a cluster of good qualty. REFERENCES [1] D. Napoleon and P. G. Lakshm, "An effcent K-Means clusterng algorthm for reducng tme complexty usng unform dstrbuton data ponts," n Trendz n Informaton Scences & Computng (TISC), 2010, pp [2] S. C. Suh, Practcal Applcatons of Data Mnng: Jones & Barlett Learnng, LLC [3] P. Berkhn, "A survey of clusterng data mnng technques," n Groupng multdmensonal data, ed: Sprnger, 2006, pp [4] B. Mrkn, Clusterng: A Data Recovery Approach: CRC Press, [5] G. M. Dayan, F. Abd, M. Khan, and A. H. Tareq, "An effcent grd algorthm for faster clusterng usng K medods approach," n Computer and Informaton Technology (ICCIT), th Internatonal Conference on, 2012, pp [6] J. Han, M. Kamber, and J. Pe, DATA MINING Concepts and Technques: Morgan Kaufmann, 3rd Edton, [7] C.-H. Ln, C.-C. Chen, H.-L. Lee, and J.-R. Lao, "Fast K-means algorthm based on a level hstogram for mage retreval," Expert Systems wth Applcatons, vol. 41, pp , [8] Z. Huang, "Extensons to the k-means algorthm for clusterng large data sets wth categorcal values," Data Mnng and Knowledge Dscovery, vol. 2, pp , [9] R. Forsat, M. Mahdav, M. Shamsfard, and M. Reza Meybod, "Effcent stochastc algorthms for document clusterng," Informaton Scences, vol. 220, pp , [10] C. Dng and T. L, "Adaptve dmenson reducton usng dscrmnant analyss and k-means clusterng," n Proceedngs of the 24th nternatonal conference on Machne learnng, 2007, pp [11] A. P. Reynolds, G. Rchards, and V. J. Rayward-Smth, "The applcaton of k-medods and pam to the clusterng of rules," n Intellgent Data Engneerng and Automated Learnng IDEAL 2004, ed: Sprnger, 2004, pp [12] S. M. Razav Zadegan, M. Mrzae, and F. Sadough, "Ranked< > k</>-medods: A fast and accurate rankbased parttonng algorthm for clusterng large datasets," Knowledge-Based Systems, vol. 39, pp , [13] R. Josh, A. Patdar, and S. Mshra, "Scalng k-medod algorthm for clusterng large categorcal dataset and ts performance analyss," n Electroncs Computer Technology (ICECT), rd Internatonal Conference on, 2011, pp [14] C. Ordonez and E. Omecnsk, "FREM: fast and robust EM clusterng for large data sets," n Proceedngs of the eleventh nternatonal conference on Informaton and knowledge management, 2002, pp [15] C. Ambrose, M. Dang, and G. Govaert, "Clusterng of spatal data by the EM algorthm," n geoenv I Geostatstcs for envronmental applcatons, ed: Sprnger, 1997, pp [16] C. Carson, S. Belonge, H. Greenspan, and J. Malk, "Blobworld: Image segmentaton usng expectatonmaxmzaton and ts applcaton to mage queryng," 25

6 Internatonal Journal of Appled Informaton Systems (IJAIS) ISSN : Foundaton of Computer Scence FCS, New York, USA Volume 7 No.7, August Pattern Analyss and Machne Intellgence, IEEE Transactons on, vol. 24, pp , [17] J. Erman, M. Arltt, and A. Mahant, "Traffc classfcaton usng clusterng algorthms," n Proceedngs of the 2006 SIGCOMM workshop on Mnng network data, 2006, pp [18] L. R. Kaufman and P. Rousseeuw, "Fndng groups n data: An ntroducton to cluster analyss," Hoboken NJ John Wley & Sons Inc, [19] S. Ben-Davd and M. Ackerman, "Measures of clusterng qualty: A workng set of axoms for clusterng," n Advances n neural nformaton processng systems, 2009, pp [20] H.-S. Park and C.-H. Jun, "A smple and fast algorthm for K-medods clusterng," Expert Systems wth Applcatons, vol. 36, pp , [21] A Cross-country Database for Sector Investment and Captal An open repostory of the World Bank: (accessed on June 23, 2014). 26