Cluster Analysis. Cluster Analysis

Transcription

1 Cluster Analyss Cluster Analyss What s Cluster Analyss? Types of Data n Cluster Analyss A Categorzaton of Maor Clusterng Methos Parttonng Methos Herarchcal Methos Densty-Base Methos Gr-Base Methos Moel-Base Clusterng Methos Outler Analyss Summary

2 What s Cluster Analyss? Cluster: a collecton of ata obects Smlar to one another wthn the same cluster Dssmlar to the obects n other clusters Cluster analyss Groupng a set of ata obects nto clusters Clusterng s unsupervse classfcaton: no preefne classes Clusterng s use: As a stan-alone tool to get nsght nto ata strbuton Vsualzaton of clusters may unvel mportant nformaton As a preprocessng step for other algorthms Effcent neng or compresson often reles on clusterng General Applcatons of Clusterng Pattern Recognton Spatal Data Analyss create thematc maps n GIS by clusterng feature spaces etect spatal clusters an eplan them n spatal ata mnng Image Processng cluster mages base on ther vsual content Economc Scence (especally market research) WWW an IR ocument classfcaton cluster Weblog ata to scover groups of smlar access patterns

3 What Is Goo Clusterng? A goo clusterng metho wll prouce hgh qualty clusters wth hgh ntra-class smlarty low nter-class smlarty The qualty of a clusterng result epens on both the smlarty measure use by the metho an ts mplementaton. The qualty of a clusterng metho s also measure by ts ablty to scover some or all of the hen patterns. Requrements of Clusterng n Data Mnng Scalablty Ablty to eal wth fferent types of attrbutes Dscovery of clusters wth arbtrary shape Mnmal requrements for oman knowlege to etermne nput parameters Able to eal wth nose an outlers Insenstve to orer of nput recors Hgh mensonalty Incorporaton of user-specfe constrants Interpretablty an usablty

4 Outlers Outlers are obects that o not belong to any cluster or form clusters of very small carnalty cluster outlers In some applcatons we are ntereste n scoverng outlers, not clusters (outler analyss) Cluster Analyss What s Cluster Analyss? Types of Data n Cluster Analyss A Categorzaton of Maor Clusterng Methos Parttonng Methos Herarchcal Methos Densty-Base Methos Gr-Base Methos Moel-Base Clusterng Methos Outler Analyss Summary

5 Data Structures ata matr (two moes) the classc ata nput ssmlarty or stance matr (one moe) the esre ata nput to some clusterng algorthms tuples/obects obects n (,) (, ) : ( n,) attrbutes/mensons f f nf obects (,) : ( n,) : p p np Measurng Smlarty n Clusterng Dssmlarty/Smlarty metrc: The ssmlarty (, ) between two obects an s epresse n terms of a stance functon, whch s typcally a metrc: (, ) (non-negatvty) (, )= (solaton) (, )= (, ) (symmetry) (, ) (, h)(h, ) (trangular nequalty) The efntons of stance functons are usually fferent for nterval-scale, boolean, categorcal, ornal an rato-scale varables. Weghts may be assocate wth fferent varables base on applcatons an ata semantcs.

6 Type of ata n cluster analyss Interval-scale varables e.g., salary, heght Bnary varables e.g., gener (M/F), has_cancer(t/f) Nomnal (categorcal) varables e.g., relgon (Chrstan, Muslm, Buhst, Hnu, etc.) Ornal varables e.g., mltary rank (soler, sergeant, lutenant, captan, etc.) Rato-scale varables populaton growth (,,,,) Varables of me types multple attrbutes wth varous types Smlarty an Dssmlarty Between Obects Dstance metrcs are normally use to measure the smlarty or ssmlarty between two ata obects The most popular conform to Mnkowsk stance: p p p / p L p (, ) = n n where = (,,, n ) an = (,,, n ) are two n-mensonal ata obects, an p s a postve nteger If p =, L s the Manhattan (or cty block) stance: L (, ) = n n

7 7 Smlarty an Dssmlarty Between Obects (Cont.) If p =, L s the Euclean stance: Propertes (,) (,) = (,) = (,) (,) (,k) (k,) Also one can use weghte stance: ) ( ), ( n n = ) ( ), ( n n n w w w = Bnary Varables A bnary varable has two states: absent, present A contngency table for bnary ata Smple matchng coeffcent stance (nvarant, f the bnary varable s symmetrc): Jaccar coeffcent stance (nonnvarant f the bnary varable s asymmetrc): c b a c b = ), ( c b a c b = ), ( p b c a sum c c b a b a sum obect obect

8 Bnary Varables Another approach s to efne the smlarty of two obects an not ther stance. In that case we have the followng: Smple matchng coeffcent smlarty: s(, ) = a a b c Jaccar coeffcent smlarty: s(, ) = a a b c Note that: s(,) = (,) Dssmlarty between Bnary Varables Eample (Jaccar coeffcent) Name Fever Cough Test- Test- Test- Test- Jack Mary Jm all attrbutes are asymmetrc bnary enotes presence or postve test enotes absence or negatve test ( ack, mary ) = =. ( ack, m ) = =.7 ( m, mary ) = =.7

9 A smpler efnton Each varable s mappe to a btmap (bnary vector) Name Fever Cough Test- Test- Test- Test- Jack Mary Jm Jack: Mary: Jm: Smple match stance: Jaccar coeffcent: number of non - common bt postons (, ) = total number of bts number of 's n (, ) = number of 's n Varables of Me Types A atabase may contan all the s types of varables symmetrc bnary, asymmetrc bnary, nomnal, ornal, nterval an rato-scale. One may use a weghte formula to combne ther effects. (, Σ ) = p f Σ δ ( f ) ( f ) = p ( f ) δ f =

10 Cluster Analyss What s Cluster Analyss? Types of Data n Cluster Analyss A Categorzaton of Maor Clusterng Methos Parttonng Methos Herarchcal Methos Densty-Base Methos Gr-Base Methos Moel-Base Clusterng Methos Outler Analyss Summary Maor Clusterng Approaches Parttonng algorthms: Construct ranom parttons an then teratvely refne them by some crteron Herarchcal algorthms: Create a herarchcal ecomposton of the set of ata (or obects) usng some crteron Densty-base: base on connectvty an ensty functons Gr-base: base on a multple-level granularty structure Moel-base: A moel s hypothesze for each of the clusters an the ea s to fn the best ft of that moel to each other

11 Cluster Analyss What s Cluster Analyss? Types of Data n Cluster Analyss A Categorzaton of Maor Clusterng Methos Parttonng Methos Herarchcal Methos Densty-Base Methos Gr-Base Methos Moel-Base Clusterng Methos Outler Analyss Summary Parttonng Algorthms: Basc Concepts Parttonng metho: Construct a partton of a atabase D of n obects nto a set of k clusters Gven a k, fn a partton of k clusters that optmzes the chosen parttonng crteron Global optmal: ehaustvely enumerate all parttons Heurstc methos: k-means an k-meos algorthms k-means (MacQueen 7): Each cluster s represente by the center of the cluster k-meos or PAM (Partton aroun meos) (Kaufman & Rousseeuw 7): Each cluster s represente by one of the obects n the cluster

12 The k-means Clusterng Metho Gven k, the k-means algorthm s mplemente n steps:. Partton obects nto k nonempty subsets. Compute see ponts as the centros of the clusters of the current partton. The centro s the center (mean pont) of the cluster.. Assgn each obect to the cluster wth the nearest see pont.. Go back to Step, stop when no more new assgnment. The k-means Clusterng Metho Eample

13 Comments on the k-means Metho Strength Relatvely effcent: O(tkn), where n s # obects, k s # clusters, an t s # teratons. Normally, k, t << n. Often termnates at a local optmum. Weaknesses Applcable only when mean s efne, then what about categorcal ata? Nee to specfy k, the number of clusters, n avance Unable to hanle nosy ata an outlers Not sutable to scover clusters wth non-conve shapes The K-Meos Clusterng Metho Fn representatve obects, calle meos, n clusters PAM (Parttonng Aroun Meos, 7) starts from an ntal set of meos an teratvely replaces one of the meos by one of the non-meos f t mproves the total stance of the resultng clusterng PAM works effectvely for small ata sets, but oes not scale well for large ata sets CLARA (Kaufmann & Rousseeuw, ) CLARANS (Ng & Han, ): Ranomze samplng

14 PAM (Parttonng Aroun Meos) (7) PAM (Kaufman an Rousseeuw, 7), bult n statstcal package S Use real obect to represent the cluster. Select k representatve obects arbtrarly. For each par of non-selecte obect h an selecte obect, calculate the total swappng cost TC h. For each par of an h, If TC h <, s replace by h Then assgn each non-selecte obect to the most smlar representatve obect. repeat steps - untl there s no change PAM Clusterng: Total swappng cost TC h = C h s a current meo, h s a nonselecte obect Assume that s replace by h n the set of meos TC h = ; For each non-selecte obect h: TC h = (,new_me )-(,prev_me ): new_me = the closest meo to after s replace by h prev_me = the closest meo to before s replace by h

15 PAM Clusterng: Total swappng cost TC h = C h 7 t h 7 t h 7 C h = (, h) - (, ) 7 C h = 7 h t 7 h t 7 C h = (, t) - (, ) 7 C h = (, h) - (, t) CLARA (Clusterng Large Applcatons) CLARA (Kaufmann an Rousseeuw n ) Bult n statstcal analyss packages, such as S It raws multple samples of the ata set, apples PAM on each sample, an gves the best clusterng as the output Strength: eals wth larger ata sets than PAM Weakness: Effcency epens on the sample sze A goo clusterng base on samples wll not necessarly represent a goo clusterng of the whole ata set f the sample s base

16 CLARANS ( Ranomze CLARA) CLARANS (A Clusterng Algorthm base on Ranomze Search) (Ng an Han ) CLARANS raws sample of neghbors ynamcally The clusterng process can be presente as searchng a graph where every noe s a potental soluton, that s, a set of k meos If the local optmum s foun, CLARANS starts wth new ranomly selecte noe n search for a new local optmum It s more effcent an scalable than both PAM an CLARA Focusng technques an spatal access structures may further mprove ts performance (Ester et al. )