Transcription

1 DRAFT! April 1, 009 Cambridge University Press Feedback welcme Flat clustering CLUSTER Clustering algrithms grup a set f dcuments int subsets r clusters The algrithms gal is t create clusters that are cherent internally, but clearly different frm each ther In ther wrds, dcuments within a cluster shuld be as similar as pssible; and dcuments in ne cluster shuld be as dissimilar as pssible frm dcuments in ther clusters Figure 161 An example f a data set with a clear cluster structure UNSUPERVISED LEARNING Clustering is the mst cmmn frm f unsupervised learning N supervisin means that there is n human expert wh has assigned dcuments t classes In clustering, it is the distributin and makeup f the data that will determine cluster membership A simple example is Figure 161 It is visually clear that there are three distinct clusters f pints This chapter and Chapter 17 intrduce algrithms that find such clusters in an unsupervised fashin The difference between clustering and classificatin may nt seem great at first After all, in bth cases we have a partitin f a set f dcuments int grups But as we will see the tw prblems are fundamentally different Classificatin is a frm f supervised learning (Chapter 13, page 56): ur gal is t replicate a categrical distinctin that a human supervisr im-

2 Flat clustering FLAT CLUSTERING HARD CLUSTERING SOFT CLUSTERING pses n the data In unsupervised learning, f which clustering is the mst imprtant example, we have n such teacher t guide us The key input t a clustering algrithm is the distance measure In Figure 161, the distance measure is distance in the D plane This measure suggests three different clusters in the figure In dcument clustering, the distance measure is ften als Euclidean distance Different distance measures give rise t different clusterings Thus, the distance measure is an imprtant means by which we can influence the utcme f clustering Flat clustering creates a flat set f clusters withut any explicit structure that wuld relate clusters t each ther Hierarchical clustering creates a hierarchy f clusters and will be cvered in Chapter 17 Chapter 17 als addresses the difficult prblem f labeling clusters autmatically A secnd imprtant distinctin can be made between hard and sft clustering algrithms Hard clustering cmputes a hard assignment each dcument is a member f exactly ne cluster The assignment f sft clustering algrithms is sft a dcument s assignment is a distributin ver all clusters In a sft assignment, a dcument has fractinal membership in several clusters Latent semantic indexing, a frm f dimensinality reductin, is a sft clustering algrithm (Chapter 18, page 417) This chapter mtivates the use f clustering in infrmatin retrieval by intrducing a number f applicatins (Sectin 161), defines the prblem we are trying t slve in clustering (Sectin 16) and discusses measures fr evaluating cluster quality (Sectin 163) It then describes tw flat clustering algrithms, K-means (Sectin 164), a hard clustering algrithm, and the Expectatin-Maximizatin (r EM) algrithm (Sectin 165), a sft clustering algrithm K-means is perhaps the mst widely used flat clustering algrithm due t its simplicity and efficiency The EM algrithm is a generalizatin f K-means and can be applied t a large variety f dcument representatins and distributins 161 Clustering in infrmatin retrieval CLUSTER HYPOTHESIS The cluster hypthesis states the fundamental assumptin we make when using clustering in infrmatin retrieval Cluster hypthesis Dcuments in the same cluster behave similarly with respect t relevance t infrmatin needs The hypthesis states that if there is a dcument frm a cluster that is relevant t a search request, then it is likely that ther dcuments frm the same cluster are als relevant This is because clustering puts tgether dcuments that share many terms The cluster hypthesis essentially is the cntiguity

3 161 Clustering in infrmatin retrieval 351 Applicatin What is Benefit Example clustered? Search result clustering search mre effective infrmatin Figure 16 results presentatin t user Scatter-Gather (subsets f) alternative user interface: Figure 163 cllectin search withut typing Cllectin clustering cllectin effective infrmatin presentatin fr explratry McKewn et al (00), brwsing Language mdeling cllectin increased precisin and/r recall Liu and Crft (004) Cluster-based retrieval cllectin higher efficiency: faster search Saltn (1971a) Table 161 Sme applicatins f clustering in infrmatin retrieval SEARCH RESULT CLUSTERING SCATTER-GATHER hypthesis in Chapter 14 (page 89) In bth cases, we psit that similar dcuments behave similarly with respect t relevance Table 161 shws sme f the main applicatins f clustering in infrmatin retrieval They differ in the set f dcuments that they cluster search results, cllectin r subsets f the cllectin and the aspect f an infrmatin retrieval system they try t imprve user experience, user interface, effectiveness r efficiency f the search system But they are all based n the basic assumptin stated by the cluster hypthesis The first applicatin mentined in Table 161 is search result clustering where by search results we mean the dcuments that were returned in respnse t a query The default presentatin f search results in infrmatin retrieval is a simple list Users scan the list frm tp t bttm until they have fund the infrmatin they are lking fr Instead, search result clustering clusters the search results, s that similar dcuments appear tgether It is ften easier t scan a few cherent grups than many individual dcuments This is particularly useful if a search term has different wrd senses The example in Figure 16 is jaguar Three frequent senses n the web refer t the car, the animal and an Apple perating system The Clustered Results panel returned by the Vivísim search engine ( can be a mre effective user interface fr understanding what is in the search results than a simple list f dcuments A better user interface is als the gal f Scatter-Gather, the secnd applicatin in Table 161 Scatter-Gather clusters the whle cllectin t get grups f dcuments that the user can select r gather The selected grups are merged and the resulting set is again clustered This prcess is repeated until a cluster f interest is fund An example is shwn in Figure 163

4 35 16 Flat clustering Figure 16 Clustering f search results t imprve recall Nne f the tp hits cver the animal sense f jaguar, but users can easily access it by clicking n the cat cluster in the Clustered Results panel n the left (third arrw frm the tp) Autmatically generated clusters like thse in Figure 163 are nt as neatly rganized as a manually cnstructed hierarchical tree like the Open Directry at Als, finding descriptive labels fr clusters autmatically is a difficult prblem (Sectin 177, page 396) But cluster-based navigatin is an interesting alternative t keywrd searching, the standard infrmatin retrieval paradigm This is especially true in scenaris where users prefer brwsing ver searching because they are unsure abut which search terms t use As an alternative t the user-mediated iterative clustering in Scatter-Gather, we can als cmpute a static hierarchical clustering f a cllectin that is nt influenced by user interactins ( Cllectin clustering in Table 161) Ggle News and its precursr, the Clumbia NewsBlaster system, are examples f this apprach In the case f news, we need t frequently recmpute the clustering t make sure that users can access the latest breaking stries Clustering is well suited fr access t a cllectin f news stries since news reading is nt really search, but rather a prcess f selecting a subset f stries abut recent events

5 161 Clustering in infrmatin retrieval 353 Figure 163 An example f a user sessin in Scatter-Gather A cllectin f New Yrk Times news stries is clustered ( scattered ) int eight clusters (tp rw) The user manually gathers three f these int a smaller cllectin Internatinal Stries and perfrms anther scattering peratin This prcess repeats until a small cluster with relevant dcuments is fund (eg, Trinidad) The furth applicatin f clustering explits the cluster hypthesis directly fr imprving search results, based n a clustering f the entire cllectin We use a standard inverted index t identify an initial set f dcuments that match the query, but we then add ther dcuments frm the same clusters even if they have lw similarity t the query Fr example, if the query is car and several car dcuments are taken frm a cluster f autmbile dcuments, then we can add dcuments frm this cluster that use terms ther than car (autmbile, vehicle etc) This can increase recall since a grup f dcuments with high mutual similarity is ften relevant as a whle Mre recently this idea has been used fr language mdeling Equatin (110), page 45, shwed that t avid sparse data prblems in the language mdeling apprach t IR, the mdel f dcument d can be interplated with a

6 Flat clustering? Exercise cllectin mdel But the cllectin cntains many dcuments with terms untypical f d By replacing the cllectin mdel with a mdel derived frm d s cluster, we get mre accurate estimates f the ccurrence prbabilities f terms in d Clustering can als speed up search As we saw in Sectin 63 (page 13) search in the vectr space mdel amunts t finding the nearest neighbrs t the query The inverted index supprts fast nearest-neighbr search fr the standard IR setting Hwever, smetimes we may nt be able t use an inverted index efficiently, eg, in latent semantic indexing (Chapter 18) In such cases, we culd cmpute the similarity f the query t every dcument, but this is slw The cluster hypthesis ffers an alternative: Find the clusters that are clsest t the query and nly cnsider dcuments frm these clusters Within this much smaller set, we can cmpute similarities exhaustively and rank dcuments in the usual way Since there are many fewer clusters than dcuments, finding the clsest cluster is fast; and since the dcuments matching a query are all similar t each ther, they tend t be in the same clusters While this algrithm is inexact, the expected decrease in search quality is small This is essentially the applicatin f clustering that was cvered in Sectin 716 (page 141) 161 Define tw dcuments as similar if they have at least tw prper names like Clintn r Sarkzy in cmmn Give an example f an infrmatin need and tw dcuments, fr which the cluster hypthesis des nt hld fr this ntin f similarity Exercise 16 Make up a simple ne-dimensinal example (ie pints n a line) with tw clusters where the inexactness f cluster-based retrieval shws up In yur example, retrieving clusters clse t the query shuld d wrse than direct nearest neighbr search 16 Prblem statement OBJECTIVE FUNCTION We can define the gal in hard flat clustering as fllws Given (i) a set f dcuments D = {d 1,, d N }, (ii) a desired number f clusters K, and (iii) an bjective functin that evaluates the quality f a clustering, we want t cmpute an assignment γ : D {1,, K} that minimizes (r, in ther cases, maximizes) the bjective functin In mst cases, we als demand that γ is surjective, ie, that nne f the K clusters is empty The bjective functin is ften defined in terms f similarity r distance between dcuments Belw, we will see that the bjective in K-means clustering is t minimize the average distance between dcuments and their centrids r, equivalently, t maximize the similarity between dcuments and their centrids The discussin f similarity measures and distance metrics

7 16 Prblem statement 355 in Chapter 14 (page 91) als applies t this chapter As in Chapter 14, we use bth similarity and distance t talk abut relatedness between dcuments Fr dcuments, the type f similarity we want is usually tpic similarity r high values n the same dimensins in the vectr space mdel Fr example, dcuments abut China have high values n dimensins like Chinese, Beijing, and Ma whereas dcuments abut the UK tend t have high values fr Lndn, Britain and Queen We apprximate tpic similarity with csine similarity r Euclidean distance in vectr space (Chapter 6) If we intend t capture similarity f a type ther than tpic, fr example, similarity f language, then a different representatin may be apprpriate When cmputing tpic similarity, stp wrds can be safely ignred, but they are imprtant cues fr separating clusters f English (in which the ccurs frequently and la infrequently) and French dcuments (in which the ccurs infrequently and la frequently) PARTITIONAL CLUSTERING EXHAUSTIVE EXCLUSIVE A nte n terminlgy An alternative definitin f hard clustering is that a dcument can be a full member f mre than ne cluster Partitinal clustering always refers t a clustering where each dcument belngs t exactly ne cluster (But in a partitinal hierarchical clustering (Chapter 17) all members f a cluster are f curse als members f its parent) On the definitin f hard clustering that permits multiple membership, the difference between sft clustering and hard clustering is that membership values in hard clustering are either 0 r 1, whereas they can take n any nn-negative value in sft clustering Sme researchers distinguish between exhaustive clusterings that assign each dcument t a cluster and nn-exhaustive clusterings, in which sme dcuments will be assigned t n cluster Nn-exhaustive clusterings in which each dcument is a member f either n cluster r ne cluster are called exclusive We define clustering t be exhaustive in this bk 161 Cardinality the number f clusters CARDINALITY A difficult issue in clustering is determining the number f clusters r cardinality f a clustering, which we dente by K Often K is nthing mre than a gd guess based n experience r dmain knwledge But fr K-means, we will als intrduce a heuristic methd fr chsing K and an attempt t incrprate the selectin f K int the bjective functin Smetimes the applicatin puts cnstraints n the range f K Fr example, the Scatter-Gather interface in Figure 163 culd nt display mre than abut K = 10 clusters per layer because f the size and reslutin f cmputer mnitrs in the early 1990s Since ur gal is t ptimize an bjective functin, clustering is essentially

8 Flat clustering a search prblem The brute frce slutin wuld be t enumerate all pssible clusterings and pick the best Hwever, there are expnentially many partitins, s this apprach is nt feasible 1 Fr this reasn, mst flat clustering algrithms refine an initial partitining iteratively If the search starts at an unfavrable initial pint, we may miss the glbal ptimum Finding a gd starting pint is therefre anther imprtant prblem we have t slve in flat clustering 163 Evaluatin f clustering INTERNAL CRITERION OF QUALITY EXTERNAL CRITERION OF QUALITY PURITY Typical bjective functins in clustering frmalize the gal f attaining high intra-cluster similarity (dcuments within a cluster are similar) and lw intercluster similarity (dcuments frm different clusters are dissimilar) This is an internal criterin fr the quality f a clustering But gd scres n an internal criterin d nt necessarily translate int gd effectiveness in an applicatin An alternative t internal criteria is direct evaluatin in the applicatin f interest Fr search result clustering, we may want t measure the time it takes users t find an answer with different clustering algrithms This is the mst direct evaluatin, but it is expensive, especially if large user studies are necessary As a surrgate fr user judgments, we can use a set f classes in an evaluatin benchmark r gld standard (see Sectin 85, page 164, and Sectin 136, page 79) The gld standard is ideally prduced by human judges with a gd level f inter-judge agreement (see Chapter 8, page 15) We can then cmpute an external criterin that evaluates hw well the clustering matches the gld standard classes Fr example, we may want t say that the ptimal clustering f the search results fr jaguar in Figure 16 cnsists f three classes crrespnding t the three senses car, animal, and perating system In this type f evaluatin, we nly use the partitin prvided by the gld standard, nt the class labels This sectin intrduces fur external criteria f clustering quality Purity is a simple and transparent evaluatin measure Nrmalized mutual infrmatin can be infrmatin-theretically interpreted The Rand index penalizes bth false psitive and false negative decisins during clustering The F measure in additin supprts differential weighting f these tw types f errrs T cmpute purity, each cluster is assigned t the class which is mst frequent in the cluster, and then the accuracy f this assignment is measured by cunting the number f crrectly assigned dcuments and dividing by N 1 An upper bund n the number f clusterings is K N /K! The exact number f different partitins f N dcuments int K clusters is the Stirling number f the secnd kind See r Cmtet (1974)

9 163 Evaluatin f clustering 357 cluster 1 cluster cluster 3 x x x x x x x x Figure 164 Purity as an external evaluatin criterin fr cluster quality Majrity class and number f members f the majrity class fr the three clusters are: x, 5 (cluster 1);, 4 (cluster ); and, 3 (cluster 3) Purity is (1/17) (5 4 3) 071 purity NMI RI F 5 lwer bund maximum value fr Figure Table 16 Figure 164 The fur external evaluatin measures applied t the clustering in Frmally: (161) purity(ω, C) = 1 N k max ω k c j j NORMALIZED MUTUAL INFORMATION where Ω = {ω 1, ω,, ω K } is the set f clusters and C = {c 1, c,, c J } is the set f classes We interpret ω k as the set f dcuments in ω k and c j as the set f dcuments in c j in Equatin (161) We present an example f hw t cmpute purity in Figure 164 Bad clusterings have purity values clse t 0, a perfect clustering has a purity f 1 Purity is cmpared with the ther three measures discussed in this chapter in Table 16 High purity is easy t achieve when the number f clusters is large in particular, purity is 1 if each dcument gets its wn cluster Thus, we cannt use purity t trade ff the quality f the clustering against the number f clusters A measure that allws us t make this tradeff is nrmalized mutual infr- Recall ur nte f cautin frm Figure 14 (page 91) when lking at this and ther D figures in this and the fllwing chapter: these illustratins can be misleading because D prjectins f length-nrmalized vectrs distrt similarities and distances between pints

10 Flat clustering (16) (163) (164) (165) (166) matin r NMI: NMI(Ω, C) = I(Ω; C) [H(Ω) H(C)]/ I is mutual infrmatin (cf Chapter 13, page 7): I(Ω; C) = k P(ω k c j ) lg P(ω k c j ) P(ω j k )P(c j ) ω k c j = lg N ω k c j N ω k j k c j where P(ω k ), P(c j ), and P(ω k c j ) are the prbabilities f a dcument being in cluster ω k, class c j, and in the intersectin f ω k and c j, respectively Equatin (164) is equivalent t Equatin (163) fr maximum likelihd estimates f the prbabilities (ie, the estimate f each prbability is the crrespnding relative frequency) H is entrpy as defined in Chapter 5 (page 99): H(Ω) = P(ω k ) lg P(ω k ) k = k ω k N lg ω k N where, again, the secnd equatin is based n maximum likelihd estimates f the prbabilities I(Ω; C) in Equatin (163) measures the amunt f infrmatin by which ur knwledge abut the classes increases when we are tld what the clusters are The minimum f I(Ω; C) is 0 if the clustering is randm with respect t class membership In that case, knwing that a dcument is in a particular cluster des nt give us any new infrmatin abut what its class might be Maximum mutual infrmatin is reached fr a clustering Ω exact that perfectly recreates the classes but als if clusters in Ω exact are further subdivided int smaller clusters (Exercise 167) In particular, a clustering with K = N nedcument clusters has maximum MI S MI has the same prblem as purity: it des nt penalize large cardinalities and thus des nt frmalize ur bias that, ther things being equal, fewer clusters are better The nrmalizatin by the denminatr [H(Ω) H(C)]/ in Equatin (16) fixes this prblem since entrpy tends t increase with the number f clusters Fr example, H(Ω) reaches its maximum lg N fr K = N, which ensures that NMI is lw fr K = N Because NMI is nrmalized, we can use it t cmpare clusterings with different numbers f clusters The particular frm f the denminatr is chsen because [H(Ω) H(C)]/ is a tight upper bund n I(Ω; C) (Exercise 168) Thus, NMI is always a number between 0 and 1

11 163 Evaluatin f clustering 359 RAND INDEX RI An alternative t this infrmatin-theretic interpretatin f clustering is t view it as a series f decisins, ne fr each f the N(N 1)/ pairs f dcuments in the cllectin We want t assign tw dcuments t the same cluster if and nly if they are similar A true psitive (TP) decisin assigns tw similar dcuments t the same cluster, a true negative (TN) decisin assigns tw dissimilar dcuments t different clusters There are tw types f errrs we can cmmit A false psitive (FP) decisin assigns tw dissimilar dcuments t the same cluster A false negative (FN) decisin assigns tw similar dcuments t different clusters The Rand index (RI) measures the percentage f decisins that are crrect That is, it is simply accuracy (Sectin 83, page 155) RI = TP TN TP FP FN TN As an example, we cmpute RI fr Figure 164 We first cmpute TP FP The three clusters cntain 6, 6, and 5 pints, respectively, s the ttal number f psitives r pairs f dcuments that are in the same cluster is: ( 6 TP FP = ) ( 6 ) ( 5 ) = 40 Of these, the x pairs in cluster 1, the pairs in cluster, the pairs in cluster 3, and the x pair in cluster 3 are true psitives: ( 5 TP = ) ( 4 ) ( 3 ) ( ) = 0 Thus, FP = 40 0 = 0 FN and TN are cmputed similarly, resulting in the fllwing cntingency table: Same cluster Different clusters Same class TP = 0 FN = 4 Different classes FP = 0 TN = 7 F MEASURE RI is then (0 7)/( ) 068 The Rand index gives equal weight t false psitives and false negatives Separating similar dcuments is smetimes wrse than putting pairs f dissimilar dcuments in the same cluster We can use the F measure (Sectin 83, page 154) t penalize false negatives mre strngly than false psitives by selecting a value β > 1, thus giving mre weight t recall P = TP TP FP R = TP TP FN F β = (β 1)PR β P R

12 Flat clustering? Exercise Based n the numbers in the cntingency table, P = 0/40 = 05 and R = 0/ This gives us F fr β = 1 and F fr β = 5 In infrmatin retrieval, evaluating clustering with F has the advantage that the measure is already familiar t the research cmmunity 163 Replace every pint d in Figure 164 with tw identical cpies f d in the same class (i) Is it less difficult, equally difficult r mre difficult t cluster this set f 34 pints as ppsed t the 17 pints in Figure 164? (ii) Cmpute purity, NMI, RI, and F 5 fr the clustering with 34 pints Which measures increase and which stay the same after dubling the number f pints? (iii) Given yur assessment in (i) and the results in (ii), which measures are best suited t cmpare the quality f the tw clusterings? 164 K-means CENTROID K-means is the mst imprtant flat clustering algrithm Its bjective is t minimize the average squared Euclidean distance (Chapter 6, page 131) f dcuments frm their cluster centers where a cluster center is defined as the mean r centrid µ f the dcuments in a cluster ω: µ(ω) = 1 ω x x ω RESIDUAL SUM OF SQUARES The definitin assumes that dcuments are represented as length-nrmalized vectrs in a real-valued space in the familiar way We used centrids fr Rcchi classificatin in Chapter 14 (page 9) They play a similar rle here The ideal cluster in K-means is a sphere with the centrid as its center f gravity Ideally, the clusters shuld nt verlap Our desiderata fr classes in Rcchi classificatin were the same The difference is that we have n labeled training set in clustering fr which we knw which dcuments shuld be in the same cluster A measure f hw well the centrids represent the members f their clusters is the residual sum f squares r RSS, the squared distance f each vectr frm its centrid summed ver all vectrs: RSS k = x ω k x µ(ω k ) (167) RSS = K RSS k k=1 RSS is the bjective functin in K-means and ur gal is t minimize it Since N is fixed, minimizing RSS is equivalent t minimizing the average squared distance, a measure f hw well centrids represent their dcuments

13 164 K-means 361 K-MEANS({ x 1,, x N }, K) 1 ( s 1, s,, s K ) SELECTRANDOMSEEDS({ x 1,, x N }, K) fr k 1 t K 3 d µ k s k 4 while stpping criterin has nt been met 5 d fr k 1 t K 6 d ω k {} 7 fr n 1 t N 8 d j arg min j µ j x n 9 ω j ω j { x n } (reassignment f vectrs) 10 fr k 1 t K 11 d µ k 1 ω k x ω k x (recmputatin f centrids) 1 return { µ 1,, µ K } Figure 165 The K-means algrithm Fr mst IR applicatins, the vectrs x n R M shuld be length-nrmalized Alternative methds f seed selectin and initializatin are discussed n page 364 SEED The first step f K-means is t select as initial cluster centers K randmly selected dcuments, the seeds The algrithm then mves the cluster centers arund in space in rder t minimize RSS As shwn in Figure 165, this is dne iteratively by repeating tw steps until a stpping criterin is met: reassigning dcuments t the cluster with the clsest centrid; and recmputing each centrid based n the current members f its cluster Figure 166 shws snapshts frm nine iteratins f the K-means algrithm fr a set f pints The centrid clumn f Table 17 (page 397) shws examples f centrids We can apply ne f the fllwing terminatin cnditins A fixed number f iteratins I has been cmpleted This cnditin limits the runtime f the clustering algrithm, but in sme cases the quality f the clustering will be pr because f an insufficient number f iteratins Assignment f dcuments t clusters (the partitining functin γ) des nt change between iteratins Except fr cases with a bad lcal minimum, this prduces a gd clustering, but runtimes may be unacceptably lng Centrids µ k d nt change between iteratins This is equivalent t γ nt changing (Exercise 165) Terminate when RSS falls belw a threshld This criterin ensures that the clustering is f a desired quality after terminatin In practice, we

14 36 16 Flat clustering selectin f seeds assignment f dcuments (iter 1) recmputatin/mvement f µ s (iter 1) µ s after cnvergence (iter 9) mvement f µ s in 9 iteratins Figure 166 A K-means example fr K = in R The psitin f the tw centrids ( µ s shwn as X s in the tp fur panels) cnverges after nine iteratins

15 164 K-means 363 need t cmbine it with a bund n the number f iteratins t guarantee terminatin Terminate when the decrease in RSS falls belw a threshld θ Fr small θ, this indicates that we are clse t cnvergence Again, we need t cmbine it with a bund n the number f iteratins t prevent very lng runtimes We nw shw that K-means cnverges by prving that RSS mntnically decreases in each iteratin We will use decrease in the meaning decrease r des nt change in this sectin First, RSS decreases in the reassignment step since each vectr is assigned t the clsest centrid, s the distance it cntributes t RSS decreases Secnd, it decreases in the recmputatin step because the new centrid is the vectr v fr which RSS k reaches its minimum (168) (169) RSS k ( v) = x ω k v x = RSS k ( v) v m = x ω k (v m x m ) M x ω k m=1 (v m x m ) where x m and v m are the m th cmpnents f their respective vectrs Setting the partial derivative t zer, we get: (1610) OUTLIER SINGLETON CLUSTER v m = 1 ω k x ω k x m which is the cmpnentwise definitin f the centrid Thus, we minimize RSS k when the ld centrid is replaced with the new centrid RSS, the sum f the RSS k, must then als decrease during recmputatin Since there is nly a finite set f pssible clusterings, a mntnically decreasing algrithm will eventually arrive at a (lcal) minimum Take care, hwever, t break ties cnsistently, eg, by assigning a dcument t the cluster with the lwest index if there are several equidistant centrids Otherwise, the algrithm can cycle frever in a lp f clusterings that have the same cst While this prves the cnvergence f K-means, there is unfrtunately n guarantee that a glbal minimum in the bjective functin will be reached This is a particular prblem if a dcument set cntains many utliers, dcuments that are far frm any ther dcuments and therefre d nt fit well int any cluster Frequently, if an utlier is chsen as an initial seed, then n ther vectr is assigned t it during subsequent iteratins Thus, we end up with a singletn cluster (a cluster with nly ne dcument) even thugh there is prbably a clustering with lwer RSS Figure 167 shws an example f a subptimal clustering resulting frm a bad chice f initial seeds

16 Flat clustering d 1 d d 3 d 4 d 5 d Figure 167 The utcme f clustering in K-means depends n the initial seeds Fr seeds d and d 5, K-means cnverges t {{d 1, d, d 3 }, {d 4, d 5, d 6 }}, a subptimal clustering Fr seeds d and d 3, it cnverges t {{d 1, d, d 4, d 5 }, {d 3, d 6 }}, the glbal ptimum fr K = Anther type f subptimal clustering that frequently ccurs is ne with empty clusters (Exercise 1611) Effective heuristics fr seed selectin include (i) excluding utliers frm the seed set; (ii) trying ut multiple starting pints and chsing the clustering with lwest cst; and (iii) btaining seeds frm anther methd such as hierarchical clustering Since deterministic hierarchical clustering methds are mre predictable than K-means, a hierarchical clustering f a small randm sample f size ik (eg, fr i = 5 r i = 10) ften prvides gd seeds (see the descriptin f the Bucksht algrithm, Chapter 17, page 399) Other initializatin methds cmpute seeds that are nt selected frm the vectrs t be clustered A rbust methd that wrks well fr a large variety f dcument distributins is t select i (eg, i = 10) randm vectrs fr each cluster and use their centrid as the seed fr this cluster See Sectin 166 fr mre sphisticated initializatins What is the time cmplexity f K-means? Mst f the time is spent n cmputing vectr distances One such peratin csts Θ(M) The reassignment step cmputes KN distances, s its verall cmplexity is Θ(KN M) In the recmputatin step, each vectr gets added t a centrid nce, s the cmplexity f this step is Θ(NM) Fr a fixed number f iteratins I, the verall cmplexity is therefre Θ(IKN M) Thus, K-means is linear in all relevant factrs: iteratins, number f clusters, number f vectrs and dimensinality f the space This means that K-means is mre efficient than the hierarchical algrithms in Chapter 17 We had t fix the number f iteratins I, which can be tricky in practice But in mst cases, K-means quickly reaches either cmplete cnvergence r a clustering that is clse t cnvergence In the latter case, a few dcuments wuld switch membership if further iteratins were cmputed, but this has a small effect n the verall quality f the clustering

17 164 K-means K-MEDOIDS MEDOID There is ne subtlety in the preceding argument Even a linear algrithm can be quite slw if ne f the arguments f Θ( ) is large, and M usually is large High dimensinality is nt a prblem fr cmputing the distance between tw dcuments Their vectrs are sparse, s that nly a small fractin f the theretically pssible M cmpnentwise differences need t be cmputed Centrids, hwever, are dense since they pl all terms that ccur in any f the dcuments f their clusters As a result, distance cmputatins are time cnsuming in a naive implementatin f K-means Hwever, there are simple and effective heuristics fr making centrid-dcument similarities as fast t cmpute as dcument-dcument similarities Truncating centrids t the mst significant k terms (eg, k = 1000) hardly decreases cluster quality while achieving a significant speedup f the reassignment step (see references in Sectin 166) The same efficiency prblem is addressed by K-medids, a variant f K- means that cmputes medids instead f centrids as cluster centers We define the medid f a cluster as the dcument vectr that is clsest t the centrid Since medids are sparse dcument vectrs, distance cmputatins are fast Cluster cardinality in K-means We stated in Sectin 16 that the number f clusters K is an input t mst flat clustering algrithms What d we d if we cannt cme up with a plausible guess fr K? A naive apprach wuld be t select the ptimal value f K accrding t the bjective functin, namely the value f K that minimizes RSS Defining RSS min (K) as the minimal RSS f all clusterings with K clusters, we bserve that RSS min (K) is a mntnically decreasing functin in K (Exercise 1613), which reaches its minimum 0 fr K = N where N is the number f dcuments We wuld end up with each dcument being in its wn cluster Clearly, this is nt an ptimal clustering A heuristic methd that gets arund this prblem is t estimate RSS min (K) as fllws We first perfrm i (eg, i = 10) clusterings with K clusters (each with a different initializatin) and cmpute the RSS f each Then we take the minimum f the i RSS values We dente this minimum by RSS min (K) Nw we can inspect the values RSS min (K) as K increases and find the knee in the curve the pint where successive decreases in RSS min becme nticeably smaller There are tw such pints in Figure 168, ne at K = 4, where the gradient flattens slightly, and a clearer flattening at K = 9 This is typical: there is seldm a single best number f clusters We still need t emply an external cnstraint t chse frm a number f pssible values f K (4 and 9 in this case)

18 Flat clustering residual sum f squares number f clusters Figure 168 Estimated minimal residual sum f squares as a functin f the number f clusters in K-means In this clustering f 103 Reuters-RCV1 dcuments, there are tw pints where the RSS min curve flattens: at 4 clusters and at 9 clusters The dcuments were selected frm the categries China, Germany, Russia and Sprts, s the K = 4 clustering is clsest t the Reuters classificatin DISTORTION MODEL COMPLEXITY (1611) A secnd type f criterin fr cluster cardinality impses a penalty fr each new cluster where cnceptually we start with a single cluster cntaining all dcuments and then search fr the ptimal number f clusters K by successively incrementing K by ne T determine the cluster cardinality in this way, we create a generalized bjective functin that cmbines tw elements: distrtin, a measure f hw much dcuments deviate frm the prttype f their clusters (eg, RSS fr K-means); and a measure f mdel cmplexity We interpret a clustering here as a mdel f the data Mdel cmplexity in clustering is usually the number f clusters r a functin theref Fr K-means, we then get this selectin criterin fr K: K = arg min[rss min (K) λk] K where λ is a weighting factr A large value f λ favrs slutins with few clusters Fr λ = 0, there is n penalty fr mre clusters and K = N is the best slutin

19 164 K-means 367 AKAIKE INFORMATION CRITERION The bvius difficulty with Equatin (1611) is that we need t determine λ Unless this is easier than determining K directly, then we are back t square ne In sme cases, we can chse values f λ that have wrked well fr similar data sets in the past Fr example, if we peridically cluster news stries frm a newswire, there is likely t be a fixed value f λ that gives us the right K in each successive clustering In this applicatin, we wuld nt be able t determine K based n past experience since K changes A theretical justificatin fr Equatin (1611) is the Akaike Infrmatin Criterin r AIC, an infrmatin-theretic measure that trades ff distrtin against mdel cmplexity The general frm f AIC is: (161) AIC: K = arg min[ L(K) q(k)] K where L(K), the negative maximum lg-likelihd f the data fr K clusters, is a measure f distrtin and q(k), the number f parameters f a mdel with K clusters, is a measure f mdel cmplexity We will nt attempt t derive the AIC here, but it is easy t understand intuitively The first prperty f a gd mdel f the data is that each data pint is mdeled well by the mdel This is the gal f lw distrtin But mdels shuld als be small (ie, have lw mdel cmplexity) since a mdel that merely describes the data (and therefre has zer distrtin) is wrthless AIC prvides a theretical justificatin fr ne particular way f weighting these tw factrs, distrtin and mdel cmplexity, when selecting a mdel Fr K-means, the AIC can be stated as fllws: (1613) AIC: K = arg min[rss min (K) MK] K Equatin (1613) is a special case f Equatin (1611) fr λ = M T derive Equatin (1613) frm Equatin (161) bserve that q(k) = KM in K-means since each element f the K centrids is a parameter that can be varied independently; and that L(K) = (1/)RSS min (K) (mdul a cnstant) if we view the mdel underlying K-means as a Gaussian mixture with hard assignment, unifrm cluster prirs and identical spherical cvariance matrices (see Exercise 1619) The derivatin f AIC is based n a number f assumptins, eg, that the data are independent and identically distributed These assumptins are nly apprximately true fr data sets in infrmatin retrieval As a cnsequence, the AIC can rarely be applied withut mdificatin in text clustering In Figure 168, the dimensinality f the vectr space is M 50,000 Thus, MK > 50,000 dminates the smaller RSS-based term ( RSS min (1) < 5000, nt shwn in the figure) and the minimum f the expressin is reached fr K = 1 But as we knw, K = 4 (crrespnding t the fur classes China,

20 Flat clustering 165? Exercise MODEL-BASED CLUSTERING Germany, Russia and Sprts) is a better chice than K = 1 In practice, Equatin (1611) is ften mre useful than Equatin (1613) with the caveat that we need t cme up with an estimate fr λ 164 Why are dcuments that d nt use the same term fr the cncept car likely t end up in the same cluster in K-means clustering? Exercise 165 Tw f the pssible terminatin cnditins fr K-means were (1) assignment des nt change, () centrids d nt change (page 361) D these tw cnditins imply each ther? Mdel-based clustering In this sectin, we describe a generalizatin f K-means, the EM algrithm It can be applied t a larger variety f dcument representatins and distributins than K-means In K-means, we attempt t find centrids that are gd representatives We can view the set f K centrids as a mdel that generates the data Generating a dcument in this mdel cnsists f first picking a centrid at randm and then adding sme nise If the nise is nrmally distributed, this prcedure will result in clusters f spherical shape Mdel-based clustering assumes that the data were generated by a mdel and tries t recver the riginal mdel frm the data The mdel that we recver frm the data then defines clusters and an assignment f dcuments t clusters A cmmnly used criterin fr estimating the mdel parameters is maximum likelihd In K-means, the quantity exp( RSS) is prprtinal t the likelihd that a particular mdel (ie, a set f centrids) generated the data Fr K-means, maximum likelihd and minimal RSS are equivalent criteria We dente the mdel parameters by Θ In K-means, Θ = { µ 1,, µ K } Mre generally, the maximum likelihd criterin is t select the parameters Θ that maximize the lg-likelihd f generating the data D: Θ = arg max Θ L(D Θ) = arg max Θ lg N n=1 P(d n Θ) = arg max Θ N n=1 lg P(d n Θ) L(D Θ) is the bjective functin that measures the gdness f the clustering Given tw clusterings with the same number f clusters, we prefer the ne with higher L(D Θ) This is the same apprach we tk in Chapter 1 (page 37) fr language mdeling and in Sectin 131 (page 65) fr text classificatin In text classificatin, we chse the class that maximizes the likelihd f generating a particular dcument Here, we chse the clustering Θ that maximizes the

21 165 Mdel-based clustering 369 likelihd f generating a given set f dcuments Once we have Θ, we can cmpute an assignment prbability P(d ω k ; Θ) fr each dcument-cluster pair This set f assignment prbabilities defines a sft clustering An example f a sft assignment is that a dcument abut Chinese cars may have a fractinal membership f 05 in each f the tw clusters China and autmbiles, reflecting the fact that bth tpics are pertinent A hard clustering like K-means cannt mdel this simultaneus relevance t tw tpics Mdel-based clustering prvides a framewrk fr incrprating ur knwledge abut a dmain K-means and the hierarchical algrithms in Chapter 17 make fairly rigid assumptins abut the data Fr example, clusters in K-means are assumed t be spheres Mdel-based clustering ffers mre flexibility The clustering mdel can be adapted t what we knw abut the underlying distributin f the data, be it Bernulli (as in the example in Table 163), Gaussian with nn-spherical variance (anther mdel that is imprtant in dcument clustering) r a member f a different family A cmmnly used algrithm fr mdel-based clustering is the Expectatin- Maximizatin algrithm r EM algrithm EM clustering is an iterative algrithm that maximizes L(D Θ) EM can be applied t many different types f prbabilistic mdeling We will wrk with a mixture f multivariate Bernulli distributins here, the distributin we knw frm Sectin 113 (page ) and Sectin 133 (page 63): EXPECTATION- MAXIMIZATION ALGORITHM (1614) (1615) P(d ω k ; Θ) = ( t m d ) ( ) q mk (1 q mk ) t m / d where Θ = {Θ 1,, Θ K }, Θ k = (α k, q 1k,, q Mk ), and q mk = P(U m = 1 ω k ) are the parameters f the mdel 3 P(U m = 1 ω k ) is the prbability that a dcument frm cluster ω k cntains term t m The prbability α k is the prir f cluster ω k : the prbability that a dcument d is in ω k if we have n infrmatin abut d The mixture mdel then is: ) P(d Θ) = ( ) ( K α k q mk (1 q mk ) k=1 t m d t m / d In this mdel, we generate a dcument by first picking a cluster k with prbability α k and then generating the terms f the dcument accrding t the parameters q mk Recall that the dcument representatin f the multivariate Bernulli is a vectr f M Blean values (and nt a real-valued vectr) 3 U m is the randm variable we defined in Sectin 133 (page 66) fr the Bernulli Naive Bayes mdel It takes the values 1 (term t m is present in the dcument) and 0 (term t m is absent in the dcument)

22 Flat clustering EXPECTATION STEP MAXIMIZATION STEP Hw d we use EM t infer the parameters f the clustering frm the data? That is, hw d we chse parameters Θ that maximize L(D Θ)? EM is similar t K-means in that it alternates between an expectatin step, crrespnding t reassignment, and a maximizatin step, crrespnding t recmputatin f the parameters f the mdel The parameters f K-means are the centrids, the parameters f the instance f EM in this sectin are the α k and q mk The maximizatin step recmputes the cnditinal parameters q mk and the prirs α k as fllws: (1616) Maximizatin step: q mk = N n=1 r nk I(t m d n ) N n=1 r nk α k = N n=1 r nk N where I(t m d n ) = 1 if t m d n and 0 therwise and r nk is the sft assignment f dcument d n t cluster k as cmputed in the preceding iteratin (We ll address the issue f initializatin in a mment) These are the maximum likelihd estimates fr the parameters f the multivariate Bernulli frm Table 133 (page 68) except that dcuments are assigned fractinally t clusters here These maximum likelihd estimates maximize the likelihd f the data given the mdel The expectatin step cmputes the sft assignment f dcuments t clusters given the current parameters q mk and α k : (1617) Expectatin step : r nk = α k( tm d n q mk )( tm / d n (1 q mk )) K k=1 α k( tm d n q mk )( tm / d n (1 q mk )) This expectatin step applies Equatins (1614) and (1615) t cmputing the likelihd that ω k generated dcument d n It is the classificatin prcedure fr the multivariate Bernulli in Table 133 Thus, the expectatin step is nthing else but Bernulli Naive Bayes classificatin (including nrmalizatin, ie dividing by the denminatr, t get a prbability distributin ver clusters) We clustered a set f 11 dcuments int tw clusters using EM in Table 163 After cnvergence in iteratin 5, the first 5 dcuments are assigned t cluster 1 (r i,1 = 100) and the last 6 t cluster (r i,1 = 000) Smewhat atypically, the final assignment is a hard assignment here EM usually cnverges t a sft assignment In iteratin 5, the prir α 1 fr cluster 1 is 5/ because 5 f the 11 dcuments are in cluster 1 Sme terms are quickly assciated with ne cluster because the initial assignment can spread t them unambiguusly Fr example, membership in cluster spreads frm dcument 7 t dcument 8 in the first iteratin because they share sugar (r 8,1 = 0 in iteratin 1) Fr parameters f terms ccurring in ambiguus cntexts, cnvergence takes lnger Seed dcuments 6 and 7

23 165 Mdel-based clustering 371 (a) dcid dcument text dcid dcument text 1 ht chclate cca beans 7 sweet sugar cca ghana africa 8 sugar cane brazil 3 beans harvest ghana 9 sweet sugar beet 4 cca butter 10 sweet cake icing 5 butter truffles 11 cake black frest 6 sweet chclate (b) Parameter Iteratin f clustering α r 1, r, r 3, r 4, r 5, r 6, r 7, r 8, r 9, r 10, r 11, q africa, q africa, q brazil, q brazil, q cca, q cca, q sugar, q sugar, q sweet, q sweet, Table 163 The EM clustering algrithm The table shws a set f dcuments (a) and parameter values fr selected iteratins during EM clustering (b) Parameters shwn are prir α 1, sft assignment scres r n,1 (bth mitted fr cluster ), and lexical parameters q m,k fr a few terms The authrs initially assigned dcument 6 t cluster 1 and dcument 7 t cluster (iteratin 0) EM cnverges after 5 iteratins Fr smthing, the r nk in Equatin (1616) were replaced with r nk ǫ where ǫ = 00001

24 37 16 Flat clustering? Exercise bth cntain sweet As a result, it takes 5 iteratins fr the term t be unambiguusly assciated with cluster (q sweet,1 = 0 in iteratin 5) Finding gd seeds is even mre critical fr EM than fr K-means EM is prne t get stuck in lcal ptima if the seeds are nt chsen well This is a general prblem that als ccurs in ther applicatins f EM 4 Therefre, as with K-means, the initial assignment f dcuments t clusters is ften cmputed by a different algrithm Fr example, a hard K-means clustering may prvide the initial assignment, which EM can then sften up 166 We saw abve that the time cmplexity f K-means is Θ(IKNM) What is the time cmplexity f EM? 166 References and further reading Berkhin (006b) gives a general up-t-date survey f clustering methds with special attentin t scalability The classic reference fr clustering in pattern recgnitin, cvering bth K-means and EM, is (Duda et al 000) Rasmussen (199) intrduces clustering frm an infrmatin retrieval perspective Anderberg (1973) prvides a general intrductin t clustering fr applicatins In additin t Euclidean distance and csine similarity, Kullback- Leibler divergence is ften used in clustering as a measure f hw (dis)similar dcuments and clusters are (Xu and Crft 1999, Muresan and Harper 004, Kurland and Lee 004) The cluster hypthesis is due t Jardine and van Rijsbergen (1971) wh state it as fllws: Assciatins between dcuments cnvey infrmatin abut the relevance f dcuments t requests Saltn (1971a; 1975), Crft (1978), Vrhees (1985a), Can and Ozkarahan (1990), Cacheda et al (003), Can et al (004), Singitham et al (004) and Altingövde et al (008) investigate the efficiency and effectiveness f cluster-based retrieval While sme f these studies shw imprvements in effectiveness, efficiency r bth, there is n cnsensus that cluster-based retrieval wrks well cnsistently acrss scenaris Clusterbased language mdeling was pineered by Liu and Crft (004) There is gd evidence that clustering f search results imprves user experience and search result quality (Hearst and Pedersen 1996, Zamir and Etzini 1999, Tmbrs et al 00, Käki 005, Tda and Kataka 005), althugh nt as much as search result structuring based n carefully edited categry hierarchies (Hearst 006) The Scatter-Gather interface fr brwsing cllectins was presented by Cutting et al (199) A theretical framewrk fr an- 4 Fr example, this prblem is cmmn when EM is used t estimate parameters f hidden Markv mdels, prbabilistic grammars, and machine translatin mdels in natural language prcessing (Manning and Schütze 1999)

25 166 References and further reading 373 ADJUSTED RAND INDEX alyzing the prperties f Scatter/Gather and ther infrmatin seeking user interfaces is presented by Pirlli (007) Schütze and Silverstein (1997) evaluate LSI (Chapter 18) and truncated representatins f centrids fr efficient K-means clustering The Clumbia NewsBlaster system (McKewn et al 00), a frerunner t the nw much mre famus and refined Ggle News ( used hierarchical clustering (Chapter 17) t give tw levels f news tpic granularity See Hatzivassilglu et al (000) fr details, and Chen and Lin (000) and Radev et al (001) fr related systems Other applicatins f clustering in infrmatin retrieval are duplicate detectin (Yang and Callan (006), Sectin 196, page 438), nvelty detectin (see references in Sectin 179, page 399) and metadata discvery n the semantic web (Alns et al 006) The discussin f external evaluatin measures is partially based n Strehl (00) Dm (00) prpses a measure Q 0 that is better mtivated theretically than NMI Q 0 is the number f bits needed t transmit class memberships assuming cluster memberships are knwn The Rand index is due t Rand (1971) Hubert and Arabie (1985) prpse an adjusted Rand index that ranges between 1 and 1 and is 0 if there is nly chance agreement between clusters and classes (similar t κ in Chapter 8, page 165) Basu et al (004) argue that the three evaluatin measures NMI, Rand index and F measure give very similar results Stein et al (003) prpse expected edge density as an internal measure and give evidence that it is a gd predictr f the quality f a clustering Kleinberg (00) and Meilă (005) present aximatic framewrks fr cmparing clusterings Authrs that are ften credited with the inventin f the K-means algrithm include Llyd (198) (first distributed in 1957), Ball (1965), MacQueen (1967), and Hartigan and Wng (1979) Arthur and Vassilvitskii (006) investigate the wrst-case cmplexity f K-means Bradley and Fayyad (1998), Pelleg and Mre (1999) and Davidsn and Satyanarayana (003) investigate the cnvergence prperties f K-means empirically and hw it depends n initial seed selectin Dhilln and Mdha (001) cmpare K-means clusters with SVD-based clusters (Chapter 18) The K-medid algrithm was presented by Kaufman and Russeeuw (1990) The EM algrithm was riginally intrduced by Dempster et al (1977) An in-depth treatment f EM is (McLachlan and Krishnan 1996) See Sectin 185 (page 417) fr publicatins n latent analysis, which can als be viewed as sft clustering AIC is due t Akaike (1974) (see als Burnham and Andersn (00)) An alternative t AIC is BIC, which can be mtivated as a Bayesian mdel selectin prcedure (Schwarz 1978) Fraley and Raftery (1998) shw hw t chse an ptimal number f clusters based n BIC An applicatin f BIC t K-means is (Pelleg and Mre 000) Hamerly and Elkan (003) prpse an alternative t BIC that perfrms better in their experiments Anther influential Bayesian apprach fr determining the number f clusters (simultane-