Data Mining Clustering. Sheets are based on the those provided by Tan, Steinbach, and Kumar. Introduction to Data Mining

Transcription

1 Data Mining Clustering Toon Calders Sheets are based on the those provided b Tan, Steinbach, and Kumar. Introduction to Data Mining

2 What is Cluster Analsis? Finding groups of objects such that the objects in a group will be similar (or related) to one another and different from (or unrelated to) the objects in other groups Intra-cluster distances are minimized Inter-cluster distances are maximized

3 Applications of Cluster Analsis Understanding Group related documents for browsing, group genes and proteins that have similar functionalit, or group stocks with similar price fluctuations Discovered Clusters Applied-Matl-DOWN,Ba-Network-Down,3-COM-DOWN, Cabletron-Ss-DOWN,CISCO-DOWN,HP-DOWN, DSC-Comm-DOWN,INTEL-DOWN,LSI-Logic-DOWN, Micron-Tech-DOWN,Texas-Inst-Down,Tellabs-Inc-Down, Natl-Semiconduct-DOWN,Oracl-DOWN,SGI-DOWN, Sun-DOWN Apple-Comp-DOWN,Autodesk-DOWN,DEC-DOWN, ADV-Micro-Device-DOWN,Andrew-Corp-DOWN, Computer-Assoc-DOWN,Circuit-Cit-DOWN, Compaq-DOWN, EMC-Corp-DOWN, Gen-Inst-DOWN, Motorola-DOWN,Microsoft-DOWN,Scientific-Atl-DOWN Fannie-Mae-DOWN,Fed-Home-Loan-DOWN, MBNA-Corp-DOWN,Morgan-Stanle-DOWN Baker-Hughes-UP,Dresser-Inds-UP,Halliburton-HLD-UP, Louisiana-Land-UP,Phillips-Petro-UP,Unocal-UP, Schlumberger-UP Industr Group Technolog1-DOWN Technolog-DOWN Financial-DOWN Oil-UP Summarization Reduce the size of large data sets Clustering precipitation in Australia

4 Notion of a Cluster can be Ambiguous How man clusters? Six Clusters Two Clusters Four Clusters

5 Tpes of Clusterings A clustering is a set of clusters Important distinction between hierarchical and partitional sets of clusters Partitional Clustering A division data objects into non-overlapping subsets (clusters) such that each data object is in exactl one subset Hierarchical clustering A set of nested clusters organized as a hierarchical tree

6 Partitional Clustering Original Points A Partitional Clustering

7 Hierarchical Clustering p1 p3 p4 p p1 p p3 p4 Traditional Hierarchical Clustering Traditional Dendrogram p1 p3 p4 p p1 p p3 p4 Non-traditional Hierarchical Clustering Non-traditional Dendrogram

8 Tpes of Clusters: Objective Function Clusters Defined b an Objective Function Finds clusters that minimize or maximize an objective function. These problems ver quickl become NP Hard Can have global or local objectives. Hierarchical clustering algorithms tpicall have local objectives Partitional algorithms tpicall have global objectives A variation of the global objective function approach is to fit the data to a parameterized model. Parameters for the model are determined from the data. Mixture models assume that the data is a mixture' of a number of statistical distributions.

9 Outline Partitional Clustering Distance-based K-means, K-medoids, Bisecting K-means Densit-based DBSCAN Hierarchical Clustering Cluster validit

10 K-means Clustering Partitional clustering approach Each cluster is associated with a centroid (center point) Each point is assigned to the cluster with the closest centroid Number of clusters, K, must be specified The basic algorithm is ver simple

11 K-means Clustering Details Initial centroids are often chosen randoml. Clusters produced var from one run to another. The centroid is tpicall the mean of the points in the cluster. Closeness is measured b Euclidean distance, cosine similarit, correlation, etc. Question: How can we evaluate a clustering? Will K-means alwas converge? What if we cannot compute the mean? Will it alwas converge to the same solution?

12 Answer: Evaluating K-means Clusters Most common measure: Sum of Squared Error (SSE) For each point, the error is the distance to the nearest cluster To get SSE, we square these errors and sum them. SSE K = = i= 1 x C i dist ( m i, x ) i x is a data point in cluster C i and m i is the representative point for cluster C i Given two clusterings, we can choose the one with the smallest error

13 Answers: Will K-means alwas converge? K-means will converge for common similarit measures mentioned above. Finite number of potential centers SSE alwas becomes smaller Most of the convergence happens in the first few Most of the convergence happens in the first few iterations. Often the stopping condition is changed to Until relativel few points change clusters Complexit is O( n * K * I * d ) n = number of points, K = number of clusters, I = number of iterations, d = number of attributes

14 Answer: What if we cannot compute mean? Happens often Onl matrix of pairwise distances has been given Data is non-numerical E.g. graph data with geodesic distance between nodes Solution (K-medoids): For ever point in the cluster: add up all distances to the other points in the cluster Point in the cluster for which this distance is the smallest, becomes the new center

15 Answer: will it alwas converge to the same clustering? No! Depends highl on the initial cluster centers. Ever equilibrium is a potential outcome XXX OOO OOO XXX XXX OOO

16 Two different K-means Clusterings Original Points x Optimal Clustering x x Sub-optimal Clustering

17 Importance of Choosing Initial Centroids 3 Iteration x

18 Importance of Choosing Initial Centroids 3 Iteration 1 3 Iteration 3 Iteration x x x 3 Iteration 4 3 Iteration 5 3 Iteration x x x

19 Importance of Choosing Initial Centroids 3 Iteration x

20 Importance of Choosing Initial Centroids 3 Iteration 1 3 Iteration x x 3 Iteration 3 3 Iteration 4 3 Iteration x x x

21 Problems with Selecting Initial Points If there are K real clusters then the chance of selecting one centroid from each cluster is small. Chance is relativel small when K is large If clusters are the same size, n, then For example, if K = 1, then probabilit = 1!/1 1 =.36 Sometimes the initial centroids will readjust themselves in right wa, and sometimes the don t Consider an example of five pairs of clusters

22 1 Clusters Example 8 Iteration x Starting with two initial centroids in one cluster of each pair of clusters

23 1 Clusters Example 8 Iteration 1 8 Iteration x Iteration x Iteration x Starting with two initial centroids in one cluster of each pair of clusters x

24 1 Clusters Example 8 Iteration x Starting with some pairs of clusters having three initial centroids, while other have onl one.

25 1 Clusters Example 8 Iteration 1 8 Iteration Iteration x 3 Iteration x x x Starting with some pairs of clusters having three initial centroids, while other have onl one.

26 Solutions to Initial Centroids Problem Multiple runs Helps, but probabilit is not on our side Sample and use hierarchical clustering to determine initial centroids Select more than k initial centroids and then select among these initial centroids Select most widel separated Postprocessing Bisecting K-means Not as susceptible to initialization issues

27 Bisecting K-means Bisecting K-means algorithm Variant of K-means that can produce a partitional or a hierarchical clustering

28 Bisecting K-means Example

29 Handling Empt Clusters Basic K-means algorithm can ield empt clusters How? Example? Several strategies Choose the point that contributes most to SSE Choose a point from the cluster with the highest SSE If there are several empt clusters, the above can be repeated several times. Wh use maximal SSE as selection criterion?

30 Updating Centers Incrementall In the basic K-means algorithm, centroids are updated after all points are assigned to a centroid An alternative is to update the centroids after each assignment (incremental approach) Each assignment updates zero or two centroids More expensive Introduces an order dependenc Never get an empt cluster Wh not?

31 Pre-processing and Post-processing Pre-processing Normalize the data Eliminate outliers Post-processing Eliminate small clusters that ma represent outliers Split loose clusters, i.e., clusters with relativel high SSE Merge clusters that are close and that have relativel low SSE

32 Limitations of K-means K-means has problems when clusters are of differing Sizes Densities Non-globular shapes K-means has problems when the data contains outliers.

33 Limitations of K-means: Differing Sizes Original Points K-means (3 Clusters)

34 Limitations of K-means: Differing Densit Original Points K-means (3 Clusters)

35 Limitations of K-means: Non-globular Shapes Original Points K-means ( Clusters)

36 Overcoming K-means Limitations Original Points K-means Clusters One solution is to use man clusters. Find parts of clusters, but need to put together.

37 Overcoming K-means Limitations Original Points K-means Clusters

38 Overcoming K-means Limitations Original Points K-means Clusters

39 Outline Partitional Clustering Distance-based K-means, K-medoids, Bisecting K-means Densit-based DBSCAN Hierarchical Clustering Cluster validit

40 Densit-based clustering

41 Densit-based clustering

42 DBSCAN DBSCAN is a densit-based algorithm. Densit = number of points within a specified radius (Eps) A point is a core point if it has more than a specified number of points (MinPts) within Eps These are points that are at the interior of a cluster A border point has fewer than MinPts within Eps, but is in the neighborhood of a core point A noise point is an point that is not a core point or a border point.

43 DBSCAN: Core, Border, and Noise Points

44 Densit-based clustering: DBSCAN A point is (ε, µ)-densit-reachable from point x if there exists a sequence c 1,, c k of (ε,µ)-core points such that: d(x,c 1 ) ε i {1,, n-1} : d(c i,c i+1 ) ε d(c n,) ε Remark: computing all pairs (x,) such that is (ε, µ)-densit-reachable from core point x = computing transitive closure.

45 DBSCAN Algorithm Eliminate noise points Perform clustering on the remaining points

46 Voorbeeld: DBSCAN Select a core point Add all densit-reachable points Select second core point Select third core point and form third cluster Form second cluster

47 DBSCAN: Core, Border and Noise Points Original Points Point tpes: core, border and noise Eps = 1, MinPts = 4

48 When DBSCAN Works Well Original Points Clusters Resistant to Noise Can handle clusters of different shapes and sizes

49 When DBSCAN Does NOT Work Well Original Points (MinPts=4, Eps=9.9) Varing densities High-dimensional data (MinPts=4, Eps=9.75).

50 DBSCAN: Determining EPS and MinPts Idea is that for points in a cluster, their k th nearest neighbors are at roughl the same distance Noise points have the k th nearest neighbor at farther distance So, plot sorted distance of ever point to its k th nearest neighbor