CLUSTERING (Segmentation)

Transcription

1 CLUSTERING (Segmentation) Dr. Saed Sayad University of Toronto

2 What is Clustering? Given a set of records, organize the records into clusters Income Age A cluster is a subset of records which are similar 2

3 Clustering Requirements The ability to discover some or all of the hidden clusters. Within-cluster similarity and between-cluster disimilarity. Ability to deal with various types of attributes. Can deal with noise and outliers. Can handle high dimensionality. Scalability, Interpretability and usability. 3

4 Similarity - Distance Measure To measure similarity or dissimilarity between objects, we need a distance measure. The usual axioms for a distance measure D are: D(x, x) = 0 D(x, y) = D(y, x) D(x, y) D(x, z) + D(z, y) the triangle inequality 4

5 Similarity - Distance Measure Euclidean k x i y i i1 2 Manhattan k i1 x i y i Minkowski x i y i k i1 q 1 q 5

6 Similarity - Correlation r xy ( x ( x i i x) x)( y 2 i ( y i y) y) 2 Similar Dissimilar Credit$ Credit$ Age Age 6

7 Similarity Hamming Distance D H k i1 x i y i Gene 1 A A T C C A G T Gene 2 T C T C A A G C Hamming Distance

8 Clustering Methods Exclusive vs. Overlapping Hierarchical vs. Partitive Deterministic vs. Probabilistic Incremental vs. Batch learning 8

9 Exclusive vs. Overlapping Income Income Age Age 9

10 Hierarchical vs. Partitive Income Age 10

11 Hierarchical Clustering Hierarchical clustering involves creating clusters that have a predetermined ordering from top to bottom. For example, all files and folders on the hard disk are organized in a hierarchy. There are two types of hierarchical clustering: Agglomerative Divisive 11

12 Hierarchical Clustering Agglomerative Divisive 12

13 Hierarchical Clustering - Agglomerative 1. Assign each observation to its own cluster. 2. Compute the similarity (e.g., distance) between each of the clusters. 3. Join the two most similar clusters. 4. Repeat steps 2 and 3 until there is only a single cluster left. 13

14 Hierarchical Clustering - Agglomerative Given: A set X of objects {x 1,...,x n } A distance function dist(c 1,c 2 ) for i = 1 to n c i = {x i } end for C = {c 1,...,c n } l = n+1 while C.size > 1 do (c min1,c min2 ) = minimum dist(c i,c j ) for all c i,c j in C remove c min1 and c min2 from C add {c min1,c min2 } to C l = l + 1 end while 14

15 Hierarchical Clustering - Agglomerative

16 Hierarchical Clustering - Divisive 1. Assign all of the observations to a single cluster. 2. Partition the cluster to two least similar clusters. 3. Proceed recursively on each cluster until there is one cluster for each observation. 16

17 Hierarchical Clustering Single Linkage r s L( r, s) min( D( x ri, xsj)) 17

18 Hierarchical Clustering Complete Linkage r s L( r, s) max( D( x ri, xsj)) 18

19 Hierarchical Clustering Average Linkage r s L( r, s) 1 n n r s n n r s i1 j1 D( x ri, x sj ) 19

20 K Means Clustering 1. Clusters the data into k groups where k is predefined. 2. Select k points at random as cluster centers. 3. Assign observations to their closest cluster center according to the Euclidean distance function. 4. Calculate the centroid or mean of all instances in each cluster (this is the mean part) 5. Repeat steps 2, 3 and 4 until the same points are assigned to each cluster in consecutive rounds. 20

21 K Means Clustering Income Age 21

22 K Means Clustering Sum of Squares function J K j1 ns j ( x n j 2 ) 22

23 Clustering Evaluation Sarle s Cubic Clustering Criterion The Pseudo-F Statistic The Pseudo-T 2 Statistic Beale s F-Type Statistic Target-based 23

24 Clustering Evaluation Target Variable Categorical Numerical Chi 2 Test K-S Test ANOVA H Test 24

25 Chi 2 Test Y Actual N Predicted Y n 11 n 12 N n 21 n 22 r c ( n ij e 2 ij ) e i1 j1 ij

26 Analysis of Variance (ANOVA) Source of Variation Sum of Squares Degree of Freedom Mean Square F P Between Groups SS B df B MS B = SS B /df B F=MS B /MS W P(F) Within Groups SS W df w MS W = SS W /df w Total SS T df T 26

27 Clustering - Applications Marketing: finding groups of customers with similar behavior. Insurance & Banking: identifying frauds. Biology: classification of plants and animals given their features. Libraries: book ordering. City-planning: identifying groups of houses according to their house type, value and geographical location. World Wide Web: document classification; clustering weblog data to discover groups with similar access patterns. 27

28 Summary Clustering is the process of organizing objects (records or variables) into groups whose members are similar in some way. Hierarchical and K-Means are the two most used clustering techniques. The effectiveness of the clustering method depends on the similarity function. The result of the clustering algorithm can be interpreted and evaluated in different ways. 28

29 29