A Micro-course on Cluster Analysis Techniques

Transcription

1 Università di Trento DiSCoF Dipartimento di Scienze della Cognizione e Formazione A Micro-course on Cluster Analysis Techniques luigi.lombardi@unitn.it Methodological course (COBRAS-DiSCoF)

2 Finding groups of objects such that the objects in a group will be similar (or related) to one another and different from (or unrelated to) the objects in other groups Intra-cluster distances are minimized Inter-cluster distances are maximized

3 We need to construct a data matrix Data matrix

4 Then we go for a dissimilarity matrix Dissimilarity matrix

5 Transforming a data matrix into a dissimilarity matrix Properties of Dissimilarity matrix

6 The property of symmetry Dissimilarity matrix

7 Types of metrics in cluster analysis Minkowski distance

8 Derivations of Minkowski distance Standard Euclidean City block Max Nominal

9 Euclidean VS City block p3 p2 c2 a b d2 a (c1 + c2) b (d1 + d2) c1 p1 d1

10 Max distance p3 c2 p2 c1 p1 d1 d2 d 12 = max {c1,c2} = c2 d 13 = max {d1,d2} = d2

11 Nominal distance p3 c2 p2 c1 p1 d1 d2 d 12 = = 2 d 13 = = 2

12 Nominal distance p3 c1 p2 p1 d1 d2 d 12 = = 1 d 13 = = 2

13 Quantitative models

14 Geometrical view (Euclidean City block) ambiguous clusters How many clusters? Six Clusters Two Clusters Four Clusters

15 Geometrical view (Euclidean City block) Types of clustering A clustering is a set of clusters Important distinction between hierarchical and partitional sets of clusters Partitional Clustering A division data objects into non-overlapping subsets (clusters) such that each data object is in exactly one subset Hierarchical clustering A set of nested clusters organized as a hierarchical tree

16 Partitional clustering Original Points A Partitional Clustering

17 Hierarchical clustering p1 p2 p3 p4 p1 p2 p3 p4 Traditional Hierarchical Clustering Traditional Dendrogram p1 p2 p3 p4 p1 p2 p3 p4 Non-traditional Hierarchical Clustering Non-traditional Dendrogram

18 Partitional clustering / K-means algorithm Partitional clustering approach Each cluster is associated with a centroid (center point) Each point is assigned to the cluster with the closest centroid Number of clusters, K, must be specified The basic algorithm is very simple

19 Partitional clustering / K-means algorithm Initial centroids are often chosen randomly. Clusters produced vary from one run to another. The centroid is (typically) the mean of the points in the cluster. Closeness is measured by Euclidean distance. K-means will converge for common similarity measures mentioned above. Most of the convergence happens in the first few iterations. Often the stopping condition is changed to Until relatively few points change clusters

20 Original Points y x y y x Optimal Clustering x Sub-optimal Clustering

21 3 Iteration 1 3 Iteration 2 3 Iteration y y y x x x 3 Iteration 4 3 Iteration 5 3 Iteration y y y x x x

22 Overall Flowchart of K-means algorithm

23 Weaknesses of K-means algorithm Unable to handle noisy data and outliers Very large or very small values could skew the mean (centroid) Not suitable to discover clusters with non-convex shapes, differing sizes, differing densities

24 Weaknesses of K-means algorithm: differing sizes Original Points K-means (3 Clusters)

25 Weaknesses of K-means algorithm: differing densities Original Points K-means (3 Clusters)

26 Weaknesses of K-means algorithm: non-globular shapes Original Points K-means (2 Clusters)

27 Hierarchical Clustering Produces a set of nested clusters organized as a hierarchical tree Can be visualized as a dendrogram A tree like diagram that records the sequences of merges or splits

28 Good properties of Hierarchical Clustering Do not have to assume any particular number of clusters Any desired number of clusters can be obtained by cutting the dendogram at the proper level They may correspond to meaningful taxonomies Example in biological sciences (e.g., animal kingdom, phylogeny reconstruction, )

29 Types of Hierarchical Clustering Two main types of hierarchical clustering Agglomerative: Start with the points as individual clusters At each step, merge the closest pair of clusters until only one cluster (or k clusters) left Divisive: Start with one, all-inclusive cluster At each step, split a cluster until each cluster contains a point (or there are k clusters) Traditional hierarchical algorithms use a similarity or distance matrix Merge or split one cluster at a time

30 Types of Hierarchical Clustering Basic algorithm is straightforward 1. Compute the proximity matrix 2. Let each data point be a cluster 3. Repeat 4. Merge the two closest clusters 5. Update the proximity matrix 6. Until only a single cluster remains Key operation is the computation of the proximity of two clusters Different approaches to defining the distance between clusters distinguish the different algorithms

31 Hierarchical Clustering: distances between clusters D(r,s)

32 Example: agglomerative algorithm + average linkage

33 Qualitative models

34 An alternative way for Clustering qualitative data (i.e. binary data) Hierarchical classes (HICLAS) models are structural models for n-way n-mode arrays. As distinct features these models include: Clustering of the elements of each mode. If-then type relations among elements of each mode. Linking structure between n clusterings.

35 objects Hierarchical classes analysis: attributes I J Binary data matrix D Approximation D = M + E objects attributes I J Binary model matrix M H m ij = a ih b jh h=1 Decomposition objects M = A B attributes Object bundles I H A Attribute bundles J H B Binary bundle matrices

36 Structural Relations: A Obj. bundles I II III Hypothetical example objects M attributes a b c d e f g h [1,2] [3,4] [5] [6,7]

37 Structural Relations: Hypothetical example attributes M a b c d e f g h objects [a,b] [c,d] [e,f,g] [h] B a b c d e f g h I II III

38 objects M Structural Relations: Hypothetical example attributes a b c d e f g h A Obj. bundles I II III H m ij = a ih b jh h=1 B a b c d e f g h I II III

39 objects M Structural Relations: Hypothetical example attributes a b c d e f g h A Obj. bundles I II III H m ij = a ih b jh h=1 B a b c d e f g h I II III

40 Graphical representation: Hypothetical example 5 5 1,2 1,2 3,4 3,4 I II III c,d c,d a,b a,b h h e,f,g 6,7 6,7 A Obj. bundles I II III B a b c d e f g h I II III