EECS 647: Introduction to Database Systems

Transcription

1 EECS 647: Introduction to Database Systems Instructor: Luke Huan Spring 009

2 Administrative Final project demonstrations are scheduled on May 6 th. Strongly encourage everyone to do a demonstration If you want, please send me the following information by May 5 th : Your team name, your and your partner name Final project report is due on May 1 th at 1:0 5//009 Luke Huan Univ. of Kansas

3 Today s Topic Discover association in your data Clustering 5//009 Luke Huan Univ. of Kansas

4 Association Rule Mining Given a set of transactions, find rules that will predict the occurrence of an item based on the occurrences of other items in the transaction Market-Basket transactions TID Items 1 Bread, Milk Bread, Diaper, Beer, Eggs Milk, Diaper, Beer, Coke 4 Bread, Milk, Diaper, Beer 5 Bread, Milk, Diaper, Coke Example of Association Rules {Diaper} {Beer}, {Milk, Bread} {Eggs,Coke}, {Beer, Bread} {Milk}, Implication means co-occurrence, not causality! 5//009 Luke Huan Univ. of Kansas 4

5 Definition: Frequent Itemset Itemset A collection of one or more items Example: {Milk, Bread, Diaper} k-itemset An itemset that contains k items Support count (σ) Frequency of occurrence of an itemset E.g. σ({milk, Bread,Diaper}) = Support Fraction of transactions that contain an itemset E.g. s({milk, Bread, Diaper}) = /5 Frequent Itemset An itemset whose support is greater than or equal to a minsup threshold TID Items 1 Bread, Milk Bread, Diaper, Beer, Eggs Milk, Diaper, Beer, Coke 4 Bread, Milk, Diaper, Beer 5 Bread, Milk, Diaper, Coke 5//009 Luke Huan Univ. of Kansas 5

6 Association Rule Definition: Association Rule An implication expression of the form X Y, where X and Y are itemsets Example: {Milk, Diaper} {Beer} Rule Evaluation Metrics Support (s) Fraction of transactions that contain both X and Y Confidence (c) Measures how often items in Y appear in transactions that contain X TID Example: Items 1 Bread, Milk Bread, Diaper, Beer, Eggs Milk, Diaper, Beer, Coke 4 Bread, Milk, Diaper, Beer 5 Bread, Milk, Diaper, Coke { Milk, Diaper} (Milk, Diaper, Beer) s = σ = T σ (Milk,Diaper,Beer) c = = σ (Milk, Diaper) 5 = Beer = //009 Luke Huan Univ. of Kansas 6

7 Mining Association Rules TID Items 1 Bread, Milk Bread, Diaper, Beer, Eggs Milk, Diaper, Beer, Coke 4 Bread, Milk, Diaper, Beer 5 Bread, Milk, Diaper, Coke Example of Rules: {Milk,Diaper} {Beer} (s=0.4, c=0.67) {Milk,Beer} {Diaper} (s=0.4, c=1.0) {Diaper,Beer} {Milk} (s=0.4, c=0.67) {Beer} {Milk,Diaper} (s=0.4, c=0.67) {Diaper} {Milk,Beer} (s=0.4, c=0.5) {Milk} {Diaper,Beer} (s=0.4, c=0.5) Observations: All the above rules are binary partitions of the same itemset: {Milk, Diaper, Beer} Rules originating from the same itemset have identical support but can have different confidence Thus, we may decouple the support and confidence requirements 5//009 Luke Huan Univ. of Kansas 7

8 An Exercise Transaction-id Items bought f, a, c, d, g, I, m, p a, b, c, f, l,m, o b, f, h, j, o b, c, k, s, p a, f, c, e, l, p, m, n The support value of pattern {acm} Transaction is database TDB Sup(acm)= The support of pattern {ac} is Sup(ac)= Given min_sup=, acm is Frequent The confidence of the rule: {ac} => {m} is 100% 5//009 Luke Huan Univ. of Kansas 8

9 Two-step approach: Mining Association Rules 1. Frequent Itemset Generation Generate all itemsets whose support minsup. Rule Generation Generate high confidence rules from each frequent itemset, where each rule is a binary partitioning of a frequent itemset Frequent itemset generation is still computationally expensive 5//009 Luke Huan Univ. of Kansas 9

10 Apriori Algorithm A level-wise, candidate-generation-and-test approach (Agrawal & Srikant 1994) TID Scan D Data base D Items a, c, d b, c, e a, b, c, e b, e Min_sup= -candidates Itemset bce Scan D Freq -itemsets Itemset bce Sup 1-candidates Itemset a b c d e Sup 1 Freq -itemsets Itemset ac bc be ce Sup Freq 1-itemsets Itemset a b c e ab ac ae bc be ce Sup Counting Itemset Sup 1 1 -candidates Itemset ab ac ae bc be ce Scan D 5//009 Luke Huan Univ. of Kansas 10

11 Frequent Itemset Generation null A B C D E AB AC AD AE BC BD BE CD CE DE ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE ABCD ABCE ABDE ACDE BCDE ABCDE Given d items, there are d possible candidate itemsets 5//009 Luke Huan Univ. of Kansas 11

12 Reducing Number of Candidates Apriori principle: If an itemset is frequent, then all of its subsets must also be frequent Apriori principle holds due to the following property of the support measure: X, Y : ( X Y ) s( X ) s( Y ) Support of an itemset never exceeds the support of its subsets This is known as the anti-monotone property of support 5//009 Luke Huan Univ. of Kansas 1

13 Illustrating Apriori Principle Found to be Infrequent Pruned supersets 5//009 Luke Huan Univ. of Kansas 1

14 An Example of Apriori Principle TID Items a, c, d b, c, e a, b, c, e b, e null A B C D E Min_sup= AB AC AD AE BC BD BE CD CE DE ABC ABD ABE ACD ACE ADE BCD BCE BDE CDE ABCD ABCE ABDE ACDE BCDE ABCDE 5//009 Luke Huan Univ. of Kansas 14

15 What is Cluster Analysis? Finding groups of objects such that the objects in a group will be similar (or related) to one another and different from (or unrelated to) the objects in other groups Intra-cluster distances are minimized Inter-cluster distances are maximized 5//009 Luke Huan Univ. of Kansas 15

16 Understanding Applications of Cluster Analysis Group related documents for browsing, group genes and proteins that have similar functionality, or group stocks with similar price fluctuations Summarization Reduce the size of large data sets 5//009 Luke Huan Univ. of Kansas 16

17 Multidisciplinary Efforts of Clustering Pattern Recognition Spatial Data Analysis Create thematic maps in GIS by clustering feature spaces Detect spatial clusters or for other spatial mining tasks Image Processing Economic Science (especially market research) WWW Document classification Cluster Weblog data to discover groups of similar access patterns Bioinfo: Phylogenetic tree Microarray analysis 5//009 Luke Huan Univ. of Kansas 17

18 Terms in Cluster Analysis Cluster: a collection of data objects Similar to one another within the same cluster Dissimilar to the objects in other clusters Cluster analysis Finding similarities between data according to the characteristics found in the data and grouping similar data objects into clusters Unsupervised learning: no predefined classes So what? Clustering can be used as a stand-alone tool to get insight into data distribution Clustering can be used as a preprocessing step for other algorithms such as discretization 5//009 Luke Huan Univ. of Kansas 18

19 Quality: What Is Good Clustering? A good clustering method will produce high quality clusters with high intra-class similarity low inter-class similarity The quality of a clustering result depends on both the similarity measure used by the method and its implementation The quality of a clustering method is also measured by its ability to discover some or all of the hidden patterns 5//009 Luke Huan Univ. of Kansas 19

20 Measure the Quality of Clustering Dissimilarity/Similarity metric: Similarity is expressed in terms of a distance function, typically metric: d(i, j) There is a separate quality function that measures the goodness of a cluster. The definitions of distance functions are usually very different for boolean, categorical, ordinal, interval, ratio, and vector variables. Weights should be associated with different variables based on applications and data semantics. It is hard to define similar enough or good enough the answer is typically highly subjective. 5//009 Luke Huan Univ. of Kansas 0

21 Types of Clusters: Well-Separated Well-Separated Clusters: A cluster is a set of points such that any point in a cluster is closer (or more similar) to every other point in the cluster than to any point not in the cluster. well-separated clusters 5//009 Luke Huan Univ. of Kansas 1

22 Center-based Types of Clusters: Center-Based A cluster is a set of objects such that an object in a cluster is closer (more similar) to the center of a cluster, than to the center of any other cluster The center of a cluster is often a centroid, the average of all the points in the cluster, or a medoid, the most representative point of a cluster 4 center-based clusters 5//009 Luke Huan Univ. of Kansas

23 Types of Clusters: Contiguity-Based Contiguous Cluster (Nearest neighbor or Transitive) A cluster is a set of points such that a point in a cluster is closer (or more similar) to one or more other points in the cluster than to any point not in the cluster. 8 contiguous clusters 5//009 Luke Huan Univ. of Kansas

24 Types of Clusters: Density-Based Density-based A cluster is a dense region of points, which is separated by low-density regions, from other regions of high density. Used when the clusters are irregular or intertwined, and when noise and outliers are present. 6 density-based clusters 5//009 Luke Huan Univ. of Kansas 4

25 Types of Clusters: Model Based Shared Property or Conceptual Clusters Finds clusters that share some common property or represent a particular model.. Viewer 1 Viewer rating Viewer Viewer Viewer 5 Movie Movie 4 Movie 4 5 Movie 4 Overlapping Circles Movie 5 movie 1 movie movie 4 movie 6 7 Movie Movie 7 1 viewer 1 viewer viewer 4 5//009 Luke Huan Univ. of Kansas 5

26 Major Clustering Approaches (I) Partitioning approach: Construct various partitions and then evaluate them by some criterion, e.g., minimizing the sum of square errors Typical methods: k-means, k-medoids, CLARANS Hierarchical approach: Create a hierarchical decomposition of the set of data (or objects) using some criterion Typical methods: Diana, Agnes, BIRCH, ROCK, CAMELEON 5//009 Luke Huan Univ. of Kansas 6

27 The K-Means Clustering Method Given k, the k-means algorithm is implemented in four steps: Partition objects into k nonempty subsets Compute seed points as the centroids of the clusters of the current partition (the centroid is the mean point, of the cluster) Assign each object to the cluster with the nearest seed point Go back to Step, stop when no more new assignment 5//009 Luke Huan Univ. of Kansas 7

28 The K-Means Clustering Method Assign each objects to most similar center reassign Update the cluster means reassign K= Arbitrarily choose K object as initial cluster center Update the cluster means //009 Luke Huan Univ. of Kansas 8

29 Comments on the K-Means Method Strength Relatively efficient: O(tkn), where n is # objects, k is # clusters, and t is # iterations. Normally, k, t << n. Comment: Often terminates at a local optimum. The global optimum may be found using techniques such as genetic algorithms 5//009 Luke Huan Univ. of Kansas 9

30 Weakness Comments on the K-Means Method Applicable only when mean is defined, then what about categorical data? Need to specify k, the number of clusters, in advance Unable to handle noisy data and outliers Not suitable to discover clusters with non-convex shapes 5//009 Luke Huan Univ. of Kansas 0

31 Variations of the K-Means Method A few variants of the k-means which differ in Selection of the initial k means Dissimilarity calculations Strategies to calculate cluster means Handling categorical data: k-modes (Huang 98) Replacing means of clusters with modes Using new dissimilarity measures to deal with categorical objects Using a frequency-based method to update modes of clusters A mixture of categorical and numerical data: k-prototype method 5//009 Luke Huan Univ. of Kansas 1

32 Sensitive to outliers A Problem of K-means Outlier: objects with extremely large values May substantially distort the distribution of the data + + K-medoids: the most centrally located object in a cluster //009 Luke Huan Univ. of Kansas

33 A Problem K-means: Differing Density Original Points K-means ( Clusters) 5//009 Luke Huan Univ. of Kansas

34 A Problem of K-means: Non-globular Shapes Original Points K-means ( Clusters) 5//009 Luke Huan Univ. of Kansas 4

35 The K-Medoids Clustering Method Find representative objects, called medoids, in clusters PAM (Partitioning Around Medoids, 1987) starts from an initial set of medoids and iteratively replaces one of the medoids by one of the non-medoids if it improves the total distance of the resulting clustering PAM works effectively for small data sets, but does not scale well for large data sets CLARA (Kaufmann & Rousseeuw, 1990) CLARANS (Ng & Han, 1994): Randomized sampling 5//009 Luke Huan Univ. of Kansas 5

36 A Typical K-Medoids Algorithm (PAM) Total Cost = K= Arbitrary choose k object as initial medoids Total Cost = 6 Assign each remainin g object to nearest medoids Randomly select a nonmedoid object,o ramdom Do loop Until no change Swapping O and O ramdom If quality is improved Compute total cost of swapping //009 Luke Huan Univ. of Kansas 6

37 Nature of the data Data types: Summary SSN Nominal Grade Ordinal Temperature (degree) Interval Length Ratio Data Quality Noise Outlier Missing/duplicated data 5//009 Luke Huan Univ. of Kansas 7

38 Association Summary Each rule: L => R has two parts: L, the left hand item set and R the right hand item set Each rule is measured by two parameters: Support Confidence 5//009 Luke Huan Univ. of Kansas 8