! Two Fundamental Methods in Machine Learning! Supervised Learning ( learn from my example )

Transcription

1 Supervised vs. Unsupervised Learning Basic Machine Learning: Clustering CS 315 Web Search and Data Mining! Two Fundamental Methods in Machine Learning! Supervised Learning ( learn from my example ) n Goal: A program that performs a task as good as humans. n TASK well defined (the target funcnon) n EXPERIENCE training data provided by a human n PERFORMANCE error/accuracy on the task! Unsupervised Learning ( see what you can find ) n Goal: To find some kind of structure in the data. n TASK vaguely defined n No EXPERIENCE n No PERFORMANCE (but, there are some evaluanons metrics) 1 2 What is Clustering? Ex1: Cluster to Improve Recall! The most common form of Unsupervised Learning! Clustering is the process of grouping a set of physical or abstract obects into classes ( clusters ) of similar obects! Cluster hypothesis: Documents with similar text are related! Thus, when a query matches a document D, also return other documents in the cluster containing D.! It can be used in IR: n To improve recall in search n For bezer naviganon of search results

2 Ex2: Cluster for Better Navigation Clustering Characteristics! Flat Clustering vs Hierarchical Clustering n Flat: ust dividing obects in groups (clusters) n Hierarchical: organize clusters in a hierarchy! Evalua:ng Clustering n Internal Criteria n w The intra- cluster similarity is high (Nghtness) w The inter- cluster similarity is low (separateness) External Criteria w Did we discover the hidden classes? (we need gold standard data for this evaluanon) 5 6 Clustering for Web IR! RepresentaNon for clustering n Document representanon n Need a nonon of similarity/distance! How many clusters? n Fixed a priori? n Completely data driven? n Avoid trivial clusters - too large or small Recall: Documents as vectors! Each doc is a vector of 8.idf values, one component for each term. n Can normalize to unit length. d w i, d = = where w n i, = tfi, idfi d i = w 1 i,! Vector space n terms are axes - aka features n N docs live in this space n even with stemming, may have 20,000+ dimensions! What makes documents related? 7 8 2

3 Intuition for relatedness What makes documents related? D3 D2 D1! Ideal: semannc similarity.! PracNcal: stansncal similarity n We will use cosine similarity.! We will describe algorithms in terms of cosine similarity. t 2 y D4 x t 1 Cosine similarity of normalized d, d : k sim( d, d ) = n w w k i = 1 i, i, k Documents that are close together in vector space talk about the same things. This is known as the normalized inner product Clustering Algorithms! Hierarchical algorithms n BoZom- up, agglomeranve clustering! ParNNoning flat algorithms n Usually start with a random (parnal) parnnoning n Refine it iteranvely! The famous k- means parnnoning algorithm: n Given: a set of n documents and the number k n Compute: a parnnon of k clusters that opnmizes the chosen parnnoning criterion K-means! Assumes documents are real- valued vectors.! Clusters based on centroids of points in a cluster, c (= the center of gravity or mean) : 1 µ(c) = x c! Reassignment of instances to clusters is based on distance to the current cluster centroids.! See AnimaNon x c

4 K-Means Algorithm K-means: Different Issues Let d be the distance measure between instances. Select k random instances {s 1, s 2, s k } as seeds. Until clustering converges or other stopping criterion: For each instance x i : Assign x i to the cluster c such that d(x i, s ) is minimal. (Update the seeds to the centroid of each cluster) For each cluster c s = µ(c ) 13! When to stop? n When a fixed number of iteranons is reached n When centroid posinons do not change! Seed Choice n Results can vary based on random seed selecnon. n Try out mulnple starnng points If you start with centroids: B and E you converge to If you start with centroids D and F you converge to: A D Example showing sensitivity to seeds B E C F 14 Hierarchical clustering Hierarchical Agglomerative Clustering! Build a tree- based hierarchical taxonomy (dendrogram) from a set of unlabeled examples. vertebrate fish reptile amphib. mammal animal invertebrate worm insect crustacean! We assume there is a similarity funcnon that determines the similarity of two instances. Algorithm: Start with all instances in their own cluster. Until there is only one cluster: Among the current clusters, determine the two clusters, c i and c, that are most similar. Replace c i and c with a single cluster c i c Watch animation of HAC

5 What is the most similar cluster? Single link clustering! Single- link n Similarity of the most cosine- similar (single- link)! Complete- link n Similarity of the furthest points, the least cosine- similar! Group- average agglomeranve clustering n Average cosine between pairs of elements! Centroid clustering n Similarity of clusters centroids 17 1) Use maximum similarity of pairs: sim( c, c i ) = max sim( x, y) x c i, y c 2) After merging c i and c, the similarity of the resulting cluster to another cluster, c k, is: sim (( ci c ), ck ) = max( sim( ci, ck ), sim( c, ck )) 18 Complete link clustering Maor issue - labeling! Amer clustering algorithm finds clusters - how can they be useful to the end user? 1) Use minimum similarity of pairs: The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your! Need a concise label for each cluster n In search results, say Animal or Car in the aguar example. n In topic trees (Yahoo), need naviganonal cues. w Omen done by hand, a posteriori. 2) After merging c i and c, the similarity of the resulting cluster to another cluster, c k, is: The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open

6 How to Label Clusters! Show Ntles of typical documents n Titles are easy to scan n Authors create them for quick scanning! n But you can only show a few Ntles which may not fully represent cluster! Show words/phrases prominent in cluster n More likely to fully represent cluster n Use disnnguishing words/phrases n But harder to scan Further issues! Complexity: n Clustering is computanonally expensive. ImplementaNons need careful balancing of needs.! How to decide how many clusters are best?! EvaluaNng the goodness of clustering n There are many techniques, some focus on implementanon issues (complexity/nme), some on the quality of