1 / 51

Clustering

Clustering. Paolo Ferragina Dipartimento di Informatica Università di Pisa. This is a mix of slides taken from several presentations, plus my touch !. Objectives of Cluster Analysis.

virgo
Download Presentation

Clustering

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Clustering Paolo Ferragina Dipartimento di Informatica Università di Pisa This is a mix of slides taken from several presentations, plus my touch !

  2. Objectives of Cluster Analysis • Finding groups of objects such that the objects in a group will be similar (or related) to one another and different from (or unrelated to) the objects in other groups Competing objectives Intra-cluster distances are minimized Inter-cluster distances are maximized The commonest form of unsupervised learning

  3. Yahoo! Hierarchy isn’t clustering but is the kind of output you want from clustering www.yahoo.com/Science … (30) agriculture biology physics CS space ... ... ... ... ... dairy botany cell AI courses crops craft magnetism HCI missions agronomy evolution forestry relativity

  4. Google News: automatic clustering gives an effective news presentation metaphor

  5. Sec. 16.1 For improving search recall • Cluster hypothesis - Documents in the same cluster behave similarly with respect to relevance to information needs • Therefore, to improve search recall: • Cluster docs in corpus a priori • When a query matches a doc D, also return other docs in the cluster containing D • Hope if we do this: The query “car” will also return docs containing automobile

  6. Sec. 16.1 For better visualization/navigation of search results

  7. Sec. 16.2 Issues for clustering • Representation for clustering • Document representation • Vector space? Normalization? • Need a notion of similarity/distance • How many clusters? • Fixed a priori? • Completely data driven?

  8. Notion of similarity/distance • Ideal: semantic similarity • Practical: term-statistical similarity • Docs as vectors • We will use cosine similarity. • For many algorithms, easier to think in terms of a distance (rather than similarity) between docs.

  9. Clustering Algorithms • Flat algorithms • Create a set of clusters • Usually start with a random (partial) partitioning • Refine it iteratively • K means clustering • Hierarchical algorithms • Create a hierarchy of clusters (dendogram) • Bottom-up, agglomerative • Top-down, divisive

  10. Hard vs. soft clustering • Hard clustering: Each document belongs to exactly one cluster • More common and easier to do • Soft clustering: Each document can belong to more than one cluster. • Makes more sense for applications like creating browsable hierarchies • News is a proper example • Search results is another example

  11. Flat & Partitioning Algorithms • Given: a set of n documents and the number K • Find: a partition in K clusters that optimizes the chosen partitioning criterion • Globally optimal • Intractable for many objective functions • Ergo, exhaustively enumerate all partitions • Locally optimal • Effective heuristic methods: K-means and K-medoids algorithms

  12. Sec. 16.4 K-Means • Assumes documents are real-valued vectors. • Clusters based on centroids (aka the center of gravity or mean) of points in a cluster, c: • Reassignment of instances to clusters is based on distance to the current cluster centroids.

  13. Sec. 16.4 K-Means Algorithm Select K random docs {s1, s2,… sK} as seeds. Until clustering converges (or other stopping criterion): For each doc di: Assign di to the cluster crsuch that dist(di, sr) is minimal. For each cluster cj sj= (cj)

  14. Sec. 16.4 Pick seeds Reassign clusters Compute centroids Reassign clusters x x Compute centroids x x x x K Means Example (K=2) Reassign clusters Converged!

  15. Sec. 16.4 Termination conditions • Several possibilities, e.g., • A fixed number of iterations. • Doc partition unchanged. • Centroid positions don’t change.

  16. Sec. 16.4 Convergence • Why should the K-means algorithm ever reach a fixed point? • K-means is a special case of a general procedure known as the Expectation Maximization (EM) algorithm • EM is known to converge • Number of iterations could be large • But in practice usually isn’t

  17. Sec. 16.4 Convergence of K-Means • Define goodness measure of cluster c as sum of squared distances from cluster centroid: • G(c,s)= Σi (di – sc)2 (sum over all di in cluster c) • G(C,s) = Σc G(c,s) • Reassignment monotonically decreases G • It is a coordinate descent algorithm (opt one component at a time) • At any step we have some value for G(C,s) 1) Fix s, optimize C  assign d to the closest centroid  G(C’,s) < G(C,s) 2) Fix C’, optimize s  take the new centroids  G(C’,s’) < G(C’,s) < G(C,s) The new cost is smaller than the original one  local minimum

  18. Sec. 16.4 Time Complexity • The centroids are K • Each doc/centroid consists of M dimensions • Computing distance btw vectors is O(M) time. • Reassigning clusters: Each doc compared with all centroids, O(KNM) time. • Computing centroids: Each doc gets added once to some centroid, O(NM) time. Assume these two steps are each done once for I iterations: O(IKNM).

  19. Sec. 16.4 Seed Choice • Results can vary based on random seed selection. • Some seeds can result in poor convergence rate, or convergence to sub-optimal clusterings. • Select good seeds using a heuristic • doc least similar to any existing centroid • According to a probability distribution that depends inversely-proportional on the distance from the other current centroids Example showing sensitivity to seeds In the above, if you start with B and E as centroids you converge to {A,B,C} and {D,E,F} If you start with D and F you converge to {A,B,D,E} {C,F}

  20. How Many Clusters? • Number of clusters K is given • Partition n docs into predetermined number of clusters • Finding the “right” number of clusters is part of the problem • Can usually take an algorithm for one flavor and convert to the other.

  21. Bisecting K-means Variant of K-means that can produce a partitional or a hierarchical clustering

  22. Bisecting K-means Example

  23. K-means Pros • Simple • Fast for low dimensional data • It can find pure sub-clusters if large number of clusters is specified (but, over-partitioning) Cons • K-Means cannot handle non-globular data of different sizes and densities • K-Means will not identify outliers • K-Means is restricted to data which has the notion of a center (centroid)

  24. Ch. 17 animal vertebrate invertebrate fish reptile amphib. mammal worm insect crustacean Hierarchical Clustering • Build a tree-based hierarchical taxonomy (dendrogram) from a set of documents • One approach: recursive application of a partitional clustering algorithm

  25. Strengths of Hierarchical Clustering • No assumption of any particular number of clusters • Any desired number of clusters can be obtained by ‘cutting’ the dendogram at the proper level • They may correspond to meaningful taxonomies • Example in biological sciences (e.g., animal kingdom, phylogeny reconstruction, …)

  26. Sec. 17.1 Hierarchical Agglomerative Clustering (HAC) • Starts with each doc in a separate cluster • Then repeatedly joins the closest pairof clusters, until there is only one cluster. • The history of mergings forms a binary tree or hierarchy.

  27. Sec. 17.2 Closest pair of clusters • Single-link • Similarity of the closest points, the most cosine-similar • Complete-link • Similarity of the farthest points, the least cosine-similar • Centroid • Clusters whose centroids are the closest (or most cosine-similar) • Average-link • Clusters whose average distance/cosine between pairs of elements is the smallest

  28. p1 p2 p3 p4 p5 . . . p1 p2 p3 p4 p5 . . . How to Define Inter-Cluster Similarity Similarity? • Single link (MIN) • Complete link (MAX) • Centroids • Average Proximity Matrix

  29. p1 p2 p3 p4 p5 . . . p1 p2 p3 p4 p5 . . . How to Define Inter-Cluster Similarity • MIN • MAX • Centroids • Average Proximity Matrix

  30. p1 p2 p3 p4 p5 . . . p1 p2 p3 p4 p5 . . . How to Define Inter-Cluster Similarity • MIN • MAX • Centroids • Average Proximity Matrix

  31. p1 p2 p3 p4 p5 . . . p1 p2 p3 p4 p5 . . . How to define Inter-Cluster Similarity   • MIN • MAX • Centroids • Average Proximity Matrix

  32. p1 p2 p3 p4 p5 . . . p1 p2 p3 p4 p5 . . . How to Define Inter-Cluster Similarity • MIN • MAX • Centroids • Average Proximity Matrix

  33. p1 p2 p3 p4 p5 . . . p1 p2 p3 p4 p5 . . . Starting Situation • Start with clusters of individual points and a proximity matrix Proximity Matrix

  34. C1 C2 C3 C4 C5 C1 C2 C3 C4 C5 Intermediate Situation • After some merging steps, we have some clusters C3 C4 C1 Proximity Matrix C5 C2

  35. C1 C2 C3 C4 C5 C1 C2 C3 C4 C5 Intermediate Situation • We want to merge the two closest clusters (C2 and C5) and update the proximity matrix. C3 C4 C1 Proximity Matrix C5 C2

  36. After Merging • The question is “How do we update the proximity matrix?” C2 U C5 C1 C3 C4 C1 ? C3 ? ? ? ? C2 U C5 C4 C3 ? ? C4 C1 Proximity Matrix C2 U C5

  37. 1 2 3 4 5 Cluster Similarity: MIN or Single Link • Similarity of two clusters is based on the two most similar (closest) points in the different clusters • Determined by one pair of points, i.e., by one link in the proximity graph.

  38. 5 1 3 5 2 1 2 3 6 4 4 Hierarchical Clustering: MIN Nested Clusters Dendrogram

  39. Two Clusters Strength of MIN Original Points • Can handle non-elliptical shapes

  40. Two Clusters Limitations of MIN Original Points • Sensitive to noise and outliers

  41. 1 2 3 4 5 Cluster Similarity: MAX or Complete Linkage • Similarity of two clusters is based on the two least similar (most distant) points in the different clusters • Determined by all pairs of points in the two clusters

  42. 4 1 2 5 5 2 3 6 3 1 4 Hierarchical Clustering: MAX Nested Clusters Dendrogram

  43. Two Clusters Strength of MAX Original Points • Less susceptible to noise and outliers

  44. Two Clusters Limitations of MAX Original Points • Tends to break large clusters • Biased towards globular clusters

  45. 1 2 4 5 3 Cluster Similarity: Average • Proximity of two clusters is the average of pairwise proximity between points in the two clusters.

  46. 5 4 1 2 5 2 3 6 1 4 3 Hierarchical Clustering: Average Nested Clusters Dendrogram

  47. 5 1 5 3 1 4 1 2 5 2 5 2 1 5 5 2 2 2 3 6 3 6 3 6 3 3 1 4 4 1 4 4 4 Hierarchical Clustering: Comparison MAX MIN Average

  48. Sec. 16.3 How to evaluate clustering quality ? Assesses a clustering with respect to ground truth … requires labeled data Produce the gold standard is costly !!

  49. A different approach: Co-Clustering • Co-occurrence Matrices Characteristics • Data sparseness • High dimension • Noise • Is it possible to combine the document and term clustering together? • Yes, Co-Clustering • Simultaneously cluster the rows and columns of the co-occurrence matrix.

  50. Information-Theoretic Co-Clustering • View co-occurrence matrix as a joint probability distribution between row & column random variables • We seek a hard-clustering of both dimensions such that loss in “Mutual Information” is minimized given a fixed no. of row & col. clusters

More Related