Discerning Linkage-Based Algorithms Among Hierarchical Clustering Methods

1. Discerning Linkage-Based Algorithms Among Hierarchical Clustering Methods Margareta Ackerman and Shai Ben-David IJCAI 2011

2. Clustering is one of the most widely used tools for exploratory data analysis. Social Sciences Biology Astronomy Computer Science …. All apply clustering to gain a first understanding of the structure of large data sets.

3. The Theory-Practice Gap “While the interest in and application of cluster analysis has been rising rapidly, the abstract nature of the tool is still poorly understood” (Wright, 1973) • “There has been relatively little work aimed at reasoning about clustering independently of any particular algorithm, objective function, or generative data model” (Kleinberg, 2002) Both statements still apply today.

4. Bridging the Theory-Practice Gap:Previous work • Axioms of clustering [(Kleinberg, NIPS 02), (Ackerman & Ben-David, NIPS 08), (Meila, NIPS 08)] • Clusterability[(Balcan, Blum, and Vempala, STOC 08), (Ackerman & Ben-David, AISTATS 09) ]

5. Bridging the Theory-Practice Gap:Clustering algorithm selection There are a wide variety of clustering algorithms, which often produce very different clusterings. How should a user decide which algorithm to use for a given application? M. Ackerman, S. Ben-David, and D. Loker

6. Our approach for clustering algorithm selection We propose a framework that lets a user utilize prior knowledge to select an algorithm • Identify properties that distinguish between the input-output behaviour of different clustering algorithms • The properties should be: 1) Intuitive and “user-friendly” 2) Useful for classifying clustering algorithms

7. Previous Work in Property-Based Framework • A property-based classification of partitional clustering algorithms (Ackerman, Ben-David, and Loker, NIPS ‘10) • A characterizing of a single-linkage with the k-stopping criteria (Zadeh and Ben-David, UAI 09) • A characterization of linkage-based clustering with the k-stopping criteria (Ackerman, Ben-David, and Loker, COLT ‘10)

8. Our contributions • Extend the above property-based framework to the hierarchical clustering setting • Propose two intuitive properties that uniquely indentify hierarchical linkage-based clustering algorithms • Show that common hierarchical algorithms, including bisecting k-means, cannot be simulated by any linkage-based algorithm

9. Outline • Define Linkage-Based clustering • Introduce two new properties of hierarchical clustering algorithms • Main result • Hierarchical clustering paradigms that are not linkage-based • Conclusions

10. Formal Setup: Dendrograms and clusterings Dendrogram: A set C_i is a clusterin a dendrogramD if there exists a node in the dendrogram so that C_iisthe set of its leaf descendents.

11. Formal Setup: Dendrograms and clusterings C = {C1, … , Ck} is a clusteringin a dendrogramD if • Ciis a cluster in D for all 1≤ i ≤ k, and • clusters are disjoint,Ci∩Cj=Ø for all 1≤ i<j ≤k.

12. Formal Setup: Hierarchical clustering algorithm AHierarchical Clustering Algorithm A maps Input: A data set Xwith a distance function d, denoted (X,d) to Output:A dendrogram of X

13. Linkage-Based Algorithm An algorithm A is Linkage-Basedif there exists a linkage-function l:{(X1, X2 ,d): d over X1uX2 }→ R+ such that for any (X,d), A(X,d) can be constructed as follows: • Create a single-node tree for every elements of X

14. Linkage-Based Algorithm An algorithm A is Linkage-Basedif there exists a linkage-function l:{(X1, X2 ,d): d over X1uX2 }→ R+ such that for any (X,d), A(X,d) can be constructed as follows: • Create a single-node tree for every elements of X • Repeat the following until a single tree remains: Merge the pair of trees whose element sets are closest according to l. Ex. Single-linkage, average-linkage, complete linkage

15. Outline • Define Linkage-Based clustering • Introduce two new properties of hierarchical clustering algorithms • Main result • Hierarchical clustering paradigms that are not linkage-based • Conclusions

16. Locality Informal Definition D = A(X,d) D’ = A(X’,d) X’={x1, …, x6} If we select a set of disjoint clusters from a dendrogram, and run the algorithm on the union of these clusters, we obtain a result that is consistent with the original dendrogram.

17. Outer Consistency A(X,d) • The outer-consistent change makes the clustering C more prominent. • If A is outer-consistent, then A(X,d’) will also include the clustering C. C C on dataset (X,d’) C on dataset (X,d) Increase pairwise between-cluster distances

18. Outline • Define Linkage-Based clustering • Introduce two new properties of hierarchical clustering algorithms • Main result • Hierarchical clustering paradigms that are not linkage-based • Conclusions

19. Our Main Result Theorem: A hierarchical clustering function is Linkage-Based if and only if it is Local and Outer-Consistent.

20. Brief Sketch of Proof Recall direction: If A satisfies Outer-Consistency and Locality, then A is Linkage-Based. Goal: Define a linkage function l so that the linkage-based clustering based on loutputs A(X,d) (for every Xand d).

21. Brief Sketch of Proof • Define an operator <A: (X,Y,d1) <A(Z,W,d2)if when we run A on (XuYuZuW,d), where d extends d1and d2, X and Y are merged before Z and W. A(X,d) • Prove that <Acan be extended to a partial ordering by proving that it is cycle-free • This implies that there exists an order preserving function l that maps pairs of data sets to R+. Z W X Y

22. Outline • Define Linkage-Based clustering • Introduce two new properties of hierarchical clustering • Main result • Hierarchical clustering paradigms that are not linkage-based • Conclusions

23. Hierarchical but Not Linkage-Based • P -Divisive algorithms construct dendrograms top-down using a partitional 2-clustering algorithm P to determine how to split nodes. • Many natural partitional 2-clustering algorithms satisfy the following property: • A partitional 2-clustering algorithm Pis • Context Sensitive if there exist d⊂d’ so that • P({x,y,z),d) = {x, {y,z}} and P({x,y,z,w} ,d’)= {{x,y}, {z,w}}. Ex. K-means, min-sum, min-diameter, and further-centroids.

24. Hierarchical but Not Linkage-Based Theorem: If P is context-sensitive, then the P –divisive algorithm fails the locality property. • The input-output behaviourof some natural divisive algorithms is distinct from that of all linkage-based algorithms. • The bisecting k-means algorithm, and other natural divisive algorithms, cannot be simulated by any linkage-based algorithm.

25. Outline • Define Linkage-Based clustering • Introduce two new properties of hierarchical clustering algorithms • Main result • Hierarchical clustering paradigms that are not linkage-based • Conclusions

26. Conclusions • We characterize hierarchical Linkage-Based clustering in terms of two intuitive properties. • Show that some natural hierarchical algorithms have different input-output behavior than any linkage-based algorithm.

27. Locality D = A(X,d) D’ = A(X’,d) X’={x1, …, x6} For any clustering C = {C1, … , Ck} in D = A(X,d), C is also a clustering in D’ = A(X’ = uCi, d) Ci roots the same sub-dendrogram in both D and D’ For all x,y in X’, x occurs below y in Diff the same holds in D’.