1 / 105

Jianping Fan CS Department UNC-Charlotte

Density-Based Data Clustering Algorithms: K-Means & Others. Jianping Fan CS Department UNC-Charlotte. http://webpages.uncc.edu/jfan/. What is Data Clustering?.

mkatie
Download Presentation

Jianping Fan CS Department UNC-Charlotte

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Density-Based Data Clustering Algorithms: K-Means & Others Jianping Fan CS Department UNC-Charlotte http://webpages.uncc.edu/jfan/

  2. What is Data Clustering? • Finding groups of objects such that the objects in a group will be similar (or related) to one another and different from (or unrelated to) the objects in other groups

  3. Applications of Data Clustering • Understanding • Group related documents for browsing, group genes and proteins that have similar functionality, or group stocks with similar price fluctuations • Summarization • Reduce the size of large data sets Why? Clustering precipitation in Australia

  4. This is the goal for visual analytics

  5. Inter-cluster distances are maximized Intra-cluster distances are minimized What are key issues for data clustering? • Similarity or distance function • Inter-cluster similarity or distance • Intra-cluster similarity or distance • Number of clusters

  6. What are key issues for data clustering? • Similarity or distance function x z Distance Similarity function

  7. What are key issues for data clustering? • Intra-cluster similarity or distance

  8. What are key issues for data clustering? • Inter-cluster similarity or distance

  9. What are key issues for data clustering? • Number of clusters K

  10. What are key issues data clustering? • Objective function for clustering Inter-Cluster Distance

  11. What are key issues data clustering? • Objective function for clustering Intra-Cluster Distance

  12. Inter-cluster distances are maximized Intra-cluster distances are minimized What are key issues data clustering? • Objective function for clustering Objective Function for Clustering

  13. K-Means (K-Centers) Clustering Cluster Centers

  14. K-Means Clustering • Similarity or distance function x z Distance Similarity function

  15. K-Means Clustering • Similarity or distance function for K-means clustering

  16. K-Means Clustering • Intra-cluster similarity or distance

  17. K-Means Clustering • Inter-cluster similarity or distance

  18. K-Means Clustering • Number of clusters K

  19. K-Means Clustering • Objective function for clustering Inter-Cluster Distance

  20. K-Means Clustering • Objective function for clustering Intra-Cluster Distance

  21. Inter-cluster distances are maximized Intra-cluster distances are minimized K-Means Clustering • Objective function for clustering Objective Function for Clustering

  22. Inter-cluster distances are maximized Intra-cluster distances are minimized K-Means Clustering • Objective Function for K-Means Clustering

  23. K-Means Clustering: Clustering for New Sample y Test Sample Test sample is assigned into the closest cluster!

  24. How many clusters? Six Clusters Two Clusters Four Clusters Notion of a Cluster can be Ambiguous Why it is ambiguous?

  25. K-Means Clustering Approach • Number of K (centers) • Similarity or distance function • Assignment rules • Determining number of centers • Putting centers in dense areas • Distance as similarity • Assign into the closest one

  26. K-Means Clustering • What are the potential problems for K-means clustering? • Number of centers? K • Where are the centers? Center selection • Inter-cluster overlapping • Data manifold

  27. K-Means Clustering • Number of centers? K • Where are the centers? Center selection

  28. K-Means Clustering • Number of centers? K • Where are the centers? Center selection

  29. K-Means Clustering • Number of centers? K • Where are the centers? Center selection If K is set too big, what we can do?

  30. K-Means Clustering • Number of centers? K • Where are the centers? Center selection If K is set too small, what we can do?

  31. K-Means Clustering • If initial K is too big, what we can do further? Merging the overlapped clusters as a bigger one Bottom-Up Approach

  32. K-Means Clustering • If initial K is too small, what we can do further? Top-Down Approach Separating diverse cluster into multiple homogeneous ones

  33. K-Means Clustering • Where are the centers? Center selection How to identify dense regions?

  34. K-Means Clustering • How to identify the dense regions for center selection? • Interactive selection via visualization When I see it, then I know it

  35. K-Means Clustering • How to identify the dense regions for center selection? • Automatic selection via DB-Scan

  36. K-Means Clustering • Inter-cluster overlapping

  37. K-Means Clustering • Data Manifold Spectral Clustering

  38. K-means Clustering • Partitional clustering approach • Each cluster is associated with a centroid (center point) • Each point is assigned to the cluster with the closest centroid • Number of clusters, K, must be specified • The basic algorithm is very simple Key Issues for K-Means: • K centroids • Similarity function • Objective function for optimization

  39. K-means Clustering – Details • Initial centroids are often chosen randomly. • Clusters produced vary from one run to another. • The centroid is (typically) the mean of the points in the cluster. • ‘Closeness’ is measured by Euclidean distance, cosine similarity, correlation, etc. • K-means will converge for common similarity measures mentioned above. • Most of the convergence happens in the first few iterations. • Often the stopping condition is changed to ‘Until relatively few points change clusters’ • Complexity is O( n * K * I * d ) • n = number of points, K = number of clusters, I = number of iterations, d = number of attributes

  40. Most common measure is Sum of Squared Error (SSE) For each point, the error is the distance to the nearest cluster To get SSE, we square these errors and sum them. x is a data point in cluster Ci and mi is the representative point for cluster Ci can show that micorresponds to the center (mean) of the cluster Given two clusters, we can choose the one with the smallest error One easy way to reduce SSE is to increase K, the number of clusters A good clustering with smaller K can have a lower SSE than a poor clustering with higher K Evaluating K-means Clusters

  41. K-means Given a set of observations (x1, x2, …, xn), where each observation is a d-dimensional real vector,  k-means clustering aims to partition the n observations into k (≤ n) sets S = {S1, S2, …, Sk} so as to minimize the within-cluster sum of squares (WCSS). In other words, its objective is to find: where μi is the mean of points in Si.

  42. K-means Given an initial set of k means m1(1),…,mk(1) , the algorithm proceeds by alternating between two steps: Assignment step: Assign each observation to the cluster whose mean yields the least within-cluster sum of squares (WCSS). Since the sum of squares is the squared Euclidean distance, this is intuitively the "nearest" mean.

  43. K-means Update step: Calculate the new means to be the  centroids of the observations in the new clusters.

  44. Optimal Clustering Sub-optimal Clustering Two different K-means Clustering Original Points

  45. Importance of Choosing Initial Centroid

  46. Importance of Choosing Initial Centroid

  47. Importance of Choosing Initial Centroid …

  48. Importance of Choosing Initial Centroid …

  49. Problems with Selecting Initial Points • If there are K ‘real’ clusters then the chance of selecting one centroid from each cluster is small. • Chance is relatively small when K is large • If clusters are the same size, n, then • For example, if K = 10, then probability = 10!/1010 = 0.00036 • Sometimes the initial centroids will readjust themselves in ‘right’ way, and sometimes they don’t • Consider an example of five pairs of clusters

  50. Solutions to Initial Centroid Problem • Multiple runs • Helps, but probability is not on your side • Sample and use hierarchical clustering to determine initial centroids • Select more than k initial centroids and then select among these initial centroids • Select most widely separated • Postprocessing • Bisecting K-means • Not as susceptible to initialization issues

More Related