Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Cluster Analysis Midterm: Monday Oct 29, 4PM Lecture Notes from Sept 5, 2007 until Oct 15, 2007. Chapters from Textbook and papers discussed in class (see below detailed list) Specific Readings Textbook: Chapter Chapter Chapter Chapter Chapter Chapter Chapter Papers: 1 2: 3: 4: 5 6: 7: 2.1- 2.4 3.1-3.4 4.1.1-4.1.2, 4.2.1 6.1-6.5, 6.9.1, 6.12, 6.13, 6.14 7.1-7.4 Apriori Paper: R. Agrawal, R. Srikant: Fast Algorithms for Mining Association Rules. VLDB 1994 MaxMiner Paper: R. J. Bayardo Jr: Efficiently Mining Long Patterns from Databases. SIGMOD 1998 SLIQ paper:M. Mehta, R. Agrawal, J. Rissanen: SLIQ: A Fast Scalable Classifier for Data Mining. EDBT 1996 Cluster Analysis What is Cluster Analysis? Types of Data in Cluster Analysis A Categorization of Major Clustering Methods Partitioning Methods Hierarchical Methods Density-Based Methods Grid-Based Methods Model-Based Clustering Methods Clustering High-Dimensional Data Clustering high-dimensional data Many applications: text documents, DNA micro-array data Major challenges: Many irrelevant dimensions may mask clusters Distance measure becomes meaningless—due to equi-distance Clusters may exist only in some subspaces Methods Feature transformation: only effective if most dimensions are relevant Feature selection: wrapper or filter approaches PCA & SVD useful only when features are highly correlated/redundant useful to find a subspace where the data have nice clusters Subspace-clustering: find clusters in all the possible subspaces CLIQUE, ProClus, and frequent pattern-based clustering The Curse of Dimensionality (graphs adapted from Parsons et al. KDD Explorations 2004) Data in only one dimension is relatively packed Adding a dimension “stretch” the points across that dimension, making them further apart Adding more dimensions will make the points further apart—high dimensional data is extremely sparse Distance measure becomes meaningless—due to equi-distance Why Subspace Clustering? (adapted from Parsons et al. SIGKDD Explorations 2004) Clusters may exist only in some subspaces Subspace-clustering: find clusters in some of the subspaces CLIQUE (Clustering In QUEst) Agrawal, Gehrke, Gunopulos, Raghavan (SIGMOD’98) Automatically identifying subspaces of a high dimensional data space that allow better clustering than original space CLIQUE can be considered as both density-based and gridbased It partitions each dimension into the same number of equal length interval It partitions an m-dimensional data space into non-overlapping rectangular units A unit is dense if the fraction of total data points contained in the unit exceeds the input model parameter A cluster is a maximal set of connected dense units within a subspace CLIQUE: The Major Steps Partition the data space and find the number of points that lie inside each cell of the partition. Identify the subspaces that contain clusters using the Apriori principle Identify clusters Determine dense units in all subspaces of interests Determine connected dense units in all subspaces of interests. Generate minimal description for the clusters Determine maximal regions that cover a cluster of connected dense units for each cluster Determination of minimal cover for each cluster =3 30 40 Vacation 20 50 Salary (10,000) 0 1 2 3 4 5 6 7 30 Vacation (week) 0 1 2 3 4 5 6 7 age 60 20 30 40 50 age 50 age 60 Strength and Weakness of CLIQUE Strength automatically finds subspaces of the highest dimensionality such that high density clusters exist in those subspaces insensitive to the order of records in input and does not presume some canonical data distribution scales linearly with the size of input and has good scalability as the number of dimensions in the data increases Weakness The accuracy of the clustering result may be degraded at the expense of simplicity of the method Frequent Pattern-Based Approach Clustering high-dimensional space (e.g., clustering text documents, microarray data) Projected subspace-clustering: which dimensions to be projected on? CLIQUE, ProClus Feature extraction: costly and may not be effective? Using frequent patterns as “features” “Frequent” are inherent features Mining freq. patterns may not be so expensive Typical methods Frequent-term-based document clustering Clustering by pattern similarity in micro-array data (pClustering) Clustering by Pattern Similarity (p-Clustering) Right: The micro-array “raw” data shows 3 genes and their values in a multi-dimensional space Difficult to find their patterns Bottom: Some subsets of dimensions form nice shift and scaling patterns Why p-Clustering? Microarray data analysis may need to Clustering on thousands of dimensions (attributes) Discovery of both shift and scaling patterns Clustering with Euclidean distance measure? — cannot find shift patterns Clustering on derived attribute Aij = ai – aj? — introduces N(N-1) dimensions Bi-cluster using transformed mean-squared residue score matrix (I, J) d ij 1 d | J | j J ij d Ij 1 d | I | i I ij d 1 d | I || J | i I , j J ij Where A submatrix is a δ-cluster if H(I, J) ≤ δ for some δ > 0 IJ Problems with bi-cluster No downward closure property, Due to averaging, it may contain outliers but still within δ-threshold p-Clustering Given objects x, y in O and features a, b in T, pCluster is a 2 by 2 matrix d xa d xb pScore( ) | (d xa d xb ) (d ya d yb ) | d d ya yb A pair (O, T) is in δ-pCluster if for any 2 by 2 matrix X in (O, T), pScore(X) ≤ δ for some δ > 0 Properties of δ-pCluster Downward closure Clusters are more homogeneous than bi-cluster (thus the name: pair-wise Cluster) Pattern-growth algorithm has been developed for efficient mining For scaling patterns, one can observe, taking logarithmic on will lead to the pScore form d xa / d ya d xb / d yb Cluster Analysis What is Cluster Analysis? Types of Data in Cluster Analysis A Categorization of Major Clustering Methods Partitioning Methods Hierarchical Methods Density-Based Methods Grid-Based Methods Model-Based Clustering Methods Model based clustering Assume data generated from K probability distributions Typically Gaussian distribution Soft or probabilistic version of K-means clustering Need to find distribution parameters. EM Algorithm EM Algorithm Initialize K cluster centers Iterate between two steps Expectation step: assign points to clusters w Pr(d P(di ck ) wk Pr( di | ck ) wk j Pr( d c ) i i |cj ) j k i N Maximation step: estimate model parameters k 1 m d i P ( d i ck ) i 1 P ( d i c j ) m k Cluster Analysis What is Cluster Analysis? Types of Data in Cluster Analysis A Categorization of Major Clustering Methods Partitioning Methods Hierarchical Methods Density-Based Methods Grid-Based Methods Model-Based Clustering Methods Cluster Validity Cluster Validity For supervised classification we have a variety of measures to evaluate how good our model is Accuracy, precision, recall For cluster analysis, the analogous question is how to evaluate the “goodness” of the resulting clusters? But “clusters are in the eye of the beholder”! Then why do we want to evaluate them? To To To To avoid finding patterns in noise compare clustering algorithms compare two sets of clusters compare two clusters 1 1 0.9 0.9 0.8 0.8 0.7 0.7 0.6 0.6 0.5 0.5 y Random Points y Clusters found in Random Data 0.4 0.4 0.3 0.3 0.2 0.2 0.1 0.1 0 0 0.2 0.4 0.6 0.8 0 1 DBSCAN 0 0.2 0.4 x 1 1 0.9 0.9 0.8 0.8 0.7 0.7 0.6 0.6 0.5 0.5 y y K-means 0.4 0.4 0.3 0.3 0.2 0.2 0.1 0.1 0 0 0.2 0.4 0.6 x 0.6 0.8 1 x 0.8 1 0 Complete Link 0 0.2 0.4 0.6 x 0.8 1 Different Aspects of Cluster Validation 1. 2. 3. Determining the clustering tendency of a set of data, i.e., distinguishing whether non-random structure actually exists in the data. Comparing the results of a cluster analysis to externally known results, e.g., to externally given class labels. Evaluating how well the results of a cluster analysis fit the data without reference to external information. - Use only the data 4. 5. Comparing the results of two different sets of cluster analyses to determine which is better. Determining the ‘correct’ number of clusters. For 2, 3, and 4, we can further distinguish whether we want to evaluate the entire clustering or just individual clusters. Measures of Cluster Validity Numerical measures that are applied to judge various aspects of cluster validity, are classified into the following three types. External Index: Used to measure the extent to which cluster labels match externally supplied class labels. Internal Index: Used to measure the goodness of a clustering structure without respect to external information. Sum of Squared Error (SSE) Relative Index: Used to compare two different clusterings or clusters. Entropy Often an external or internal index is used for this function, e.g., SSE or entropy Sometimes these are referred to as criteria instead of indices However, sometimes criterion is the general strategy and index is the numerical measure that implements the criterion. Measuring Cluster Validity Via Correlation Two matrices Proximity Matrix “Incidence” Matrix Compute the correlation between the two matrices One row and one column for each data point An entry is 1 if the associated pair of points belong to the same cluster An entry is 0 if the associated pair of points belongs to different clusters Since the matrices are symmetric, only the correlation between n(n-1) / 2 entries needs to be calculated. High correlation indicates that points that belong to the same cluster are close to each other. Not a good measure for some density or contiguity based clusters. Measuring Cluster Validity Via Correlation Correlation of incidence and proximity matrices for the K-means clusterings of the following two data sets. 1 1 0.9 0.9 0.8 0.8 0.7 0.7 0.6 0.6 0.5 0.5 y y 0.4 0.4 0.3 0.3 0.2 0.2 0.1 0.1 0 0 0.2 0.4 0.6 x Corr = -0.9235 0.8 1 0 0 0.2 0.4 0.6 x Corr = -0.5810 0.8 1 Using Similarity Matrix for Cluster Validation Order the similarity matrix with respect to cluster labels and inspect visually. 1 1 0.9 0.8 0.7 Points 0.6 y 0.5 0.4 0.3 0.2 0.1 0 10 0.9 20 0.8 30 0.7 40 0.6 50 0.5 60 0.4 70 0.3 80 0.2 90 0.1 100 0 0.2 0.4 0.6 x 0.8 1 20 40 60 Points 80 0 100 Similarity Using Similarity Matrix for Cluster Validation Clusters in random data are not so crisp 1 10 0.9 0.9 20 0.8 0.8 30 0.7 0.7 40 0.6 0.6 50 0.5 0.5 60 0.4 0.4 70 0.3 0.3 80 0.2 0.2 90 0.1 0.1 100 20 40 60 80 0 100 Similarity Points y Points 1 0 0 0.2 0.4 0.6 x DBSCAN 0.8 1 Using Similarity Matrix for Cluster Validation Clusters in random data are not so crisp 1 10 0.9 0.9 20 0.8 0.8 30 0.7 0.7 40 0.6 0.6 50 0.5 0.5 60 0.4 0.4 70 0.3 0.3 80 0.2 0.2 90 0.1 0.1 100 20 40 60 80 0 100 Similarity y Points 1 0 0 0.2 0.4 0.6 x Points K-means 0.8 1 Using Similarity Matrix for Cluster Validation Clusters in random data are not so crisp 1 10 0.9 0.9 20 0.8 0.8 30 0.7 0.7 40 0.6 0.6 50 0.5 0.5 60 0.4 0.4 70 0.3 0.3 80 0.2 0.2 90 0.1 0.1 100 20 40 60 80 0 100 Similarity y Points 1 0 0 Points 0.2 0.4 0.6 x Complete Link 0.8 1 Using Similarity Matrix for Cluster Validation 1 0.9 500 1 2 0.8 6 0.7 1000 3 0.6 4 1500 0.5 0.4 2000 0.3 5 0.2 2500 0.1 7 3000 DBSCAN 500 1000 1500 2000 2500 3000 0 Internal Measures: SSE Clusters in more complicated figures aren’t well separated Internal Index: Used to measure the goodness of a clustering structure without respect to external information SSE SSE is good for comparing two clusterings or two clusters (average SSE). Can also be used to estimate the number of clusters 10 9 6 8 7 4 6 SSE 2 5 4 0 3 -2 2 1 -4 0 -6 5 10 15 2 5 10 15 K 20 25 30 Internal Measures: SSE SSE curve for a more complicated data set 1 2 6 3 4 5 7 SSE of clusters found using K-means Framework for Cluster Validity Need a framework to interpret any measure. For example, if our measure of evaluation has the value, 10, is that good, fair, or poor? Statistics provide a framework for cluster validity The more “atypical” a clustering result is, the more likely it represents valid structure in the data Can compare the values of an index that result from random data or clusterings to those of a clustering result. If the value of the index is unlikely, then the cluster results are valid These approaches are more complicated and harder to understand. For comparing the results of two different sets of cluster analyses, a framework is less necessary. However, there is the question of whether the difference between two index values is significant Internal Measures: Cohesion and Separation Cluster Cohesion: Measures how closely related are objects in a cluster Example: SSE Cluster Separation: Measure how distinct or wellseparated a cluster is from other clusters Example: Squared Error Cohesion is measured by the within cluster sum of squares (SSE) 2 WSS ( x mi ) i xC i Separation is measured by the between cluster sum of squares BSS Ci (m mi )2 i Where |Ci| is the size of cluster i Internal Measures: Cohesion and Separation A proximity graph based approach can also be used for cohesion and separation. Cluster cohesion is the sum of the weight of all links within a cluster. Cluster separation is the sum of the weights between nodes in the cluster and nodes outside the cluster. cohesion separation Internal Measures: Silhouette Coefficient Silhouette Coefficient combine ideas of both cohesion and separation, but for individual points, as well as clusters and clusterings For an individual point, i Calculate a = average distance of i to the points in its cluster Calculate b = min (average distance of i to points in another cluster) The silhouette coefficient for a point is then given by s = 1 – a/b if a < b, (or s = b/a - 1 if a b, not the usual case) b a Typically between 0 and 1. The closer to 1 the better. Can calculate the Average Silhouette width for a cluster or a clustering Final Comment on Cluster Validity “The validation of clustering structures is the most difficult and frustrating part of cluster analysis. Without a strong effort in this direction, cluster analysis will remain a black art accessible only to those true believers who have experience and great courage.” Algorithms for Clustering Data, Jain and Dubes