Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
DATA MINING van data naar informatie Ronald Westra Dep. Mathematics Maastricht University CLUSTERING AND CLUSTER ANALYSIS Data Mining Lecture IV [Chapter 8: sections 8.4 and Chapter 9 from Principles of Data Mining by Hand,, Manilla, Smyth ] 1. Clustering versus Classification • • • • classification: give a pre-determined label to a sample clustering: provide the relevant labels for classification from structure in a given dataset clustering: maximal intra-cluster similarity and maximal inter-cluster dissimilarity Objectives: - 1. segmentation of space - 2. find natural subclasses Examples of Clustering and Classification 1. Computer Vision Examples of Clustering and Classification: 1. Computer Vision Example of Clustering and Classification: 1. Computer Vision Examples of Clustering and Classification: 2. Types of chemical reactions Examples of Clustering and Classification: 2. Types of chemical reactions Voronoi Clustering Georgy Fedoseevich Voronoy 1868 - 1908 Voronoi Clustering A Voronoi diagram (also called a Voronoi tessellation, Voronoi decomposition, Dirichlet tessellation), is a special kind of decomposition of a metric space determined by distances to a specified discrete set of objects in the space, e.g., by a discrete set of points. Voronoi Clustering Voronoi Clustering Voronoi Clustering Voronoi Clustering Voronoi Clustering Voronoi Clustering Voronoi Clustering Partitional Clustering [book section 9.4] score-functions centroid intra-cluster distance inter-cluster distance C-means [book page 303] k-means clustering (also: C-means) The k-means algorithm assigns each point to the cluster whose center (also called centroid) is nearest. The center is the average of all the points in the cluster, ie its coordinates is the arithmetic mean for each dimension separately for all the points in the cluster. k-means clustering (also: C-means) Example: The data set has three dimensions and the cluster has two points: X = (x1, x2, x3) and Y = (y1, y2, y3). Then the centroid Z becomes Z = (z1, z2, z3), where z1 = (x1 + y1)/2 and z2 = (x2 + y2)/2 and z3 = (x3 + y3)/2 k-means clustering (also: C-means) This is the basic structure of the algorithm (J. MacQueen, 1967): •Randomly generate k clusters and determine the cluster centers or directly generate k seed points as cluster centers •Assign each point to the nearest cluster center. •Recompute the new cluster centers. •Repeat until some convergence criterion is met (usually that the assignment hasn't changed). C-means [book page 303] while changes in cluster Ck % form clusters for k=1,…,K do Ck = {x | ||x – rk|| < || x – rl|| } end % compute new cluster centroids for k=1,…,K do rk = mean({x | x Ck }) end end k-means clustering (also: C-means) The main advantages of this algorithm are its simplicity and speed, which allows it to run on large datasets. Yet it does not systematically yield the same result with each run of the algorithm. Rather, the resulting clusters depend on the initial assignments. The kmeans algorithm maximizes inter-cluster (or minimizes intra-cluster) variance, but does not ensure that the solution given is not a local minimum of variance. k-means clustering k-means clustering (also: C-means) k-means clustering (also: C-means) k-means clustering (also: C-means) Fuzzy c-means One of the problems of the k-means algorithm is that it gives a hard partitioning of the data, that is to say that each point is attributed to one and only one cluster. But points on the edge of the cluster, or near another cluster, may not be as much in the cluster as points in the center of cluster. Fuzzy c-means Therefore, in fuzzy clustering, each point does not pertain to a given cluster, but has a degree of belonging to a certain cluster, as in fuzzy logic. For each point x we have a coefficient giving the degree of being in the k-th cluster uk(x). Usually, the sum of those coefficients has to be one, so that uk(x) denotes a probability of belonging to a certain cluster: Fuzzy c-means With fuzzy c-means, the centroid of a cluster is computed as being the mean of all points, weighted by their degree of belonging to the cluster, that is: Fuzzy c-means The degree of being in a certain cluster is related to the inverse of the distance to the cluster then the coefficients are normalized and fuzzyfied with a real parameter m > 1 so that their sum is 1. So : Fuzzy c-means For m equal to 2, this is equivalent to normalising the coefficient linearly to make their sum 1. When m is close to 1, then cluster center closest to the point is given much more weight than the others, and the algorithm is similar to k-means. Fuzzy c-means The fuzzy c-means algorithm is greatly similar to the k-means algorithm : Fuzzy c-means •Choose a number of clusters •Assign randomly to each point coefficients for being in the clusters •Repeat until the algorithm has converged (that is, the coefficients' change between two iterations is no more than ε, the given sensitivity threshold) : •Compute the centroid for each cluster, using the formula above •For each point, compute its coefficients of being in the clusters, using the formula above Fuzzy C-means uij is membership of sample i to custer j ck is centroid of custer i while changes in cluster Ck % compute new memberships for k=1,…,K do for i=1,…,N do ujk = f(xj – ck) end end % compute new cluster centroids for k=1,…,K do % weighted mean ck = SUMj jkxk xj /SUMj ujk end end Fuzzy c-means The fuzzy c-means algorithm minimizes intracluster variance as well, but has the same problems as k-means, the minimum is local minimum, and the results depend on the initial choice of weights. Fuzzy c-means Fuzzy c-means Fuzzy c-means Centroids Trajectory of Fuzzy MultiVariate Centroids Fuzzy c-means 1 0.9 0.8 0.7 1 0.6 2 453 0.5 0.4 2 4 0.3 0.2 3 5 0.1 0 0.8 1 -0.1 0 0.2 0.4 0.6 0.8 1 Hierarchical Clustering [book section 9.5] One major problem with partitional clustering is that the number of clusters (= #classes) must be pre-specified !!! This poses the question: what IS the real number of clusters in a given set of data? Answer: it depends! • Agglomerative methods: bottom-up • Divisive methods: top-down Hierarchical Clustering Agglomerative hierarchical clustering Hierarchical Clustering Hierarchical Clustering Hierarchical Clustering Hierarchical Clustering Example of Clustering and Classification 1. Clustering versus Classification • • • • classification: give a pre-determined label to a sample clustering: provide the relevant labels for classification from structure in a given dataset clustering: maximal intra-cluster similarity and maximal inter-cluster dissimilarity Objectives: - 1. segmentation of space - 2. find natural subclasses DATA ANALYSIS AND UNCERTAINTY Data Mining Lecture V [Chapter 4, Hand, Manilla, Smyth ] RANDOM VARIABLES Random Variables [4.3] multivariate random variables marginal density conditional density & dependency: p(x|y) = p(x,y) / p(y) * example: supermarket purchases RANDOM VARIABLES Example: supermarket purchases X = n customers x p products; X(i,j) = Boolean variable: “Has customer #i bought a product of type p ?” nA = sum(X(:,A)) is number of customers that bought product A nB = sum(X(:,B)) is number of customers that bought product B nAB = sum(X(:,A).*X(:,B)) is number of customers that bought product B *** Demo: matlab: conditionaldensity RANDOM VARIABLES (conditionally) independent: p(x,y) = p(x)*p(y) i.e. : p(x|y) = p(x) RANDOM VARIABLES Simpson’s paradox Observation of two different treatments for several categories of patients Category Treatment A Treatment B Old 0.33 0.20 Young 1.00 0.53 Total 0.50 0.40 RANDOM VARIABLES Explanation: the sets are POOLED: Category Treatment A Old 2 / 10 0.20 Young 48 / 90 0.53 Total 50 / 100 0.50 Treatment B 30 / 90 0.33 10 / 10 1.00 40 / 100 0.40 Both for Old and Young individually, treatment B seems best, but for total Treatment A seems superior. RANDOM VARIABLES - 1st order Markov processes [example: text analysis and reconstruction with Markov chains] Demo: dir *.txt type 'werther.txt' [m0,m1,m2] = Markov2Analysis('werther.txt'); CreateMarkovText(m0,m1,m2,3,80,20); Nederlands, duits, frans, engels, matlabs, europees Correlation, dependency, causation Example 4.3 SAMPLING Sampling [4.4] - Large quantities of observations: central limit theorem : normally distributed Small sample size: modeling: OK, pattern recognition: not Figure 4.1 Role of model parameters Probability of observing data D = { x(1), x(1), …, x(n)} with independent probabilities: n P( D | M , ) p(x(i ) | M , ) i 1 with fixed parameter-set Θ. ESTIMATION Estimation [4.5] - Properties 1. ˆ is estimator of θ – depends on sample so stochastic variable with E[ˆ] etc. 2. Bias: bias (ˆ) E[ˆ] unbiased: bias (ˆ) 0 , i.e. no systematic bias 3. consistent estimator : asymptotically unbiased 4. Variance: var(ˆ) E[ˆ E[ˆ]] 2 5. Best unbiased estimator: unbiased estimator with smallest variance Maximum Likelihood Estimation - Maximum Likelihood Estimation n - Expression 4.7: L( | D) f (x(i ) | ) i 1 scalar function in θ drop all terms not containing θ value of θ with highest L(θ) is maximum likelihood estimator MLE: ˆMLE arg max L( ) - Example 4.4 + figure 4.2 Demo: MLbinomial Maximum Likelihood Estimation - Example 4.5: Normal distribution, log L(θ) Example 4.6: Sufficient statistic: nice idea but practical infeasible MLE: biased bur consistent O(1/n) In practice log- likelihood l(θ) = log L(θ) is more attractive Multiple parameters: complex but relevant: EM-algorithm [treated later] Determination of confidence levels: example 4.8 Maximum Likelihood Estimation - Sampling using central limit theorem (CLT) with large samples: Bootstrap method: Example 4.9: use observed data as i. The real distribution F, ii. To generate many sub-samples, iii. Estimate properties in these sub-samples using the “known” distribution F, iv. Compare sub-sample estimate and “real” estimate and apply CLT. BAYESIAN ESTIMATION - Bayesian estimation - Here not frequentist approach but subjective approach: D is given and parameters θ are random variables. p(θ) represents our degree of belief in the value of θ in practice this means that we have to make all sorts of (wild) assumptions about p. prior distributon: p(θ) posterior distributon: p(θ|D) Modification of prior to posterior by Bayes theorem: p( D | ) p( ) p ( D | ) p( ) p( | D) p( D) p( D | ) p( )d - BAYESIAN ESTIMATION - MAP: Maximum a Posteriori method: < θ> : mean or mode of the posterior distribution MAP relates to MLE Example 4.10 Bayesian approach: rather than point-estimates keep full knowledge of uncertainties in model(s) This causes massive computation -> therefore only recent decade popular BAYESIAN ESTIMATION - Sequential updating: p( | D) p( D | ) p( ) (equation 4.10) p( | D1 , D2 ) p( D2 | ) p( D1 | ) p( ) (equation 4.15) Reminds of Markov Process Algorithm: * start with p(θ) * new data D: use eq. 4.10 to obtain posterior distribution * new data D2 : use eq. 4.15 to obtain new posterior distribution * repeat ad nauseam BAYESIAN ESTIMATION - Large Problem: different observers may define different p(θ); the choice is subjective! hence: different results! Analysis depends on choice of p(θ) - Often a consensus prior e.g. Jeffrey’s prior (equations 4.16 and 4.17) - Computation of credibility interval - Markov Chain Monte Carlo (MCMC) methods for estimating posterior distributions - Similarly: Bayesian Belief Networks (BBN) BAYESIAN ESTIMATION - Bayesian Approach: Uncertainty_Model + Uncertainty_Params MLE: aim is: a point estimate Bayesian: aim is: posterior distribution Bayesian estimate = weighted estimate over all models M and all parameters θ where the weights are the likelihoods of the parameters in the different models PROBLEM: these weights are difficult to estimate PROBABILISTIC MODEL-BASED CLUSTERING USING MIXTURE MODELS Data Mining Lecture VI [4.5, 8.4, 9.2, 9.6, Hand, Manilla, Smyth ] Probabilistic Model-Based Clustering using Mixture Models A probability mixture model A mixture model is a formalism for modeling a probability density function as a sum of parameterized functions. In mathematical terms: A probability mixture model where pX(x) is the modeled probability distribution function, K is the number of components in the mixture model, and ak is mixture proportion of component k. By definition, 0 < ak < 1 for all k = 1…K and: A probability mixture model h(x | λk) is a probability distribution parameterized by λk. Mixture models are often used when we know h(x) and we can sample from pX(x), but we would like to determine the ak and λk values. Such situations can arise in studies in which we sample from a population that is composed of several distinct subpopulations. A common approach for ‘decomposing’ a mixture model It is common to think of mixture modeling as a missing data problem. One way to understand this is to assume that the data points under consideration have "membership" in one of the distributions we are using to model the data. When we start, this membership is unknown, or missing. The job of estimation is to devise appropriate parameters for the model functions we choose, with the connection to the data points being represented as their membership in the individual model distributions. Probabilistic Model-Based Clustering using Mixture Models The EM-algorithm [book section 8.4] Mixture Decomposition: The ‘Expectation-Maximization’ Algorithm The Expectation-maximization algorithm computes the missing memberships of data points in our chosen distribution model. It is an iterative procedure, where we start with initial parameters for our model distribution (the ak's and λk's of the model listed above). The estimation process proceeds iteratively in two steps, the Expectation Step, and the Maximization Step. The ‘Expectation-Maximization’ Algorithm The expectation step With initial guesses for the parameters in our mixture model, we compute "partial membership" of each data point in each constituent distribution. This is done by calculating expectation values for the membership variables of each data point. The ‘Expectation-Maximization’ Algorithm The maximization step With the expectation values in hand for group membership, we can recompute plug-in estimates of our distribution parameters. For the mixing coefficient of this is simply the fractional membership of all data points in the second distribution. EM-algorithm for Clustering The Suppose we have data D with a model with parameters and hidden parameters H Interpretation: H = the class label Log-likelihood of observed data: l ( ) log p( ) log p( D, H | ) H EM-algorithm for Clustering With p the probability over the data D. Let Q be the unknown distribution over the hidden parameters H Then the log-likelihood is: Then the log-likelihood is: l ( ) log p( D, H | ) H p ( D, H | ) l ( ) log Q( H ) Q( H ) H p ( D, H | ) l ( ) Q( H ). log Q( H ) H [*Jensen’s inequality] 1 l ( ) Q( H ). log p( D, H | ) Q( H ). log F (Q, ) Q( H ) H H Jensen’s inequality for a concave-down function, the expected value of the function is less than the function of the expected value. The gray rectangle along the horizontal axis represents the probability distribution of x, which is uniform for simplicity, but the general idea applies for any distribution EM-algorithm So: F(Q,) is a lower-bound on the log-likelihood function l(Q,) . EM alternates between: E-step: maximising F to Q with fixed , and: M-step: maximising F to with fixed Q . EM-algorithm E-step: M-step: Q k 1 k 1 arg max F (Q , ) k k Q arg max F (Q k 1 , ) k Probabilistic Model-Based Clustering using Gaussian Mixtures Probabilistic Model-Based Clustering using Mixture Models Probabilistic Model-Based Clustering using Mixture Models Probabilistic Model-Based Clustering using Mixture Models Probabilistic Model-Based Clustering using Mixture Models Gaussian Mixture Decomposition Gaussian mixture Decomposition is a good classificator. It allows supervised as well as unsupervised learning (find how many classes is optimal, how they should be defined,...). But training is iterative and time consuming. Idea is to set position and width of gaussian distribution(s) to optimize the coverage of learning samples. Probabilistic Model-Based Clustering using Mixture Models Probabilistic Model-Based Clustering using Mixture Models Probabilistic Model-Based Clustering using Mixture Models Probabilistic Model-Based Clustering using Mixture Models Probabilistic Model-Based Clustering using Mixture Models Probabilistic Model-Based Clustering using Mixture Models Probabilistic Model-Based Clustering using Mixture Models Probabilistic Model-Based Clustering using Mixture Models Probabilistic Model-Based Clustering using Mixture Models Probabilistic Model-Based Clustering using Mixture Models The End