Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Intro. ANN & Fuzzy Systems Lecture 20 Clustering (1) Intro. ANN & Fuzzy Systems Outline • Unsupervised learning (Competitive Learning) and Clustering • K-Means Clustering Algorithm (C) 2001 by Yu Hen Hu 2 Intro. ANN & Fuzzy Systems Unsupervised Learning • Data Mining – Understand internal/hidden structure of data distribution • Labeling (Target value, teaching input) Cost is High – Large amount of feature vectors – Sampling may involve costly experiments – Data label may not be available at all • Pre-processing for classification – features within the same cluster are similar, and – often belong to the same class (C) 2001 by Yu Hen Hu 3 Intro. ANN & Fuzzy Systems Competitive Learning • A form of unsupervised learning. • Neurons compete against each other with their activation values. The winner(s) reserve the privilege to update their weights. The losers may even be punished by updating their weights in opposite direction. • Competitive and Cooperative Learning: Competitive: Only one neuron's activation can be reinforced. Cooperative: Several neurons' activation can be reinforced. (C) 2001 by Yu Hen Hu 4 Intro. ANN & Fuzzy Systems Competitive Learning Rule • A neuron WINS the competition if its output is largest among all neurons for the same input x(n). • The weights of the winning neuron (k-th) is adjusted: D wk(n) [x(n) – wk(n)] The positions of losing neurons remain unchanged. • If the weights of a neuron represents its POSITION. If the output of a neuron is inversely proportional to the distance between x(n) and wk(n), then Competitive Learning = CLUSTERING! (C) 2001 by Yu Hen Hu 5 Intro. ANN & Fuzzy Systems Competitive Learning Example initial after 25 iterations 2 2 1 1 0 0 -1 -1 -2 -2 -1 0 1 2 -2 -2 after 75 iterations 2 1 1 0 0 -1 -1 -1 0 1 0 1 2 at end of 100 iterations 2 -2 -2 -1 2 -2 -2 -1 0 1 2 learncl1.m (C) 2001 by Yu Hen Hu 6 Intro. ANN & Fuzzy Systems What is “Clustering”? 1 What can we learn from these “unlabeled” data samples? 0.5 0 0 20 40 60 80 100 1.5 1 0.5 0 -0.5 (C) 2001 by Yu Hen Hu 0 0.5 1 – Structures: Some samples are closer to each other than other samples – The closeness between samples are determined using a “similarity measure” – The number of samples per unit volume is related to the concept of “density” or “distribution” 1.5 7 Intro. ANN & Fuzzy Systems Clustering Problem Statement • Given a set of vectors {xk; 1 k K}, find a set of M clustering centers {w(i); 1 i c} such that each xk is assigned to a cluster, say, w(i*), according to a distance (distortion, similarity) measure d(xk, w(i)) such that the average distortion 1 D K c K I ( x , i)d ( x ,W (i)) i 1 k 1 k k is minimized. • I(xk,i) = 1 if x is assigned to cluster i with cluster center w(I); and = 0 otherwise -- indicator function. (C) 2001 by Yu Hen Hu 8 Intro. ANN & Fuzzy Systems k-means Clustering Algorithm Initialization: Initial cluster center w(i); 1 i c, D(–1)= 0, I(xk,i) = 0, 1 i c, 1 k K; Repeat (A) Assign cluster membership (Expectation step) Evaluate d(xk, w(i)); 1 i c, 1 k K I(xk,i) = 1 if d(xk, w(i)) < d(xk, w(j)), j i; = 0; otherwise. 1kK N (B) Evaluate distortion D: D(iter ) I ( x , i )d ( x , w(i )) 1 k K k 1 k k (C) Update code words according to new assignment (Maximization) N W (i ) I ( xk , i ) xk , k 1 N N i I ( xk , i ), 1 i c k 1 (D) Check for convergence if 1–D(Iter–1)/D(Iter) < e , then convergent = TRUE, (C) 2001 by Yu Hen Hu 9 Intro. ANN & Fuzzy Systems A Numerical Example x = {-1, -2,0,2,3,4}, W={2.1, 2.3} 1. Assign membership 2.1: {-1, -2, 0, 2} 2.3: {3, 4} 2. Distortion D = (-1-2.1)2 + (-2-2.1)2 + (0-2.1)2 + (2-2.1)2 + (3-2.3)2 + (4-2.3)2 (C) 2001 by Yu Hen Hu 3. Update W to minimize distortion W1 = (-1-2+0+2)/4 = -.25 W2 = (3+4)/2 = 3.5 4. Reassign membership -.25: {-1, -2, 0} 3.5: {2, 3, 4} 5. Update W: w1 = (-1-2+0)/3 = -1 w2 = (2+3+4)/3 = 3. Converged. 10 Intro. ANN & Fuzzy Systems Kmeans Algorithm Demonstration 2.5 2 data points True cluster centers data samples converged centers true centers cluster boundary 2 1.5 1.5 1 1 0.5 0.5 0 0 -0.5 -0.5 -1 -0.5 (C) 2001 by Yu Hen Hu 0 0.5 1 1.5 -1 -2 -1 0 1 2 3 Clusterdemo.m 11