Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Topic 5: Data Mining II Classification and Prediction The classification model assumes a set of predefined classes and aims to classify a large collection of tuples/samples to these classes A B C 1 1 2 2 2 3 1 2 1 1 2 2 3 2 1 2 1 2 C 1 1 2 2 2 3 1 2 1 1 2 2 3 2 1 2 1 2 Dr. N. Mamoulis C predicts categorical class labels classifies data (constructs a model) based on the training set and the values (class labels) in a class attribute and uses it in classifying new data Prediction: class Y Clustering is else called unsupervised classification; the aim is to divide the data tuples into non-predefined classes. cluster X A B class X A Classification: cluster Y models continuous-valued functions, i.e., predicts unknown or missing values Typical Applications credit approval target marketing medical diagnosis treatment effectiveness analysis B Advanced Database Technologies 1 Classification—A Two-Step Process Dr. N. Mamoulis Advanced Database Technologies 2 Classification Process (1): Model Construction Model construction: describing a set of predetermined classes Each tuple/sample is assumed to belong to a predefined class, as determined by the class label attribute The set of tuples used for model construction: training set The model is represented as classification rules, decision trees, or mathematical formulae Model usage: for classifying future or unknown objects NAME M ik e M ary B ill J im D ave Anne Estimate accuracy of the model The known label of test sample is compared with the classified result from the model Accuracy rate is the percentage of test set samples that are correctly classified by the model Test set is independent of training set, otherwise over-fitting will occur Dr. N. Mamoulis Advanced Database Technologies Classification Algorithms Training Data 3 Classification Process (2): Use the Model in Prediction RANK YEARS TENURED A ssistan t P ro f 3 no A ssistan t P ro f 7 yes P ro fesso r 2 yes A sso c iate P ro f 7 yes A ssistan t P ro f 6 no A sso c iate P ro f 3 no Dr. N. Mamoulis Classifier (Model) IF rank = ‘professor’ OR years > 6 THEN tenured = ‘yes’ Advanced Database Technologies 4 Classification by Decision Tree Induction Decision tree Classifier Testing Data Unseen Data Decision tree generation consists of two phases (Jeff, Professor, 4) NAME Tom M erlisa G eorge Joseph RANK YEARS TENURED Assistant Prof 2 no Associate Prof 7 no Professor 5 yes Assistant Prof 7 yes Dr. N. Mamoulis Advanced Database Technologies Tenured? A flow-chart-like tree structure Internal node denotes a test on an attribute Branch represents an outcome of the test Leaf nodes represent class labels or class distribution Tree construction At start, all the training examples are at the root Partition examples recursively based on selected attributes Tree pruning Identify and remove branches that reflect noise or outliers Use of decision tree: Classifying an unknown sample 5 Test the attribute values of the sample against the decision tree Dr. N. Mamoulis Advanced Database Technologies 6 1 Training Dataset This follows an example from Quinlan’s ID3 age <=30 <=30 31…40 >40 >40 >40 31…40 <=30 <=30 >40 <=30 31…40 31…40 >40 Dr. N. Mamoulis income high high high medium low low low medium low medium medium medium high medium Output: A Decision Tree for “buys_computer” student no no no no yes yes yes no yes yes yes no yes no credit_rating fair excellent fair fair fair excellent excellent fair fair fair excellent excellent fair excellent Advanced Database Technologies 7 How is the decision tree constructed? Advanced Database Technologies What is Cluster Analysis? Cluster: a collection of data objects Similar to one another within the same cluster Dissimilar to the objects in other clusters Grouping a set of data objects into clusters As a stand-alone tool to get insight into data distribution As a preprocessing step for other algorithms credit rating? no yes excellent fair no yes no yes Classification rules: IF age = “<=30” AND student = “no” THEN buys_computer = “no” … Dr. N. Mamoulis Advanced Database Technologies 8 Assign to sample X the class label C such that P(C|X) is maximal Naïve Bayes Classifier: Uses Bayes-theorem to estimate probabilities, assuming attributes are conditionally independent Classification by backpropagation (Neural Networks) Classification based on association rules mining k-nearest neighbor classifier case-based reasoning Rough set approach Fuzzy set approaches 9 Dr. N. Mamoulis Advanced Database Technologies 10 General Applications of Clustering Pattern Recognition Spatial Data Analysis Cluster analysis Clustering is unsupervised classification: no predefined classes Typical applications >40 Genetic algorithms All samples for a given node belong to the same class There are no remaining attributes for further partitioning – majority voting is employed for classifying the leaf There are no samples left Dr. N. Mamoulis yes student? Bayesian Classification At start, all the training examples are at the root. Attributes are categorical (if continuous-valued, they are discretized in advance) The attribute with the highest information gain is selected, and their values formulate partitions. The examples are then partitioned and the tree is constructed recursively by selecting the attribute with the highest information gain at the next level. The information gain of an attribute is a probabilistic measure, that reflects the “least randomness” in the partitions. Conditions for stopping partitioning/recursion 30..40 Other Classification Methods Basic algorithm (a greedy algorithm) age? <=30 create thematic maps in GIS by clustering feature spaces detect spatial clusters and explain them in spatial data mining Image Processing Economic Science (especially market research) WWW Document classification Cluster Weblog data to discover groups of similar access patterns Dr. N. Mamoulis Advanced Database Technologies 12 2 Requirements of Clustering Clustering Visualized Input of Clustering Scalability Ability to deal with different types of attributes Discovery of clusters with arbitrary shape Minimal requirements for domain knowledge to determine input parameters Able to deal with noise and outliers High dimensionality Incorporation of user-specified constraints Interpretability and usability Advanced Database Technologies A B C 1 1 2 2 2 3 1 2 1 1 2 2 3 2 1 2 1 2 A cluster X C cluster Y outliers B Output of Clustering Insensitive to order of input records Dr. N. Mamoulis A large data set with n attributes (dimensions). Its tuples can be modeled as points in a high dimensional space 13 Major Clustering Approaches A set of k clusters, where the distance between points in the same cluster is small and the distance between points in different clusters is large A set of outliers, i.e., points which do not belong to a cluster; they form clusters of small cardinality and thus small interest. Dr. N. Mamoulis Advanced Database Technologies 14 Partitioning Algorithms: Basic Concept Partitioning algorithms: Construct various partitions and then evaluate them by some criterion Hierarchy algorithms: Create a hierarchical decomposition of the set of data (or objects) using some criterion Density-based: based on connectivity and density functions Grid-based: based on a multiple-level granularity structure Model-based: A model is hypothesized for each of the clusters and the idea is to find the best fit of that model to Partitioning method: Construct a partition of a database D of n objects into a set of k clusters Given a k, find a partition of k clusters that optimizes the chosen partitioning criterion Global optimal: exhaustively enumerate all partitions Heuristic methods: k-means and k-medoids algorithms k-means: Each cluster is represented by the center of the cluster k-medoids or PAM (Partition around medoids): Each cluster is represented by one of the objects in the cluster each other Dr. N. Mamoulis Advanced Database Technologies 15 The K-Means Clustering Method Dr. N. Mamoulis Advanced Database Technologies The K-Means Clustering Method Example for k=2 Given k, the k-means algorithm is implemented in 4 steps: 1. Partition objects into k non-empty subsets 2. Compute seed points as the centroids of the clusters of the current partition. The centroid is the center (mean point) of the cluster. 3. Assign each object to the cluster with the nearest seed point. 4. Go back to Step 2, stop when no more new assignment. 10 10 9 9 8 8 7 7 6 6 5 5 4 4 3 3 2 2 1 1 0 0 0 1 2 3 4 5 6 7 8 9 10 10 9 9 8 8 7 7 6 6 5 5 4 4 3 Dr. N. Mamoulis 2 3 4 5 6 7 8 9 10 2 1 0 0 17 1 3 2 0 Advanced Database Technologies 0 10 1 Dr. N. Mamoulis 16 1 2 3 4 5 6 7 8 9 10 0 1 2 Advanced Database Technologies 3 4 5 6 7 8 9 10 18 3 PAM (Partitioning Around Medoids) Comments on the K-Means Method Strength Relatively efficient: O(tkn), where n is # objects, k is # clusters, and t is # iterations. Normally, k, t << n. Often terminates at a local optimum. The global optimum may be found using techniques such as: deterministic annealing and genetic algorithms Weakness Applicable only when mean is defined, then what about categorical data? Need to specify k, the number of clusters, in advance Unable to handle noisy data and outliers Not suitable to discover clusters with non-convex shapes starts from an initial set of medoids and iteratively replaces one of the medoids by one of the non-medoids if it improves the total distance of the resulting clustering\Use real object to represent the cluster: 1. Select k representative objects arbitrarily 2. For each pair of non-selected object h and selected object i, calculate the total swapping cost TCih For each pair of i and h, 3. If TCih < 0, i is replaced by h Then assign each non-selected object to the most similar representative object 4. Dr. N. Mamoulis Advanced Database Technologies 19 PAM Clustering: Total swapping cost TCih=∑jCjih 10 7 7 j h 5 i 4 3 6 5 h i 4 3 2 2 1 1 0 0 0 1 2 3 4 5 6 7 8 9 10 Cjih = d(j, h) - d(j, i) 0 1 2 3 4 5 6 Step 0 7 8 9 Cjih = 0 a b 9 9 h 8 7 8 j 6 5 i 5 4 t 3 h t 3 2 2 1 1 0 1 2 3 4 5 6 7 8 9 Cjih = d(j, t) - d(j, i) 10 0 1 2 3 4 5 6 8 9 Cjih = d(j, h) - d(j, t) Advanced Database Technologies 21 do not scale well: time complexity of at least O(n2), where n is the number of total objects can never undo what was done previously Integration of hierarchical with distance-based clustering Step 4 BIRCH (1996): uses CF-tree and incrementally adjusts the quality of sub-clusters CURE (1998): selects well-scattered points from the cluster and then shrinks them towards the center of the cluster by a specified fraction CHAMELEON (1999): hierarchical clustering using dynamic modeling Advanced Database Technologies Step 2 Step 1 Step 0 Advanced Database Technologies divisive (DIANA) 22 Use links to measure similarity/proximity Links: The number of common neighbors for the two points {1,2,3}, {1,2,4}, {1,2,5}, {1,3,4}, {1,3,5} {1,4,5}, {2,3,4}, {2,3,5}, {2,4,5}, {3,4,5} 3 {1,2,3} {1,2,4} Algorithm Dr. N. Mamoulis Step 3 Dr. N. Mamoulis Clustering Categorical Data (ROCK) Major weakness of agglomerative clustering methods de 10 Hierarchical Clustering (cont’d) abcde cde e 7 agglomerative (AGNES) ab d j 0 0 Step 2 Step 3 Step 4 c 7 6 i 4 Step 1 10 10 10 Dr. N. Mamoulis t 8 6 20 Use distance matrix as clustering criteria. This method does not require the number of clusters k as an input, but needs a termination condition j 9 t 8 Advanced Database Technologies Hierarchical Clustering 10 9 repeat steps 2-3 until there is no change Dr. N. Mamoulis 23 Draw random sample Cluster with links Dr. N. Mamoulis Advanced Database Technologies 24 4 DBSCAN: Density Based Spatial Clustering of Applications with Noise Density-Based Clustering Methods Relies on a density-based notion of cluster: A cluster is defined as a maximal set of density-connected points Discovers clusters of arbitrary shape in spatial databases with noise Clustering based on density (local cluster criterion), such as density-connected points Major features: Discover clusters of arbitrary shape Handle noise One scan Need density parameters as termination condition Outlier Several interesting studies: Border DBSCAN: Ester, et al. (KDD’96) OPTICS: Ankerst, et al (SIGMOD’99). DENCLUE: Hinneburg & D. Keim (KDD’98) CLIQUE: Agrawal, et al. (SIGMOD’98) Dr. N. Mamoulis Advanced Database Technologies 25 If the number of attributes is large (high dimensional space), clustering can be meaningless, just like nearest neighbor search. This is because of the “dimensionality curse”; a point could be as close to its cluster as to the other clusters. For such cases, clustering could be more meaningful in a subset of the full-dimensionality, where the rest of the dimensions are “noise” to the specific cluster. Some clustering techniques (e.g., CLIQUE, PROCLUS) discover clusters in subsets of the full dimensional space. Advanced Database Technologies Dr. N. Mamoulis MinPts = 5 Advanced Database Technologies 26 The Role of DB Research in Classification and Clustering Sub-space clustering Dr. N. Mamoulis Eps = 1cm Core 27 Classification is a classic problem in Statistics and Machine Learning. ML has mainly focused on improving the accuracy of classification. On the other hand, database research mainly focuses on improving the scalability of clustering/classification methods. We would like to cluster/classify data sets with millions of tuples and hundreds of attributes (dimensions) with reasonable speed. The next presentations will describe two clustering methods and a scalable decision-tree classifier. Dr. N. Mamoulis Advanced Database Technologies 28 5