Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Data Mining Chapter 24 of book Dr Eamonn Keogh Computer Science & Engineering Department University of California - Riverside Riverside,CA 92521 [email protected] Don’t bother reading 24.3.7 or 24.3.8 What is Data Mining? Data Mining has been defined as “The nontrivial extraction of implicit, previously unknown, and potentially useful information from data”. data visualization statistics Data Mining databases artificial intelligence Informally, data mining is the extraction of interesting knowledge (rules, regularities, patterns, constraints) from data in large databases. Data Mining Broadly speaking, data mining is the process of semi- automatically analyzing large databases to find useful patterns Like knowledge discovery in artificial intelligence data mining discovers statistical rules and patterns Differs from machine learning in that it deals with large volumes of data stored primarily on disk. Some types of knowledge discovered from a database can be represented by a set of rules. • e.g.,: “Young women with annual incomes greater than $50,000 are most likely to buy sports cars” Other types of knowledge represented by equations, or by prediction functions, or by clusters Some manual intervention is usually required • Pre-processing of data, choice of which type of pattern to find, postprocessing to find novel patterns Applications of Data Mining Prediction based on past history • Predict if a credit card applicant poses a good credit risk, based on some attributes (income, job type, age, ..) and past history • Predict if a customer is likely to switch brand loyalty • Predict if a customer is likely to respond to “junk mail” • Predict if a pattern of phone calling card usage is likely to be fraudulent Some examples of prediction mechanisms: • Classification • Given a training set consisting of items belonging to different classes, and a new item whose class is unknown, predict which class it belongs to • Regression formulae • given a set of parameter-value to function-result mappings for an unknown function, predict the function-result for a new parameter-value Applications of Data Mining (Cont.) Descriptive Patterns • Associations • Find books that are often bought by the same customers. If a new customer buys one such book, suggest that he buys the others too. • Other similar applications: camera accessories, clothes, etc. • Associations may also be used as a first step in detecting causation • E.g. association between exposure to chemical X and cancer, or new medicine and cardiac problems • Clusters • E.g. typhoid cases were clustered in an area surrounding a contaminated well • Detection of clusters remains important in detecting epidemics Classification Rules Classification rules help assign new objects to a set of classes. E.g., given a new automobile insurance applicant, should he or she be classified as low risk, medium risk or high risk? Classification rules for above example could use a variety of knowledge, such as educational level of applicant, salary of applicant, age of applicant, etc. • person P, P.degree = masters and P.income > 75,000 P.credit = excellent • person P, P.degree = bachelors and (P.income 25,000 and P.income 75,000) P.credit = good Rules are not necessarily exact: there may be some misclassifications Classification rules can be compactly shown as a decision tree. Decision Tree person P, P.degree = masters and P.income > 75,000 P.credit = excellent Construction of Decision Trees Training set: a data sample in which the grouping for each tuple is already known. Consider credit risk example: Suppose degree is chosen to partition the data at the root. • Since degree has a small number of possible values, one child is created for each value. At each child node of the root, further classification is done if required. Here, partitions are defined by income. • Since income is a continuous attribute, some number of intervals are chosen, and one child created for each interval. Different classification algorithms use different ways of choosing which attribute to partition on at each node, and what the intervals, if any, are. In general • Different branches of the tree could grow to different levels. • Different nodes at the same level may use different partitioning attributes. Construction of Decision Trees (Cont.) Greedy top down generation of decision trees. • Each internal node of the tree partitions the data into groups based on a partitioning attribute, and a partitioning condition for the node • More on choosing partitioning attribute/condition shortly • Algorithm is greedy: the choice is made once and not revisited as more of the tree is constructed • The data at a node is not partitioned further if either • all (or most) of the items at the node belong to the same class, or • all attributes have been considered, and no further partitioning is possible. • Such a node is a leaf node. • Otherwise the data at the node is partitioned further by picking an attribute for partitioning data at the node. Best Splits Idea: evaluate different attributes and partitioning conditions and pick the one that best improves the “purity” of the training set examples • The initial training set has a mixture of instances from different classes and is thus relatively impure • E.g. if degree exactly predicts credit risk, partitioning on degree would result in each child having instances of only one class • I.e., the child nodes would be pure The purity of a set S of training instances can be measured quantitatively in several ways. Notation: number of classes = k, number of instances = |S|, fraction of instances in class i = pi. The Gini measure of purity is defined as [ k Gini (S) = 1 - p2i i- 1 • When all instances are in a single class, the Gini value is 0, while it reaches its maximum (of 1 –1 /k) if each class the same number of instances. Best Splits (Cont.) Another measure of purity is the entropy measure, which is defined as k entropy (S) = – i- 1 pilog2 pi When a set S is split into multiple sets Si, I=1, 2, …, r, we can measure the purity of the resultant set of sets as: r purity(S1, S2, ….., Sr) = |Si| i= 1 |S| purity (Si) The information gain due to particular split of S into Si, i = 1, 2, …., r Information-gain (S, {S1, S2, …., Sr) = purity(S) – purity (S1, S2, … Sr) Best Splits (Cont.) Measure of “cost” of a split: Information-content(S, {S1, S2, ….., Sr})) = – r |Si| i- 1 |S| log2 |Si| |S| Information-gain ratio = Information-gain (S, {S1, S2, ……, Sr}) Information-content (S, {S1, S2, ….., Sr}) The best split for an attribute is the one that gives the maximum information gain ratio Continuous valued attributes • • Can be ordered in a fashion meaningful to classification • e.g. integer and real values Categorical attributes • Cannot be meaningfully ordered (e.g. country, school/university, item-color, .): Finding Best Splits • Categorical attributes: • Multi-way split, one child for each value • may have too many children in some cases • Binary split: try all possible breakup of values into two sets, and pick the best • Continuous valued attribute • Binary split: • Sort values in the instances, try each as a split point • E.g. if values are 1, 10, 15, 25, split at 1, 10, 15 • Pick the value that gives best split • Multi-way split: more complicated, see bibliographic notes • A series of binary splits on the same attribute has roughly equivalent effect Decision-Tree Construction Algorithm I Procedure Grow.Tree(S) Partition(S); Procedure Partition (S) if (purity(S) > p or |S| < s) then return; for each attribute A evaluate splits on attribute A; Use best split found (across all attributes) to partition S into S1, S2, …., Sr, for i = 1, 2, ….., r Partition(Si); Decision-Tree Construction Algorithm II • Variety of algorithms have been developed to • Reduce CPU cost and/or • Reduce IO cost when handling datasets larger than memory • Improve accuracy of classification • Decision tree may be overfitted, i.e., overly tuned to given training set • Pruning of decision tree may be done on branches that have too few training instances • When a subtree is pruned, an internal node becomes a leaf • and its class is set to the majority class of the instances that map to the node • Pruning can be done by using a part of the training set to build tree, and a second part to test the tree • prune subtrees that increase misclassification on second part Shoe Size A visual intuition of the classification problem 10 9 8 7 6 5 4 3 2 1 Given a database (called a training database) of labeled examples, predict future unlabeled examples… What is the class of Homer? 1 2 3 4 5 6 7 8 9 10 Blood Sugar name Health status Shoe Size Blood Sugar Edna ?????? 7 9.2 name Health status Shoe Size Blood Sugar Lisa Healthy 2 2.2 Bart Sick 6 7.1 Apu Sick 11 9.3 Joe Healthy 3 3.1 Tom Sick 8 7.9 Sue Healthy 7 2.8 Decision-Tree Shoe Size A visual intuition 10 9 8 7 6 5 4 3 2 1 Blood Sugar > 3.5? 1 2 3 4 5 6 7 8 9 10 Blood Sugar no yes Healthy Sick White Cell Count 10 9 8 7 6 5 4 3 2 1 Blood Sugar > 3.9? no yes White Cell > 4.3? 1 2 3 4 5 6 7 8 9 10 Blood Sugar 10 9 8 7 6 5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 yes no Healthy sick sick Other Types of Classifiers • • Further types of classifiers • Neural net classifiers • Bayesian classifiers Neural net classifiers use the training data to train artificial neural nets • • Widely studied in AI, won’t cover here Bayesian classifiers use Bayes theorem, which says • p(cj | d) = p(d | cj ) p(cj) • p(d) where p(cj | d) = probability of instance d being in class cj, • p(d | cj) = probability of generating instance d given class cj, • p(cj) = probability of occurrence of class cj, and • p(d) = probability of instance d occurring For more details see: Keogh, E. & Pazzani, M. (1999). Learning augmented Bayesian classifiers: A comparison of distribution-based and classification-based approaches. In Uncertainty 99, 7th. Int'l Workshop on AI and Statistics, Ft. Lauderdale, FL, pp. 225--230. Naïve Bayesian Classifiers • Bayesian classifiers require • computation of p(d | cj) • precomputation of p(cj) • p(d) can be ignored since it is the same for all classes • To simplify the task, naïve Bayesian classifiers assume attributes have independent distributions, and thereby estimate • p(d|cj) = p(d1|cj) * p(d2|cj) * ….* (p(dn|cj) • Each of the p(di|cj) can be estimated from a histogram on di values for each class cj • the histogram is computed from the training instances • Histograms on multiple attributes are more expensive to compute and store Naïve Bayesian Classifiers Visual Intuition I 6 foot 6 5 foot 8 4 foot 8 5 foot 8 Naïve Bayesian Classifiers Visual Intuition II p(cj | d) = probability of instance d being in class cj, P(male | 5 foot 8 ) = 10 / (10 + 2) = 0.833 P(female | 5 foot 8 ) = 2 / (10 + 2) = 0.166 10 2 5 foot 8 Clustering Clustering: Intuitively, finding clusters of points in the given data such that similar points lie in the same cluster Can be formalized using distance metrics in several ways E.g. Group points into k sets (for a given k) such that the average distance of points from the centroid of their assigned group is minimized • Centroid: point defined by taking average of coordinates in each dimension. • Another metric: minimize average distance between every pair of points in a cluster Has been studied extensively in statistics, but on small data sets • Data mining systems aim at clustering techniques that can handle very large data sets • E.g. the Birch clustering algorithm (more shortly) What is Clustering? Also called unsupervised learning, sometimes called classification by statisticians and sorting by psychologists and segmentation by people in marketing • Organizing data into classes such that there is • high intra-class similarity • low inter-class similarity • Finding the class labels and the number of classes directly from the data (in contrast to classification). • More informally, finding natural groupings among objects. (I.e east coast cities, west coast cities) What is a natural grouping among these objects? What is a natural grouping among these objects? Clustering is subjective Simpson's Family School Employees Females Males What is Similarity? The quality or state of being similar; likeness; resemblance; as, a similarity of features. Webster's Dictionary Similarity is hard to define, but… “We know it when we see it” The real meaning of similarity is a philosophical question. We will take a more pragmatic approach. Similarity Measures For the moment assume that we can measure the similarity between any two objects. (we will cover this in detail later). One intuitive example is to measure the distance between two cities and call it the similarity. For example we have D(LA,San Diego) = 110, and D(LA,New York) = 3,000. This would allow use to make (subjectively correct) statements like “LA is more similar to San Francisco that it is to New York”. MACSTEEL USA LOCATIONS Defining Distance Measures Definition: Let O1 and O2 be two objects from the universe of possible objects. The distance (dissimilarity) between O1 and O2 is a real number denoted by D(O1,O2) Peter Piotr 0.23 3 342.7 Peter Piotr d('', '') = 0 d(s, '') = d('', s) = |s| -- i.e. length of s d(s1+ch1, s2+ch2) = min( d(s1, s2) + if ch1=ch2 then 0 else 1 fi, d(s1+ch1, s2) + 1, d(s1, s2+ch2) + 1 ) When we peek inside one of these black boxes, we see some function on two variables. These functions might very simple or very complex. In either case it is natural to ask, what properties should these functions have? 3 What properties should a distance measure have? • D(A,B) = D(B,A) • D(A,A) = 0 • D(A,B) = 0 IIf A= B • D(A,B) D(A,C) + D(B,C) Symmetry Constancy of Self-Similarity Positivity (Separation) Triangular Inequality Intuitions behind desirable distance measure properties D(A,B) = D(B,A) Symmetry Otherwise you could claim “Alex looks like Bob, but Bob looks nothing like Alex.” D(A,A) = 0 Constancy of Self-Similarity Otherwise you could claim “Alex looks more like Bob, than Bob does.” D(A,B) = 0 IIf A=B Positivity (Separation) Otherwise there are objects in your world that are different, but you cannot tell apart. D(A,B) D(A,C) + D(B,C) Triangular Inequality Otherwise you could claim “Alex is very like Bob, and Alex is very like Carl, but Bob is very unlike Carl.” Two Types of Clustering • Partitional algorithms: Construct various partitions and then evaluate them by some criterion (we will see an example called BIRCH) • Hierarchical algorithms: Create a hierarchical decomposition of the set of objects using some criterion Hierarchical Partitional A Useful Tool for Summarizing Similarity Measurements In order to better appreciate and evaluate the examples given in the early part of this talk, we will now introduce the dendrogram. Terminal Branch Root Internal Branch Internal Node Leaf The similarity between two objects in a dendrogram is represented as the height of the lowest internal node they share. Note that hierarchies are commonly used to organize information, for example in a web portal. Yahoo’s hierarchy is manually created, we will focus on automatic creation of hierarchies in data mining. Business & Economy B2B Finance Aerospace Agriculture… Shopping Banking Bonds… Jobs Animals Apparel Career Workspace (Bovine:0.69395,(Gibbon:0.36079,(Orangutan:0.3 3636,(Gorilla:0.17147,(Chimp:0.19268,Human:0. 11927):0.08386):0.06124):0.15057):0.54939); Desirable Properties of a Clustering Algorithm • Scalability (in terms of both time and space) • Ability to deal with different data types • Minimal requirements for domain knowledge to determine input parameters • Able to deal with noise and outliers • Insensitive to order of input records • Incorporation of user-specified constraints • Interpretability and usability Hierarchical Clustering The number of dendrograms with n leafs = (2n -3)!/[(2(n -2)) (n -2)!] Number of Leafs 2 3 4 5 ... 10 Number of Possible Dendrograms 1 3 15 105 … 34,459,425 Since we cannot test all possible trees we will have to heuristic search of all possible trees. We could do this.. Bottom-Up (agglomerative): Starting with each item in its own cluster, find the best pair to merge into a new cluster. Repeat until all clusters are fused together. Top-Down (divisive): Starting with all the data in a single cluster, consider every possible way to divide the cluster into two. Choose the best division and recursively operate on both sides. We begin with a distance matrix which contains the distances between every pair of objects in our database. 0 D( , ) = 8 D( , ) = 1 8 8 7 7 0 2 4 4 0 3 3 0 1 0 Bottom-Up (agglomerative): Starting with each item in its own cluster, find the best pair to merge into a new cluster. Repeat until all clusters are fused together. Consider all possible merges… … Choose the best Bottom-Up (agglomerative): Starting with each item in its own cluster, find the best pair to merge into a new cluster. Repeat until all clusters are fused together. Consider all possible merges… Consider all possible merges… … … Choose the best Choose the best Bottom-Up (agglomerative): Starting with each item in its own cluster, find the best pair to merge into a new cluster. Repeat until all clusters are fused together. Consider all possible merges… Consider all possible merges… Consider all possible merges… Choose the best … … … Choose the best Choose the best Bottom-Up (agglomerative): Starting with each item in its own cluster, find the best pair to merge into a new cluster. Repeat until all clusters are fused together. Consider all possible merges… Consider all possible merges… Consider all possible merges… Choose the best … … … Choose the best Choose the best We know how to measure the distance between two objects, but defining the distance between an object and a cluster, or defining the distance between two clusters is non obvious. • Single linkage (nearest neighbor): In this method the distance between two clusters is determined by the distance of the two closest objects (nearest neighbors) in the different clusters. • Complete linkage (furthest neighbor): In this method, the distances between clusters are determined by the greatest distance between any two objects in the different clusters (i.e., by the "furthest neighbors"). • Group average: In this method, the distance between two clusters is calculated as the average distance between all pairs of objects in the two different clusters. Summary of Hierarchal Clustering Methods • No need to specify the number of clusters in advance. • Hierarchal nature maps nicely onto human intuition for some domains • They do not scale well: time complexity of at least O(n2), where n is the number of total objects. • Like any heuristic search algorithms, local optima are a problem. • Interpretation of results is subjective. Partitional Clustering Algorithms Clustering algorithms have been designed to handle very large datasets E.g. the Birch algorithm • Main idea: use an in-memory R-tree to store points that are being clustered • Insert points one at a time into the R-tree, merging a new point with an existing cluster if is less than some distance away • If there are more leaf nodes than fit in memory, merge existing clusters that are close to each other • At the end of first pass we get a large number of clusters at the leaves of the R-tree Merge clusters to reduce the number of clusters Partitional Clustering Algorithms The Birch algorithm We need to specify the number of clusters in advance, I have chosen 2 R10 R11 R10 R11 R12 R1 R2 R3 R12 R4 R5 R6 R7 R8 R9 Data nodes containing points Partitional Clustering Algorithms The Birch algorithm R10 R11 R10 R11 R12 {R1,R2} R3 R4 R5 R6 R12 R7 R8 R9 Data nodes containing points Partitional Clustering Algorithms The Birch algorithm R10 R11 R12 Up to this point we have simply assumed that we can measure similarity, but How do we measure similarity? Peter Piotr 0.23 3 342.7 A generic technique for measuring similarity To measure the similarity between two objects, transform one of the objects into the other, and measure how much effort it took. The measure of effort becomes the distance measure. The distance between Patty and Selma. Change dress color, 1 point Change earring shape, 1 point Change hair part, 1 point D(Patty,Selma) = 3 The distance between Marge and Selma. Change dress color, Add earrings, Decrease height, Take up smoking, Lose weight, D(Marge,Selma) = 5 1 1 1 1 1 point point point point point This is called the “edit distance” or the “transformation distance” Edit Distance Example It is possible to transform any string Q into string C, using only Substitution, Insertion and Deletion. Assume that each of these operators has a cost associated with it. How similar are the names “Peter” and “Piotr”? Assume the following cost function Substitution Insertion Deletion 1 Unit 1 Unit 1 Unit D(Peter,Piotr) is 3 The similarity between two strings can be defined as the cost of the cheapest transformation from Q to C. Peter Note that for now we have ignored the issue of how we can find this cheapest transformation Substitution (i for e) Piter Insertion (o) Pioter Deletion (e) Piotr Association Rules (market basket analysis) Retail shops are often interested in associations between different items that people buy. • Someone who buys bread is quite likely also to buy milk • A person who bought the book Database System Concepts is quite likely also to buy the book Operating System Concepts. Associations information can be used in several ways. • E.g. when a customer buys a particular book, an online shop may suggest associated books. Association rules: bread milk DB-Concepts, OS-Concepts Networks • Left hand side: antecedent, right hand side: consequent • An association rule must have an associated population; the population consists of a set of instances • E.g. each transaction (sale) at a shop is an instance, and the set of all transactions is the population Association Rule Definitions Set of items: I={I1,I2,…,Im} Transactions: D={t1,t2, …, tn}, tj I Itemset: {Ii1,Ii2, …, Iik} I Support of an itemset: Percentage of transactions which contain that itemset. Large (Frequent) itemset: Itemset whose number of occurrences is above a threshold. Association Rules Example I = { Beer, Bread, Jelly, Milk, PeanutButter} Support of {Bread,PeanutButter} is 60% Association Rule Definitions Association Rule (AR): implication X Y where X,Y I and X Y = the null set; Support of AR (s) X Y: Percentage of transactions that contain X Y Confidence of AR (a) X Y: Ratio of number of transactions that contain X Y to the number that contain X Association Rules Ex (cont’d) Association Rules Ex (cont’d) Of 5 transactions, 3 involve both Bread and PeanutButter, 3/5 = 60% Of the 4 transactions that involve Bread, 3 of them also involve PeanutButter 3/4 = 75% Association Rule Problem Given a set of items I={I1,I2,…,Im} and a database of transactions D={t1,t2, …, tn} where ti={Ii1,Ii2, …, Iik} and Iij I, the Association Rule Problem is to identify all association rules X Y with a minimum support and confidence (supplied by user). NOTE: Support of X Y is same as support of X Y. Association Rule Algorithm (Basic Idea) 1. Find Large Itemsets. 2. Generate rules from frequent itemsets. This is the simple naïve algorithm, better algorithms exist. Association Rule Algorithm We are generally only interested in association rules with reasonably high support (e.g. support of 2% or greater) Naïve algorithm 1. Consider all possible sets of relevant items. 2. For each set find its support (i.e. count how many transactions purchase all items in the set). • • Large itemsets: sets with sufficiently high support Use large itemsets to generate association rules. • From itemset A generate the rule A - {b} b for each b A. • Support of rule = support (A). • Confidence of rule = support (A ) / support (A - {b}) • From itemset A generate the rule A - {b} b for each b A. • Support of rule = support (A). • Confidence of rule = support (A ) / support (A - {b}) Lets say itemset A = {Bread, Butter, Milk} Then A - {b} b for each b A includes 3 possibilities {Bread, Butter} Milk {Bread, Milk} Butter {Butter, Milk} Bread Apriori Large Itemset Property: Any subset of a large itemset is large. Contrapositive: If an itemset is not large, none of its supersets are large. Large Itemset Property Large Itemset Property If B is not frequent, then none of the supersets of B can be frequent. If {ACD} is frequent, then all subsets of {ACD} ({AC}, {AD}, {CD}) must be frequent.