Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Algorithms for Big Data Classification Dr. Jianye Hao Associate Professor School of Software Review • Decision Tree Classifier – decision tree induction algorithm • a greedy approach • how to specify the attribute splitting condition? – Binary vs multi-way splitting • how to select the optimal splitting? – Impurity measures: Gini index, entropy, classification error • When to stop splitting? – Overfitting vs underfitting Review • How to evaluate and compare the performance of a classifier? – Accuracy, precision, recall… – Confidence interval • the accuracy rate of a classifier • the accuracy difference between two models (statistically significant?) • the accuracy difference between two algorithms (statistically significant?) Classical classification methods • Bayes Classifier max(P(wi|x)) • Linear discriminant classifier • Quadratic discriminant classifier • Minimum Distance Classifier min(Dj(x)) • Minimum Error Rate Classifier min(ri(x)) • K-Nearest Neighbor Classifier Joint probability & conditional probability • If we have two variables, X and Y, some probabilities involve both X and Y • Joint Probability: p(x,y) is the probability that two things happen together • Conditional Probability: p(x|y) the conditional probability that X takes the value x given that Y takes the value y. • The relationship: p(x,y) = p(x|y)p(y) Bayes rule • From the formula relating joint probability with conditional probability, we know p(x|y)p(y) = p(x,y) = p(y|x)p(x) So, p( x | y ) p( y ) p( y | x ) p( x ) This is the Bayes Rule. Using that we can express p(y|x) in terms of p(x|y). An example of Bayes reasoning We know rain wet ground; and wet ground rain (ground can be wet by pouring water) But usually we say it is raining when we see the ground is wet. Why do we have this prediction? An example of Bayes reasoning • Bayes rule: =1 (always true) P( rain ) P( rain | wet ground ) P( wet ground | rain ) P( wet ground ) • In fact – If P(wet ground)>>P(rain), for instance, somebody frequently pours water on the ground, we will not say it is raining when seeing the wet ground. – If P(wet ground) P(rain), we may conclude that it is raining when seeing the wet ground – This is exactly what Bayes rule tells us! Another example of Bayes reasoning • Given: – A doctor knows that meningitis causes stiff neck 50% of the time – Prior probability of any patient having meningitis is 1/50,000 – Prior probability of any patient having stiff neck is 1/20 • If a patient has stiff neck, what’s the probability he/she has meningitis? P( S | M ) P( M ) 0.5 1 / 50000 P( M | S ) 0.0002 P( S ) 1 / 20 Bayes Classifiers • Consider the attribute and the class label as random variables x and w • Given the attribute x – Goal is to predict class w – Specifically, we want to find value of i such that maximizes P(wi|x), say k, because given x, wk is most probably true • Can we estimate P(wi|x) directly from data? Bayes Classifiers • Approach: – Compute the posterior probability P(wi|x) for all values of wi using the Bayes rule prior posterior P( wi | x ) P( x | wi ) P( wi ) P( x ) – Choose value of i that maximized P(wi|x) – Equivalent to choosing value of i that maximizes P(x|wi)P(wi) because P(x) is a constant • P(x|wi) and P(wi) are given or can be easily estimate from data An Illustrating Example • • • • • In US, 0.8% of the people has this form of cancer When this disease is present, the test returns a correct POS result 98% of the times It returns a correct NE result 97% of the times when the disease is not present Suppose Anne comes to the doctor’s office and run a blood test and the test result is POS What is the likelihood that Anne has this form of cancer or not? – This is not good for Anne. After all, the test is 98% accurate An Illustrating Example • • • • • In US, 0.8% of the people has this form of cancer When this disease is present, the test returns a correct POS result 98% of the times It returns a correct NE result 97% of the times when the disease is not present Suppose Anne comes to the doctor’s office and run a blood test and the test result is POS What is the likelihood that Anne has this form of cancer or not? P(C | POS ) P( POS | C ) P(C ) 0.98 0.008 0.0078 P( POS ) P( POS ) P( POS ) P( NC | POS ) • P( POS | NC ) P( NC ) 0.03 0.92 0.0298 P( POS ) P( POS ) P( POS ) We can classify Anne is not having cancer. How tol Estimate Probabilities s al a u c c i i r r uo o o from Data? n s i g g t s e e t t n a l Tid 10 ca ca co Refund Marital Status Taxable Income Evade c 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married 60K No 7 Yes Divorced 220K No 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes • Class: P(C) = Nc/N – e.g., P(No) = 7/10, P(Yes) = 3/10 • For discrete attributes: P(Ai | Ck) = |Aik|/ Nc – where |Aik| is number of instances having attribute Ai and belongs to class Ck – Examples: P(Status=Married|No) = 4/7 P(Refund=Yes|Yes)=0 How to handle continuous attributes? • For continuous attributes: – Discretize the range into bins • one ordinal attribute per bin – Two-way split: (A < v) or (A > v) • choose only one of the two splits as new attribute – Example: • Age (<18, 18-22, 23-30,30-40,>40) • Annual Salary (>200,000, 120,000-200,000, 60,000-120,000, <60,000) How to handle continuous attributes? • For continuous attributes: – Probability density estimation: • Assume attribute follows a normal distribution • Use data to estimate parameters of distribution – Mean : sample mean x 1 xi d i – standard deviation: sample standard deviation sd 2 ( x x) i i d 1 • Once probability distribution is known, can use it to estimate the conditional probability P(Ai|c) How to handle continuous attributes? c Tid at Refund o g e l a ric c at o g e Marital Status l a ric co in t n Taxable Income u s u o as l c Evade 1 Yes Single 125K No 2 No Married 100K No 3 No Single 70K No 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married 60K No 7 Yes Divorced 220K No 8 No Single 85K Yes 9 No Married 75K No 10 No Single 90K Yes s • Normal distribution: 1 P( A | c ) e 2 i j ( Ai ij ) 2 2 ij2 2 ij – One for each (Ai,ci) pair • For (Income, Class=No): – If Class=No • sample mean = 110 • sample variance = 2975 10 1 P( Income 120 | No) e 2 (54.54) ( 120110 ) 2 2 ( 2975 ) 0.0072 Example of Naive Bayes Classifier Given a Test Record: X (Refund No, Married, Income 120K) naive Bayes Classifier: P(Refund=Yes|No) = 3/7 P(Refund=No|No) = 4/7 P(Refund=Yes|Yes) = 0 P(Refund=No|Yes) = 1 P(Marital Status=Single|No) = 2/7 P(Marital Status=Divorced|No)=1/7 P(Marital Status=Married|No) = 4/7 P(Marital Status=Single|Yes) = 2/7 P(Marital Status=Divorced|Yes)=1/7 P(Marital Status=Married|Yes) = 0 For taxable income: If class=No: sample mean=110 sample variance=2975 If class=Yes: sample mean=90 sample variance=25 P(X|Class=No) = P(Refund=No|Class=No) P(Married| Class=No) P(Income=120K| Class=No) = 4/7 4/7 0.0072 = 0.0024 P(X|Class=Yes) = P(Refund=No| Class=Yes) P(Married| Class=Yes) P(Income=120K| Class=Yes) = 1 0 1.2 10-9 = 0 Since P(X|No)P(No) > P(X|Yes)P(Yes) Therefore P(No|X) > P(Yes|X) => Class = No Bayes Classifiers • Robust to isolated noise points • Handle missing values by ignoring the instance during probability estimate calculations • Robust to irrelevant attributes • Bayes Classification Decision boundary Quadratic Discriminant Classifier • Using the Bayes classifier, assign x to class j: j arg max P(wi | x) arg max P( x | wi ) P(wi ) i i • Assume data points in each class follow the multivariate Gaussian distribution, • Then the Bayes classifier is reduced to assign x to class j: j arg max d i ( x ), i n 1 1 1 d i ( x ) ln P( wi ) ln 2 ln | Ci | ( x mi )T Ci ( x mi ) 2 2 2 • di(x) is a quadratic function of x, which is also called quadratic discriminant functions (QDA) Demo of decision boundaries • Consider a simple case – x is two-dimensional – Ci is diagonal – with two classes • Then the decision boundary is a quadratic equation of x1 and x2, and the coefficient of x1x2 is zero Demo of decision boundaries Linear Discriminant Classifier • If we assume that the covariance matrix is the same for all classes, then 1 2 1 2 k ( x) log k ( x k )T k 1 ( x k ) log k 1 T 1 k ( x) x k k k log k 2 T 1 • The decision boundary function can be defined as follows. Linear Discriminant Classifier • In practice the parameters of the Gaussian distribution can be estimated using the training data, ˆ N k / N where N k is the number of instance in class k uˆk x / N gi k i K k ˆ ( xi ˆ k )( xi ˆ k )T /( N K ) k 1 g i k Minimum Distance Classifier • Compute the mean vector i of each class i • Given a new instance x, calculate the Euclidean distance between x and each mean vector d x i ( x i )T ( x i ) Minimum Distance Classifier • Can be considered as a special case of linear discriminant classifier when the covariance matrix 1 2 , and the prior probability of each class is the same. 1 T 1 k ( x) x k k k log k 2 T 1 1 T k ( x) x k k k 2 linear discriminant function T MDC Decision function Minimum error rate classifiers • Given item x, the probability that item x belongs to class wi is P(wi|x). • If item x is wrongly classified as class wj (it actually belongs to class wk), then the loss incurs is Lkj Minimum error rate classifiers • Assign x to class j which minimizes the conditional average risk (loss): M j arg min ri ( x ) arg min Lki P( wk | x ) i i k 1 • We can see the expected risk is minimized • If , we have r i (x) = 1-P(wi|x) • Then minimizing ri (x) is equivalent to Bayes classifier Overview of the classification methods Minimum Error Rate Classifiers Bayes Classifiers Gaussian distribution Hyperquadrics as the Decision boundaries The same Ci for all i Linear discriminant classifiers Minimum distance classifiers Ci = σ2I, and P(wi)=P(wj) Instance-Based Classifiers Set of Stored Cases Atr1 ……... AtrN Class A • Store the training records • Use training records to predict the class label of unseen cases B B C A C B Unseen Case Atr1 ……... AtrN Instance Based Classifiers • Examples: – Rote-learner • Memorizes entire training data and performs classification only if attributes of record match one of the training examples exactly – Nearest neighbor • Uses k “closest” points (nearest neighbors) for performing classification Nearest-Neighbor Classifiers • Basic idea: – If it walks like a duck, quacks like a duck, then it’s probably a duck Compute Distance Training Records Choose k of the “nearest” records Test Record Nearest-Neighbor Classifiers Unknown record • Requires three things – The set of stored records – Distance Metric to compute distance between records – The value of k, the number of nearest neighbors to retrieve • To classify an unknown record: – Compute distance to other training records – Identify k nearest neighbors – Use class labels of nearest neighbors to determine the class label of unknown record (e.g., by taking majority vote) Definition of Nearest Neighbor X (a) 1-nearest neighbor X X (b) 2-nearest neighbor (c) 3-nearest neighbor K-nearest neighbors of a record x are data points that have the k smallest distance to x 1 nearest-neighbor Voronoi Diagram Nearest Neighbor Classification • Compute distance between two points: – Euclidean distance d ( p, q ) n 2 ( p q ) i i i 1 • Determine the class from nearest neighbor list – take the majority vote of class labels among the knearest neighbors – Weigh the vote according to distance • weight factor, w = 1/d2 Nearest Neighbor Classification • Choosing the value of k: – If k is too small, sensitive to noise points – If k is too large, neighborhood may include points from other classes X Nearest Neighbor Classification • Scaling issues – Attributes may have to be scaled to prevent distance measures from being dominated by one of the attributes – Example: • height of a person may vary from 1.5m to 1.8m • weight of a person may vary from 90lb to 300lb • income of a person may vary from $10K to $1M Nearest Neighbor Classification • Problem with Euclidean measure: – High dimensional data • curse of dimensionality – Can produce counter-intuitive results 111111111110 100000000000 vs 011111111111 000000000001 d = 1.4142 • Solutions: • • Normalize the vectors to unit length Cosine similarity d = 1.4142 Example: PEBLS • PEBLS: Parallel Examplar-Based Learning System (Cost & Salzberg) – Works with both continuous and nominal features • For nominal features, distance between two nominal values is computed using modified value difference metric (MVDM) – Each record is assigned a weight factor – Number of nearest neighbor, k = 1 Example: PEBLS Tid Refund Marital Status Taxable Income Cheat 1 Yes Single 125K No d(Single,Married) 2 No Married 100K No = | 2/4 – 0/4 | + | 2/4 – 4/4 | = 1 3 No Single 70K No d(Single,Divorced) 4 Yes Married 120K No 5 No Divorced 95K Yes 6 No Married No d(Married,Divorced) 7 Yes Divorced 220K No = | 0/4 – 1/2 | + | 4/4 – 1/2 | = 1 8 No Single 85K Yes d(Refund=Yes,Refund=No) 9 No Married 75K No = | 0/3 – 3/7 | + | 3/3 – 4/7 | = 6/7 10 No Single 90K Yes 60K Distance between nominal attribute values: = | 2/4 – 1/2 | + | 2/4 – 1/2 | = 0 10 Marital Status Class Refund Single Married Divorced Yes 2 0 1 No 2 4 1 Class Yes No Yes 0 3 No 3 4 d (V1 ,V2 ) i n1i n2i n1 n2 Example: PEBLS Tid Refund Marital Status Taxable Income Cheat X Yes Single 125K No Y No Married 100K No 10 Distance between record X and record Y: d ( X , Y ) wX wY d ( X i , Yi )2 i 1 where: Number of times X is used for prediction wX Number of times X predicts correctly wX 1 if X makes accurate prediction most of the time wX > 1 if X is not reliable for making predictions Collaborative Filtering • Making recommendation based on nearest neighbor approach. • Basic Idea: to recommend a music to Anne, find the person A that are the most similar to Anne and recommend to Anne the music that person A is listening. • We use Pearson Correlation Coefficient to compute the similarity (the opposite of distance) between them. Collaborative Filtering • Suppose Amanda, Eric and Sally rated the band, the Grey Warden, and their similarity with Anne is as follows, Person Grey Wardens Rating Amanda 4.5 Eric 5 Sally 3.5 Person Pearson Sally 0.8 Eric 0.7 Amanda 0.5 Collaborative Filtering Person Grey Wardens Rating Influence Amanda 4.5 25% Eric 5 35% Sally 3.5 40% • K = 3: the projected rating for Anne= (4.5*0.25)+(5*0.35)+(3.5*0.4)=4.275 • K=2: the projected rating for Anne = (3.5*0.8/1.5)+(5*0.7/1.5)=4.2 Nearest neighbor Classification • k-NN classifiers are lazy learners – It does not build models explicitly – Unlike eager learners such as decision tree induction and rule-based systems – Classifying unknown records are relatively expensive Classification methods • The methods are related – – – – – Minimum distance classifier Linear discriminant classifier Quadratic discriminant classifier Bayes classifier Minimum error rate classifier • The methods based on the statistical property of the data set Automated Negotiation • A negotiation consists of – A number of agents (agent space) – A negotiation domain D (outcome space) D = {I1, I2, … In} and each issue consists of k values Ii = {v1,v2,…vk} – A number of utility space (preference profiles) • A Laptop negotiation domain – – – – Two negotiation agents: agent A and agent B Three issues: brand, hard disk, monitor Each issue contains a number of discrete values One bidding instance is a Dell laptop with 80 Gb and 17’inch monitor. Automated Negotiation • Utility Space – Specify a preference of each agent for each outcome Pareto optimal bids • The optimal bid of an agent is the bid that gives the maximum utility to that agent • The utilities of negotiation agents are often contradictory, i.e., one agent’s gain is another agent’s pain Automated Negotiation • Optimal goal of a negotiation – Maximizing individual payoff – Maximizing social welfare (the sum of the payoffs of all partied involved in the negotiation) Automated Negotiation • Negotiation protocol – Defines the rules to regulate how the negotiation proceeds between negotiation agents. – Agents are obliged to follow the protocol, and any deviation from the protocol will be penalized. • Negotiation Strategy – Specify how an agent should behave during a negotiation under the regulation of a negotiation protocol. Negotiation Protocol • Bilateral Negotiation Protocol (Alternating Offer Protocol) – – – – Involves two parties- agent A and B One agent (e.g., agent A) starts the negotiation first Each agent takes turn to negotiate At its turn, each agent is allowed to present one of the following three options • Accept – accept the current proposal from the negotiation partner • Offer – propose a new offer to the partner • EndNegotiation – choose to terminate the negotiation without reaching an agreement • Reservation Value – The value that an agent obtains if no agreement is reached by the end of negotiation or the negotiation terminates earlier • Time pressure – The utility decreases with the passing of negotiation time Overall Structure of a Negotiation Strategy “Decoupling Negotiating Agents to Explore the Space of Negotiation Strategies”, Novel Insights in Agent-based Complex Automated Negotiation,2014 Design Effective Negotiation Strategies using Mining Techniques • Benefits of predicting the opponent’s behavior – Cooperative environments ->better coordination with others – Competitive environments -> opportunity of taking exploitative actions to maximize its own payoff. • How to predict the opponent’s behavior? – Regression techniques (non-linear regression, Guassian process regression) predicting the opponent’s concession degree [Using gaussian processes to optimise concession in complex negotiations against unknown opponents, IJCAI, 2011] – Chebychev Polynomials – predict the opponent’s decision function [Modeling opponent decision in repeated one-shot negotiations. Proceedings of the fourth international joint conference on Autonomous agents and multiagent systems. ACM, 2005: 397-403.] – Neural Network [Predicting Opponent’s Moves in Electronic Negotiations Using Neural Networks, Group Decision and Negotiation Conference,2006] Design Effective Negotiation Strategies using Mining Techniques • Benefits of predicting the opponent’s preference – Increase the chance of reaching win-win negotiation outcomes – Better understanding of the opponent’s behaviors • How to predict the opponent’s preference? – Bayesian Learning - Predict the opponent’s issue weight and the evaluation function [Opponent modelling in automated multiissue negotiation using bayesian learning, AAMAS, 2008] – Bayesian Learning – predict the opponent’s reservation value [Sycara K, Zeng D. Benefits of learning in negotiation, AAAI. 1997.] Bayesian Learning in Negotiation • A Buyer and a Seller • Task: learn the reservation value of the seller using Bayesian learning • Two hypotheses on the seller’s reservation value: H1 = 100, H2 =130 • Prior knowledge of the buyer: P(H1) = P(H2) = 0.5 • Domain Knowledge of the buyer: a seller usually offers a price which is above their reservation price by 17% – P(152|H2) = 0.95, P(152|H1) = 0.75 • Given a new price offered by the seller, apply Bayesian learning P ( H 1 | e) P(e | H 1) P( H 1) P(e | H 1) P( H 1) P (e) P(e | H 1) P( H 1) P(e | H 2) P( H 2) P ( H 2 | e) P(e | H 2) P( H 2) P(e | H 2) P( H 2) P (e) P(e | H 1) P( H 1) P(e | H 2) P( H 2) Select Effective Negotiation Strategies using Mining Techniques • Given a new domain, predicting which existing negotiation strategy performs best? – Artificial Neural Network – Decision Tree – Linear/Logistic Regression Basic Procedures • • • • Extract the features of a domain Build the training set Choose a data mining algorithm and train the model Apply the model to a testing set and evaluate the performance of the prediction results. Discounting factor Avg No. of values in each issue Average utility overall all proposals No. of Issue Domain size Class 0.3 0.3 0.1 3 1000 Strategy 1 0.8 0.45 0.35 10 2888 Strategy 2 0.1 0.32 0.56 21 33 Strategy 1 0.2 0.76 0.8 4 200 Strategy 3 0.3 0.678 0.95 8 34 Strategy 5 Project Introduction Genius Negotiation Platform A negotiation environment for heterogeneous negotiation agents A set of negotiation problems (domains) A set of negotiation agents (strategies) A set of analytical tools to evaluate an agent’s performance Webpage: http://ii.tudelft.nl/genius/ User manual: https://ii.tudelft.nl/genius/sites/default/files/userguide.pdf Resources: http://mmi.tudelft.nl/negotiation/index.php/Automated_Ne gotiating_Agents_Competition_(ANAC) A Remainder • Start earlier ! • Send the group information to [email protected] ASAP Exercise II • Using Bayes classier and k-NN to classify the following task and compare their performance. • Pima Indians Diabetes – – – – – – – – – the number of times pregnant, plasma glucose concentration, blood pressure, triceps skin fold thickness, serum insulin level, body mass index, diabetes pedigree function, age, Class label: ‘1’-diabetes; ‘0’ - not.