Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Document
Document related concepts
Transcript
Data Preprocessing Classification & Regression Supervised Learning Regression Given the value of an input π, the output π belongs to the set of real values π . The goal is to predict output accurately for a new input. Classification The predictions or outputs π¦ are categorical, while π₯ can take any set of values (real or categorical). The goal is to select the correct class for a new instance. 1 Data Preprocessing Classification & Regression Classification The task of assigning objects to one of several predefined categories. 2 Data Preprocessing Classification & Regression Classification Feature set (π) 3 Classification Model Class label (π¦) Data Preprocessing Classification & Regression Definition of Classification β’ The input data for a classification task is a collection of records, each known as an instance, composed of a feature set and a class label. β’ Classification, then, is the task of learning a target function π that maps each feature set π₯ to one of the predefined class labels π¦. β’ The target function is also known informally as the classification model. A classification model is useful 4 for several purposes. Data Preprocessing Classification & Regression Types of Classification Models Two purposes of classification models: 5 1 Descriptive Modeling 2 Predictive Modeling A classification model can serve as an explanatory tool to distinguish between objects of different classes. A classification model can also be used to predict the class label of unknown records. Data Preprocessing Classification & Regression Classification Models: An Example Input Features: X Instances 6 Class: Y Make Cylinders Length Weight Style Honda Four 150 1956 Hatchback Toyota Four 167.9 2280 Wagon BMW Six 176.8 2765 Sedan Descriptive modeling: Explain what features define a hatchback, wagon, or sedan. Predictive modeling: Given another car, determine the style to which it belongs. Data Preprocessing Classification & Regression Building a Classification Model β’ A classification technique (or classifier) is a systematic approach to building classification models from an input dataset. β’ Each technique employs a learning algorithm to identify a model that best fits the relationship between the features and class of the input data. β’ A key objective of the learning algorithm is to build models with good generalization capability. 7 Data Preprocessing Classification & Regression Building a Classification Model Training Set Learning Algorithm Learn Model Model Data Apply Model Test Set 8 Types of Learners β’ Eager learners learn a model that maps the input features to the class label a soon as the training data becomes available. β’ Lazy learners delay modeling the training data until it is needed to classify the test examples. β An example of a lazy leaner is a rote classifier, which memorizes the entire training data and performs classification only if a test instance matches one of the training examples exactly. 9 Data Preprocessing Classification & Regression Rote Learning 10 Make Cylinders Length Weight Style Honda Four 150 1956 Hatchback Toyota Four 167.9 2280 Wagon BMW Six 176.8 2765 Sedan BMW Six 176.8 2765 Sedan Nearest Neighbor Classifier The Idea: Find which training data is closest to the test instance, and classify the test instance as that class. Problems: β Computationally expensive. β Not interpretable. β Very sensitive to noise/outliers. 11 Data Preprocessing Classification & Regression Nearest Neighbor Learning 12 Make Cylinders Length Weight Style Honda Four 150 1956 Hatchback Toyota Four 167.9 2280 Wagon BMW Six 176.8 2765 Sedan BMW Six 174.2 2456 Sedan π-Nearest Neighbor (kNN) Classifier The Idea: Find the π training instances that are closest to the test instance, and classify the test instance as the majority class of the k nearest training instances. βIf it walks like a duck, quacks like a duck, and looks like a duck, then itβs probably a duck.β 13 Data Preprocessing Classification & Regression π-Nearest Neighbor (kNN) Classifier Training New Instance 14 ? Classification Data Preprocessing Classification & Regression The kNN Algorithm The π-nearest neighbor classification algorithm. 1: Let π be the number of nearest neighbors and π· be the set of training examples. 2: for each test example π§ = π± β² , π¦ β² do 3: Compute π π± β² , π± , the distance between π§ and every example, π±, π¦ β π·. 4: 5: Select π·π§ β π·, the set of π closest training examples to π§. π¦ β² = argmax π₯π ,π¦π βπ·π§ πΌ π£ = π¦π 6: end for 15 π£ Data Preprocessing Classification & Regression Nearest Neighbors 1-nearest neighbor 2-nearest neighbor 3-nearest neighbor The π nearest neighbors of an example π₯ are the data points that have the π smallest distances to π₯. 16 Data Preprocessing Classification & Regression Choosing the k for kNN β’ If π is too small, then the nearest-neighbor classifier may be susceptible to overfitting because of noise in the training data. β’ If π is too large, the nearest-neighbor classifier may misclassify the test instance because its list of nearest neighbors may include data points that are located far away from its neighborhood. 17 Data Preprocessing Classification & Regression Choosing the k for kNN π-nearest neighbor classification with small π. 18 π-nearest neighbor classification with large π. Data Preprocessing Classification & Regression kNN Prediction: Classification Classification: Take the majority vote of class labels among the k-nearest neighbors: Majority voting: π¦ = argmax π£ πΌ π£ = π¦π π± π ,π¦π βπ·π§ where π£ is a class label, π¦π is the class label for one of the nearest neighbors, and πΌ β is an indicator function that returns the value 1 if its argument is true and 0 otherwise. 19 Data Preprocessing Classification & Regression kNN Prediction: Regression Regression: Predict the average value of the class value of the k-nearest neighbors. Average value: π¦= 1 π±π , π¦π β π·π§ π¦π π± π ,π¦π βπ·π§ where π£ is a class label and π¦π is the class label for one of the nearest neighbors. 20 Data Preprocessing Classification & Regression kNN Prediction: Distance-Weighted β’ In the majority voting approach, every neighbor has the same impact on the classification. β’ Instead, the influence of each nearest neighbor π₯π can be weighted according to distance: π€π = 1 π π±β², π±π 2 . β’ Then, the class label can be determined as follows: π¦ = argmax π£ 21 π€π × πΌ π£ = π¦π . π± π ,π¦π βπ·π§ Data Preprocessing Classification & Regression kNN Efficiency β’ Very efficient in model induction (training) β Only store the training data. β’ Not particularly efficient in testing β Computation of distance measure to every training instance β’ Note that 1NN and πNN are equally efficient β Retrieving the π nearest neighbors is (almost) no more expensive than retrieving a single nearest neighbor β π nearest neighbors can be maintained in a queue. 22 Data Preprocessing Classification & Regression kNN Summary β’ Nearest-neighbor classification is part of a more general technique known as instance-based learning. β’ Lazy learners such as nearest-neighbor classifiers do not require model building. β’ Generate their predictions based on local information. β’ Can produce arbitrarily shaped decision boundaries. β’ Can easily produce wrong predictions without appropriate data preprocessing. 23 Data Preprocessing Classification & Regression Back to Building a Classification Model Learning Algorithm Training Set Learn Model Model Data Apply Model Test Set 24 For Nearest Neighbors Classification, the model is the training set itself. Data Preprocessing Classification & Regression Letβs see how a model that actually learns works. 25 Components of Learning β’ Suppose that a bank wants to automate the process of evaluating credit card applications. β Input π₯ (customer information that is used to make a credit application). β Target function π: π β π (ideal formula for credit approval), where π and π are the input and output space, respectively. β Dataset π· of input-output examples π₯1 , π¦1 , β¦ , π₯π , π¦π . β Hypothesis (skill) with hopefully good performance: π: π β π (βlearnedβ formula to be used) 26 Data Preprocessing Classification & Regression Components of Learning Unknown Target Function π: π β π (ideal credit approval formula) Training Examples π₯1 , π¦1 , β¦ , π₯π , π¦π (historical records of credit customers) Hypothesis Set π» Learning Algorithm π΄ Final Hypothesis πβπ (learned credit approval formula) (set of candidate formulas) Use data to compute hypothesis π that approximates target π 27 Simple Learning Model: The Perceptron For π± = π₯1 , β¦ , π₯π (βfeatures of the customerβ), compute a weighted score and: π Approve credit if π€π π₯π > threshold , π=1 π Deny credit if π€π π₯π < threshold . π=1 28 Simple Learning Model: The Perceptron This formula can be written more compactly as π π π± = sign π€π π₯π β threshold , π=1 where π π± = +1 means βapprove creditβ and π π± = β 1 means βdeny creditβ; sign π = +1 if π > 0 and sign π = β1 if π < 0. This model is called a perceptron. 29 Data Preprocessing Classification & Regression Simple Learning Model: The Perceptron 30 π₯1 π₯2 π₯3 π¦ 1 0 0 -1 1 0 1 1 1 1 0 1 1 1 1 1 0 0 1 -1 0 1 0 -1 0 1 1 1 0 0 0 -1 Input Nodes π₯1 π₯2 π₯3 0.3 0.3 Output Node Ξ£ 0.3 π‘ = 0.4 π¦ 2 Perceptrons (in β ) π π± = π πππ π€0 + π€1 π₯1 + π€2 π₯2 The perceptron is a linear (binary) classifier: 31 β’ Customer features x: points on the plane β’ Labels π¦: (+1), (-1) β’ Hypothesis π: line (divide positive and negative) Data Preprocessing Classification & Regression And nowβ¦ Letβs see some classifying! 32
