Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
26.10.2015 Getting to Know Your Data • Data Objects and Attribute Types CENG 464 Introduction to Data Mining • Basic Statistical Descriptions of Data • Data Visualization • Measuring Data Similarity and Dissimilarity • Summary Data Visualization • • Why data visualization? – Gain insight into an information space by mapping data onto graphical primitives – Provide qualitative overview of large data sets – Search for patterns, trends, structure, irregularities, relationships among data – Help find interesting regions and suitable parameters for further quantitative analysis – Provide a visual proof of computer representations derived Categorization of visualization methods: – Pixel-oriented visualization techniques – Geometric projection visualization techniques – Icon-based visualization techniques – Hierarchical visualization techniques – Visualizing complex data and relations Pixel-Oriented Visualization Techniques • For a data set of m dimensions, create m windows on the screen, one for each dimension • The m dimension values of a record are mapped to m pixels at the corresponding positions in the windows • The colors of the pixels reflect the corresponding values (a) Income (b) Credit Limit (c) transaction volume (d) age 1 26.10.2015 Scatterplot Matrices Getting to Know Your Data Used by ermission of M. Ward, Worcester Polytechnic Institute • Data Objects and Attribute Types • Basic Statistical Descriptions of Data • Data Visualization • Measuring Data Similarity and Dissimilarity • Summary Matrix of scatterplots (x-y-diagrams) of the k-dim. data [total of (k2/2-k) scatterplots] Similarity and Dissimilarity • Similarity – Numerical measure of how alike two data objects are – Value is higher when objects are more alike Data Matrix and Dissimilarity Matrix • Data matrix – n data points with p dimensions – Two modesrows&columns – Often falls in the range [0,1] • Dissimilarity (e.g., distance) – Numerical measure of how different two data objects are – Lower when objects are more alike – Minimum dissimilarity is often 0 – Upper limit varies • Proximity refers to a similarity or dissimilarity • Dissimilarity matrix – n data points, but registers only the distance – A triangular matrix – d(i,j) is the distance between objects i and j – Nonzero value x11 ... x i1 ... x n1 x1p ... ... x ip ... ... ... x np ... x1f ... ... ... ... ... x if ... ... ... x nf 0 d(2,1) 0 d(3,1) d ( 3,2) 0 : : : d ( n,1) d ( n,2) ... ... 0 2 26.10.2015 Proximity Measure for Binary Attributes Proximity Measure for Nominal Attributes • Can take 2 or more states, e.g., red, yellow, blue, green (generalization of a binary attribute) • Method 1: Simple matching – m: # of matches, p: total # of variables m d (i, j) p p • Method 2: Use a large number of binary attributes – creating a new binary attribute for each of the M nominal states Dissimilarity between Binary Variables Object j • Example Name Jack Mary Jim Fever Y Y Y Cough N N P Test-1 P P N Test-2 N N N Test-3 N P N q, r, s, t refer to total # of attributes Object i • Distance measure for symmetric binary variables: • Distance measure for asymmetric binary variables: #of negative matches considered unimportant • Jaccard coefficient (similarity measure for asymmetric binary variables): Note: Jaccard coefficient is the same as “coherence”: Dissimilarity between Binary Variables Name Jack Mary Jim Test-4 N N N – Gender is a symmetric attribute – The remaining attributes are asymmetric binary – Let the values Y and P be 1, and the value N 0 01 0.33 2 01 11 d ( jack , jim ) 0.67 111 1 2 d ( jim , mary ) 0.75 11 2 d ( jack , mary ) A contingency table for binary data: • Example Object i Gender M F M Object j • – – – – Gender M F M Fever Y Y Y Cough N N P Test-1 P P N Test-2 N N N Test-3 N P N Test-4 N N N Gender is a symmetric attribute The remaining attributes are asymmetric binary Let the values Y and P be 1, and the value N 0 Dissimilarity based on asymmetric attributes: 01 0.33 2 01 11 0.67 111 1 2 d ( jim , mary ) 0.75 11 2 d ( jack , mary ) d ( jack , jim ) 3 26.10.2015 Distance on Numeric Data: Minkowski Distance Special Cases of Minkowski Distance • Minkowski distance: A popular distance measure • h = 1: Manhattan (city block, L1 norm) distance – E.g., the Hamming distance: the number of bits that are different between two binary vectors d (i, j) | x x | | x x | ... | x x | i1 j1 i2 j 2 ip jp where i = (xi1, xi2, …, xip) and j = (xj1, xj2, …, xjp) are two pdimensional data objects, and h is the order (the distance so defined is also called L-h norm) • h = 2: (L2 norm) Euclidean distance d (i, j) (| x x |2 | x x |2 ... | x x |2 ) i1 j1 i2 j 2 ip jp • Properties • – d(i, j) > 0 if i ≠ j, and d(i, i) = 0 (Positive definiteness) – d(i, j) = d(j, i) (Symmetry) h . “supremum” (Lmax norm, L norm) distance. – This is the maximum difference between any component (attribute) of the vectors – d(i, j) d(i, k) + d(k, j) (Triangle Inequality) • A distance that satisfies these properties is a metric Example: Data Matrix and Dissimilarity Matrix Example: Minkowski Distance Dissimilarity Matrices point x1 x2 x3 x4 Data Matrix point x1 x2 x3 x4 attribute1 attribute2 1 2 3 5 2 0 4 5 x2 0 3.61 2.24 4.24 x3 0 5.1 1 x1 0 5 3 6 x2 x3 x4 0 6 1 0 7 0 x1 x3 x4 x2 0 3.61 2.24 4.24 0 5.1 1 0 5.39 0 Supremum x4 0 5.39 L x1 x2 x3 x4 L2 x1 x2 x3 x4 Dissimilarity Matrix x1 Manhattan (L1) Euclidean (L2) (with Euclidean Distance) x1 x2 x3 x4 attribute 1 attribute 2 1 2 3 5 2 0 4 5 0 L x1 x2 x3 x4 x1 x2 0 3 2 3 x3 0 5 1 x4 0 5 0 4 26.10.2015 Attributes of Mixed Type Ordinal Variables • A database may contain all attribute types – Nominal, symmetric binary, asymmetric binary, numeric, ordinal • One may use a weighted formula to combine their effects • An ordinal variable can be discrete or continuous • Order is important, e.g., rank • Can be treated like interval-scaled – replace xif by their rank rif {1,..., M f } d (i, j) – map the range of each variable onto [0, 1] by replacing i-th object in the f-th variable by zif pf 1 ij( f )dij( f ) pf 1 ij( f ) – =0 if xif or xjf is missing OR Xif or xjf =0 OR f is asymmetric – f is binary or nominal: dij(f) = 0 if xif = xjf , or dij(f) = 1 otherwise – f is numeric: use the normalized distance – f is ordinal zif r 1 • Compute ranks rif and M 1 • Treat zif as interval-scaled rif 1 M f 1 – compute the dissimilarity using methods for interval-scaled variables if f Cosine Similarity • • • • A document can be represented by thousands of attributes, each recording the frequency of a particular word (such as keywords) or phrase in the document. Other vector objects: gene features in micro-arrays, … Applications: information retrieval, biologic taxonomy, gene feature mapping, ... Cosine measure: If d1 and d2 are two vectors (e.g., term-frequency vectors), then cos(d1, d2) = (d1 d2) /||d1|| ||d2|| , where indicates vector dot product, ||d||: the length of vector d Example: Cosine Similarity • cos(d1, d2) = (d1 d2) /||d1|| ||d2|| , where indicates vector dot product, ||d|: the length of vector d • Ex: Find the similarity between documents 1 and 2. d1 = (5, 0, 3, 0, 2, 0, 0, 2, 0, 0) d2 = (3, 0, 2, 0, 1, 1, 0, 1, 0, 1) d1d2 = 5*3+0*0+3*2+0*0+2*1+0*1+0*1+2*1+0*0+0*1 = 25 ||d1||= (5*5+0*0+3*3+0*0+2*2+0*0+0*0+2*2+0*0+0*0)0.5=(42)0.5 = 6.481 ||d2||= (3*3+0*0+2*2+0*0+1*1+1*1+0*0+1*1+0*0+1*1)0.5=(17)0.5 = 4.12 cos(d1, d2 ) = 0.94 5 26.10.2015 Summary • Data attribute types: nominal, binary, ordinal, interval-scaled, ratio-scaled • Many types of data sets, e.g., numerical, text, graph, Web, image. • Gain insight into the data by: – Basic statistical data description: central tendency, dispersion, graphical displays – Data visualization: map data onto graphical primitives – Measure data similarity • Above steps are the beginning of data preprocessing • Many methods have been developed but still an active area of research Data Preprocessing • Data Preprocessing: An Overview – Data Quality – Major Tasks in Data Preprocessing • Data Cleaning • Data Integration • Data Reduction • Data Transformation and Data Discretization • Summary Issues of Data Quality • Why is quality important? – “Garbage in, garbage out!” Quality decisions must be based on quality data – For data mining, tackling the quality issue at the data source cannot be always expected • By cleaning the data as much as possible • By developing and using more tolerate mining solutions – Data quality is relevant to the intended purpose of data mining, e.g. Do spelling errors in student names really matter when only the increase/decrease of student numbers in particular subject areas over the years is of interest? Why Is Data Dirty? • Incomplete data may come from – “Not applicable” data value when collected – Different considerations between the time when the data was collected and when it is analyzed. – Human/hardware/software problems • Noisy data (incorrect values) may come from – Faulty data collection instruments – Human or computer error at data entry – Errors in data transmission • Inconsistent data may come from – Different data sources – Functional dependency violation (e.g., modify some linked data) • Duplicate records 6 26.10.2015 Measures for data quality • Measures for data quality: A multidimensional view – Accuracy: correct or wrong, accurate or not – Completeness: not recorded, unavailable, … – Consistency: some modified but some not, dangling, … – Timeliness: timely update? – Believability: how trustable the data are correct? – Interpretability: how easily the data can be understood? Major Tasks in Data Preprocessing • Data cleaning – Fill in missing values, smooth noisy data, identify or remove outliers, and resolve inconsistencies • Data integration – Integration of multiple databases, data cubes, or files • Data reduction – Dimensionality reduction – Numerosity reduction – Data compression • Data transformation and data discretization – Normalization – Concept hierarchy generation Data Cleaning Data Preprocessing • Data Preprocessing: An Overview – Data Quality – Major Tasks in Data Preprocessing • Data Cleaning • Data Integration • Data Reduction • Data Transformation and Data Discretization • Data in the Real World Is Dirty: Lots of potentially incorrect data, e.g., instrument faulty, human or computer error, transmission error – incomplete: lacking attribute values, lacking certain attributes of interest, or containing only aggregate data • e.g., Occupation = “ ” (missing data) – noisy: containing noise, errors, or outliers • e.g., Salary = “−10” (an error) – inconsistent: containing discrepancies in codes or names, e.g., • Age = “42”, Birthday = “03/07/2010” • Was rating “1, 2, 3”, now rating “A, B, C” • discrepancy between duplicate records – Intentional (e.g., disguised missing data) • Jan. 1 as everyone’s birthday? • Summary 7 26.10.2015 Incomplete (Missing) Data Missing values - example • Data is not always available – E.g., many tuples have no recorded value for several attributes, such as customer income in sales data • Missing data may be due to – equipment malfunction – inconsistent with other recorded data and thus deleted – data not entered due to misunderstanding – certain data may not be considered important at the time of entry – not register history or changes of the data Hospital Check-in Database • Value may be missing because it is unrecorded or because it is inapplicable • In medical data, value for Pregnant? attribute for Jane is missing, while for Joe or Anna should be considered Not applicable • Some programs can infer missing values Name Age Sex Pregnant? .. Mary 25 F N Jane 27 F - Joe 30 M - Anna 2 F - • Missing data may need to be inferred 30 How to Handle Missing Data? • Ignore the tuple: usually done when class label is missing (when doing classification)—not effective when the % of missing values per attribute varies considerably • Fill in the missing value manually: tedious + infeasible? • Fill in it automatically with – a global constant : e.g., “unknown”, a new class?! – the attribute mean – the attribute mean for all samples belonging to the same class: smarter Noisy Data • Noise: random error or variance in a measured variable • Incorrect attribute values may be due to – faulty data collection instruments – data entry problems – data transmission problems – technology limitation – inconsistency in naming convention – the most probable value: inference-based such as Bayesian formula or decision tree 8 26.10.2015 Noise: example Examples: distortion of a person’s voice when talking on a poor phone and “snow” on television screen Two Sine Waves How to Handle Noisy Data? • Binning: smooth data value by consulting its neighborhood – first sort data and partition into (equal-frequency) bins – then one can smooth by bin means, smooth by bin median, smooth by bin boundaries, etc. • Regression – smooth by fitting the data into regression functions • Clustering-Outlier analysis – detect and remove outliers • Combined computer and human inspection – detect suspicious values and check by human (e.g., deal with possible outliers) Two Sine Waves + Noise 34 Outliers • Outliers are data objects with characteristics that are considerably different than most of the other data objects in the data set • Data cleaning – Smoothing outliers Duplicate Data • Data set may include data objects that are duplicates, or almost duplicates of one another – Major issue when merging data from heterogonous sources • Examples: – Same person with multiple email addresses 9 26.10.2015 Data Cleaning as a Process Forms of Data Preprocessing • • • Data discrepancy detection – Use metadata (e.g., domain, range, dependency, distribution) – Check field overloading – Check uniqueness rule, consecutive rule and null rule – Use commercial tools • Data scrubbing: use simple domain knowledge (e.g., postal code, spell-check) to detect errors and make corrections • Data auditing: by analyzing data to discover rules and relationship to detect violators (e.g., correlation and clustering to find outliers) Data migration and integration – Data migration tools: allow transformations to be specified – ETL (Extraction/Transformation/Loading) tools: allow users to specify transformations through a graphical user interface Integration of the two processes – Iterative and interactive (e.g., Potter’s Wheels) discrepancy detection and transformation 37 38 Data Integration Data Preprocessing • Data Preprocessing: An Overview • Data integration: • Schema integration: e.g., A.cust-id B.cust-# • Entity identification problem: – Combines data from multiple sources into a coherent store – Data Quality – Integrate metadata from different sources – Major Tasks in Data Preprocessing • Data Cleaning – Identify real world entities from multiple data sources, e.g., Bill Clinton = William Clinton • Data Integration • • Data Reduction Detecting and resolving data value conflicts – For the same real world entity, attribute values from different sources are different • Data Transformation and Data Discretization – Possible reasons: different representations, different scales, e.g., metric vs. British units • Summary 39 39 40 40 10 26.10.2015 Handling Redundancy in Data Integration Correlation Analysis (Nominal Data) • • Redundant data occur often when integration of multiple databases Χ2 (chi-square) test: given two attributes meaures how strongly one attribute implies the other 2 – Object identification: The same attribute or object may have different names in different databases – Derivable data: One attribute may be a “derived” attribute in another table, e.g., annual revenue • Redundant attributes may be able to be detected by correlation analysis and covariance analysis (Observed freq . Expected freq ) 2 Expected freq • The larger the Χ2 value, the more likely the variables are related • Expected freq= (ai*bj)/total • The cells that contribute the most to the Χ2 value are those whose actual count is very different from the expected count • Correlation does not imply causality – # of hospitals and # of car-theft in a city are correlated • Careful integration of the data from multiple sources may help reduce/avoid redundancies and inconsistencies and improve mining speed and quality – Both are causally linked to the third variable: population 41 41 42 Chi-Square Calculation: An Example male female Sum (row) Like science fiction 250(90) 200(360) 450 Not like science fiction 50(210) 1000(840) 1050 Sum(col.) 300 1200 1500 Correlation Analysis (Numeric Data) • Correlation coefficient (also called Pearson’s product moment coefficient): tells also about the degree of correlation n • Are gender and preferred_reading correlated? • Χ2 (chi-square) calculation (numbers in parenthesis are expected counts calculated based on the data distribution in the two categories) 2 rA, B (250 90) 2 (50 210) 2 (200 360) 2 (1000 840) 2 507.93 90 210 360 840 • Since value to reject the hpothesis (gender and prefered reading are independent) )is 10.828 for 1 degree of freedom test, • resuIt shows that like_science_fiction and gender are correlated • • 43 i 1 (ai A)(bi B) (n 1) A B n i 1 (ai bi ) n AB (n 1) A B where n is the number of tuples, A and B are the respective means of A and B, σA and σB are the respective standard deviation of A and B, and Σ(aibi) is the sum of the AB cross-product. If rA,B > 0, A and B are positively correlated (A’s values increase as B’s). The higher, the stronger correlation. rA,B = 0: independent; rAB < 0: negatively correlated 44 11 26.10.2015 Visually Evaluating Correlation Covariance (Numeric Data) • Covariance is similar to correlation describes how two attributes change together Correlation coefficient: Scatter plots showing the similarity from –1 to 1. • • • where n is the number of tuples, A and B are the respective mean or expected values of A and B, σA and σB are the respective standard deviation of A and B Positive covariance: If CovA,B > 0, then A and B both tend to be larger than their expected values Negative covariance: If CovA,B < 0 then if A is larger than its expected value, B is likely to be smaller than its expected value Independence: CovA,B = 0 but the converse is not true: – Some pairs of random variables may have a covariance of 0 but are not independent. Only under some additional assumptions (e.g., the data follow multivariate normal distributions) does a covariance of 0 imply independence 45 46 Co-Variance: An Example • It can be simplified in computation as • Suppose two stocks A and B have the following values in one week: (2, 5), (3, Data Preprocessing • Data Preprocessing: An Overview – Data Quality 8), (5, 10), (4, 11), (6, 14). • Question: If the stocks are affected by the same industry trends, will their • Data Cleaning prices rise or fall together? • Data Integration – E(A) =avg= (2 + 3 + 5 + 4 + 6)/ 5 = 20/5 = 4 – E(B) =avg= (5 + 8 + 10 + 11 + 14) /5 = 48/5 = 9.6 – Cov(A,B) = (2×5+3×8+5×10+4×11+6×14)/5 − 4 × 9.6 = 4 • – Major Tasks in Data Preprocessing Thus, A and B rise together since Cov(A, B) > 0. • Data Reduction • Data Transformation and Data Discretization • Summary 48 48 12 26.10.2015 Data Reduction Strategies • • • • Data reduction: Obtain a reduced representation of the data set that is much smaller in volume but yet produces the same (or almost the same) analytical results Why data reduction? — A database/data warehouse may store terabytes of data. Complex data analysis may take a very long time to run on the complete data set. Curse of dimensionality – When dimensionality increases, data becomes increasingly sparse – Density and distance between points, which is critical to clustering, outlier analysis, becomes less meaningful – The possible combinations of subspaces will grow exponentially Dimensionality reduction – Avoid the curse of dimensionality – Help eliminate irrelevant features and reduce noise – Reduce time and space required in data mining – Allow easier visualization • Data Reduction 1: Dimensionality Reduction Data reduction strategies – Dimensionality reduction, e.g., remove unimportant , irrelevant, redundant attributes • Wavelet transforms • Principal Components Analysis (PCA) • Feature subset selection, feature creation – Numerosity reduction (some simply call it: Data Reduction) • Regression and Log-Linear Models • Histograms, clustering, sampling • Data cube aggregation – Data compression (lossless or lossy) 50 49 Mapping Data to a New Space Wavelet Transformation Haar2 Wavelet transform Two Sine Waves Daubechie4 • Discrete wavelet transform (DWT) for linear signal processing, multi-resolution analysis • Compressed approximation: store only a small fraction of the strongest of the wavelet coefficients • Similar to discrete Fourier transform (DFT), but better lossy compression, localized in space • Method: Two Sine Waves + Noise – Length, L, must be an integer power of 2 (padding with 0’s, when necessary) – Each transform has 2 functions: smoothing, difference – Applies to pairs of data, resulting in two set of data of length L/2 – Applies two functions recursively, until reaches the desired length Frequency 51 52 13 26.10.2015 Principal Component Analysis (PCA) Why Wavelet Transform? • Effective removal of outliers – Insensitive to noise, insensitive to input order • Multi-resolution – Detect arbitrary shaped clusters at different scales • Efficient – Complexity O(N) • Only applicable to low dimensional data • Find a projection that captures the largest amount of variation in data • The original data are projected onto a much smaller space, resulting in dimensionality reduction. • Better than wavelength transform at handling sparse data x2 e 53 x1 Attribute Subset Selection 54 Heuristic Search in Attribute Selection • There are 2d possible attribute combinations of d attributes • Typical heuristic (greedy) attribute selection methods: make the best locally optimal choice at the moment hoping it will lead to globally optimal solution – Best single attribute under the attribute independence assumption: choose by significance tests – Best step-wise forward selection: • The best single-attribute is picked first • Then next best attribute condition to the first, ... – Step-wise backward elimination: • Repeatedly eliminate the worst attribute – Best combined forward selection and backward elimination – Decision Tree Induction • It is also called feature subset selection in ML • Another way to reduce dimensionality of data • Aim is to find a minimum set of attributes such that the resulting probability distribution of the data classes is as close as possible to the original distribution obtained using all attributes • Redundant attributes – Duplicate much or all of the information contained in one or more other attributes – E.g., purchase price of a product and the amount of sales tax paid • Irrelevant attributes – Contain no information that is useful for the data mining task at hand – E.g., students' ID is often irrelevant to the task of predicting students' GPA • Forward selection, backward elimination, decision tree induction 55 56 14 26.10.2015 Attribute Creation (Feature Generation) Heuristic Search in Attribute Selection • Create new attributes (features) that can capture the important information in a data set more effectively than the original ones 57 58 Parametric Data Reduction: Regression and Log-Linear Models Data Reduction 2: Numerosity Reduction • Linear regression – Data modeled to fit a straight line – Often uses the least-square method to fit the line • Multiple regression – Allows a response variable Y to be modeled as a linear function of multidimensional feature vector • Log-linear model – Approximates discrete multidimensional probability distributions based on a smaller subset of dimensional combinations • Reduce data volume by choosing alternative, smaller forms of data representation • Parametric methods (e.g., regression) – Assume the data fits some model, estimate model parameters, store only the parameters, and discard the data (except possible outliers) – Ex.: regression, Log-linear models—used to obtain approximate data • Non-parametric methods – Do not assume models – Major families: histograms, clustering, sampling, … 59 60 15 26.10.2015 y Regression Analysis Regress Analysis and Log-Linear Models Y1 • • Regression analysis: A collective name for Y1’ techniques for the modeling and analysis of – Two regression coefficients, w and b, specify the line and are to be estimated by using the data at hand y=x+1 numerical data consisting of values of a – Using the least squares criterion to the known values of Y1, Y2, …, X1, X2, …. dependent variable (also called response variable or measurement) and of one or more x X1 independent variables (aka. explanatory variables or predictors) • The parameters are estimated so as to give a "best fit" of the data • Most commonly the best fit is evaluated by using the least squares method, but other Linear regression: Y = w X + b • Multiple linear regression: Y = b0 + b1 X1 + b2 X2 • Log-linear models: – Many nonlinear functions can be transformed into the above • Used for prediction (including forecasting of time-series data), inference, hypothesis testing, and modeling of causal relationships – Approximate discrete multidimensional probability distributions – Estimate the probability of each point (tuple) in a multi-dimensional space for a set of discretized attributes, based on a smaller subset of dimensional combinations – Useful for dimensionality reduction and data smoothing criteria have also been used 61 Histogram Analysis • Divide data into buckets and store average (sum) for each bucket • Partitioning rules: 62 Clustering • Partition data set into clusters based on similarity (distance), and store cluster representation (e.g., centroid and diameter) only 40 35 30 • Can be very effective if data is clustered but not if data is “smeared” 25 – Equal-width: equal bucket 20 range • Can have hierarchical clustering and be stored in multidimensional index tree structures – Equal-frequency (or equal- 15 depth) 10 • There are many choices of clustering definitions and clustering algorithms 5 • Cluster analysis will be studied in depth later 0 10000 30000 50000 70000 90000 63 16 26.10.2015 Sampling • Sampling: obtaining a small sample s to represent the whole data set N • Key principle: Choose a representative subset of the data – Simple random sampling may have very poor performance in the presence of skew – Develop adaptive sampling methods, e.g., stratified sampling Sampling: With or without Replacement Types of Sampling • Simple random sampling – There is an equal probability of selecting any particular item • Sampling without replacement – Once an object is selected, it is removed from the population • Sampling with replacement – A selected object is not removed from the population • Stratified sampling: – Partition the data set, and draw samples from each partition (proportionally, i.e., approximately the same percentage of the data) – Used in conjunction with skewed data Sampling: Cluster or Stratified Sampling Raw Data Cluster/Stratified Sample Raw Data 17 26.10.2015 Data Compression Chapter 3: Data Preprocessing • Data Preprocessing: An Overview – Data Quality Compressed Data Original Data lossless – Major Tasks in Data Preprocessing • Data Cleaning • Data Integration • Data Reduction Original Data Approximated • Data Transformation and Data Discretization • Summary Data Transformation • A function that maps the entire set of values of a given attribute to a new set of replacement values such that each old value can be identified with one of the new values • Methods Normalization • v' – Attribute/feature construction • Z-score normalization (μ: mean, σ: standard deviation): – Aggregation: Summarization, data cube construction v' – Normalization: Scaled to fall within a smaller, specified range • normalization by decimal scaling – Discretization: Concept hierarchy climbing v A A • min-max normalization • z-score normalization v minA (new _ maxA new _ minA) new _ minA maxA minA – Ex. Let income range $12,000 to $98,000 normalized to [0.0, 1.0]. 73,600 12,000 (1.0 0) 0 0.716 Then $73,000 is mapped to 98,000 12,000 – Smoothing: Remove noise from data • New attributes constructed from the given ones Min-max normalization: to [new_minA, new_maxA] – Ex. Let μ = 54,000, σ = 16,000. Then • 73,600 54,000 1.225 16,000 Normalization by decimal scaling v' v 10 j Where j is the smallest integer such that Max(|ν’|) < 1 18 26.10.2015 Data Discretization Methods Discretization • Three types of attributes – Nominal—values from an unordered set, e.g., color, profession – Ordinal—values from an ordered set, e.g., military or academic rank – Numeric—real numbers, e.g., integer or real numbers • Discretization: Divide the range of a continuous attribute into intervals – Interval labels can then be used to replace actual data values – Reduce data size by discretization – Supervised vs. unsupervised – Split (top-down) vs. merge (bottom-up) Simple Discretization: Binning • Equal-width (distance) partitioning – Divides the range into N intervals of equal size: uniform grid – if A and B are the lowest and highest values of the attribute, the width of intervals will be: W = (B –A)/N. – The most straightforward, but outliers may dominate presentation – Skewed data is not handled well • Equal-depth (frequency) partitioning – Divides the range into N intervals, each containing approximately same number of samples – Good data scaling – Managing categorical attributes can be tricky • Typical methods: All the methods can be applied recursively – Binning • Top-down split, unsupervised • Sensitive to user specified number of bins – Histogram analysis • Top-down split, unsupervised – Clustering analysis (unsupervised, top-down split or bottom-up merge) – Decision-tree analysis (supervised, top-down split) – Correlation (e.g., 2) analysis (unsupervised, bottom-up merge) Binning Methods for Data Smoothing Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29, 34 Equal-frequency bins of size 3 * Partition into equal-frequency (equi-depth) bins: - Bin 1: 4, 8, 9, 15 - Bin 2: 21, 21, 24, 25 - Bin 3: 26, 28, 29, 34 * Smoothing by bin means: - Bin 1: 9, 9, 9, 9 - Bin 2: 23, 23, 23, 23 - Bin 3: 29, 29, 29, 29 * Smoothing by bin boundaries: - Bin 1: 4, 4, 4, 15 - Bin 2: 21, 21, 25, 25 - Bin 3: 26, 26, 26, 34 19 26.10.2015 Discretization by Classification & Correlation Analysis • Classification (e.g., decision tree analysis) – Supervised: Given class labels, e.g., cancerous vs. benign – Using entropy to determine split point (discretization point) – Top-down, recursive split – Details to be covered later • Correlation analysis (e.g., Chi-merge: χ2-based discretization) – Supervised: use class information – Bottom-up merge: find the best neighboring intervals (those having similar distributions of classes, i.e., low χ2 values) to merge Concept Hierarchy Generation • Concept hierarchy organizes concepts (i.e., attribute values) hierarchically and is usually associated with each dimension in a data warehouse • Concept hierarchy formation: Recursively reduce the data by collecting and replacing low level concepts (such as numeric values for age) by higher level concepts (such as youth, adult, or senior) • Concept hierarchies can be explicitly specified by domain experts and/or data warehouse designers • Concept hierarchy can be automatically formed for both numeric and nominal data—For numeric data, use discretization methods – Merge performed recursively, until a predefined stopping condition Summary Chapter 3: Data Preprocessing • Data Preprocessing: An Overview – Data Quality – Major Tasks in Data Preprocessing • Data Cleaning • Data Integration • Data Reduction • Data Transformation and Data Discretization • Summary • Data quality: accuracy, completeness, consistency, timeliness, believability, interpretability • Data cleaning: e.g. missing/noisy values, outliers • Data integration from multiple sources: – Entity identification problem; Remove redundancies; Detect inconsistencies • Data reduction – Dimensionality reduction; Numerosity reduction; Data compression • Data transformation and data discretization – Normalization; Concept hierarchy generation 20 26.10.2015 WEKA • A collection of open source of many data mining and machine learning algorithms, including – pre-processing on data – classification – clustering – association rule extraction • Created by researchers at the University of Waikato in New Zealand • Java based (also open source). Installation Download Weka (the stable version) from http://www.cs.waikato.ac.nz/ml/weka/ – Choose a self-extracting executable (including Java VM) – (If you are interested in modifying/extending weka there is a developer version that includes the source code) • After download is completed, run the self extracting file to install Weka, and use the default set-ups. WEKA Main features • 49 data preprocessing tools • 76 classification/regression algorithms • 8 clustering algorithms • 15 attribute/subset evaluators + 10 search algorithms for feature selection. • 3 algorithms for finding association rules • 3 graphical user interfaces – “The Explorer” (exploratory data analysis) – “The Experimenter” (experimental environment) – “The KnowledgeFlow” (new process model inspired interface) WEKA From windows desktop, – click “Start”, choose “All programs”, – Choose “Weka 3.7.10” to start Weka – Then the first interface window appears: Weka GUI Chooser 21 26.10.2015 WEKA applications WEKA: The ARFF format % % ARFF file for weather data with some numeric features % @relation weather • Explorer – preprocessing, attribute selection, learning, visualization • Experimenter – testing and evaluating machine learning algorithms • Knowledge Flow – visual design of KDD process – Explorer • Simple Command-line – A simple interface for typing commands @attribute @attribute @attribute @attribute @attribute @data sunny, 85, 85, false, no sunny, 80, 90, true, no overcast, 83, 86, false, yes ... Lines starting with : % are comments @relation names the data set @attribute specifies the attribute names and their data @data specifies the start of data section Weka: A Brief Introduction Data Exploration in Weka Explorer • ARFF file format • Weka Simple CLI Data set name Schema section outlook {sunny, overcast, rainy} temperature numeric humidity numeric windy {true, false} play? {yes, no} – Weka facilities as Java classes – Calling the Java functions as commands Numeric attribute names and types Categorical attribute name and values Data section One data record per line; Values separated by “,”; “?” represents unknown. 22 26.10.2015 Weka: A Brief Introduction Weka: A Brief Introduction • Weka KnowledgeFlow • Weka Experimenter – Comparing performances of different classification solutions on a collection of data sets – Setting up a flow of knowledge discovery in a diagram – Overview of the entire discovery project Data Exploration in Weka Explorer Data Exploration in Weka Explorer • Glance of an opened data set • Visualisation in Weka (limited) Summary statistics Visualisation of value distribution 23 26.10.2015 Data Exploration in Weka Explorer *Exploring data with WEKA • Filters for pre-processing – – – – Many filters Supervised/unsupervised Attribute/instance Choose followed by parameter setting in command line • Use Weka to explore – – – • Weather data Iris data (+ visualization) Labor negotiation Filters: – – – – Copy Make_indicator Nominal to binary Merge-two-values 94 24