Download CENG 464 Introduction to Data Mining Getting to Know Your Data

26.10.2015
Getting to Know Your Data
• Data Objects and Attribute Types
CENG 464
Introduction to Data Mining
• Basic Statistical Descriptions of Data
• Data Visualization
• Measuring Data Similarity and Dissimilarity
• Summary
Data Visualization
•
•
Why data visualization?
– Gain insight into an information space by mapping data onto graphical
primitives
– Provide qualitative overview of large data sets
– Search for patterns, trends, structure, irregularities, relationships among
data
– Help find interesting regions and suitable parameters for further
quantitative analysis
– Provide a visual proof of computer representations derived
Categorization of visualization methods:
– Pixel-oriented visualization techniques
– Geometric projection visualization techniques
– Icon-based visualization techniques
– Hierarchical visualization techniques
– Visualizing complex data and relations
Pixel-Oriented Visualization Techniques
• For a data set of m dimensions, create m windows on the screen, one
for each dimension
• The m dimension values of a record are mapped to m pixels at the
corresponding positions in the windows
• The colors of the pixels reflect the corresponding values
(a) Income
(b) Credit Limit
(c) transaction volume
(d) age
1
26.10.2015
Scatterplot Matrices
Getting to Know Your Data
Used by ermission of M. Ward, Worcester Polytechnic Institute
• Data Objects and Attribute Types
• Basic Statistical Descriptions of Data
• Data Visualization
• Measuring Data Similarity and Dissimilarity
• Summary
Matrix of scatterplots (x-y-diagrams) of the k-dim. data [total of (k2/2-k) scatterplots]
Similarity and Dissimilarity
• Similarity
– Numerical measure of how alike two data objects are
– Value is higher when objects are more alike
Data Matrix and Dissimilarity Matrix
• Data matrix
– n data points with p
dimensions
– Two modesrows&columns
– Often falls in the range [0,1]
• Dissimilarity (e.g., distance)
– Numerical measure of how different two data objects are
– Lower when objects are more alike
– Minimum dissimilarity is often 0
– Upper limit varies
• Proximity refers to a similarity or dissimilarity
• Dissimilarity matrix
– n data points, but
registers only the
distance
– A triangular matrix
– d(i,j) is the distance
between objects i and j
– Nonzero value
 x11

 ...
x
 i1
 ...
x
 n1
x1p 

... 
... x ip 

...
... 
... x np 

...
x1f
...
...
...
...
...
x if
...
...
... x nf
 0
 d(2,1)
0

 d(3,1) d ( 3,2) 0

:
:
 :
d ( n,1) d ( n,2) ...






... 0
2
26.10.2015
Proximity Measure for Binary Attributes
Proximity Measure for Nominal Attributes
• Can take 2 or more states, e.g., red, yellow, blue,
green (generalization of a binary attribute)
• Method 1: Simple matching
– m: # of matches, p: total # of variables
m
d (i, j)  p 
p
• Method 2: Use a large number of binary attributes
– creating a new binary attribute for each of the M
nominal states
Dissimilarity between Binary Variables
Object j
• Example
Name
Jack
Mary
Jim
Fever
Y
Y
Y
Cough
N
N
P
Test-1
P
P
N
Test-2
N
N
N
Test-3
N
P
N
q, r, s, t refer to total # of attributes
Object i
•
Distance measure for symmetric binary
variables:
•
Distance measure for asymmetric
binary variables: #of negative matches
considered unimportant
•
Jaccard coefficient (similarity measure
for asymmetric binary variables):

Note: Jaccard coefficient is the same as “coherence”:
Dissimilarity between Binary Variables
Name
Jack
Mary
Jim
Test-4
N
N
N
– Gender is a symmetric attribute
– The remaining attributes are asymmetric binary
– Let the values Y and P be 1, and the value N 0
01
 0.33
2 01
11
d ( jack , jim ) 
 0.67
111
1 2
d ( jim , mary ) 
 0.75
11 2
d ( jack , mary ) 
A contingency table for binary data:
• Example
Object i
Gender
M
F
M
Object j
•
–
–
–
–
Gender
M
F
M
Fever
Y
Y
Y
Cough
N
N
P
Test-1
P
P
N
Test-2
N
N
N
Test-3
N
P
N
Test-4
N
N
N
Gender is a symmetric attribute
The remaining attributes are asymmetric binary
Let the values Y and P be 1, and the value N 0
Dissimilarity based on asymmetric attributes:
01
 0.33
2 01
11
 0.67
111
1 2
d ( jim , mary ) 
 0.75
11 2
d ( jack , mary ) 
d ( jack , jim ) 
3
26.10.2015
Distance on Numeric Data: Minkowski Distance
Special Cases of Minkowski Distance
• Minkowski distance: A popular distance measure
•
h = 1: Manhattan (city block, L1 norm) distance
– E.g., the Hamming distance: the number of bits that are different
between two binary vectors
d (i, j) | x  x |  | x  x | ... | x  x |
i1 j1
i2 j 2
ip jp
where i = (xi1, xi2, …, xip) and j = (xj1, xj2, …, xjp) are two pdimensional data objects, and h is the order (the distance
so defined is also called L-h norm)
•
h = 2: (L2 norm) Euclidean distance
d (i, j)  (| x  x |2  | x  x |2 ... | x  x |2 )
i1 j1
i2 j 2
ip
jp
• Properties
•
– d(i, j) > 0 if i ≠ j, and d(i, i) = 0 (Positive definiteness)
– d(i, j) = d(j, i) (Symmetry)
h  . “supremum” (Lmax norm, L norm) distance.
– This is the maximum difference between any component (attribute)
of the vectors
– d(i, j)  d(i, k) + d(k, j) (Triangle Inequality)
• A distance that satisfies these properties is a metric
Example:
Data Matrix and Dissimilarity Matrix
Example: Minkowski Distance
Dissimilarity Matrices
point
x1
x2
x3
x4
Data Matrix
point
x1
x2
x3
x4
attribute1 attribute2
1
2
3
5
2
0
4
5
x2
0
3.61
2.24
4.24
x3
0
5.1
1
x1
0
5
3
6
x2
x3
x4
0
6
1
0
7
0
x1
x3
x4
x2
0
3.61
2.24
4.24
0
5.1
1
0
5.39
0
Supremum
x4
0
5.39
L
x1
x2
x3
x4
L2
x1
x2
x3
x4
Dissimilarity Matrix
x1
Manhattan (L1)
Euclidean (L2)
(with Euclidean Distance)
x1
x2
x3
x4
attribute 1 attribute 2
1
2
3
5
2
0
4
5
0
L
x1
x2
x3
x4
x1
x2
0
3
2
3
x3
0
5
1
x4
0
5
0
4
26.10.2015
Attributes of Mixed Type
Ordinal Variables
• A database may contain all attribute types
– Nominal, symmetric binary, asymmetric binary, numeric,
ordinal
• One may use a weighted formula to combine their effects
• An ordinal variable can be discrete or continuous
• Order is important, e.g., rank
• Can be treated like interval-scaled
– replace xif by their rank
rif {1,..., M f }
d (i, j) 
– map the range of each variable onto [0, 1] by replacing i-th
object in the f-th variable by
zif 
 pf  1 ij( f )dij( f )
 pf  1 ij( f )
–  =0 if xif or xjf is missing OR Xif or xjf =0 OR f is asymmetric
– f is binary or nominal:
dij(f) = 0 if xif = xjf , or dij(f) = 1 otherwise
– f is numeric: use the normalized distance
– f is ordinal
zif  r  1
• Compute ranks rif and
M 1
• Treat zif as interval-scaled
rif 1
M f 1
– compute the dissimilarity using methods for interval-scaled
variables
if
f
Cosine Similarity
•
•
•
•
A document can be represented by thousands of attributes, each recording the
frequency of a particular word (such as keywords) or phrase in the document.
Other vector objects: gene features in micro-arrays, …
Applications: information retrieval, biologic taxonomy, gene feature mapping, ...
Cosine measure: If d1 and d2 are two vectors (e.g., term-frequency vectors),
then
cos(d1, d2) = (d1  d2) /||d1|| ||d2|| ,
where  indicates vector dot product, ||d||: the length of vector d
Example: Cosine Similarity
•
cos(d1, d2) = (d1  d2) /||d1|| ||d2|| ,
where  indicates vector dot product, ||d|: the length of vector d
•
Ex: Find the similarity between documents 1 and 2.
d1 = (5, 0, 3, 0, 2, 0, 0, 2, 0, 0)
d2 = (3, 0, 2, 0, 1, 1, 0, 1, 0, 1)
d1d2 = 5*3+0*0+3*2+0*0+2*1+0*1+0*1+2*1+0*0+0*1 = 25
||d1||= (5*5+0*0+3*3+0*0+2*2+0*0+0*0+2*2+0*0+0*0)0.5=(42)0.5 = 6.481
||d2||= (3*3+0*0+2*2+0*0+1*1+1*1+0*0+1*1+0*0+1*1)0.5=(17)0.5
= 4.12
cos(d1, d2 ) = 0.94
5
26.10.2015
Summary
• Data attribute types: nominal, binary, ordinal, interval-scaled,
ratio-scaled
• Many types of data sets, e.g., numerical, text, graph, Web, image.
• Gain insight into the data by:
– Basic statistical data description: central tendency, dispersion,
graphical displays
– Data visualization: map data onto graphical primitives
– Measure data similarity
• Above steps are the beginning of data preprocessing
• Many methods have been developed but still an active area of
research
Data Preprocessing
• Data Preprocessing: An Overview
– Data Quality
– Major Tasks in Data Preprocessing
• Data Cleaning
• Data Integration
• Data Reduction
• Data Transformation and Data Discretization
• Summary
Issues of Data Quality
• Why is quality important?
– “Garbage in, garbage out!” Quality decisions must be
based on quality data
– For data mining, tackling the quality issue at the data
source cannot be always expected
• By cleaning the data as much as possible
• By developing and using more tolerate mining solutions
– Data quality is relevant to the intended purpose of data
mining, e.g. Do spelling errors in student names really matter
when only the increase/decrease of student numbers in particular
subject areas over the years is of interest?
Why Is Data Dirty?
• Incomplete data may come from
– “Not applicable” data value when collected
– Different considerations between the time when the data was collected
and when it is analyzed.
– Human/hardware/software problems
• Noisy data (incorrect values) may come from
– Faulty data collection instruments
– Human or computer error at data entry
– Errors in data transmission
• Inconsistent data may come from
– Different data sources
– Functional dependency violation (e.g., modify some linked data)
• Duplicate records
6
26.10.2015
Measures for data quality
• Measures for data quality: A multidimensional view
– Accuracy: correct or wrong, accurate or not
– Completeness: not recorded, unavailable, …
– Consistency: some modified but some not, dangling, …
– Timeliness: timely update?
– Believability: how trustable the data are correct?
– Interpretability: how easily the data can be
understood?
Major Tasks in Data Preprocessing
• Data cleaning
– Fill in missing values, smooth noisy data, identify or remove
outliers, and resolve inconsistencies
• Data integration
– Integration of multiple databases, data cubes, or files
• Data reduction
– Dimensionality reduction
– Numerosity reduction
– Data compression
• Data transformation and data discretization
– Normalization
– Concept hierarchy generation
Data Cleaning
Data Preprocessing
• Data Preprocessing: An Overview
– Data Quality
– Major Tasks in Data Preprocessing
• Data Cleaning
• Data Integration
• Data Reduction
• Data Transformation and Data Discretization
•
Data in the Real World Is Dirty: Lots of potentially incorrect data, e.g.,
instrument faulty, human or computer error, transmission error
– incomplete: lacking attribute values, lacking certain attributes of
interest, or containing only aggregate data
• e.g., Occupation = “ ” (missing data)
– noisy: containing noise, errors, or outliers
• e.g., Salary = “−10” (an error)
– inconsistent: containing discrepancies in codes or names, e.g.,
• Age = “42”, Birthday = “03/07/2010”
• Was rating “1, 2, 3”, now rating “A, B, C”
• discrepancy between duplicate records
– Intentional (e.g., disguised missing data)
• Jan. 1 as everyone’s birthday?
• Summary
7
26.10.2015
Incomplete (Missing) Data
Missing values - example
• Data is not always available
– E.g., many tuples have no recorded value for several
attributes, such as customer income in sales data
• Missing data may be due to
– equipment malfunction
– inconsistent with other recorded data and thus deleted
– data not entered due to misunderstanding
– certain data may not be considered important at the time
of entry
– not register history or changes of the data
Hospital Check-in Database
• Value may be missing
because it is unrecorded or
because it is inapplicable
• In medical data, value for
Pregnant? attribute for Jane
is missing, while for Joe or
Anna should be considered
Not applicable
• Some programs can infer
missing values
Name Age Sex Pregnant? ..
Mary
25
F
N
Jane
27
F
-
Joe
30
M
-
Anna
2
F
-
• Missing data may need to be inferred
30
How to Handle Missing Data?
• Ignore the tuple: usually done when class label is missing (when
doing classification)—not effective when the % of missing values
per attribute varies considerably
• Fill in the missing value manually: tedious + infeasible?
• Fill in it automatically with
– a global constant : e.g., “unknown”, a new class?!
– the attribute mean
– the attribute mean for all samples belonging to the same
class: smarter
Noisy Data
• Noise: random error or variance in a measured
variable
• Incorrect attribute values may be due to
– faulty data collection instruments
– data entry problems
– data transmission problems
– technology limitation
– inconsistency in naming convention
– the most probable value: inference-based such as Bayesian
formula or decision tree
8
26.10.2015
Noise: example
Examples: distortion of a person’s voice when
talking on a poor phone and “snow” on television
screen
Two Sine Waves
How to Handle Noisy Data?
• Binning: smooth data value by consulting its neighborhood
– first sort data and partition into (equal-frequency) bins
– then one can smooth by bin means, smooth by bin median,
smooth by bin boundaries, etc.
• Regression
– smooth by fitting the data into regression functions
• Clustering-Outlier analysis
– detect and remove outliers
• Combined computer and human inspection
– detect suspicious values and check by human (e.g., deal
with possible outliers)
Two Sine Waves + Noise
34
Outliers
• Outliers are data objects with
characteristics that are
considerably different than
most of the other data
objects in the data set
• Data cleaning
– Smoothing outliers
Duplicate Data
• Data set may include data objects that are duplicates, or
almost duplicates of one another
– Major issue when merging data from heterogonous sources
• Examples:
– Same person with multiple email addresses
9
26.10.2015
Data Cleaning as a Process
Forms of Data Preprocessing
•
•
•
Data discrepancy detection
– Use metadata (e.g., domain, range, dependency, distribution)
– Check field overloading
– Check uniqueness rule, consecutive rule and null rule
– Use commercial tools
• Data scrubbing: use simple domain knowledge (e.g., postal code,
spell-check) to detect errors and make corrections
• Data auditing: by analyzing data to discover rules and relationship to
detect violators (e.g., correlation and clustering to find outliers)
Data migration and integration
– Data migration tools: allow transformations to be specified
– ETL (Extraction/Transformation/Loading) tools: allow users to specify
transformations through a graphical user interface
Integration of the two processes
– Iterative and interactive (e.g., Potter’s Wheels) discrepancy detection and
transformation
37
38
Data Integration
Data Preprocessing
• Data Preprocessing: An Overview
•
Data integration:
•
Schema integration: e.g., A.cust-id  B.cust-#
•
Entity identification problem:
– Combines data from multiple sources into a coherent store
– Data Quality
– Integrate metadata from different sources
– Major Tasks in Data Preprocessing
• Data Cleaning
– Identify real world entities from multiple data sources, e.g., Bill Clinton =
William Clinton
• Data Integration
•
• Data Reduction
Detecting and resolving data value conflicts
– For the same real world entity, attribute values from different sources are
different
• Data Transformation and Data Discretization
– Possible reasons: different representations, different scales, e.g., metric
vs. British units
• Summary
39
39
40
40
10
26.10.2015
Handling Redundancy in Data Integration
Correlation Analysis (Nominal Data)
•
• Redundant data occur often when integration of multiple
databases
Χ2 (chi-square) test: given two attributes meaures how strongly one
attribute implies the other
2  
– Object identification: The same attribute or object may
have different names in different databases
– Derivable data: One attribute may be a “derived” attribute
in another table, e.g., annual revenue
• Redundant attributes may be able to be detected by
correlation analysis and covariance analysis
(Observed freq .  Expected freq ) 2
Expected freq
•
The larger the Χ2 value, the more likely the variables are related
•
Expected freq= (ai*bj)/total
•
The cells that contribute the most to the Χ2 value are those whose actual
count is very different from the expected count
•
Correlation does not imply causality
– # of hospitals and # of car-theft in a city are correlated
• Careful integration of the data from multiple sources may help
reduce/avoid redundancies and inconsistencies and improve
mining speed and quality
– Both are causally linked to the third variable: population
41
41
42
Chi-Square Calculation: An Example
male
female
Sum (row)
Like science fiction
250(90)
200(360)
450
Not like science fiction
50(210)
1000(840)
1050
Sum(col.)
300
1200
1500
Correlation Analysis (Numeric Data)
• Correlation coefficient (also called Pearson’s product moment
coefficient): tells also about the degree of correlation

n
•
Are gender and preferred_reading correlated?
•
Χ2 (chi-square) calculation (numbers in parenthesis are expected counts calculated
based on the data distribution in the two categories)
2 
rA, B 
(250  90) 2 (50  210) 2 (200  360) 2 (1000  840) 2



 507.93
90
210
360
840
•
Since value to reject the hpothesis (gender and prefered reading are independent) )is
10.828 for 1 degree of freedom test,
•
resuIt shows that like_science_fiction and gender are correlated
•
•
43
i 1
(ai  A)(bi  B)
(n  1) A B

n

i 1
(ai bi )  n AB
(n  1) A B
where n is the number of tuples, A and B are the respective
means of A and B, σA and σB are the respective standard deviation of A
and B, and Σ(aibi) is the sum of the AB cross-product.
If rA,B > 0, A and B are positively correlated (A’s values increase as B’s). The
higher, the stronger correlation.
rA,B = 0: independent; rAB < 0: negatively correlated
44
11
26.10.2015
Visually Evaluating Correlation
Covariance (Numeric Data)
•
Covariance is similar to correlation describes how two attributes change
together
Correlation coefficient:
Scatter plots
showing the
similarity from
–1 to 1.
•
•
•
where n is the number of tuples, A and B are the respective mean or
expected values of A and B, σA and σB are the respective standard deviation
of A and B
Positive covariance: If CovA,B > 0, then A and B both tend to be larger than their
expected values
Negative covariance: If CovA,B < 0 then if A is larger than its expected value, B is
likely to be smaller than its expected value
Independence: CovA,B = 0 but the converse is not true:
– Some pairs of random variables may have a covariance of 0 but are not independent.
Only under some additional assumptions (e.g., the data follow multivariate normal
distributions) does a covariance of 0 imply independence
45
46
Co-Variance: An Example
•
It can be simplified in computation as
•
Suppose two stocks A and B have the following values in one week: (2, 5), (3,
Data Preprocessing
• Data Preprocessing: An Overview
– Data Quality
8), (5, 10), (4, 11), (6, 14).
•
Question: If the stocks are affected by the same industry trends, will their
• Data Cleaning
prices rise or fall together?
• Data Integration
– E(A) =avg= (2 + 3 + 5 + 4 + 6)/ 5 = 20/5 = 4
– E(B) =avg= (5 + 8 + 10 + 11 + 14) /5 = 48/5 = 9.6
– Cov(A,B) = (2×5+3×8+5×10+4×11+6×14)/5 − 4 × 9.6 = 4
•
– Major Tasks in Data Preprocessing
Thus, A and B rise together since Cov(A, B) > 0.
• Data Reduction
• Data Transformation and Data Discretization
• Summary
48
48
12
26.10.2015
Data Reduction Strategies
•
•
•
•
Data reduction: Obtain a reduced representation of the data set that is much
smaller in volume but yet produces the same (or almost the same) analytical
results
Why data reduction? — A database/data warehouse may store terabytes of
data. Complex data analysis may take a very long time to run on the complete
data set.
Curse of dimensionality
– When dimensionality increases, data becomes increasingly sparse
– Density and distance between points, which is critical to clustering, outlier
analysis, becomes less meaningful
– The possible combinations of subspaces will grow exponentially
Dimensionality reduction
– Avoid the curse of dimensionality
– Help eliminate irrelevant features and reduce noise
– Reduce time and space required in data mining
– Allow easier visualization
•
Data Reduction 1: Dimensionality
Reduction
Data reduction strategies
– Dimensionality reduction, e.g., remove unimportant , irrelevant,
redundant attributes
• Wavelet transforms
• Principal Components Analysis (PCA)
• Feature subset selection, feature creation
– Numerosity reduction (some simply call it: Data Reduction)
• Regression and Log-Linear Models
• Histograms, clustering, sampling
• Data cube aggregation
– Data compression (lossless or lossy)
50
49
Mapping Data to a New Space
Wavelet Transformation
Haar2

Wavelet transform
Two Sine Waves
Daubechie4
• Discrete wavelet transform (DWT) for linear signal processing,
multi-resolution analysis
• Compressed approximation: store only a small fraction of the
strongest of the wavelet coefficients
• Similar to discrete Fourier transform (DFT), but better lossy
compression, localized in space
• Method:
Two Sine Waves + Noise
– Length, L, must be an integer power of 2 (padding with 0’s, when
necessary)
– Each transform has 2 functions: smoothing, difference
– Applies to pairs of data, resulting in two set of data of length L/2
– Applies two functions recursively, until reaches the desired length
Frequency
51
52
13
26.10.2015
Principal Component Analysis (PCA)
Why Wavelet Transform?
• Effective removal of outliers
– Insensitive to noise, insensitive to input order
• Multi-resolution
– Detect arbitrary shaped clusters at different scales
• Efficient
– Complexity O(N)
• Only applicable to low dimensional data
•
Find a projection that captures the largest amount of variation in data
•
The original data are projected onto a much smaller space, resulting in
dimensionality reduction.
•
Better than wavelength transform at handling sparse data
x2
e
53
x1
Attribute Subset Selection
54
Heuristic Search in Attribute Selection
• There are 2d possible attribute combinations of d attributes
• Typical heuristic (greedy) attribute selection methods: make the
best locally optimal choice at the moment hoping it will lead to
globally optimal solution
– Best single attribute under the attribute independence
assumption: choose by significance tests
– Best step-wise forward selection:
• The best single-attribute is picked first
• Then next best attribute condition to the first, ...
– Step-wise backward elimination:
• Repeatedly eliminate the worst attribute
– Best combined forward selection and backward elimination
– Decision Tree Induction
• It is also called feature subset selection in ML
• Another way to reduce dimensionality of data
• Aim is to find a minimum set of attributes such that the resulting
probability distribution of the data classes is as close as possible to
the original distribution obtained using all attributes
• Redundant attributes
– Duplicate much or all of the information contained in one or
more other attributes
– E.g., purchase price of a product and the amount of sales tax
paid
• Irrelevant attributes
– Contain no information that is useful for the data mining task at
hand
– E.g., students' ID is often irrelevant to the task of predicting
students' GPA
• Forward selection, backward elimination, decision tree
induction
55
56
14
26.10.2015
Attribute Creation (Feature
Generation)
Heuristic Search in Attribute Selection
• Create new attributes (features) that can capture the important
information in a data set more effectively than the original ones
57
58
Parametric Data Reduction: Regression and
Log-Linear Models
Data Reduction 2: Numerosity Reduction
• Linear regression
– Data modeled to fit a straight line
– Often uses the least-square method to fit the line
• Multiple regression
– Allows a response variable Y to be modeled as a linear
function of multidimensional feature vector
• Log-linear model
– Approximates discrete multidimensional probability
distributions based on a smaller subset of dimensional
combinations
• Reduce data volume by choosing alternative, smaller forms of
data representation
• Parametric methods (e.g., regression)
– Assume the data fits some model, estimate model
parameters, store only the parameters, and discard the
data (except possible outliers)
– Ex.: regression, Log-linear models—used to obtain
approximate data
• Non-parametric methods
– Do not assume models
– Major families: histograms, clustering, sampling, …
59
60
15
26.10.2015
y
Regression Analysis
Regress Analysis and Log-Linear Models
Y1
•
•
Regression analysis: A collective name for
Y1’
techniques for the modeling and analysis of
– Two regression coefficients, w and b, specify the line and are to be
estimated by using the data at hand
y=x+1
numerical data consisting of values of a
– Using the least squares criterion to the known values of Y1, Y2, …, X1, X2,
….
dependent variable (also called response
variable or measurement) and of one or more
x
X1
independent variables (aka. explanatory
variables or predictors)
•
The parameters are estimated so as to give a
"best fit" of the data
•
Most commonly the best fit is evaluated by
using the least squares method, but other
Linear regression: Y = w X + b
•
Multiple linear regression: Y = b0 + b1 X1 + b2 X2
•
Log-linear models:
– Many nonlinear functions can be transformed into the above
•
Used for prediction (including
forecasting of time-series data),
inference, hypothesis testing,
and modeling of causal
relationships
– Approximate discrete multidimensional probability distributions
– Estimate the probability of each point (tuple) in a multi-dimensional
space for a set of discretized attributes, based on a smaller subset of
dimensional combinations
– Useful for dimensionality reduction and data smoothing
criteria have also been used
61
Histogram Analysis
• Divide data into buckets and
store average (sum) for each
bucket
• Partitioning rules:
62
Clustering
• Partition data set into clusters based on similarity (distance),
and store cluster representation (e.g., centroid and diameter)
only
40
35
30
• Can be very effective if data is clustered but not if data is
“smeared”
25
– Equal-width: equal bucket
20
range
• Can have hierarchical clustering and be stored in multidimensional index tree structures
– Equal-frequency (or equal- 15
depth)
10
• There are many choices of clustering definitions and
clustering algorithms
5
• Cluster analysis will be studied in depth later
0
10000
30000
50000
70000
90000
63
16
26.10.2015
Sampling
• Sampling: obtaining a small sample s to represent the whole
data set N
• Key principle: Choose a representative subset of the data
– Simple random sampling may have very poor performance in
the presence of skew
– Develop adaptive sampling methods, e.g., stratified sampling
Sampling: With or without Replacement
Types of Sampling
• Simple random sampling
– There is an equal probability of selecting any particular item
• Sampling without replacement
– Once an object is selected, it is removed from the population
• Sampling with replacement
– A selected object is not removed from the population
• Stratified sampling:
– Partition the data set, and draw samples from each partition
(proportionally, i.e., approximately the same percentage of
the data)
– Used in conjunction with skewed data
Sampling: Cluster or Stratified Sampling
Raw Data
Cluster/Stratified Sample
Raw Data
17
26.10.2015
Data Compression
Chapter 3: Data Preprocessing
• Data Preprocessing: An Overview
– Data Quality
Compressed
Data
Original Data
lossless
– Major Tasks in Data Preprocessing
• Data Cleaning
• Data Integration
• Data Reduction
Original Data
Approximated
• Data Transformation and Data Discretization
• Summary
Data Transformation
•
A function that maps the entire set of values of a given attribute to a new
set of replacement values such that each old value can be identified with
one of the new values
•
Methods
Normalization
•
v' 
– Attribute/feature construction
•
Z-score normalization (μ: mean, σ: standard deviation):
– Aggregation: Summarization, data cube construction
v' 
– Normalization: Scaled to fall within a smaller, specified range
• normalization by decimal scaling
– Discretization: Concept hierarchy climbing
v  A

A
• min-max normalization
• z-score normalization
v  minA
(new _ maxA  new _ minA)  new _ minA
maxA  minA
– Ex. Let income range $12,000 to $98,000 normalized to [0.0, 1.0].
73,600  12,000
(1.0  0)  0  0.716
Then $73,000 is mapped to
98,000  12,000
– Smoothing: Remove noise from data
• New attributes constructed from the given ones
Min-max normalization: to [new_minA, new_maxA]
– Ex. Let μ = 54,000, σ = 16,000. Then
•
73,600  54,000
 1.225
16,000
Normalization by decimal scaling
v' 
v
10 j
Where j is the smallest integer such that Max(|ν’|) < 1
18
26.10.2015
Data Discretization Methods
Discretization
• Three types of attributes
– Nominal—values from an unordered set, e.g., color, profession
– Ordinal—values from an ordered set, e.g., military or academic rank
– Numeric—real numbers, e.g., integer or real numbers
• Discretization: Divide the range of a continuous attribute into intervals
– Interval labels can then be used to replace actual data values
– Reduce data size by discretization
– Supervised vs. unsupervised
– Split (top-down) vs. merge (bottom-up)
Simple Discretization: Binning
•
Equal-width (distance) partitioning
– Divides the range into N intervals of equal size: uniform grid
– if A and B are the lowest and highest values of the attribute, the width of
intervals will be: W = (B –A)/N.
– The most straightforward, but outliers may dominate presentation
– Skewed data is not handled well
•
Equal-depth (frequency) partitioning
– Divides the range into N intervals, each containing approximately same
number of samples
– Good data scaling
– Managing categorical attributes can be tricky
•
Typical methods: All the methods can be applied recursively
– Binning
• Top-down split, unsupervised
• Sensitive to user specified number of bins
– Histogram analysis
• Top-down split, unsupervised
– Clustering analysis (unsupervised, top-down split or bottom-up merge)
– Decision-tree analysis (supervised, top-down split)
– Correlation (e.g., 2) analysis (unsupervised, bottom-up merge)
Binning Methods for Data Smoothing
 Sorted data for price (in dollars): 4, 8, 9, 15, 21, 21, 24, 25, 26, 28, 29, 34
 Equal-frequency bins of size 3
* Partition into equal-frequency (equi-depth) bins:
- Bin 1: 4, 8, 9, 15
- Bin 2: 21, 21, 24, 25
- Bin 3: 26, 28, 29, 34
* Smoothing by bin means:
- Bin 1: 9, 9, 9, 9
- Bin 2: 23, 23, 23, 23
- Bin 3: 29, 29, 29, 29
* Smoothing by bin boundaries:
- Bin 1: 4, 4, 4, 15
- Bin 2: 21, 21, 25, 25
- Bin 3: 26, 26, 26, 34
19
26.10.2015
Discretization by Classification & Correlation
Analysis
• Classification (e.g., decision tree analysis)
– Supervised: Given class labels, e.g., cancerous vs. benign
– Using entropy to determine split point (discretization point)
– Top-down, recursive split
– Details to be covered later
• Correlation analysis (e.g., Chi-merge: χ2-based discretization)
– Supervised: use class information
– Bottom-up merge: find the best neighboring intervals (those having
similar distributions of classes, i.e., low χ2 values) to merge
Concept Hierarchy Generation
• Concept hierarchy organizes concepts (i.e., attribute values)
hierarchically and is usually associated with each dimension in a data
warehouse
• Concept hierarchy formation: Recursively reduce the data by collecting
and replacing low level concepts (such as numeric values for age) by
higher level concepts (such as youth, adult, or senior)
• Concept hierarchies can be explicitly specified by domain experts
and/or data warehouse designers
• Concept hierarchy can be automatically formed for both numeric and
nominal data—For numeric data, use discretization methods
– Merge performed recursively, until a predefined stopping condition
Summary
Chapter 3: Data Preprocessing
• Data Preprocessing: An Overview
– Data Quality
– Major Tasks in Data Preprocessing
• Data Cleaning
• Data Integration
• Data Reduction
• Data Transformation and Data Discretization
• Summary
• Data quality: accuracy, completeness, consistency, timeliness,
believability, interpretability
• Data cleaning: e.g. missing/noisy values, outliers
• Data integration from multiple sources:
– Entity identification problem; Remove redundancies; Detect
inconsistencies
• Data reduction
– Dimensionality reduction; Numerosity reduction; Data
compression
• Data transformation and data discretization
– Normalization; Concept hierarchy generation
20
26.10.2015
WEKA
• A collection of open source of many data mining and machine
learning algorithms, including
– pre-processing on data
– classification
– clustering
– association rule extraction
• Created by researchers at the University of Waikato in New
Zealand
• Java based (also open source).
Installation
Download Weka (the stable version) from
http://www.cs.waikato.ac.nz/ml/weka/
– Choose a self-extracting executable (including Java VM)
– (If you are interested in modifying/extending weka there is a developer
version that includes the source code)
• After download is completed, run the self extracting file to install Weka,
and use the default set-ups.
WEKA Main features
• 49 data preprocessing tools
• 76 classification/regression algorithms
• 8 clustering algorithms
• 15 attribute/subset evaluators + 10 search
algorithms for feature selection.
• 3 algorithms for finding association rules
• 3 graphical user interfaces
– “The Explorer” (exploratory data analysis)
– “The Experimenter” (experimental environment)
– “The KnowledgeFlow” (new process model inspired
interface)
WEKA
From windows desktop,
– click “Start”, choose “All programs”,
– Choose “Weka 3.7.10” to start Weka
– Then the first interface
window appears:
Weka GUI Chooser
21
26.10.2015
WEKA applications
WEKA: The ARFF format
%
% ARFF file for weather data with some numeric features
%
@relation weather
•
Explorer
– preprocessing, attribute selection, learning, visualization
• Experimenter
– testing and evaluating machine learning algorithms
• Knowledge Flow
– visual design of KDD process
– Explorer
• Simple Command-line
– A simple interface for typing commands
@attribute
@attribute
@attribute
@attribute
@attribute
@data
sunny, 85, 85, false, no
sunny, 80, 90, true, no
overcast, 83, 86, false, yes
...
Lines starting with :
% are comments
@relation names the data set
@attribute specifies the attribute names
and their data
@data specifies the start of data section
Weka: A Brief Introduction
Data Exploration in Weka Explorer
• ARFF file format
• Weka Simple CLI
Data set name
Schema section
outlook {sunny, overcast, rainy}
temperature numeric
humidity numeric
windy {true, false}
play? {yes, no}
– Weka facilities as Java classes
– Calling the Java functions as commands
Numeric attribute
names and types
Categorical attribute
name and values
Data section
One data record per line;
Values separated by “,”;
“?” represents unknown.
22
26.10.2015
Weka: A Brief Introduction
Weka: A Brief Introduction
• Weka KnowledgeFlow
• Weka Experimenter
– Comparing performances of different classification solutions
on a collection of data sets
– Setting up a flow of knowledge discovery in a diagram
– Overview of the entire discovery project
Data Exploration in Weka Explorer
Data Exploration in Weka Explorer
• Glance of an opened data set
• Visualisation in Weka (limited)
Summary
statistics
Visualisation of
value distribution
23
26.10.2015
Data Exploration in Weka Explorer
*Exploring data with WEKA
• Filters for pre-processing
–
–
–
–
Many filters
Supervised/unsupervised
Attribute/instance
Choose followed by
parameter setting in
command line
•
Use Weka to explore
–
–
–
•
Weather data
Iris data (+ visualization)
Labor negotiation
Filters:
–
–
–
–
Copy
Make_indicator
Nominal to binary
Merge-two-values
94
24