Download Data Preprocessing

COMP9318: Data Warehousing
and Data Mining
— L3: Data Preprocessing and Data Cleaning —
COMP9318: Data Warehousing and Data Mining
1
n 
Why preprocess the data?
COMP9318: Data Warehousing and Data Mining
2
Why Data Preprocessing?
n 
Data in the real world is dirty
n  incomplete: lacking attribute values, lacking certain
attributes of interest, or containing only aggregate
data
n 
n 
noisy: containing errors or outliers
n 
n 
e.g., occupation=“”
e.g., Salary=“-10”
inconsistent: containing discrepancies in codes or
names
n 
n 
n 
e.g., Age=“42” Birthday=“03/07/1997”
e.g., Was rating “1,2,3”, now rating “A, B, C”
e.g., discrepancy between duplicate records
COMP9318: Data Warehousing and Data Mining
3
Why Is Data Dirty?
n 
Incomplete data comes from
n 
n 
n 
n 
n 
n/a data value when collected
different consideration between the time when the data was
collected and when it is analyzed.
human/hardware/software problems
Noisy data comes from the process of data
n 
collection
n 
entry
n 
transmission
Inconsistent data comes from
n 
Different data sources
n 
Functional dependency violation
COMP9318: Data Warehousing and Data Mining
4
Why Is Data Preprocessing Important?
n 
No quality data, no quality mining results!
n 
Quality decisions must be based on quality data
n 
n 
n 
n 
e.g., duplicate or missing data may cause incorrect or even
misleading statistics.
Data warehouse needs consistent integration of quality
data
Data extraction, cleaning, and transformation comprises
the majority of the work of building a data warehouse. —
Bill Inmon
Also a critical step for data mining.
COMP9318: Data Warehousing and Data Mining
5
Major Tasks in Data Preprocessing
n 
Data cleaning
n 
n 
Data integration
n 
n 
Normalization and aggregation
Data reduction
n 
n 
Integration of multiple databases, data cubes, or files
Data transformation
n 
n 
Fill in missing values, smooth noisy data, identify or remove
outliers, and resolve inconsistencies
Obtains reduced representation in volume but produces the
same or similar analytical results
Data discretization & Data Type Conversion
COMP9318: Data Warehousing and Data Mining
6
n 
Data cleaning
COMP9318: Data Warehousing and Data Mining
7
Data Cleaning
n 
n 
Importance
n  “Data cleaning is one of the three biggest problems
in data warehousing”—Ralph Kimball
n  “Data cleaning is the number one problem in data
warehousing”—DCI survey
Data cleaning tasks
n 
Fill in missing values
n 
Identify outliers and smooth out noisy data
n 
Correct inconsistent data
n 
Resolve redundancy caused by data integration
COMP9318: Data Warehousing and Data Mining
8
Missing Data
n 
Data is not always available
n 
n 
Missing data may be due to
n 
equipment malfunction
n 
inconsistent with other recorded data and thus deleted
n 
data not entered due to misunderstanding
n 
n 
n 
E.g., many tuples have no recorded value for several
attributes, such as customer income in sales data
certain data may not be considered important at the time of
entry
not register history or changes of the data
Missing data may need to be inferred.
n 
Many algorithms need a value for all attributes
n 
Tuples with missing values may have different true values
9
How to Handle Missing Data?
n 
Ignore the tuple: usually done when class label is missing (assuming
the tasks in classification—not effective when the percentage of
missing values per attribute varies considerably.
n 
Fill in the missing value manually: tedious + infeasible?
n 
Fill in it automatically with
n 
a global constant : e.g., “unknown”, a new class?!
n 
the attribute mean
n 
the attribute mean for all samples belonging to the same class:
smarter
n 
the most probable value: inference-based such as Bayesian
formula or decision tree
COMP9318: Data Warehousing and Data Mining
10
Noisy Data
n 
n 
n 
Noise: random error or variance in a measured variable
Incorrect attribute values may due to
n  faulty data collection instruments
n  data entry problems
n  data transmission problems
n  technology limitation
n  inconsistency in naming convention
Other data problems which requires data cleaning
n  duplicate records
n  incomplete data
n  inconsistent data
COMP9318: Data Warehousing and Data Mining
11
How to Handle Noisy Data?
n 
n 
n 
n 
To be
discussed in
discretization
Binning method:
n  first sort data and partition into (equi-depth) bins
n  then one can smooth by bin means, smooth by bin
median, smooth by bin boundaries, etc.
Clustering
n  detect and remove outliers
Combined computer and human inspection
n  detect suspicious values and check by human (e.g.,
deal with possible outliers)
Regression
n  smooth by fitting the data into regression functions
COMP9318: Data Warehousing and Data Mining
12
Regression
Suburb
#Residents
Usage
Charge
Kingsford
2
1502
3047
Kensington
3
987
265.6
Maroubra
1
568
198.3
…
…
…
…
X2
6
X2 = 1.1X1 + 0.7
2.9
2
X1
13
n 
Data integration and transformation
COMP9318: Data Warehousing and Data Mining
14
Data Integration
n 
n 
n 
Data integration:
n  combines data from multiple sources into a coherent
store
Schema integration
n  integrate metadata from different sources
n  Entity identification problem: identify real world entities
from multiple data sources, e.g., A.cust-id ≡ B.cust-#
Detecting and resolving data value conflicts
n  for the same real world entity, attribute values from
different sources are different
n  possible reasons: different representations, different
scales, e.g., metric vs. British units
COMP9318: Data Warehousing and Data Mining
15
Example
n 
n 
n 
Data source 1:
n  Book(bid, title, isbn)
n  Author(aid, fname, lname, birthdate)
n  Writes(bid, aid, order)
Data source 2:
n  Book(isbn, title, year, author1, author2, …,
author10)
Data source 3:
n  Author(name, bornInYear, description, book1,
book2, …, book5)
COMP9318: Data Warehousing and Data Mining
16
Handling Redundancy in Data Integration
n 
Redundant data occur often when integration of multiple
databases
n 
n 
n 
n 
The same attribute may have different names in
different databases
One attribute may be a “derived” attribute in another
table, e.g., annual revenue
Redundant data may be able to be detected by
correlational analysis
Careful integration of the data from multiple sources may
help reduce/avoid redundancies and inconsistencies and
improve mining speed and quality
COMP9318: Data Warehousing and Data Mining
17
Also see other transformations later in the Clustering part
Data Transformation
n 
Smoothing: remove noise from data
n 
Aggregation: summarization, data cube construction
n 
Generalization: concept hierarchy climbing
n 
n 
Normalization: scaled to fall within a small, specified
range
n 
min-max normalization
n 
z-score normalization
n 
normalization by decimal scaling
Attribute/feature construction
n 
New attributes constructed from the given ones
COMP9318: Data Warehousing and Data Mining
18
Data Transformation: Normalization
n 
MinMaxScaler
min-max normalization
v − min A
v' =
(new _ max A − new _ min A ) + new _ min A
max A − min A
StandardScaler; sklearn uses
biased variance estimate
n 
z-score normalization
n 
normalization by decimal scaling
v
v'= j
10
Where j is the smallest integer such that max( v ' ) < 1
In scikit-learn, they are called Scaling.
Normalization means converting vectors to unit vectors.
19
n 
Data reduction
COMP9318: Data Warehousing and Data Mining
20
Data Reduction Strategies
n 
n 
n 
Modern datasets may be very large
n  Ratings of millions of customers on millions of items
n  Many ML algorithms have high time and space complexities.
n  Even learned models could be very large.
n  E.g., learned word embeddings (300 dims) for 1M words è at least
1.2GB memory
Data reduction
n  Obtain a reduced representation of the data set that is much smaller
in volume but yet produce the same (or almost the same) analytical
results
Data reduction strategies
n  Dimensionality reduction—remove unimportant attributes
n  Data Compression
n  Numerosity reduction—fit data into models
n  Discretization and concept hierarchy generation
21
High-dimensional Features
n 
It is common for many datasets to contain many
features
n  More is better at data capturing/creation
n 
n 
n 
561 features for human activity recognition using
smartphone dataset:
https://archive.ics.uci.edu/ml/datasets/Human
+Activity+Recognition+Using+Smartphones
GIST: 128 dimensional feature
Mandated by some model
n 
A document is converted into a high-dimensional
feature vector. #dims = |vocabulary|
COMP9318: Data Warehousing and Data Mining
22
The Curse of Dimensionality
n 
n 
n 
n 
Data in only one dimension is relatively
packed
Adding a dimension “stretches” the points
across that dimension, making them
further apart
Adding more dimensions will make the
points further apart—high dimensional
data is extremely sparse è hard to
learn
Distance measure tends to become
meaningless
(graphs from Parsons et al. KDD Explorations 2004)
High-dimensional space
n 
n 
High-dimensional space is totally different from lowdimensional space (e.g., 3D)
Many counter-intuitive facts about the high-dimensional
space
n  Two random vectors are almost surely orthogonal
n  Random sample n points within a unit hypercube è
most points are on a thin layer of the surface (annulus)
http://www.visiondummy.com/2014/04/curse-dimensionality-affect-classification/
24
Goals
n 
Reduce dimensionality of the data, yet still
maintain the meaningfulness of the data
Dimensionality reduction
n 
n 
n 
Dataset X consisting of n points in a ddimensional space
Data point xi є Rd (d-dimensional real vector):
xi = [xi1, xi2,…, xid]T
Dimensionality reduction methods:
n  Feature selection: choose a subset of the
features
n  Feature extraction: create new features by
combining new ones
Feature Selection
n 
n 
Feature selection (i.e., attribute subset selection):
n  Select a minimum set of features such that the
probability distribution of different classes given the
values for those features is as close as possible to the
original distribution given the values of all features
n  reduce # of patterns in the patterns, easier to
understand
Heuristic methods (due to exponential # of choices):
n  step-wise forward selection
n  step-wise backward elimination
n  combining forward selection and backward elimination
n  decision-tree induction
COMP9318: Data Warehousing and Data Mining
27
Heuristic Feature Selection Methods
n 
n 
There are 2d possible sub-features of d features
Several heuristic feature selection methods:
n  Best single features under the feature independence
assumption: choose by significance tests.
n  Best step-wise feature selection:
n  The best single-feature is picked first
n  Then next best feature condition to the first, ...
n  Step-wise feature elimination:
n  Repeatedly eliminate the worst feature
n  Best combined feature selection and elimination:
n  Optimal branch and bound:
n  Use feature elimination and backtracking
COMP9318: Data Warehousing and Data Mining
28
Principal Component Analysis (PCA)
n 
Original dataset: N d-dimensional vectors X = {xi}i=1..n
n 
n 
n 
n 
Find k ≤ d orthogonal basis vectors that can be
best used to represent data
Preserves maximum “information” (i.e., variance under
the orthogonal constraint) if projected onto these k
basis vectors
Reduced data set: Project each xi to the k basis vectors
(aka., principal components)
b1
T
n  xi’ = [b1…bk] xi
b2
x1 x2 x3
T
…
n  X’ = [b1…bk] X (en masse)
bk
Closed related to Singular Vector Decomposition (SVD)
29
https://gist.github.com/anonymous/7d888663c6ec679ea65428715b99bfdd
Projection
n 
n 
bT x: projection of x onto the basis vector b
What about x’ = BT x, where B consists of another set of
d-dim basis vectors?
30
JL Lemma
n 
Johnson-Lindenstrauss Flattening Lemma ‘84:
n  Given ε>0, and an integer n, let k be a positive
integer such that k ≥ k0=O(ε-2 logn). For every
set X of n points in Rd there exists F: Rd à Rk
such that for all xi, xj in X
kF (xi )
n 
F (xj )k2 2 (1 ± ")kxi
xj k2
What is the intuitive interpretation of the Lemma?
COMP9318: Data Warehousing and Data Mining
31
Distributional JL Lemma
n 
Given ε in (0, ½], δ > 0, there is a random linear
mapping F: Rd à Rk with k = O(ε-2 logδ-1) such
that for any unit vector x in Rd,
P r[kF (x)k2 2 1 ± "] 1
Take δ = n-2, so k = O(ε-2 log(n)), and then for
for all xi, xj є X,
P r[kF (xi ) F (xj )k2 2 (1 ± ")kxi xj k2 ] 1
n 
n 
Hence, by a simple union
✓ ◆bound, the same
n
statement holds for all 2 pairs from X
simultaneously with probability at least ½.
n 
There exists a deterministic linear mapping which is an
approximate isometry. ç Why/How?
1
n2
32
Explicit Mapping
n 
F(x) = k-½ * Ux, where Uij ~ N(0, 1), i.e., i.i.d. samples
from the standard Gaussian distribution.
U*1
U*1
F(x) =
x = y1 y2 … yk
…
U*k
Quick proof:
yj = hU⇤j , xi =
kyk2 ⇠ kxk2 ·
d
X
i=1
2
k
xi Uij ⇠ N (0, kxk2 )
E[kyk2 ] = kxk2 · k
V ar[kyk2 ] = 2k
Concentration bound of chisquared distribution: ✓
◆
z
3 2
2
P r[|
1| < "] 1 exp
k"
if z ⇠ k
k
16
33
Approximating Inner Product
n 
n 
20 news groups
Origin dim: 5000
34
Non-linear Dimensionality Reduction
n 
There are many advanced non-linear dimensionality
reduction methods
n  Hypothesis: real high-dimensional data live in a
manifold with low intrinsic dimensionality
COMP9318: Data Warehousing and Data Mining
35
Digits dataset (d = 64, Class = 0..5)
COMP9318: Data Warehousing and Data Mining
36
PCA (time = 0.01s)
COMP9318: Data Warehousing and Data Mining
37
t-SNE (time = 5.69s)
COMP9318: Data Warehousing and Data Mining
38
Data Compression
n 
n 
n 
String compression
n  There are extensive theories and well-tuned algorithms
n  Typically lossless
n  But only limited manipulation is possible without
expansion
Audio/video compression
n  Typically lossy compression, with progressive
refinement
n  Sometimes small fragments of signal can be
reconstructed without reconstructing the whole
Time sequence is not audio
n  Typically short and vary slowly with time
39
Numerosity Reduction
n 
Parametric methods
n 
n 
n 
Assume the data fits some model, estimate model
parameters, store only the parameters, and discard
the data (except possible outliers)
Log-linear analysis: obtain value at a point in m-D
space as the product on appropriate marginal
subspaces
Non-parametric methods
n 
n 
Do not assume models
Major families: histograms (binning), clustering,
sampling
COMP9318: Data Warehousing and Data Mining
40
Random Sampling
n 
Allow a mining algorithm to run in complexity that is
potentially sub-linear to the size of the data
n  For approximately evaluating models/parameters, etc.
n  Then run the “best” model/parameters on large
dataset
8000 points
2000 Points
COMP9318: Data Warehousing and Data Mining
500 Points
41
Other Sampling Methods
n 
n 
n 
Simple random sampling may have very poor
performance in the presence of skew
Adaptive sampling methods
n  Stratified sampling:
n  Approximate the percentage of each class (or
subpopulation of interest) in the overall database
n  Used in conjunction with skewed data
Sketch/synopsis based methods
n  E.g., count-min sketch
n 
A simple and versatile data structure to remember the
frequency of elements approximately
42
n 
Conversion of data types:
n 
Discretization
n 
Kernel density estimation
COMP9318: Data Warehousing and Data Mining
43
Discretization
n 
Three types of simple attributes:
n  Nominal/categorical — values from an unordered set
n 
n 
Profession: clerk, driver, teacher, …
Ordinal — values from an ordered set
n 
WAM: HD, D, CR, PASS, FAIL
Continuous — real numbers, including Boolean values
Other types:
n  Array
n  String
n  Objects
n 
n 
COMP9318: Data Warehousing and Data Mining
44
Discrete values è Continuous values
n 
Here we focus on
n  Continuous values è discrete values
n 
n 
n 
n 
n 
Removes noise
Some ML methods only work with discrete valued features
Reduce the number of distinct values on features, which may
improve the performance of some ML models
Reduce data size
Discrete values è continuous values
n 
n 
Smooth the distribution
Reconstruct probability density distribution from samples,
which helps generalization
45
Discretization
n 
Discretization
n 
n 
reduce the number of values for a given continuous
attribute by dividing the range of the attribute into
intervals. Interval labels can then be used to replace
actual data values
Methods
n 
Binning/Histogram analysis
n 
Clustering analysis
n 
Entropy-based discretization
COMP9318: Data Warehousing and Data Mining
46
Simple Discretization Methods: Binning
n 
n 
Equal-width (distance) partitioning:
n  Divides the range into N intervals of equal size:
uniform grid
n  if A and B are the lowest and highest values of the
attribute, the width of intervals will be: W = (B –A)/N.
n  The most straightforward, but outliers may dominate
presentation
n  Skewed data is not handled well.
Equal-depth (frequency) partitioning:
n  Divides the range into N intervals, each containing
approximately same number of samples
n  Good data scaling
n  Managing categorical attributes can be tricky.
COMP9318: Data Warehousing and Data Mining
47
Optimal Binning Problem
n 
n 
n 
n 
After binning, the educated guess or the smoothed value
is E(xi), where xi are all the values in the same bin
cost(bin) = SSE([x1, …, xm]) = ∑i=1m (xi – E(xi))2
cost of B bins = sum(cost(bin1), …, cost(binB))
Problem: find the B-1 bin boundaries such that the cost of
the resulting bins is minimized
n  Alg( {x1, ..., xn}, B )
n  Optimal Binning: Solve the problem optimally in
O(B*n2) time and O(n2) space.
n  MaxDiff: Solve the problem heuristically in O(n*log(n))
time and O(n) space.
n  Note: both algorithms do not sort input data
n 
Send in sorted({x1, ..., xn}) if necessary
48
For simplicity, we use integers, rather than real numbers, for the examples here.
Recursive Formulation
• 
• 
Observation
OPT(x[1.. n], B) = mini in [n]{SSE(x[1 .. i]) +
OPT(x[i+1 .. n], B-1)}
x[1]
x[2]
X[3]
x[4]
Example
7
9
13
5
•  n=4, B=3
(7)1+(9)1+(13,5)1
(7,9,13,5)3
2
what about
(7,9, 13)1+(5)2
(7)1+(9,13,5)2
0 + (42+42)
0+8
(7)1+(9,13)1+(5)1
(22+22) + 0
(7,9)1+(13,5)2
2+12) + 0
(1
COMP9318: Data Warehousing and Data Mining
(7,9)1+(13)1+(5)1
0+0
49
Problem Caused by Overlapping Subproblems
n 
n 
Consider calculating Fibonacci function
n  fib(0)=0
Boundary Condition
n  fib(1)=1
n  fib(n) = fib(n-1) + fib(n-2), for n>1
Naïve D&C implementation is
in efficient
fib(4)
fib(3)
fib(2)
fib(1)
22/03/17
fib(1)
fib(2)
fib(1)
fib(0)
fib(0)
50
Memoization
n 
n 
Remember solutions of all the sub-problems
Trade space for time
Sub-problem
22/03/17
fib(4)
solution
fib(4)
3
fib(3)
2
fib(2)
1
fib(1)
1
fib(0)
0
fib(3)
fib(2)
fib(1)
fib(0)
51
Dynamic Programming
n 
Ideas
n  Ensure all needed recursive calls are already computed
and memorized è a good schedule of computation
n  (Optional) Reused space to store previous recursive call
results
fib(4)
Sub-problem
solution
fib(3)
fib(2)
fib(1)
fib(1)
fib(0)
fib(2)
fib(1)
fib(0)
2D Dynamic Programming
n 
n 
n 
OPT(x[1.. n], B) = mini in [n]{SSE(x[1 .. i]) +
OPT(x[i+1 .. n], B-1)}
OPT(S1, B) = mini in [n]{… + OPT(Si+1, B-1)}
Goal:
How to schedule the computations?
OPT(S1, )
…
OPT(Sj, )
OPT(Sn, )
…
OPT(Sj+1, )
…
OPT(Sn, )
53
Pseudocode
OPT(S1, )
…
OPT(Sj, )
OPT(Sn, )
…
OPT(Sj+1, )
…
OPT(Sn, )
54
Example
n 
X = [7, 9, 13, 5], B = 3
B
S1
S2
1
2
3
n 
??
***
S3
S4
32
0
0
-
-
-
(B=2, S2)
n  What’s the problem?
n  How to calculate it?
COMP9318: Data Warehousing and Data Mining
55
MaxDiff
• 
• 
Complexity of the DP algorithm:
2
•  O(n *B) running time!
Consider a heuristic method: MaxDiff
•  Idea: use the top-(B-1) max “gaps” in the data as the
bin/bucket boundary
•  Example:
n=4, B=3
x[1]
x[2]
X[3]
x[4]
7
9
13
5
gap(1-2)
gap(2-3)
gap(3-4)
2
4
8
(7,9)1 (13)1 (5)1
COMP9318: Data Warehousing and Data Mining
56
Discretization via Clustering
n 
Can consider multiple attributes together
An example where univariate discretization does not work well
57
Supervised Discretization Methods
n 
MDLPC [Fayyad & Irani, 1993]
COMP9318: Data Warehousing and Data Mining
58
Entropy measures uncertainty
n 
n 
Two classes:
n  Give a set S of instances with binary classes {+,-}.
Let the proportions of + and – be p+ and p-.
+
+
n  Ent(S) = - (p log2 p + p log2 p )
(Note: log 0 ≣ 0)
m classes:
m
Ent(S)
X
From information
1
theory, number of
Ent(S) =
pi log pi
bits to encode the
class label.
i=1
n 
Consider drawing a random
sample from S. What can you
tell about its label?
n  If Ent(S) = 0:
n  If Ent(S) = log(m):
p+
0
0.5
1
59
Can be generalized to n-ary split
Entropy After Splitting T
n 
n 
Split S into two subsets: S1 and S2.
What about the label of a random sample given that you
know which subset it is drawn from?
n 
n 
n 
n 
|S1 |
|S2 |
E(T ; S) =
Ent(S1 ) +
Ent(S2 )
Define
|S|
|S|
Gain(T ) = Ent(S) E(T ; S)
Intuitive meaning of Gain?
60
These functions are also used in Decision Trees.
Entropy-Based Discretization: MDLPC
n 
Given a set of samples S, if S is partitioned into two
intervals S1 and S2 using boundary T, the entropy after
partitioning is
|S1 |
|S2 |
E(T ; S) =
n 
n 
|S|
Ent(S2 )
The boundary that minimizes the entropy function over all
possible boundaries is selected as a binary discretization.
The process is recursively applied to partitions obtained
until some stopping criterion is met, e.g.,
before
n 
|S|
Ent(S1 ) +
Ent ( S ) − E (T , S ) > δ
after
Experiments show that it may reduce data size and
improve classification accuracy
COMP9318: Data Warehousing and Data Mining
61
Not needed. Just for the sake of completeness.
Stopping Condition of MDLPC
n 
Stop when
COMP9318: Data Warehousing and Data Mining
62
Comments
n 
n 
n 
n 
Understanding the underlying data distribution is
important
n  e.g., via visualization
Supervised methods usually works well
More advanced methods exist
After learning some ML models, you need to
rethink
n  Why discretization?
n  Why different method/parameters affects the
model performance?
63
Continuous values è Discrete values
n 
n 
n 
After repeating the experiments (e.g., measuring
customers arriving 3-4pm), we observed the
following random variable xi (e.g., #customers):
n  xi = 2, 3, 3, 3, 3, 1, 5
n  What’s the probability to see x = 3 in a new
experiment? What about x = 4?
Naive estimation
n  P(x = 3) = 4/7
P(x = 4) = 0 / 7
Assume x follows the Poisson distribution
x
e
P (x; ) =
n  MLE estimation of
x!
n  MAP estimation with a prior (typically Gamma)
64
Non-parametric Estimation
n 
Kernel density estimation (KDE)
n  Let {xi}i=1:n be n i.i.d. sample of an unknown
f(x)
✓
n
X
1
x
K
n  We can estimate f(x) as f (x; h) =
n
n 
n 
h
K(z) controls the weight given to f(xi) to influence
f(x)
n 
n 
i=1
xi
◆
May think K(x, xi) as measuring their similarity
h is the bandwidth parameter
Gaussian kernel: K(x; h) / exp
✓
2
x
2h2
◆
65
Impact of h
COMP9318: Data Warehousing and Data Mining
66
Categorical Values
n 
One hot encoding is widely used in ML
n  Let there be m distinct values for the attribute
n  The i-th (category) value is converted into a mdimensional binary vector v, where
n 
n 
vj = 0, if j !=1
vj = 1, otherwise
scikit-learn: OneHotEncoder
Disadvantages:
n  Ignores similarity between values
Embedding-based methods can learn better (real)
vectors
n 
n 
n 
67
n 
Case Study
n  c.f., TUN_datacleansing.ppt
COMP9318: Data Warehousing and Data Mining
68