Download w - Mining of Massive Datasets

Note to other teachers and users of these slides: We would be delighted if you found this our
material useful in giving your own lectures. Feel free to use these slides verbatim, or to modify
them to fit your own needs. If you make use of a significant portion of these slides in your own
lecture, please include this message, or a link to our web site: http://www.mmds.org
Mining of Massive Datasets
Jure Leskovec, Anand Rajaraman, Jeff Ullman
Stanford University
http://www.mmds.org
High dim.
data
Graph
data
Infinite
data
Machine
learning
Apps
Locality
sensitive
hashing
PageRank,
SimRank
Filtering
data
streams
SVM
Recommen
der systems
Clustering
Community
Detection
Web
advertising
Decision
Trees
Association
Rules
Dimensional
ity
reduction
Spam
Detection
Queries on
streams
Perceptron,
kNN
Duplicate
document
detection
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
2
Example: Spam filtering
Instance space x ∈ X (|X|= n data points)
Binary or real-valued feature vector x of
word occurrences
d features (words + other things, d~100,000)
Class y ∈ Y
y: Spam (+1), Ham (-1)
Goal: Estimate a function f(x) so that y = f(x)
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
3
Would like to do prediction:
estimate a function f(x) so that y = f(x)
Where y can be:
Real number: Regression
Categorical: Classification
Complex object:
Ranking of items, Parse tree, etc.
Data is labeled:
Have many pairs {(x, y)}
x … vector of binary, categorical, real valued features
y … class ({+1, -1}, or a real number)
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
4
Task: Given data (X,Y) build a model f() to
predict Y’ based on X’
Strategy: Estimate Training
X
on , .
data
Hope that the same also
Test
X’
works to predict unknown ’ data
Y
Y’
The “hope” is called generalization
Overfitting: If f(x) predicts well Y but is unable to predict Y’
We want to build a model that generalizes
well to unseen data
But Jure, how can we well on data we have
never seen before?!?
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
5
Idea: Pretend we do not know the data/labels
we actually do know
Build the model f(x) on
the training data
See how well f(x) does on
the test data
Training
set
Validation
set
Test
set
X
Y
X’
If it does well, then apply it also to X’
Refinement: Cross validation
Splitting into training/validation set is brutal
Let’s split our data (X,Y) into 10-folds (buckets)
Take out 1-fold for validation, train on remaining 9
Repeat this 10 times, report average performance
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
6
Binary classification:
f (x) =
{
+1
-1
if w(1) x(1) + w(2) x(2) +. . .w(d) x(d) ≥ θ
otherwise
Input: Vectors xj and labels yj
Vectors xj are real valued where Goal: Find vector w = (w(1), w(2) ,... , w(d) )
Each w(i) is a real number
w⋅x=0
-- - - -- - - -w
-J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
Decision
boundary
is linear
Note:
x → x, 1
∀x
w → w, − θ
7
(Very) loose motivation: Neuron
Inputs are feature values
Each feature has a weight wi
(1)
w
(1)
Activation is the sum:
x
w(2)
∑ ⋅ If the f(x) is:
Positive: Predict +1
Negative: Predict -1
x(2)
x(3)
x(4)
w⋅⋅x=0
w(3)
w(4)
nigeria
∑
≥ 0?
x1
Spam=1
x2
w
Ham=-1
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
viagra
8
Note that the Perceptron is
a conservative algorithm: it
ignores samples that it
classifies correctly.
Perceptron: y’ = sign(w⋅ x)
How to find parameters w?
Start with w0 = 0
Pick training examples xt one by one
Predict class of xt using current wt
y’ = sign(wt⋅ xt)
If y’ is correct (i.e., yt = y’)
No change: wt+1 = wt
η⋅y
η⋅ t⋅xt
If y’ is wrong: Adjust wt
wt+1 = wt + η ⋅ yt ⋅ xt
η is the learning rate parameter
xt is the t-th training example
yt is true t-th class label ({+1, -1})
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
wt
wt+1
xt, yt=1
9
Good: Perceptron convergence theorem:
If there exist a set of weights that are consistent
(i.e., the data is linearly separable) the Perceptron
learning algorithm will converge
Bad: Never converges:
If the data is not separable
weights dance around
indefinitely
Bad: Mediocre generalization:
Finds a “barely” separating solution
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
10
Perceptron will oscillate and won’t converge
So, when to stop learning?
(1) Slowly decrease the learning rate η
A classic way is to: η = c1/(t + c2)
But, we also need to determine constants c1 and c2
(2) Stop when the training error stops chaining
(3) Have a small test dataset and stop when the
test set error stops decreasing
(4) Stop when we reached some maximum
number of passes over the data
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
11
Want to separate “+” from “-” using a line
Data:
+
Training examples:
+
+
(x1, y1) … (xn, yn)
Each example i:
+
+
-
+
-
- -
xi = ( xi(1),… , xi(d) )
xi(j) is real valued
yi ∈ { -1, +1 }
Inner product:
⋅ ∑ ⋅ Which is best linear separator (defined by w)?
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
13
A
C
+
+
+
+
+
+
B
+
-
-
+
-
+
-
-
-
-
Distance from the
separating
hyperplane
corresponds to
the “confidence”
of prediction
Example:
We are more sure
about the class of
A and B than of C
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
14
Margin : Distance of closest example from
the decision line/hyperplane
The reason we define margin this way is due to theoretical convenience and existence of
generalization error bounds that depend on the value of margin.
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
15
Remember: Dot product
⋅ ⋅ ⋅ #$%
||
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
! "
"
16
Dot product
⋅ What is ⋅ , ⋅ ?
x2
+ +x
1
In this case
& x2
+ +x
1
x2
+ +x
1
In this case
& So, roughly corresponds to the margin
Bigger bigger the separation
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
17
Distance from a point to a line
w
A (xA(1), xA(2))
+
H
(0,0)
M (x1, x2)
L
Note we assume
Let:
Line L: w∙x+b =
w(1)x(1)+w(2)x(2)+b=0
w = (w(1), w(2))
Point A = (xA(1), xA(2))
Point M on a line = (xM(1), xM(2))
d(A, L) = |AH|
= |(A-M) · w|
= |(xA(1) – xM(1)) w(1) + (xA(2) – xM(2)) w(2)|
= xA(1) w(1) + xA(2) w(2) + b
=w·A+b
Remember xM(1)w(1) + xM(2)w(2) = - b
since M belongs to line L
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
18
+
+ +
+
-
+
+
+
- -
-
Prediction = sign(w⋅x + b)
“Confidence” = (w⋅ x + b) y
For i-th datapoint:
⋅ ' ( Want to solve:
)*+ ),- Can rewrite as
max γ
w,γ
s.t.∀i, yi ( w ⋅ xi + b) ≥ γ
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
19
Maximize the margin:
+
Good according to intuition,
theory (VC dimension) &
+ +
+
practice
+
max γ
+
s.t.∀i, yi ( w ⋅ xi + b) ≥ γ
+
+
-
γ
γ
w,γ
is margin … distance from
the separating hyperplane
w⋅⋅x+b=0
γ
- -
Maximizing the margin
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
20
Separating hyperplane
is defined by the
support vectors
Points on +/- planes
from the solution
If you knew these
points, you could
ignore the rest
Generally,
d+1 support vectors (for d dim. data)
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
22
Problem:
Let ⋅ ' ( then ⋅ ' ( Scaling w increases margin!
x2
Solution:
x1
Work with normalized w:
⋅ ' ( w
|| w ||
Let’s also require support vectors "
to be on the plane defined by: ⋅
" ' ( .
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
|w|
! 0
23
Want to maximize margin !
What is the relation
between x1 and x2?
' ||||
x2
2γγ
We also know:
x1
⋅ ' ( '
⋅ ' ( 1
w
|| w ||
So:
⋅ ' ( '
' ||||
⋅ ' ( ' -1
' ( '
⋅
'
w
1
⇒γ =
=
w⋅ w w
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
Note:
2
w⋅w = w
24
We started with
max w ,γ γ
s.t .∀ i , yi ( w ⋅ xi + b ) ≥ γ
But w can be arbitrarily large!
2γγ
We normalized and...
arg max γ = arg max
x2
1
= arg min w = arg min
w
1
2
x1
w
2
Then:
min
1
w 2
|| w ||
w
|| w ||
2
s.t.∀i, yi ( w ⋅ xi + b) ≥ 1
This is called SVM with “hard” constraints
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
25
If data is not separable introduce penalty:
min
1
w 2
2
w + C ⋅ (# number of mistakes)
Minimize ǁwǁ2 plus the
number of training mistakes
Set C using cross validation
How to penalize mistakes?
+
+
s.t .∀i, yi ( w ⋅ xi + b ) ≥ 1
+
+ +
+
-
-
+
-
-
All mistakes are not
equally bad!
+
-
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
26
Introduce slack variables ξi
1
w ,b ,ξ i ≥ 0 2
min
n
w + C ⋅ ∑ ξi
2
i =1
s.t .∀i, yi ( w ⋅ xi + b ) ≥ 1 − ξ i
+
If point xi is on the wrong
side of the margin then
get penalty ξi
+
+
+
+
+
ξi
ξj
+
-
+
- -
For each data point:
If margin ≥ 1, don’t care
If margin < 1, pay linear penalty
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
27
min
1
w 2
2
w + C ⋅ (# number of mistakes)
s.t .∀i , yi ( w ⋅ xi + b ) ≥ 1
What is the role of slack penalty C: small C
“good” C
+
C=∞
∞: Only want to w, b
+
that separate the data
+
C=0: Can set ξi to anything,
+
+
big C
then w=0 (basically
+
ignores the data)
+
+ - (0,0)
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
28
SVM in the “natural”n form
arg min
w ,b
1
2
w ⋅ w + C ⋅ ∑ max {0,1 − yi ( w ⋅ xi + b )}
i =1
Margin
Regularization
parameter
SVM uses “Hinge Loss”:
min
penalty
Empirical loss L (how well we fit training data)
w ,b
1
2
n
w + C ∑ ξi
2
i =1
s.t.∀i, yi ⋅ ( w ⋅ xi + b ) ≥ 1 − ξ i
0/1 loss
Hinge loss: max{0, 1-z}
-1
0
1
2
z = yi ⋅ ( xi ⋅ w + b)
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
29
n
min
w ,b
1
2
w ⋅ w + C ⋅ ∑ ξi
i =1
s.t .∀i , yi ⋅ ( xi ⋅ w + b ) ≥ 1 − ξ i
Want to estimate and (!
Standard way: Use a solver!
Solver: software for finding solutions to
“common” optimization problems
Use a quadratic solver:
Minimize quadratic function
Subject to linear constraints
Problem: Solvers are inefficient for big data!
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
31
n
Want to estimate w, b!
Alternative approach:
min
w ,b
1
2
w ⋅ w + C ∑ ξi
i =1
s.t .∀i, yi ⋅ ( xi ⋅ w + b ) ≥ 1 − ξ i
Want to minimize f(w,b):
d


( j) ( j)
1
f ( w, b) = 2 w ⋅ w + C ⋅ ∑ max0,1 − yi (∑ w xi + b)
i =1
j =1


n
Side note:
How to minimize convex functions 23
?
g(z)
Use gradient descent: minz g(z)
Iterate: zt+1 ← zt – η ∇g(zt)
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
z
32
Want to minimize f(w,b):
d
( )
f ( w, b) = 12 ∑ w
j =1
( j) 2
d


( j) ( j)
+ C ∑ max0,1 − yi (∑ w xi + b)
i =1
j =1


n
Empirical loss 4 Compute the gradient ∇(j) w.r.t. w(j)
n
∂
f
(
w
,
b
)
∂L( xi , yi )
( j)
( j)
∇f =
= w + C∑
( j)
( j)
∂w
∂
w
i =1
∂L ( xi , y i )
=0
if y i (w ⋅ xi + b ) ≥ 1
( j)
∂w
= − y i xi( j ) else
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
33
Gradient descent:
Iterate until convergence:
• For j = 1 … d
n
∂
f
(
w
,
b
)
∂L( xi , yi )
( j)
( j)
• Evaluate:∇f =
= w + C∑
( j)
( j)
∂w
∂w
i =1
• Update:
w(j) ← w(j) - η∇f
η∇ (j)
η…learning rate parameter
C… regularization parameter
Problem:
Computing ∇f(j) takes O(n) time!
n … size of the training dataset
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
34
We just had:
Stochastic Gradient Descent
n
∇f
( j)
=w
( j)
+ C∑
i =1
∂L( xi , yi )
∂w( j )
Instead of evaluating gradient over all examples
evaluate it for each individual training example
∂L( xi , yi )
∇f ( xi ) = w + C ⋅
( j)
∂w
Stochastic gradient descent:
( j)
( j)
Notice: no summation
over i anymore
Iterate until convergence:
• For i = 1 … n
• For j = 1 … d
• Compute: ∇f(j)(xi)
• Update: w(j) ← w(j) - η ∇f(j)(xi)
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
35
Example by Leon Bottou:
Reuters RCV1 document corpus
Predict a category of a document
One vs. the rest classification
n = 781,000 training examples (documents)
23,000 test examples
d = 50,000 features
One feature per word
Remove stop-words
Remove low frequency words
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
37
Questions:
(1) Is SGD successful at minimizing f(w,b)?
(2) How quickly does SGD find the min of f(w,b)?
(3) What is the error on a test set?
Training time
Value of f(w,b)
Test error
Standard SVM
“Fast SVM”
SGD SVM
(1) SGD-SVM is successful at minimizing the value of f(w,b)
(2) SGD-SVM is super fast
(3) SGD-SVM test set error is comparable
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
38
SGD SVM
Conventional
SVM
Optimization quality: | f(w,b) – f (wopt,bopt) |
For optimizing f(w,b) within reasonable quality
SGD-SVM is super fast
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
39
SGD on full dataset vs. Conjugate Gradient on
a sample of n training examples
Theory says: Gradient
descent converges in
linear time 5. Conjugate
gradient converges in 5.
Bottom line: Doing a simple (but fast) SGD update
many times is better than doing a complicated (but
slow) CG update a few times
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
5… condition number
40
Need to choose learning rate η and t0
∂L( xi , yi ) 
ηt 
wt +1 ← wt −
 wt + C

∂w 
t + t0 
Leon suggests:
Choose t0 so that the expected initial updates are
comparable with the expected size of the weights
Choose η:
Select a small subsample
Try various rates η (e.g., 10, 1, 0.1, 0.01, …)
Pick the one that most reduces the cost
Use η for next 100k iterations on the full dataset
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
41
Sparse Linear SVM:
Feature vector xi is sparse (contains many zeros)
Do not do: xi = [0,0,0,1,0,0,0,0,5,0,0,0,0,0,0,…]
But represent xi as a sparse vector xi=[(4,1), (9,5), …]
Can we do the SGD update more efficiently?
Approximated in 2 steps:
∂L( xi , yi ) cheap: xi is sparse and so few
w ← w − ηC
coordinates j of w will be updated
∂w
w ← w(1 − η )
expensive: w is not sparse, all
coordinates need to be updated
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
42
Solution 1: % ⋅ 6
Represent vector w as the
product of scalar s and vector v
Then the update procedure is:
(1) 6 6 1
Two step update procedure:
∂L( xi , yi )
∂w
(2) w ← w(1 − η )
(1) w ← w − ηC
94 ,
78
9
(2) % % 1 7
Solution 2:
Perform only step (1) for each training example
Perform step (2) with lower frequency
and higher η
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
43
Stopping criteria:
How many iterations of SGD?
Early stopping with cross validation
Create a validation set
Monitor cost function on the validation set
Stop when loss stops decreasing
Early stopping
Extract two disjoint subsamples A and B of training data
Train on A, stop by validating on B
Number of epochs is an estimate of k
Train for k epochs on the full dataset
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
44
Idea 1:
One against all
Learn 3 classifiers
+ vs. {o, -}
- vs. {o, +}
o vs. {+, -}
Obtain:
w+ b+, w- b-, wo bo
How to classify?
Return class c
arg maxc wc x + bc
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
45
Idea 2: Learn 3 sets of weights simoultaneously!
For each class c estimate wc, bc
Want the correct class to have highest margin:
wy xi + by ≥ 1 + wc xi + bc ∀c ≠ yi , ∀i
i
i
(xi, yi)
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
46
Optimization problem:
min
w ,b
1
2
∑w
c
c
2
n
+ C ∑ ξi
i =1
wyi ⋅ xi + byi ≥ wc ⋅ xi + bc + 1 − ξ i
∀ c ≠ yi , ∀ i
ξ i ≥ 0, ∀ i
To obtain parameters wc , bc (for each class c)
we can use similar techniques as for 2 class SVM
SVM is widely perceived a very powerful
learning algorithm
J. Leskovec, A. Rajaraman, J. Ullman: Mining of Massive Datasets, http://www.mmds.org
47