Download Support 9ector Machines

*!!3
z
Find a linear hyperplane (decision boundary) that will separate the data
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
53
*!!3
z
One Possible Solution
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
54
*!!3
z
Another possible solution
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
55
*!!3
z
Other possible solutions
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
56
*!!3
z
z
Which one is better? B1 or B2?
How do you define better?
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
57
*!!3
z
Find hyperplane maximizes the margin => B1 is better than B2
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
58
*!!3
& &
w• x + b = 0
& &
w • x + b = +1
& &
w • x + b = −1
­ 1
&
f (x) = ®
¯− 1
& &
if w • x + b ≥ 1
& &
if w • x + b ≤ − 1
© Tan,Steinbach, Kumar
Introduction to Data Mining
& & &
w • ( x s − xc ) = 2
2
d = Margin = &
w
4/18/2004
59
*!!3
z
We want to maximize:
2
Margin = &
|| w ||
& 2
|| w ||
– Which is equivalent to minimizing: L( w) =
2
– But subjected to the following constraints:
& &
if w • x i + b ≥ 1
­ 1
&
f ( xi ) = ®
& &
¯ − 1 if w • x i + b ≤ − 1
‹
This is a constrained optimization problem
– Numerical approaches to solve it (e.g., quadratic programming)
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
60
*!!3
z
What if the problem is not linearly separable?
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
61
*!!3
z
What if the problem is not linearly separable?
– Introduce slack variables
& 2
‹ Need to minimize:
|| w ||
§ N k·
+ C ¨ ¦ ξi ¸
L( w ) =
2
© i =1 ¹
‹
Subject to:
­ 1
&
f ( xi ) = ®
¯− 1
© Tan,Steinbach, Kumar
& &
if w • x i + b ≥ 1 - ξ i
& &
if w • x i + b ≤ − 1 + ξ i
Introduction to Data Mining
4/18/2004
62
0
*!!3
z
What if decision boundary is not linear?
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
63
0
*!!3
z
Transform data into higher dimensional space
© Tan,Steinbach, Kumar
Introduction to Data Mining
4/18/2004
64