Welcome!
Essential of Machine Learning
cloudthat
move up.
Topic : Support vector machine
We will begin at 9:10 am
Supp ort Vecto r Mach ine
Support vector machines (SVMs) are a popular linear classifier.
SVMs are supervised learning models, meaning sample data
must be labeled, that can be applied to almost any type of
data.
[Link]
Suppor t Vector Machin e
They are especially effective at classification, numeral
prediction, and pattern recognition tasks. SVMs find a line (or
hyperplane in dimensions greater than 2) in between different
classes of data such that the distance on either side of that line
or hyperplane to the next-closest data points is maximized.
[Link]
Support Vector Machine
In other words, support vector machines calculate a
maximum-margin boundary that leads to a homogeneous
partition of all data points. This classifies an SVM as a
maximum margin classifier.
Finds +~e
c1~,se~
Svpport
vtc.+ors ', ' Dec:i,ion
b041nd o.r~
0 '
' '-<
ChrisAlbon '
~ i%ififij@1
cloud that
Linear Separation
•••••
I
• •• I
It fairly separates the two classes. Any point I
that is left of line falls into black circle class
and on right falls into blue square class.
Separation of classes.
••
That's what SVM does. It finds out a line/ •• •
hyper-plane (in multidimensional space that I
separate outs classes).
• I I
I
I
Non-Linear Separation
There is no line that can separate the two classes in this x-y plane. So
what do we do? We apply transformation and add one more dimension
as we call it z-axis.
■ -
Z
. .. •• t· •••
11 I 11 I
I
I I I I
I I
II \I
... ·:\ ___
I
_ I
11111
I
I
I
••
•
•
• ••
••• I
I
I
I
I
I ----- . .. ••
...• .... y
;...._
• \ : :• J ••
"-.:,/
I
I II I I
• • I •• • •
[Link]
Non-Linear Separation
Original data Data with kernel
...• • •
§
• No good linear ·-c:;:;
N • •• decision boundary .2
~
QI
-i~
>
• • Linear decision
variable 1 hyperplane
i
QI
..
lli
•••
-~ •••
• •
variable 1
Non-Linear Separation
What if data plot overlaps? 0 r, what in case some of the black points
are inside the blue ones? The Second image tolerates some outlier
points. The Third image is trying to achieve 0 tolerance with perfect
partition.
•
•• I
•• ti ■ II
••
• II
I
•
•• I
•• t i I . I
• • •I I
I I I I
• • • oI
I
I I I
II I I I I I
II I II
[Link]
Supp ort Vect or
The points closest to the
hyperplane are called as
the support vector points
and the distance of the
vectors from the
hyperplane are called the
margins.
www .[Link]
Hyperplane(Decision surface )
The hyperplane is a function which is
used to differentiate between features.
In 2-D, the function used to classify +
0
between features is a line whereas, the
function used to classify the features in a 0
0
3-D is called as a plane similarly the
function which classifies the point in
higher dimension is called as a
Fig. I
hyperplane.
[Link]
Identify Rig ht Hyperplane
Here, we have three hyper-planes (A, A B
y
B and C). Now, identify the right hyper-
plane to classify star and circle. *
You need to remember a thumb rule to
** * C
identify the right hyper-plane: "Select
the hyperplane which segregates the
••
two classes better". In this scenario,
hyper-plane "B" has excellently
performed this job.
[Link] [Link]
Identify Right Hyperplane
Here, we have three hyper-planes
(A, B and C) and all are segregating
the classes well. Now, How can we y 1 A
identify the right hyper-plane?
**
Above, you can see that the margin * *
for hyper-plane C is high as ** • ••
compared to both A and B. Hence,
we name the right hyper-plane as C. •
X
[Link].c
Tuning parameters: Regularization {Penalty )
It is also known as Penalty Parameter. It determines influence of misclassification.
It controls cost of misclassification on the training data. If the value of C is high, the
model will choose more data points as support vector while low value of C will
make the model to choose less data points as support vector.
Large Value of parameter C => small margin I Small Value of parameter C =>
Large ma··-·'·- ~ - - - - - - - ~
4 00 400
._ ·. ••7, r: . .. ,, ...·
300 . : ....
,., ..,, ;:;,,-• , : -'
.. 300
. . .~:' ,," . ..
,,--
2 00 • • • • • ,,' ,,"" • • • >OO •
• ,,,.' : ,
.. I •
. ••
,.~...
•,'
100 •
,,
,,' • ,/lit .. -·· '
.
0
,.,,,,•
,,
: .., ' z ••
. . .. :-.
0 100 200 300 400 0 100 200 300 4 00
C•1 C • 100
[Link]
Tuning par ame ters :Ga mm a
• • High Gam ma
Basically, gamm a is a param eter in Only nearby points are
-----1:....L.~'---■-----➔
which we tell the SVM whic h point s considered.
to consider when it make s the • • •
decision boundary. High Gamma
If we set gamm a to a high value that means
that we want the SVM to look at the closes t LowG amm a
Far away points are also
points to make a decision. But, if we set ------i~........,,.::..;.......-■
--- considered.
gamm a to a low value we're basically telling
the SVM to consid er the points which are the . .-. •
farthe st away. Here's an examp le of high Low Gamma
gamm a vs low gamm a.
[Link] [Link]
I
Tunin g param eters: Kernel
SVC with linear kernel
·- --- - ---·~,...,.
RBF: Radial Basis Kernel Function is the set ·<:~i:i . ""'
:-Jt:\ :~l, ••
of mathematical function. These are suitable {(;~}-::,: .
.,, . ; •'-'
when number of observation are larger than y'
S e pal length
number of features. SVC with RBF kernel
Linear Kernels: These are used when number ••
of features is more than number of
observations.
[Link]
Support Vector Machine - Code
from [Link] import SVC
model = SVC(kernel='rbf)
model. fit(x_ train, y_ train)
pred= [Link](x_ test)
Hyper Parameter
kernel = 'rbf' , 'linear'
C=0.1, 1,10,100
gamma= 10, 1 ,0.1,0.01
www .cloudthat.c o
References
1. https:/[Link]/@[Link]/support-vector-machines-explained-73f4ec363fl 3
2. [Link] al 07
3. [Link]
46 71 e2cf3755
4. [Link]
5. https:/[Link],achine-learning-101 /chapter-2-svm-support-vector-machine-theory-
f0812effc72
[Link].
Thank You !!!