Support Vector Machine
(SVM)
Introduction
• Support Vector Machine(SVM) is a supervised machine learning
algorithm used for both classification and regression. Though we
say regression problems as well its best suited for classification.
• The goal of the SVM algorithm is to create the best line or
decision boundary that can segregate n-dimensional space into
classes so that we can easily put the new data point in the correct
category in the future. This best decision boundary is called a
hyperplane.
• When we add the new testing data, whatever side of the
hyperplane it goes will eventually decide the class that we assign
to it.
L1.2
Introduction
• In the above diagram blue line (hyperplane) is separating two class of
star and circle. Support vector machine make sure when you are
creating this hyperplane it also create two margin lines. These two
margin line will having some distance. And these two margin lines
will be parallel to our hyperplane.
• SVM chooses the extreme points/vectors that help in creating this
hyperplane. These extreme cases are called as support vectors.
L1.3
Introduction
• The second line, the SVM hyperplane (B), likewise separates the
two clusters, but from a position of maximum distance between
itself and the two clusters.
• These margin lines will be passing through the nearest data point
of both classes.
• You will also notice a gray area that denotes margin, which is
the distance between the hyperplane and the nearest data point,
multiplied by two.
• The margin is a key feature of SVM and is important because it
offers additional support to cope with new data points that may
infringe on a logistic regression hyperplane.
• Whenever we create a hyperplane we should select a hyperplane
that maximize the distance between two classes Or that has
maximum margin.
L1.4
SVM Types
• Linear SVM: Linear SVM is used for linearly separable data,
which means if a dataset can be classified into two classes by
using a single straight line, then such data is termed as linearly
separable data, and classifier is used called as Linear SVM
classifier.
• Non-linear SVM: Non-Linear SVM is used for non-linearly
separated data, which means if a dataset cannot be classified by
using a straight line, then such data is termed as non-linear data
and classifier used is called as Non-linear SVM classifier.
L1.5
Identify Hyperplane (Linear)
• We need to remember a thumb rule to identify the right hyper-
plane:
1. Select the hyper-plane which segregates the two classes better.
2. Whenever we create a hyperplane we should select a
hyperplane that maximize the distance between two classes Or
that has maximum margin.
L1.6
Identify the right hyper-plane
• Here, we have three hyper-planes (A, B, and C) and all are
segregating the classes well. Now, How can we identify the right
hyper-plane?
• Here, maximizing the distances between nearest data point
(either class) and hyper-plane will help us to decide the right
hyper-plane. This distance is called as Margin.
L1.7
Identify the right hyper-plane
• Below, you can see that the margin for hyper-plane C is high as
compared to both A and B. Hence, we name the right hyper plane
as C. Another lightning reason for selecting the hyper-plane with
higher margin is robustness. If we select a hyper-plane having
low margin like A or B then there is high chance of miss
classification or our test data will give us lots of error .
L1.8
Anomalies in SVM
• Another useful application case of SVM is for mitigating anomalies. A
limitation of standard logistic regression is that it goes out of its way to fit
anomalies (as seen in the scatterplot with the star in the bottom right
corner in Figure).
• SVM, however, is less sensitive to such data points and actually minimizes
their impact on the final location of the boundary line. In Figure, we can
see that Line B (SVM hyperplane) is less sensitive to the anomalous star
on the right-hand side. SVM can thus be used as one method to fight
anomalies.
The SVM algorithm has a feature
to ignore outliers and find the hyper
-plane that has the maximum margin.
Hence, we can say, SVM
classification is robust to outliers.
L1.9
Non-Linear SVM
• In the scenario below, we can’t have linear hyper-plane between
the two classes, if we try to apply linear SVM on it, accuracy will
be less that 50%.
• To solve these kind of problems SVM used a technique called
SVM kernel.
L1.10
Non-Linear SVM
• However, SVM’s real strength is found in high dimensional data
and handling multiple features.
• SVM has numerous variations available to classify high-
dimensional data, known as “kernels,” including linear SVC (seen
in Figure 12), polynomial SVC,z and the Kernel Trick.
• The Kernel Trick is an advanced solution to map data from a low
dimensional to a high-dimensional space. Transitioning from a
two-dimensional to a three-dimensional space allows you to use a
linear plane to split the data within a 3-D space.
• So to separate these data points, we need to add one more
dimension. For linear data, we have used two dimensions x and y,
so for non-linear data, we will add a third dimension z. It can be
calculated as:
z=x2 +y2 L1.11
Non-Linear SVM
• By adding the third dimension, the sample space will become as
below figure.
• In above plot, points to consider are:
– All values for z would be positive always because z is the squared
sum of both x and y
– In the original plot, red circles appear close to the origin of x and y
axes, leading to lower value of z and star relatively away from the
origin result to higher value of z.
L1.12
Applications of SVM
• Sentiment analysis.
• Spam Detection.
• Handwritten digit recognition.
• Image recognition challenges
L1.13