Support Vector
Machine
Dr. Ananya Choudhury
Asst. Prof, Dept. of ETE, AEC
Support Vector Machine (SVM)
• Can perform linear classification as well as regression
• Based on the concept of a surface, called a hyperplane, which draws a
boundary between data instances plotted in the multi-dimensional feature
space.
• The output of an SVM is one of two conceivable classes which are defined
in the training data
• Goal: builds an N-dimensional hyperplane that assigns future instances
into one of the two possible output classes
Classification
using hyperplanes
SVM - a hyperplane is built to discriminate
the data instances belonging to different
classes. for the sake of simplicity
Assume: data instances are linearly
separable. So, the data instances will fall in
different sides of a straight line drawn in the
2D space
For multidimensional feature space→
straight line transforms to a hyperplane
Contd..
• SVM model → representation of the input instances as points in the
feature space, such that an apparent gap between them divides the
instances of separate classes.
• Goal of the SVM analysis → to find a plane (or a hyperplane), which
separates the instances.
• New examples (or instances) are then mapped into that space and
predicted to belong to a class on the basis of which side of the gap the
new instance will fall on.
• There may be many possible hyperplanes, and one of the challenges
with the SVM model is to find the optimal hyperplane.
Some Definitions
• Support Vectors: the data points (representing classes), which are near the identified set of
lines (hyperplane). If support vectors are removed, they will alter the position of the dividing
hyperplane.
• Hyperplane and Margin: For an N-dimensional feature space, hyperplane is a flat subspace
of dimension (N−1) that separates and classifies a set of data. For example, if we consider a
2D feature space, a hyperplane will be a 1D subspace or a straight line.
Mathematically, in a 2D space, hyperplane can be defined by the equation
c0 + c1 X1 + c2 X 2 = 0
For N-dimensional space, hyperplane can be defined by
c0 + c1 X1 + c2 X 2 + ... + cN X N = 0
Contd..
• In short, can be represented as
c . X + c0 = 0
• The more the distance of the data points from the
hyperplane→ the more confident we can be about
correct categorization.
• New test data point → the side of the hyperplane it
lands on will decide the class that we assign to it.
• The distance between hyperplane and data points
is known as margin.
Contd..
Maximum Margin Hyperplane (MMH) →
identifying the hyperplane which has the
largest separation with the data
instances of the two classes
Support vectors : are data instances
from the two classes which are closest to
the MMH.
≈ that there should be at least one
support vector from each class
Identifying the correct
hyperplane in SVM
• Scenario 1: Maximizing the distances between the nearest data
points of both the classes and hyperplane will help us decide the
correct hyperplane. This distance is called as margin.
• If we select a hyperplane having a lower margin (distance), then
there is a high probability of misclassification
Margin for hyperplane A > both B and
C. Hence, hyperplane A is the correct
hyperplane.
Contd.
Scenario 2: SVM selects the hyperplane
which classifies the classes accurately
before maximizing the margin
Hyperplane B has a classification error,
and A has classified all data instances
correctly. Therefore, A will be the correct
hyperplane.
Contd..
Scenario 3: Is not possible to distinctly
segregate the two classes by using a
straight line, as one data instance
belonging to triangle classes lies in the
territory of another class (circle) as an
outlier. (non-linearly separable data)
SVM has a feature to ignore outliers and
find the hyperplane that has the
maximum margin (hyperplane A)
SVM is robust to outliers
Identifying the MMH for linearly
separable data
1st approach: using convex hull method
- An outer boundary is drawn for the
data instances belonging to the different
classes (known as convex hull).
- The MMH is then drawn as the
perpendicular bisector of the shortest line
between the convex hulls.
• 2nd approach –
- An N dimensional feature space can be represented by the equation:
c . X + c0 = 0
Using this eq., the objective is to find a set of values for the vector c such that two hyperplanes can
be specified by the equations below
c . X + c0 +1
c . X + c0 −1
- The distance of these planes should be 2/c . In order to maximize the distance between hyperplanes,
the value of c should be minimized.
1
- Task of SVM is to solve the optimization problem: min c
2
Identifying the MMH for
non-linearly separable
data
1st approach:
• Slack variable ξ → provides some soft margin for
data instances in one class that fall on the wrong
side of the hyperplane
• A cost value ‘C’ is imposed on all such data instances
that fall on the wrong side of the hyperplane.
• Task of SVM is to minimize the total cost due to such
data instances & solve the optimization problem:
1 2 N
min c + C i
2 i =1
Non-linearly separable data
Kernel technique
2nd Approach:
- Kernel: are functions which can
transform lower dimensional input space
to a higher dimensional space.
- it converts linearly non-separable data
to a linearly separable data.
- Used when data instances of the classes
are closer to each other
Kernel functions
Some common kernel functions for
transforming from a lower dimension ‘i’ to
a higher dimension ‘j’ -
Strengths and Weakness
Strengths of SVM: Weaknesses of SVM:
• can be used for both classification and • SVM is applicable only for binary
regression. classification,
• robust, i.e. not much impacted by data • The SVM model is very complex
with noise or outliers.
• It is slow for a large dataset
• prediction results are very promising.
• It is quite memory-intensive.
Thank you