Support Vector Machine
Support Vector Machines
• Support Vector Machines are a set of supervised learning methods
used for classification, regression, and outliers detection problems.
• Support vector machines are different from other classification
algorithms in the way they choose the decision boundary that
maximizes the distance from the nearest data points of all the
categories. The decision boundary created by SVM is called the
maximum margin classifier or the maximum margin hyperplane.
• This approach helps SVMs make robust and accurate predictions.
Important terms in Support
Vector Machine
• Hyperplane − A hyperplane is a decision boundary that differentiates the two classes in
SVM. A data point falling on either side of the hyperplane can be attributed to different
classes. The dimension of the hyperplane depends on the number of input features in the
dataset.
• Support Vectors − Datapoints that are closest to the hyperplane/Maximum margin are
called support vectors. The separating line will be defined with the help of these data
points.
• Kernel — A kernel is a function used in SVM for helping to solve problems. They provide
shortcuts to avoid complex calculations.
• Margin − It may be defined as the gap between two lines on the closet data points of
different classes. A large margin is considered a good margin and a small margin is
considered as a bad margin.
Important terms in Support
Vector Machine
Working of SVM
• A simple linear SVM classifier works by making a straight line between
two groups of data. That means all of the data points on one side of
the line will represent a particular category and the data points on the
other side of the line will be put into a different category. This means
there can be an infinite number of lines for us to choose from.
• What makes the linear SVM algorithm better than some of the other
algorithms, like k-nearest neighbors, is that it chooses the best line to
classify the data points. The algorithm chooses the line that
separates the data and is the furthest away from the closest data
points as possible. If we select a hyperplane having a low margin
then there is a high chance of misclassification.
SVM kernel
• An important function in the implementation of the SVM algorithm is the SVM kernel.
• It is a function that takes low dimensional input space and transforms it into a higher dimensional
space.
• Explanation:
• Sometimes, in the original dataset (low-dimensional space), the data is not linearly separable (you
can’t draw a straight line to separate the classes).
• A kernel function helps by mapping the data into a higher-dimensional space where separation
becomes possible.
• 👉 Example:
• Imagine data in 2D shaped like two concentric circles (inner circle = Class A, outer circle = Class B).
• In 2D, no straight line can separate them.
• But if we transform the data into 3D (using a kernel), the circles become separable by a plane.
• This is the power of kernel functions.
•Kernel = mathematical trick to map inputs into higher dimensions without explicitly calculating all new coordinates
•Useful mainly for non-linear problems where data cannot be separated with a straight line.
Types of Support vector
Machine
• Linear SVM
• Non-Linear SVM
Linear SVM
• Linear SVM is used in the case of linearly separable data. It means if a dataset can be classified into two classes by
using a straight line, then such data is termed linearly separable data, and we can use the Linear SVM classifier.
• In a simple case where the data is linearly separable, SVM finds the hyperplane that best separates the classes.
• The equation of a hyperplane in a two-dimensional space is given as
• The formula of the linear kernel says that the dot product between two vectors is the sum of the multiplication of
each pair of input values.
• The dot product of two vectors is the sum of the products of their corresponding components.
Linear SVM
• The working of the Linear SVM
algorithm can be understood by
using an example.
• Suppose we have a dataset that has
two categories of data represented
by blue and red.
• We want a classifier that can classify
the pair of coordinates in either red
or blue, which is plotted as follows.
Non-linear SVM
• Non-Linear SVM is used in the case of non-linearly separated data.
• It means if a straight line cannot classify a dataset, then such data is
termed non-linear data, and we can use the Non-linear SVM classifier.
Non-linear SVM
• Suppose we have a dataset that has
two categories of data represented by
blue and red. We want a classifier
that can classify the pair of
coordinates in either red or blue.
• Radial Basis Kernel(RBF) is a kernel
function that is used to find a non-
linear classifier or regression line.
• RBF kernel, mostly used in SVM
classification, maps input space in
indefinite dimensional space.
What happens when a data point from the
test data set falls inside the above margin ?
How does the SVM model classify such a
data point ?
• Hard margin SVM expects all the +ve points to be on or above the +ve
support vector line and all -ve points to be on or below the -ve support
vector line but soft margin SVM modifies the hard margin constraint so we
can work with almost linearly separable data points - so soft margin SVM
will classify those points correctly.
• Hard margin and soft margin comes into picture while training the model
on the train data set. Having a hard margin or soft margin lets us decide
how much error on the training data sets are we going to allow. It is
decided by the parameter C, which when very high results into an overfit
model and when very low results into an underfit model. So having a
hard/soft margin is all about choosing what kind of Bias / Variance we want
from our model.
• All this is done while training on the known data.
Regularization (C Parameter)
• The SVM algorithm introduces a regularization parameter 'C' to control the trade-
off between maximizing the margin and minimizing classification errors.
• A smaller 'C' value creates a larger margin but may allow for some
misclassification, while a larger 'C' value may lead to a narrower margin but fewer
misclassifications.
Total error of a SVM
• When a misclassification occurs, it is because a given point is on the
wrong side of the separating hyperplane, and that's called a
classification error.
• Whenever a point is inside the margin, that counts as a margin error.
The total error of a SVM, is the sum of the classification error and the
margin error.
Advantages of SVM
• A support vector machine uses a subset of training points in the
decision function called support vectors which makes it memory
efficient.
• It is effective in cases where the number of features is greater than
the number of data points.
• Support vector machine is effective on datasets with multiple
features.
• In this algorithm, different kernel functions can be specified for the
decision function. We can also use custom kernels.
Disadvantages of SVM
• The support vector machine algorithm works best on small data sets
because of its high training time.
• If the number of features is a lot bigger than the number of data
points, avoiding over-fitting when choosing kernel functions and
regularization term is crucial.
• SVM doesn’t provide probability estimates, they are calculated using
cross-validation.