SUPPORT VECTOR

MACHINE
ALGORITHM
WHAT IS SVM
Support Vector Machine or SVM is one of the most popular Supervised
Learning algorithms, which is mainly used for classification problems. The
primary objective of the SVM algorithm is to identify the optimal hyperplane in
an N-dimensional space that can effectively separate data points into different
classes in the feature space.
The algorithm ensures that the margin between the closest points of different
classes, known as support vectors, is maximized.
TYPES OF SVM
SVM are two types –

• Linear SVM : – Linear SVM is used for linearly separable

data, which means if a dataset can be classified into two
classes by using a single straight line, and the classifier is
used called as Linear SVM classifier.

• Non-Linear SVM : – Non-Linear SVM is used for non-linearly

separated data, which means if a dataset cannot be classified
by using a straight line, then such data is termed as non-
linear data and classifier used is called as Non-linear SVM
classifier.
SVM TERMINOLOGIES
• Hyperplane: The hyperplane is the decision boundary used to separate data points of different
classes in a feature space. For linear classification, this is a linear equation represented as mx
+ c = 0.

• Support Vectors: Support vectors are the closest data points to the hyperplane. These points
are play a vital role in determining the hyperplane and the margin in Support Vector Machine
(SVM).

• Margin: The margin refers to the distance between the support vector and the hyperplane.
The primary goal of the SVM algorithm is to maximize this margin, as a wider margin typically
results in better classification performance.
SVM TERMINOLOGIES
• Kernel: The kernel is a mathematical function used in SVM to map input data into a feature
space. It’s allows the SVM to find a hyperplane in where data points are not linearly separable.
Common kernel functions include linear, Radial basis function(RBF), polynomial, and sigmoid.

• Hard Margin: A hard margin refers to the maximum-margin hyperplane that perfectly
separates the data points of different classes without any misclassifications.

• Soft Margin: When data contains outliers or is not perfectly separable, SVM uses the soft
margin technique. This method introduces a slack variable for each data point to allow some
misclassifications while balancing between maximizing the margin and minimizing violations.
SVM TERMINOLOGIES
• C: The C parameter in SVM is a regularization term that balances margin maximization and the
penalty for misclassifications.
• A low value of C means the training model has some misclassifications and hence creates a larger
margin hyperplane also known as Soft Margin. Lower values of C lead to Underfitting resulting in a high
bias and low variance.

• A high value of C means the training model is focused on achieving better accuracy by avoiding errors
or misclassification and resulting in a smaller margin hyperplane also known as Hard Margin. Higher
values of C lead to Overfitting resulting in a low bias and high variance.

• Hinge Loss: The hinge loss is a common loss function in SVMs. It penalizes misclassified points
or margin violations and is often combined with a regularization term in the objective function.
SVM TERMINOLOGIES
• Gamma: In the context of Support Vector Machines (SVM) the parameter γ (gamma) plays a
crucial role in defining the behavior of the decision boundary. Gamma is used when we use the
Gaussian RBF kernel. if you use linear or polynomial kernel then you do not need gamma only
you need C hypermeter.

• Gamma high means more curvature.

• Gamma low means less curvature.
KERNELS IN SVM
Linear
kernel
Linear kernel: K(w, b) = wT x + b

Polynomial kernel: K(w, x) = (γ wT x + b)d

Gaussian RBF: K(w, x) = exp( –γ || xi – xj ||2)

Sigmoid kernel: K(w, x) = tanh (w xiT xj + b) Polynomial kernel

RBF kernel Sigmoid
kernel
KERNEL TRICK IN SVM
The kernel trick is a method used in SVMs to enable them to classify non-
linear data using a linear classifier. It is a technique used to transform low-
dimensional data into higher-dimensional data to find a linear decision
boundary. It avoids the computational complexity that arises when
explicitly mapping the data to a higher dimension.
 HOW DOES LINEAR SVM
WORKS
The working of the SVM algorithm can be understood by using an example. Suppose we have a dataset
that has two tags (green and blue), and the dataset has two features x1 and x2. We want a classifier
that can classify the pair(x1, x2) of coordinates in either green or blue.
So as it is 2-d space so by just using a straight line, we can easily separate these two classes. But there
can be multiple lines that can separate these classes.

Hence, the SVM algorithm helps to find the best line or decision boundary, this best boundary or region
is called as a hyperplane. SVM algorithm finds the closest point of the lines from both the classes.
HOW DOES NON-LINEAR SVM WORKS
If data is linearly arranged, then we can separate it by using a straight line, but for non-linear data, we
cannot draw a single straight line. Consider the below image:

So to separate these data points, we need to add one more dimension.

For linear data, we have used two dimensions x and y, so for non-linear data,
we will add a third dimension z. It can be calculated as z = x2 + y2. By adding the third dimension,
the sample space will become as this image ->:

So now, SVM will divide the datasets into classes in the following way.
Since we are in 3-d Space, hence it is looking like a plane parallel to the x-axis.
If we convert it in 2d space with z=1, then it will become as:
MATHEMATICAL EQUATIONS
Consider a binary classification problem with two classes, labeled as +1 and -1.
We have a training dataset consisting of input feature vectors X and their
corresponding class labels Y.
The equation for the linear hyperplane can be written as: wT x + b = 0 ----------
(i)
The equation for the Positive hyperplane can be written as: wT x2 + b = 1 --------
(ii) w T x2 + b =
(-)1wT x1 + b =
_____________can be written as: wT x + b = -1
The equation for the Negative hyperplane
-1
wT (x2 – x1) 1
------- (iii)
=2
w T
(x –
Combining (ii) and_____________
(iii) we
2 x
get
1) 2
– ______
=
||w|| ||w||

Then the distance

where ||w|| represents the Euclidean norm of the weight vector w.

FINDING BEST HYPERPLANE
The hyperplane function for two variables is = b + w1x1 + w2x2
There are four possible hyperplanes =
5 + 2x1 + 5x2 -----------------------(i)
5 + 20x1 + 50x2 --------------------(ii)
14 + 10x1 + 30x2 -----------------(iii)
7 + 8x1 + 20x2 ---------------------(iv)

The distance error for no(i) = w12 + w22 = 22 + 52 = 5.38

The distance error for no(ii) = w12 + w22 = 202 + 502 = 53.85
The distance error for no(iii) = w12 + w22 = 102 + 302 = 31.62
The distance error for no(iv) = w12 + w22 = 82 + 202 = 21.54
FINDING BEST HYPERPLANE
The distance error for no(i) = 5.38
The distance error for no(ii) = 53.85
The distance error for no(iii) = 31.62
The distance error for no(iv) = 21.54

2
______ 2
______
The distance given by no (i) classifier = = = 0.37
||w|| 5.38

2 2
______
The distance given by no (ii) classifier = ______ = = 0.037
||w|| 53.85

2
______ 2
______
The distance given by no (iii) classifier = = = 0.063
||w|| 31.62

2
______= 2
______
The distance given by no (iii) classifier = = 0.092
||w|| 21.54
ADVANTAGES OF SVM

• High-Dimensional Performance: SVM excels in high-dimensional spaces, making it

suitable for image classification analysis.

• Versatile: SVM can be applied to both classification and regression problems. They
support different kernel functions, enabling flexibility in capturing complex relationships
in the data.

• Resistant to Overfitting: SVM are less prone to overfitting compared to other

algorithms, like decision trees -- overfitting is where a model performs extremely well on
the training data but becomes too specific to that data and can't generalize to new data.

• Ability to Handle Nonlinear Data: SVMs can implicitly handle non-linearly separable
data by using kernel functions. The kernel trick enables SVMs to transform the input
space into a higher-dimensional feature space, making it possible to find linear decision
boundaries.

• Memory Efficiency: SVM focuses on support vectors, making it memory efficient

compared to other algorithms.
 DISADVANTAGES OF SVM
• Slow Training: SVM can be slow for large datasets, affecting performance in SVM in
data mining tasks.

• Parameter Tuning Difficulty: Selecting the right kernel and adjusting parameters like
C requires careful tuning, impacting SVM algorithms.

• Difficulty in Interpreting Complex Models: SVMs can create complex decision

boundaries, especially when using nonlinear kernels. This complexity may make it
challenging to interpret the model and understand the underlying patterns in the data.

• Noise Sensitivity: SVM struggles with noisy datasets and overlapping classes, limiting
effectiveness in real-world scenarios.

• Scalability Issues: SVMs may face scalability issues when applied to extremely large
data sets. Training an SVM on millions of samples can become impractical due to
 APPLICATION AREAS

• Face Detection
• Classification of Images
• Handwriting Recognition
• Text and Hypertext Categorization
• Geo and Environmental Sciences
THANK
YOU