Support Vector Machine (SVM)

Prepared By
Archana
AP/PE
IIT(ISM), Dhanbad
Support Vector Machine (SVM)
 It is a supervised machine learning algorithm used for classification and regression
tasks.
 It tries to find the best boundary known as hyperplane that separates different
classes in the data. It is useful when you want to do binary classification.
Support Vector Machine (SVM)
The main goal of SVM is to maximize the margin between the two
classes. The larger the margin the better the model performs on new
and unseen data.
Key Concepts of Support Vector Machine
Hyperplane: A decision boundary separating different classes in feature
space and is represented by the equation wx + b = 0 in linear
classification.
Support Vectors: The closest data points to the hyperplane, crucial for
determining the hyperplane and margin in SVM.
Margin: The distance between the hyperplane and the support vectors.
SVM aims to maximize this margin for better classification performance.
Kernel: A function that maps data to a higher-dimensional space enabling SVM to
handle non-linearly separable data.
Hard Margin: A maximum-margin hyperplane that perfectly separates the data
without misclassifications.
Soft Margin: Allows some misclassifications by introducing slack variables, balancing
margin maximization and misclassification penalties when data is not perfectly
separable.
C: A regularization term balancing margin maximization and misclassification
penalties. A higher C value forces stricter penalty for misclassifications.
Hinge Loss: A loss function penalizing misclassified points or margin violations and is
combined with regularization in SVM.
Dual Problem: Involves solving for Lagrange multipliers associated with support
vectors, facilitating the kernel trick and efficient computation.
How does SVM classify the data?
The blue ball in the boundary of red ones is an outlier of blue balls. The SVM algorithm has the
characteristics to ignore the outlier and finds the best hyperplane that maximizes the margin. SVM is
robust to outliers.

A soft margin allows for some misclassifications or violations of the margin to improve generalization. The
SVM optimizes the following equation to balance margin maximization and penalty minimization:
The penalty used for violations is often hinge loss which has the
following behavior:
If a data point is correctly classified and within the margin there
is no penalty (loss = 0).
If a point is incorrectly classified or violates the margin the hinge
loss increases proportionally to the distance of the violation.
if data is not linearly separable
When data is not linearly separable i.e. it can't be divided by a
straight line, SVM uses a technique called kernels to map the data
into a higher-dimensional space where it becomes separable.
This transformation helps SVM find a decision boundary even for
non-linear data.
Linear SVM is a classifier that uses a straight hyperplane to separate linearly
separable data, while non-linear SVM is used for non-linearly separable data
by employing kernel functions to transform the data into a higher-
dimensional space where a linear separation is possible.
The key difference lies in the nature of the decision boundary: a straight line
for linear SVM versus a curved one enabled by kernel mapping for non-linear
SVM.
Linear SVM
Data Type: Ideal for datasets where data points can be cleanly separated by a
single straight line (or hyperplane in higher dimensions).
Decision Boundary: Finds a straight-line hyperplane that maximizes the
margin between the classes.
Mechanism: Directly applies the classification function to the input data to
find the optimal separating hyperplane.
Non-Linear SVM
Data Type: Used when the data is not linearly separable and cannot be divided by
a straight line.
Decision Boundary: Creates a curved or complex decision boundary by
transforming the data.
Mechanism: Employs kernel tricks (like the Radial Basis Function or RBF kernel) to
map the original data into a higher-dimensional feature space where it becomes
linearly separable.
The algorithm then finds a linear hyperplane in this new, higher-dimensional
space, which corresponds to a non-linear boundary in the original space.
When to Use Which
Choose Linear SVM: when the data shows a clear, linear trend, allowing for
efficient classification with a straight-line boundary. Choose Non-Linear SVM: for
more complex datasets, such as those with circular or intertwined patterns, where
a simple straight line cannot effectively separate the classes
How SVMs Work
Data Transformation: For non-linearly separable data, SVMs use kernel functions
(like linear, quadratic, or radial basis functions) to transform the original data into a
higher-dimensional "feature space".
This transformation makes it easier to find a linear boundary that separates the
classes.
Finding the Hyperplane: Within this higher-dimensional space, the algorithm seeks
a hyperplane—a boundary that best separates the data points into their respective
classes.
Maximizing the Margin: The "optimal" hyperplane is the one with the largest
possible margin.
Identifying Support Vectors: The data points that are closest to the hyperplane and
define the boundaries of the margin are called support vectors.
Classification: Once the hyperplane is established, SVMs can classify new, unknown
data points by determining which side of the hyperplane they fall on.
Margin in Support Vector Machines (SVMs)
 It aim to find a decision boundary (hyperplane) that
maximally separates data points belonging to different
classes.
 In SVMs, the main idea is the margin.
 The margin signifies the distance between the decision
boundary and the closest data points from each class.
 Maximizing this margin is essential as it provides a measure
of robustness against noise and aids in better generalization
to unseen data.
 Two approaches to margins in SVMs are Hard Margin and Soft
Margin.
Hard Margin SVM
 In a hard margin SVM, the objective is to identify a hyperplane that
completely separates data points belonging to different classes,
ensuring a clear demarcation with the utmost margin width possible.
 This margin is the distance between the hyperplane and the nearest
data point, also known as the support vectors.
 The hyperplane equation plays a crucial role in hard margin SVMs
because it defines the boundary that separates the classes. Ideally,
we want this boundary to have a maximum margin from the nearest
data points of each class.
 The objective function in hard margin SVM aims to find the weight
vector and bias term that maximize this margin while ensuring that all
data points are correctly classified.
Hard Margin SVM
The decision boundary is solely determined by these support
vectors, and any data points falling on the wrong side of the
hyperplane contribute to the margin violation.
In the below image, the hyperplane is defined by black solid
line and the dashed lines on both the sides of hyperplane are
margin.
The data points falling on the margin are support vectors. The
image displays a hard margin scenario, here there is no data
point falling between the margins hence, ensuring perfect
separation.
Mathematically, for a linearly separable dataset, the decision function of a hard margin SVM
can be expressed as:
 The decision boundary is determined by the
hyperplane equation 𝑥 𝑇 𝜔 + 𝑏 = 0.
 This equation essentially states that any data point
lying on the hyperplane satisfies this equation.
 In other words, the dot product of the feature vector
and the weight vector, shifted by the bias term,
equals zero for points on the decision boundary.
 The margin is given by the distance between the
hyperplane and the closest data point of each class, which
can be computed as:

 where ||w|| denotes the norm (magnitude) of the weight

vector.
 A larger weight vector norm corresponds to a wider
margin, indicating a clearer separation between classes.
Soft Margin SVM
 Soft Margin SVM introduces flexibility by allowing some margin violations
(misclassifications) to handle cases where the data is not perfectly
separable.
 Suitable for scenarios where the data may contain noise or outliers. It
Introduces a penalty term for misclassifications, allowing for a trade-off
between a wider margin and a few misclassifications.
 Soft margin SVM allows for some margin violations, meaning that it
permits certain data points to fall within the margin or even on the wrong
side of the decision boundary.
 This adaptability is managed by a factor called C, also called the
"regularization parameter," which helps find a balance between making
the gap as big as possible and reducing mistakes in grouping things.
Soft Margin SVM
 In the image, we can observe some data points falling within the margin and
also on the wrong side of the hyperplane, these violations are permissible in
a soft margin setting.
 Despite these violations, the data points situated on the margins or
transgressing them remain pivotal and are typically identified as support
vectors.
 Mathematically, decision function of a soft margin SVM can defined as:

 The equation introduces slack variables 𝜉𝑖 to allow for some margin violations
(misclassifications).
 The term 1 − 𝜉𝑖 represents the minimum required margin for each data point.
 If a point falls within the margin or on the wrong side, its corresponding value becomes
positive, reflecting the amount of violation.
 The objective function of a soft margin SVM combines the margin maximization with a
penalty term for margin violations, minimizing:

Where:
 The parameter C in Support Vector Machines serves as a regularization parameter,
dictating the balance between the width of the margin and the tolerance for
classification errors.
 N is the number of data points.
 The equation combines the objectives of maximizing the margin (represented by the
first term) and minimizing the penalty for margin violations (represented by the second
term).
 The regularization parameter (C) controls the trade-off between these objectives. A
higher C value prioritizes a wider margin, even if it allows some
misclassifications. Conversely, a lower C value allows for more margin violations to
achieve a smoother decision boundary.
Support Vector Machine (SVM)
High C Value:
Effect: Prioritizes minimizing misclassifications on the training data.
Margin: The margin becomes narrower to accommodate more points
correctly.
Decision Boundary: The decision boundary becomes more complex and
can become very tight around the data points, leading to a higher risk of
overfitting.
Bias-Variance: Decreases bias but increases variance.
Low C Value:
 Effect: Allows for more misclassifications on the training
data.
 Margin: The margin becomes wider, leading to a more
generalized decision boundary.
 Decision Boundary: The decision boundary is smoother and
less sensitive to individual data points and outliers.
 Bias-Variance: Increases bias but reduces variance, which helps
to prevent overfitting.
Choosing the Right C:
 The appropriate value of C depends on the specific dataset.
 If your data is clean and well-separated, a higher C might work.
 If your data is noisy or has outliers, a lower C is often better to
achieve a more generalized and robust model.
 Proper tuning of C is a vital step in best practice for SVMs to avoid
both underfitting and overfitting
THANK YOU