BE COMPUTER

SUBJECT : MACHINE
LEARNING
[Link]. [Link] Mhatre
Module 4 – Learning with
Classification
 Support Vector Machine (SVM) Algorithm

● Support Vector Machine (SVM) is a powerful machine learning algorithm used

for linear or nonlinear classification, regression, and even outlier detection
tasks.
● SVMs can be used for a variety of tasks, such as text classification, image
classification, email spam detection, handwriting identification, face detection,
and anomaly detection.
● SVMs are adaptable and efficient in a variety of applications because they can
manage high-dimensional data and nonlinear relationships.
● SVM algorithms are very effective as we try to find the maximum separating
hyperplane between the different classes available in the target feature.
● SVM used for both classification and regression. Though we say regression
problems as well it’s best suited for classification.
Support Vector Machine (SVM)
An SVM is a powerful supervised learning algorithm used
mostly for classification. It works by finding a line or
hyperplane that separates different classes with the highest
possible margin.

• Optimal Decision Boundary & Margins: The decision

boundary is the line (or hyperplane in higher dimensions)
that splits the data classes. The margin is the distance
between this boundary and the closest data points from
either class. SVM tries to maximize this margin to ensure
the model generalizes well to new data.
• Support Vectors: These are the critical data points that
lie closest to the decision boundary. If you remove or
move these points, the position of the boundary
changes. They literally "support" and define the margin.
• Constrained Optimization & Quadratic Programming:
Finding the maximum margin requires solving a
mathematical optimization problem. Because the
objective function is quadratic (squaring the weights to
find distance) and the data separation conditions act as
linear constraints, it is solved using a technique called
Quadratic Programming (QP).
• Linear vs. Nonlinear Classification: Linear SVM works
when data can be neatly split by a straight line. When
data is tangled or overlapping, a straight line won't work,
requiring nonlinear classification methods.
• Basics of Kernel Trick: To handle nonlinear data, the
Kernel Trick mathematically projects the
low-dimensional data into a much higher-dimensional
space where it becomes linearly separable. It does this
efficiently without ever explicitly calculating the
coordinates in that high-dimensional space.
Extensions of SVM
• Support Vector Regression (SVR): While traditional SVM
predicts distinct categories (like "Red" vs. "Blue"), SVR
adapts the same margin-maximization principles to predict
continuous numerical values (like housing prices or stock
trends).
• Multiclass Classification: Standard SVMs only inherently
split data into two groups (binary). To handle multiple
classes, strategies like One-vs-One (OvO) or
One-vs-Rest (OvR) are used to combine multiple binary
SVM models together.
Example:
● Suppose we see a strange cat that also has some features of dogs, so if we want a model that
can accurately identify whether it is a cat or dog, so such a model can be created by using the
SVM algorithm.
● We will first train our model with lots of images of cats and dogs so that it can learn about
different features of cats and dogs, and then we test it with this strange creature.
● So as support vector creates a decision boundary between these two data (cat and dog) and
choose extreme cases (support vectors), it will see the extreme case of cat and dog.
● On the basis of the support vectors, it will classify it as a cat. Consider the below diagram:
● The main objective of the SVM algorithm is to find the optimal
hyperplane in an N-dimensional space that can separate the data
points in different classes in the feature space.
● The hyperplane tries that the margin between the closest points of
different classes should be as maximum as possible.
● The dimension of the hyperplane depends upon the number of
features.
○ If the number of input features is two, then the hyperplane is
just a line.
○ If the number of input features is three, then the hyperplane
becomes a 2-D plane.
○ It becomes difficult to imagine when the number of features
exceeds three. you can call it a hyperplane
Let’s consider two independent variables x1, x2, and one dependent
variable which is either a blue circle or a red circle.

● From the figure above it’s very clear

that there are multiple lines (our
hyperplane here is a line because we
are considering only two input
features x1, x2) that segregate our
data points or do a classification
between red and blue circles.
● So how do we choose the best line
or in general the best hyperplane
that segregates our data points?
 Types of SVM Non-linear SVM: Non-Linear SVM is used
Linear SVM: Linear SVM is used for linearly for non-linearly separated data, which means if
separable data, which means if a dataset can be a dataset cannot be classified by using a
classified into two classes by using a single straight
line, then such data is termed as linearly separable straight line, then such data is termed as
data, and classifier is used called as Linear SVM non-linear data and classifier used is called as
classifier. Non-linear SVM classifier.
Hyperplane and Support Vectors in the SVM algorithm:
Hyperplane:
There can be multiple lines/decision boundaries to segregate the classes in n-dimensional space, but we
need to find out the best decision boundary that helps to classify the data points. This best boundary is
known as the hyperplane of SVM.
The dimensions of the hyperplane depend on the features present in the dataset, which means if there are
2 features (as shown in image), then hyperplane will be a straight line. And if there are 3 features, then
hyperplane will be a 2-dimension plane.
We always create a hyperplane that has a maximum margin, which means the maximum distance
between the data points.

Support Vectors:
The data points or vectors that are the closest to
the hyperplane and which affect the position of the
hyperplane are termed as Support Vector. Since
these vectors support the hyperplane, hence called
a Support vector.
How does it work: So we choose the hyperplane whose distance
One reasonable choice as the best hyperplane is from it to the nearest data point on each side is
the one that represents the largest separation maximized.
or margin between the two classes.
If such a hyperplane exists it is known as the
maximum-margin hyperplane/hard margin. So
from the above figure,
Here we have one blue ball in the boundary of
the red ball. So how does SVM classify the ● So in this type of data point what SVM does
data? It’s simple! The blue ball in the is, finds the maximum margin as done with
boundary of red ones is an outlier of blue previous data sets along with that it adds a
balls. The SVM algorithm has the penalty each time a point crosses the margin.
characteristics to ignore the outlier and finds ● So the margins in these types of cases are
the best hyperplane that maximizes the
called soft margins.
margin. SVM is robust to outliers.
● When there is a soft margin to the data set,
the SVM tries to minimize (1/margin+∧
(∑penalty)).
● Hinge loss is a commonly used penalty.
● If no violations no hinge loss.
Support Vector Machine Terminology
● Hyperplane: Hyperplane is the decision ● Kernel: Kernel is the mathematical function,
boundary that is used to separate the data which is used in SVM to map the original
points of different classes in a feature space. input data points into high-dimensional
In the case of linear classifications, it will be feature spaces, so, that the hyperplane can
a linear equation i.e. wx+b = 0. be easily found out even if the data points
● Support Vectors: Support vectors are the are not linearly separable in the original
closest data points to the hyperplane, which input space. Some of the common kernel
makes a critical role in deciding the
hyperplane and margin. functions are linear, polynomial, radial basis
● Margin: Margin is the distance between the function(RBF), and sigmoid.
support vector and hyperplane. The main ● Hard Margin: The maximum-margin
objective of the support vector machine hyperplane or the hard margin hyperplane is
algorithm is to maximize the margin. The a hyperplane that properly separates the data
wider margin indicates better classification points of different categories without any
performance. misclassifications.
 Support Vector Machine Terminology
● Soft Margin: When the data is not perfectly separable or contains
outliers, SVM permits a soft margin technique. Each data point has a
slack variable introduced by the soft-margin SVM formulation, which
softens the strict margin requirement and permits certain
misclassifications or violations. It discovers a compromise between
increasing the margin and reducing violations.
● Hinge Loss: A typical loss function in SVMs is hinge loss. It punishes
incorrect classifications or margin violations. The objective function in
SVM is frequently formed by combining it with the regularisation term.
Let’s consider a two-dimensional space. The two-dimensional linearly separable
data can be separated by the equation of a line—with the data points lying on either
sides representing the respective classes.

The function of the line is y=ax+by. Considering x and y as features and naming
them as x1,x2….xn, it can be re-written as:

If we define x = (x1, x2) and w = (a, −1), we get:

This equation is derived from two-dimensional vectors. But in fact, it also works
for any number of dimensions. This is the equation for a hyperplane:
The equation for the hyperplane can be written as:

● The vector W represents the normal vector to the hyperplane. i.e the direction perpendicular to the
hyperplane.
● The parameter b in the equation represents the offset or distance of the hyperplane from the origin
along the normal vector w.

The distance between a data point x_i and the decision boundary
can be calculated as:

● where ||w|| represents the Euclidean form of the weight vector w.

The hypothesis function h is defined as:

The point above or on the hyperplane will be classified as class +1, and the point below the
hyperplane will be classified as class -1.
SVM implementation in Python

Steps
● Load the breast cancer dataset from [Link]
● Separate input features and target variables.
● BuilD and train the SVM classifiers using RBF kernel.
● Plot the scatter plot of the input features.
● Plot the decision boundary.
● Plot the decision boundary
 # Plot Decision Boundary
# Load the important packages
DecisionBoundaryDisplay.from_estimator(
from [Link] import
svm,
load_breast_cancer
X,
import [Link] as plt
response_method="predict",
from [Link] import
cmap=[Link],
DecisionBoundaryDisplay
alpha=0.8,
from [Link] import SVC
xlabel=cancer.feature_names[0],
# Load the datasets
ylabel=cancer.feature_names[1],
cancer = load_breast_cancer()
)
X = [Link][:, :2]
y = [Link]
# Scatter plot
#Build the model
[Link](X[:, 0], X[:, 1],
svm = SVC(kernel="rbf",
c=y,
gamma=0.5, C=1.0)
s=20, edgecolors="k")
# Trained the model
[Link]()
svm.fit(X, y)
Kernel Trick / Kernel Function

● Kernel Function is a method used to take data as input and transform it

into the required form of processing data. “Kernel” is used due to a set of
mathematical functions used in Support Vector Machine providing the
window to manipulate the data.
● So, Kernel Function generally transforms the training set of data so that a
non-linear decision surface is able to transform to a linear equation in a
higher number of dimension spaces.
● Basically, It returns the inner product between two points in a standard
feature dimension.
● In SVMs, there can be a lot of new dimensions, each of them
possibly involving a complicated calculation.
● Doing this for every vector in the dataset can be a lot of work.
● Here’s a trick: SVM doesn’t need the actual vectors to work its
magic—it actually can get by with only the dot products
between them.
Imagine the new space we want is:
● We need to figure out what the dot product in that space looks
like:
● a · b = xa · xb + ya · yb + za · zb
● a · b = xa · xb + ya · yb + (xa² + ya²) · (xb² + yb²)
● A linear kernel is a simple dot product between two input
vectors,
● while a non-linear kernel is a more complex function that can
capture more intricate patterns in the data.
● The choice of kernel depends on the data’s characteristics and
the task’s complexity.
● The scikit-learn library in Python provides us a lot of options to
choose as our kernel, and we can design our own custom
kernels too.
● Kernel Function is a method used to take data as input and transform it into the
required form of processing data.
● “Kernel” is used due to a set of mathematical functions used in Support Vector
Machine providing the window to manipulate the data.
● So, Kernel Function generally transforms the training set of data so that a
non-linear decision surface is able to transform to a linear equation in a higher
number of dimension spaces.
● Basically, It returns the inner product between two points in a standard feature
dimension.
Standard Kernel Function Equation :
1
0 (means its parallel to y-axis)

0
1(means its parallel to x-axis)

1
1(means its 45 degree with respect to x and y-axis)
Support vector regression (SVR)
● Support vector regression (SVR) is a type of support vector machine
(SVM) that is used for regression tasks.
● It tries to find a function that best predicts the continuous output value for
a given input value.