12/5/25, 3:54 PM Support Vector Machine (SVM) Algorithm - GeeksforGeeks

We tested this page and blocked content coming from

Search... potentially dangerous or risky sites. Allow this content ViewSign
all In
block
only if you're sure it comes from safe sites.

Support Vector Machine (SVM) Algorithm

Last Updated : 13 Nov, 2025

Support Vector Machine (SVM) is a supervised machine learning

algorithm used for classification and regression tasks. It tries to find
the best boundary known as hyperplane that separates different
classes in the data. It is useful when you want to do binary
classification like spam vs. not spam or cat vs. dog.

1/3

The main goal of SVM is to maximize the margin between the two
classes. The larger the margin the better the model performs on new
and unseen data.

Key Concepts of Support Vector Machine

Hyperplane: A decision boundary separating different classes in
feature space and is represented by the equation wx + b = 0 in
linear classification.
Support Vectors: The closest data points to the hyperplane, crucial
for determining the hyperplane and margin in SVM.
Margin: The distance between the hyperplane and the support
vectors. SVM aims to maximize this margin for better classification

[Link] 1/15
12/5/25, 3:54 PM Support Vector Machine (SVM) Algorithm - GeeksforGeeks

performance. We tested this page and blocked content coming from

potentially dangerous or risky sites. Allow this content
Kernel: A only maps
function that if you'redata
sure ittocomes from safe sites.
a higher-dimensional space
enabling SVM to handle non-linearly separable data.
Hard Margin: A maximum-margin hyperplane that perfectly
separates the data without misclassifications.
Soft Margin: Allows some misclassifications by introducing slack
variables, balancing margin maximization and misclassification
penalties when data is not perfectly separable.
C: A regularization term balancing margin maximization and
misclassification penalties. A higher C value forces stricter penalty
for misclassifications.
Hinge Loss: A loss function penalizing misclassified points or
margin violations and is combined with regularization in SVM.
Dual Problem: Involves solving for Lagrange multipliers associated
with support vectors, facilitating the kernel trick and efficient
computation.

How does Support Vector Machine Algorithm Work?

The key idea behind the SVM algorithm is to find the hyperplane that
best separates two classes by maximizing the margin between them.
This margin is the distance from the hyperplane to the nearest data
points (support vectors) on each side.

Multiple hyperplanes separate the data from two classes

The best hyperplane also known as the "hard margin" is the one that
maximizes the distance between the hyperplane and the nearest data
[Link] 2/15
12/5/25, 3:54 PM Support Vector Machine (SVM) Algorithm - GeeksforGeeks

points from both We

classes.
tested This ensures
this page a clearcontent
and blocked separation between
coming from the
potentially dangerous or risky sites. Allow this content
classes. So from only if figure,
the above you're sure
weitchoose
comes from
L2 as safe [Link].
hard Let's
consider a scenario like shown below:

Selecting hyperplane for data with outlier

Here, we have one blue ball in the boundary of the red ball.

How does SVM classify the data?

The blue ball in the boundary of red ones is an outlier of blue balls.
The SVM algorithm has the characteristics to ignore the outlier and
finds the best hyperplane that maximizes the margin. SVM is robust to
outliers.

Hyperplane which is the most optimized one

[Link] 3/15
12/5/25, 3:54 PM Support Vector Machine (SVM) Algorithm - GeeksforGeeks

A soft margin allows for some

We tested misclassifications
this page or violations
and blocked content coming fromof the
potentially dangerous or risky sites. Allow this content
margin to only if you're sure
improve generalization. Theit SVM
comesoptimizes
from safe sites.
the following
equation to balance margin maximization and penalty minimization:
1
Objective Function = ( margin ) + λ ∑ penalty
​

The penalty used for violations is often hinge loss which has the
following behavior:

If a data point is correctly classified and within the margin there is

no penalty (loss = 0).
If a point is incorrectly classified or violates the margin the hinge
loss increases proportionally to the distance of the violation.

Till now we were talking about linearly separable data that seprates
group of blue balls and red balls by a straight line/linear line.

What if data is not linearly separable?

When data is not linearly separable i.e it can't be divided by a straight
line, SVM uses a technique called kernels to map the data into a
higher-dimensional space where it becomes separable. This
transformation helps SVM find a decision boundary even for non-
linear data.

Original 1D dataset for classification

A kernel is a function that maps data points into a higher-dimensional

space without explicitly computing the coordinates in that space. This
allows SVM to work efficiently with non-linear data by implicitly
[Link] 4/15
12/5/25, 3:54 PM Support Vector Machine (SVM) Algorithm - GeeksforGeeks

performing the mapping.

We testedFor
this example consider
page and blocked datacoming
content pointsfrom
that are not
potentially dangerous or risky sites. Allow this content
linearly only if you're
separable. By applying a sure it comes
kernel from SVM
function safe sites.
transforms the
data points into a higher-dimensional space where they become
linearly separable.

Linear Kernel: For linear separability.

Polynomial Kernel: Maps data into a polynomial space.
Radial Basis Function (RBF) Kernel: Transforms data into a space
based on distances between data points.

Mapping 1D data to 2D to become able to separate the two classes

In this case the new variable y is created as a function of distance from

the origin.

Mathematical Computation of SVM

Consider a binary classification problem with two classes, labeled as
+1 and -1. We have a training dataset consisting of input feature
vectors X and their corresponding class labels Y. The equation for the
linear hyperplane can be written as:

wT x + b = 0
Where:

w is the normal vector to the hyperplane (the direction

perpendicular to it).
b is the offset or bias term representing the distance of the
hyperplane from the origin along the normal vector w .

[Link] 5/15
12/5/25, 3:54 PM Support Vector Machine (SVM) Algorithm - GeeksforGeeks

Distance from aWe tested

Data this page
Point and blocked
to the Hyperplanecontent coming from
potentially dangerous or risky sites. Allow this content
only if you're sure it comes from safe sites.

The distance between a data point xi and the decision boundary can ​

be calculated as:

w T xi +b
di = ​

∣∣w∣∣
​

where ||w|| represents the Euclidean norm of the weight vector w.

Linear SVM Classifier

Distance from a Data Point to the Hyperplane:

1 : wT x + b ≥ 0
y^ = {
−1 : wT x + b < 0
​ ​ ​

^ is the predicted label of a data point.

Where y ​

Optimization Problem for SVM

For a linearly separable dataset the goal is to find the hyperplane that
maximizes the margin between the two classes while ensuring that all
data points are correctly classified. This leads to the following
optimization problem:

minimize 12 ​ ​ ∥w ∥2
w,b

Subject to the constraint:

yi (w T xi + b) ≥ 1 f or i = 1, 2, 3, ⋯ , m
​ ​

Where:

yi ​is the class label (+1 or -1) for each training instance.
​

xi ​is the feature vector for the i-th training instance.

​

m is the total number of training instances.

The condition yi (w T xi ​ + b) ≥ 1 ensures that each data point is

correctly classified and lies outside the margin.

[Link] 6/15
12/5/25, 3:54 PM Support Vector Machine (SVM) Algorithm - GeeksforGeeks

We tested
Soft Margin in Linear this page
SVM and blocked content coming from
Classifier
potentially dangerous or risky sites. Allow this content
only if you're sure it comes from safe sites.

In the presence of outliers or non-separable data the SVM allows

some misclassification by introducing slack variables ζi ​. The ​

optimization problem is modified as:

m
minimize 12 ∥w∥2 ​ ​
+ C ∑i=1 ζi ​ ​

w,b

Subject to the constraints:

yi (w T xi + b) ≥ 1 − ζi
​ ​ ​
and ζi ≥ 0
​
for i =
1, 2, … , m
Where:

C is a regularization parameter that controls the trade-off between

margin maximization and penalty for misclassifications.
ζi ​are slack variables that represent the degree of violation of the
​

margin by each data point.

Dual Problem for SVM

The dual problem involves maximizing the Lagrange multipliers

associated with the support vectors. This transformation allows
solving the SVM optimization using kernel functions for non-linear
classification.

The dual objective function is given by:

m m m
maximize 12 ​ ​
∑i=1 ∑j=1 αi αj ti tj K(xi , xj ) − ∑i=1 αi
​ ​ ​ ​ ​ ​ ​ ​ ​ ​

Where:

αi ​are the Lagrange multipliers associated with the ith training

​

sample.
ti ​is the class label for the ith -th training sample.
​

K(xi , xj ) is the kernel function that computes the similarity

​ ​

between data points xi ​and xj ​. The kernel allows SVM to handle

​ ​

non-linear classification problems by mapping data into a higher-

dimensional space.

[Link] 7/15
12/5/25, 3:54 PM Support Vector Machine (SVM) Algorithm - GeeksforGeeks

We tested
The dual formulation this page
optimizes theand blocked content
Lagrange coming
multipliers αfrom
i ​and the
potentially dangerous or risky sites. Allow this content
​

only if
support vectors are those you're sure
training it comeswhere
samples from safe
αi sites.
> 0. ​

SVM Decision Boundary

Once the dual problem is solved, the decision boundary is given by:
m
w = ∑i=1 αi ti K(xi , x) + b
​ ​ ​ ​

Where w is the weight vector, x is the test data point and b is the bias
term. Finally the bias term b is determined by the support vectors,
which satisfy:

ti (w T xi − b) = 1
​ ​ ⇒ b = w T xi − ti ​ ​

Where xi ​is any support vector.

​

This completes the mathematical framework of the Support Vector

Machine algorithm which allows for both linear and non-linear
classification using the dual problem and kernel trick.

Types of Support Vector Machine

Based on the nature of the decision boundary, Support Vector
Machines (SVM) can be divided into two main parts:

Linear SVM: Linear SVMs use a linear decision boundary to

separate the data points of different classes. When the data can be
precisely linearly separated, linear SVMs are very suitable. This
means that a single straight line (in 2D) or a hyperplane (in higher
dimensions) can entirely divide the data points into their respective
classes. A hyperplane that maximizes the margin between the
classes is the decision boundary.

Non-Linear SVM: Non-Linear SVM can be used to classify data

when it cannot be separated into two classes by a straight line (in
the case of 2D). By using kernel functions, nonlinear SVMs can
handle nonlinearly separable data. The original input data is
transformed by these kernel functions into a higher-dimensional
feature space where the data points can be linearly separated. A

[Link] 8/15
12/5/25, 3:54 PM Support Vector Machine (SVM) Algorithm - GeeksforGeeks

linear SVM is used to locate

We tested a nonlinear
this page decision
and blocked boundary
content coming fromin this
potentially dangerous or risky sites. Allow this content
modified space. only if you're sure it comes from safe sites.

Implementing SVM Algorithm Using Scikit-Learn

We will predict whether cancer is Benign or Malignant using historical
data about patients diagnosed with cancer. This data includes
independent attributes such as tumor size, texture, and others. To
perform this classification, we will use an SVM (Support Vector
Machine) classifier to differentiate between benign and malignant
cases effectively.

load_breast_cancer(): Loads the breast cancer dataset (features

and target labels).
SVC(kernel="linear", C=1): Creates a Support Vector Classifier
with a linear kernel and regularization parameter C=1.
[Link](X, y): Trains the SVM model on the feature matrix X and
target labels y.
DecisionBoundaryDisplay.from_estimator(): Visualizes the
decision boundary of the trained model with a specified color map.
[Link](): Creates a scatter plot of the data points, colored by
their labels.
[Link](): Displays the plot to the screen.

from [Link] import load_breast_cancer

import [Link] as plt
from [Link] import DecisionBoundaryDisplay
from [Link] import SVC

cancer = load_breast_cancer()
X = [Link][:, :2]
y = [Link]

svm = SVC(kernel="linear", C=1)

[Link](X, y)

DecisionBoundaryDisplay.from_estimator(
svm,
X,
response_method="predict",
alpha=0.8,
cmap="Pastel1",
xlabel=cancer.feature_names[0],
ylabel=cancer.feature_names[1],
)
[Link] 9/15
12/5/25, 3:54 PM Support Vector Machine (SVM) Algorithm - GeeksforGeeks

We tested this page and blocked content coming from

potentially
[Link](X[:, 0], X[:, 1], dangerous or risky sites. Allow this content
c=y, only if you're sure it comes from safe sites.
s=20, edgecolors="k")
[Link]()

Output:

SVM

Advantages of Support Vector Machine (SVM)

1. High-Dimensional Performance: SVM excels in high-dimensional
spaces, making it suitable for image classification and gene
expression analysis.
2. Nonlinear Capability: Utilizing kernel functions like RBF and
polynomial SVM effectively handles nonlinear relationships.
3. Outlier Resilience: The soft margin feature allows SVM to ignore
outliers, enhancing robustness in spam detection and anomaly
detection.
4. Binary and Multiclass Support: SVM is effective for both binary
classification and multiclass classification suitable for applications
in text classification.
5. Memory Efficiency: It focuses on support vectors making it memory
efficient compared to other algorithms.

[Link] 10/15
12/5/25, 3:54 PM Support Vector Machine (SVM) Algorithm - GeeksforGeeks

We tested this page and blocked content coming from

Disadvantages of Support
potentially dangerousVector Machine
or risky sites. (SVM)
Allow this content
only if you're sure it comes from safe sites.
1. Slow Training: SVM can be slow for large datasets, affecting
performance in SVM in data mining tasks.
2. Parameter Tuning Difficulty: Selecting the right kernel and
adjusting parameters like C requires careful tuning, impacting SVM
algorithms.
3. Noise Sensitivity: SVM struggles with noisy datasets and
overlapping classes, limiting effectiveness in real-world scenarios.
4. Limited Interpretability: The complexity of the hyperplane in
higher dimensions makes SVM less interpretable than other
models.
5. Feature Scaling Sensitivity: Proper feature scaling is essential,
otherwise SVM models may perform poorly.

Support Vector Regression Intuition Visit Course

Comment A aswat… Follow 102

Article Tags : Machine Learning AI-ML-DS python

[Link] 11/15