Rajiv Gandhi University of Knowledge Technologies

Catering to the Educational Needs of Gifted Rural Youth of Andhra Pradesh

(Established by the Govt. of Andhra Pradesh and recognized as per Section 2(f) of UGC Act, 1956)
Rajiv Knowledge Valley Campus

Department of Computer Science and Engineering

Machine Learning
Day-3
Presented by
R Sreenivas
Assistant Professor
RGUKT RK Valley
© Copyright 2025 ® All rights are reserved
 Agenda

01 Support Vector Machine

02 Formulation of SVM

03 SVM Classifier Algorithm

04 Hard Margin Vs Soft Margin

04 Kernel function and types

 Support Vector Machine
➢ Support Vector Machine (SVM) is supervised learning algorithm ML
which used for classification as well as regression problems
➢ However, mainly , it is used for classification problems in ML
ML
➢ The goal of SVM algorithm is to create the best line or decision
boundary that can be segregate n-dimensional space into classes
so that we can easily put the new data point in the correct
ML
category in the feature.
 Support Vector Machine
➢ This best decision boundary is called a “Hyper plane” ML
➢ We always create a hyper plane that has maximum margin which
means the maximum distance between support vectors.
ML
➢ The closest data points to the hyper plane is called “Support
Vectors”.
➢ The dimensions of the hyper plane depends on the features
ML
present in the dataset which means , if there are two features
then hyper plane will be straight line
Support Vector Machine

Hyper plane

Support Vectors

Margin
 Types of SVM
Linear SVM Non - Linear SVM ML

➢ Linear SVM is used for linearly separable data, which means if

dataset can be classified into two classes by using a single straight
ML
line , then such data is termed as linearly separable data and
classifier is used called “Linear SVM classifier”
➢ Non-linear SVM is used for non-linearly separable data, which ML
means if dataset cannot be classified by using a straight line , then
such data is termed as non-linear data and classifier is used called
“Non-linear SVM classifier”
Linear SVM Non - Linear SVM
 Formulation of SVM
➢ Assume that we have a two-class training dataset of N points of the
form(x⃗ 1, y1), (x⃗2, y2), . . . , (x⃗N, yN).
where the yi’s – is output labels. Each x⃗i is a n-dimensional real vector.
➢ We assume that the dataset is linearly separable.
➢ Any hyperplane can be written as the set of points x⃗ = (x 1, . . . , xn)
satisfying an equation of
the form w⃗ ⋅ x⃗ − b = 0
➢ Since the training data is linearly separable, we can select two parallel
hyperplanes that separate the two classes of data, so that the distance
between them is as large as possible
w⃗ ⋅ x⃗ − b = +1 ------------> 1
w⃗ ⋅ x⃗ − b = −1 ------------> 2
 Formulation of SVM

w⃗ ⋅ x⃗ − b = +1
w⃗ ⋅ x⃗ − b = 0
w⃗ ⋅ x⃗ − b = -1
➢ For any point on or “above” the hyperplane -> 1 , the class label is +1
w⃗ ⋅ x⃗ i − b ≥ +1, if yi = +1 ---------> 3
➢ For any point on or “below” the hyperplane -> 2, the class label is +1
w⃗ ⋅ x⃗ i − b ≤ +1, if yi = -1 ---------> 4
from 3 and 4 , we get yi(w⃗ ⋅ x⃗i − b) ≥ 1, for all 1 ≤ i ≤ N
 Formulation of SVM
x⃗

d⃗
x p⃗

Now , we find the distance between two hyperplanes.

d⃗ = α. w⃗
xp⃗ = x⃗ - d⃗
T
w . xp⃗ - b = 0
The distance between the two hyperplanes is 2/∥ w⃗∥
 Formulation of SVM

To maximize the distance between the planes we have to minimize ∥ w⃗∥ .

∥ w⃗∥ is minimum when 1/ 2 ∥ w⃗∥2 is minimum
Finally SVM is optimization problem
find vectors w⃗ and a number b which
minimize 1/2∥ w⃗∥2
subject to yi(w⃗ ⋅ x⃗i − b) ≥ 1, for i = 1, . . . N
Solution
 Formulation of SVM

Solution:
The vector w⃗ and the scalar b are given by
N
w⃗ = ∑ αi yi xi⃗
i=1

b = 1/2 ( min (w⃗ ⋅ x⃗i) + max (w⃗ ⋅ x⃗i))

i:y I = +1 i:yi = -1

where α⃗ = (α 1 , α 2, . . . , αN) is a vector which maximizes

N N
Ф(αi ) = ∑αi −1/2 ∑ αiαj yiyj(x⃗i ⋅ x⃗j)
i=1 i=1,j=1
subject to
N
∑ αiyi = 0
i=1
αi > 0 for i = 1, 2, . . . , N
 SVM Classifier Algorithm
1. Given a two-class linearly separable dataset of N points of the form
(x⃗ 1 , y 1), (x⃗ 2, y 2), . . . , (x⃗N, yN)

2. Find α⃗ = (α 1 , α 2, . . . , αN) is a vector which maximizes

Ф(αi ) = ∑αi −1/2 ∑ αiαj yiyj(x⃗i ⋅ x⃗j)
subject to
∑ αiyi = 0
αi > 0 for i = 1, 2, . . . , N

3. Compute w⃗ = ∑ αi yi xi⃗

4. Compute b = 1/2 ( min (w⃗ ⋅ x⃗i) + max (w⃗ ⋅ x⃗i))

5. Form the SVM classifier function is f(x) = w⃗ ⋅ x⃗ − b = 0,

where αi is non-zero only if x⃗ is support vector

 SVM classifier example
Example1
Using SVM algorithm , find the SVM classifier for the below data
Example X1 X2 Class
1 2 1 +1
2 4 3 -1

Example2
Using SVM algorithm , find the SVM classifier for the below data
Example X1 X2 Class
1 2 2 -1
2 4 5 +1
3 7 4 +1
 Hard Margin Vs Soft Margin

The goal is to find the hyperplane The soft margin is concept that allows for
some degree of misclassification or
that can perfectly separate the data overlap between classes in the training
into two classes without any data. To handle this SVM introducing
slack variables that allow some data
misclassification points to be on the wrong side of the
margin.
 Hard Margin Vs Soft Margin
find vectors w⃗ and a number b which find vectors w⃗ and ξ⃗ and a number b which
minimize 1/2∥ w⃗∥2 Minimize 1/2∥ w⃗∥ ^2 + C ∑ξi
subject to subject to yi(w⃗ ⋅ x⃗i − b) ≥ 1 − ξi,
yi(w⃗ ⋅ x⃗i − b) ≥ 1, for i = 1, . . . N for i = 1, . . . N, ξi ≥ 0, for i = 1, . . . , N

The hard margin SVM looks for the the soft margin SVM finds the hyperplane
widest margin possible that maximizes the margin while minimizing
the penalty for misclassification.
 SVM Kernel Function

A kernel function is a function of the form K(x,⃗ y⃗), where x⃗ and y⃗ are
n-dimensional vectors, having a special property. These functions are used to
obtain SVM-like classifiers for two-class datasets which are not linearly
separable.
Let x⃗ and y⃗ be arbitrary vectors in the n-dimensional vector space Rn. Let φ
be a mapping from Rn to some vector space. A function K(x,⃗ y⃗) is called a
kernel function if there is a function φ such that K(x,⃗ y⃗) = φ(x⃗) ⋅ φ(y⃗).
 Some of the SVM Kernel Function
we assume that x⃗ = (x1, x2, . . . , xn) and y⃗ = (y1, y2, . . . , yn).
1. Homogeneous polynomial kernel
K(x,⃗ y⃗) = (x⃗ ⋅ y⃗)^d
where d is some positive integer.
2. Non-homogeneous polynomial kernel
K(x,⃗ y⃗) = (x⃗ ⋅ y⃗ + θ)^d
where d is some positive integer and θ is a real constant.
3. Radial basis function (RBF) kernel
K( x,⃗ y⃗) = e^− ∥ x⃗−y⃗∥ ^2/ 2σ ^2
This is also called the Gaussian radial function kernel
4. Laplacian kernel function
K( x,⃗ y⃗) = e^−∥ x⃗−y⃗∥/σ
5. Hyperbolic tangent kernel function (Sigmoid kernel function)
K( x,⃗ y⃗) = tanh( α( x⃗ ⋅ y⃗) + c)
 Kernel trick ( method)
1. Choose an appropriate kernel function K(x,⃗ y⃗).
2. Formulate and solve the optimization problem obtained by replacing
each inner product x⃗ ⋅ y⃗ by K(x,⃗ y⃗) in the SVM optimization problem.
3. In the formulation of the classifier function for the SVM problem using
the inner products of unclassified data z⃗ and input vectors x⃗i, replace
each inner product z⃗ ⋅ x⃗i with K(z⃗, x⃗i) to obtain the new classifier
function.