CH-4

Learning with Classifiation

[Link] Vector Machine
4.2 SVM Regression,
Constraint Optimization
P—rocess of optimizing an objective function with respect to some
variables.
O—bjective function can be cost or energy function which is to be
minimized
—Objective function can be reward or utility function which is to be
maximised.
Eg finding best weight of Chair
Designing the fastest vehicle with a constraint on fuel efficiency
 Solution Methods
1. Equality Constraint- find the maxima of f(x1,x2,x3) subject to
x1+x2+x3=1
. Eg length of the wire given, what will be figure that have maximum
or minimum area

1. Inequality Constraint- minimize function f(x1,x2,x3) subject to

x1+x2+x3<1. Eg designing fastest vehicle with fuel consumption less
than

target value
 Supervised vs. Unsupervised
■ Learning
Supervised learning (classification)
■ Supervision: The training data (observations, measurements,
etc.) are accompanied by labels indicating the class of the
observations

■ New data is classified based on the training set.

■ Unsupervised learning (clustering)
■ The class labels of training data is unknown
■ Given a set of measurements, observations, etc. with the aim of
establishing the existence of classes or clusters in the data
5
Support Vector Machine(SVM)
⚫ Support Vector Machine (SVM) is a supervised machine
learning algorithm used for classification and regression.
⚫ The main objective of the SVM algorithm is to find the
optimal hyper-plane in an N-dimensional space that can
separate the data points in different classes in the feature
space.
⚫ The hyper-plane tries that the margin between the closest
points of different classes should be as maximum as
possible.
n-dimensional space: A hyperplane exists within an n-dimensional space (like a
line in 2D, a plane in 3D, etc.).

Normal vector (w): This vector is perpendicular to the hyperplane.

Point on the hyperplane (x): Any point that satisfies the equation lies on the
hyperplane.

Offset (b): This scalar determines the position of the hyperplane relative to the
origin.
 SVM/Maximal Margin Classifier
Support Vectors: Support vectors are the data points that lie farthest to the decision surface (or
hyperplane). They are the data points which are most difficult to classify. They have a direct impact on the
optimum location of the decision surface.

Hyperplane: A hyperplane in Machine Learning is a crucial deciding boundary that partitions the input space into two
or
more sections, with each section corresponding to a unique class or output label. A hyperplane manifests as a straight line
that bifurcates the space into two equal halves in a 2D space.
Margin: In Support Vector Machines (SVMs), the margin is the distance between the decision boundary

(hyperplane) and the closest data points from each class.

In one-dimensional space, the separating hyperplane is a point
In two dimensions, the hyperplane that separates class A and class B is a line:
 And in three dimensions, the separating hyperplane is a plane.
Similarly in N dimensions the separating hyperplane will be an (N-1)-
dimensional subspace.
 Hard Margins and Soft Margins
● The margin does not have any misclassified data point. Such a classifier is called a hard margin
classifier.
● Classifier have a misclassified data point inside the margin. Such a classifier that allows minimal
misclassifications is a soft margin classifier.
 Linearity and Non-Linearity: On a Plane, a linear function is a function
where the graph is a straight line. The line can go in any direction, but it’s
always a straight line. However, a non-linear function has a shape that is not a
straight line.
 Kernel Tricks
● SVM has a technique called the kernel trick. These are functions that take low
dimensional input space and transform it into a higher-dimensional space i.e.
it converts not separable problem to separable problem.

● To deal with non-linear data, one approach could be to map the input data into
a higher-dimensional space where it is linearly separable. This mapping involves
transforming the data into a new space (feature space) where the separation
between the data points is clearer.

● The kernel trick is a method used in SVMs to enable them to classify non-linear
data using a linear classifier. By applying a kernel function, SVMs can implicitly map
input data into a higher-dimensional space
Types of Kernel Functions/Kernal Tricks

● Linear Kernel: No mapping is needed as the data is already assumed to be linearly

separable.
● Polynomial Kernel: Maps inputs into a polynomial feature space, enhancing the
classifier’s ability to capture interactions between features. It uses
Quardratic equations
● Radial Basis Function (RBF) Kernel: Also known as the Gaussian kernel, it is
useful for capturing complex regions by considering the distance between points in the
input space.
Polynomial Kernel/Quadratic Function
Polynomial Kernel/Quadratic Function
 Radial Basis Function
RBF kernels are the most generalized form of kernelization and are one of the most widely
used kernels due to their similarity (how close they are to each other) to the Gaussian
distribution. Mathematically, RBF
 Radial Basis Function
● RBF is a function that takes a vector and output a scalar based on vector distance. This
distance can be either from (0,0) or from another vector.
● This gaussian version maps the data from its feature space to higher feature space(
Infinite dimension space)
SVM as Constrained Optimisation
SVM as Constrained Optimisation
SVM as Constrained Optimisation
SVM as Constrained Optimisation
SVM as Constrained Optimisation
SVM as Constrained Optimisation
SVM as Constrained Optimisation
SVM as Constrained Optimisation
Support Vector Regression
Support Vector Regression
Support Vector Regression
Multi-class classification
—multiclass classification or multinomial
classification is the problem of classifying
instances into one of three or more classes
(classifying instances into one of two classes
is called binary classification).
Multi-class classification Approach
L—oad dataset from source
L—oad dataset into training and test data
T—rain decision tree, SVM and KNN classifiers
on the training data
U—se above classifiers to predict labels for the
test data.
—Measure accuracy and visualize classification
Categories of Multi-class

multi-class classification techniques can be

categorised into
t—ransformation to
binary e—xtension from
binary h—ierarchical
classification
 Transformation to binary
R—educing the problem of multiclass
classification to multiple binary classification
problems. It can be categorized into
i) one vs rest
ii) one vs one
No. of Classifiers/Model=n(n-1)/2
● Multiclass classification is a machine learning task where the goal is to assign
instances to one of multiple predefined classes or categories, where each instance
belongs to exactly one class.
● Whereas multilabel classification is a machine learning task where each instance can
be associated with multiple labels simultaneously, allowing for the assignment of
multiple binary labels to the instance.
For example, if you look in the above image, then you may see that this movie has been rated as ‘U/A’ (meaning
‘Parental Guidance for children below the age of 12 years) certificate. This is not the only type of certificate

but there are other types of certificates classes such as,

● ‘A’ (Restricted to adults), or

● ‘U’ (Unrestricted Public Exhibition),

but while categorizing the movies based on this, it is sure that each movie can only be categorized with only one
out of those three types of certificates. In short, there are multiple categories (i.e, multiple certificates assigned

to the movie) but each instance is assigned only one (i.e, each movie is assigned with only one certificate at

once)
multi-class classification problem statement.

Again, if you see carefully the image, then this movie has been categorized into the comedy and romance
genres. But this time there is a difference that each of the movies can fall into one or more different sets of

categories (i.e, have more than one genre). Therefore, each instance can be assigned with multiple categories

(i.,e multiple genres), so these types of problems are categorized under the multi-label classification problem

statement, where we have a set of target labels for each of the samples.