Supervised Learning

(Regression/Classification)
Unit - II

Dr. P. Sivakumar
Professor, CSE
 Distance Metrics
❖ Euclidean Distance
❖ Manhattan Distance (Taxicab or City Block Distance)
❖ Minkowski Distance (Generalized Distance Metric)
❖ Cosine Similarity
❖ Hamming Distance (For Categorical or Binary Data)
❖ Mahalanobis Distance (For Correlated Data Features)
 Topics
❖ Basic Methods, Distance based Methods
❖ Nearest Neighbours
❖ Decision Trees
❖ Naive Bayes
❖ Linear Models: Linear Regression, Logistic Regression
❖ Generalized Linear Models
❖ Support Vector Machines
❖ Binary Classification
❖ Multiclass/Structured outputs
❖ MNIST, Ranking.
 Euclidean Distance
❖ It is the straight-line distance between two points in Euclidean space.
❖ Most Machine learning algorithms including K- Means Clustering uses
Euclidean distance in order to calculate the similarity between two data
points or observations.
❖ Formula
Moreover, Euclidean distance is used when we are calculating
the distance between two sets of data that have numerical
data type [Link] or float.
Point to keep in mind that if your data have features with
different scales it is a must to normalize or standardize the
features across all columns before calculation of Euclidean
distance.
If scaling of features is not performed then large values in
features will dominate the distance metrics.
Euclidean distance looks very similar to the Pythagorean
theorem.
It is the most intuitive type of distance measure which can be
used to calculate distances between two different points.
 Manhattan Distance
❖ It is the sum of the absolute differences between
corresponding coordinates.
Manhattan Distance also known as City Block Distance or
Taxicab Distance calculate the distance between two
real-valued vectors.
Mathematically Manhattan distance is calculated as the
sum of absolute distances between two different data
points.
It is mostly used when two vector lies in a uniform grid-like
chess board or city block.
The name taxicab creates an intuition for what measures
the shortest path that a taxicab would take between city
blocks which are coordinates on the grid.
Euclidean distance between two different Manhattan distance between two points
points A and B
 Minkowski Distance
❖ A generalization of both Euclidean and Manhattan
distances.
Minkowski Distance generalizes Euclidean and Manhattan
Distance.
It is also called p-norm vector as it adds a parameter
called the “p” that allows different distance measures to
be calculated.

For Different values of p Minkowski will be-

p=1, the distance measure is the Manhattan measure.
p=2, the distance measure is the Euclidean measure.
p = ∞, the distance measure is the Chebyshev measure
 Cosine Similarity
❖ Measures the cosine of the angle between two vectors.
Cosine similarity basically measures the similarity between
two non-zero vectors.
It is basically the cosine angle between two vectors that are
most similar.
Most similar vectors will have 0 degrees between them hence
the value of cos 0 is 1.
Moreover the vectors opposite to each other have a value of -1
i.e. cos(180deg).
Hence it infers that cosine similarity ranges from -1 to 1
Cosine metric is mainly used in Collaborative Filtering based
recommendation systems to offer future recommendations to
users.
Cosine distance & Cosine Similarity metric is mainly used
to find similarities between two data points.
As the cosine distance between the data points increases,
the cosine similarity, or the amount of similarity decreases,
and vice versa.
Thus, Points closer to each other are more similar than
points that are far away from each other.
Cosine similarity is given by Cos θ, and cosine distance is
1- Cos θ.
In the above image, there are two data points shown in blue, the angle Now if the angle between the two points is 0 degrees in the above
between these points is 90 degrees, and Cos 90 = 0. Therefore, the figure, then the cosine similarity, Cos 0 = 1 and Cosine distance is 1-
shown two points are not similar, and their cosine distance is 1 — Cos 90 Cos 0 = 0. Then we can interpret that the two points are 100%
=1 similar to each other.

In the above figure, imagine the value of θ to be 60 degrees, then

by cosine similarity formula, Cos 60 =0.5 and Cosine distance is
1- 0.5 = 0.5. Therefore the points are 50% similar to each other.
Cosine similarity only cares about the angle between two
vectors hence not the distance between them.

cosine distance = 1-cosine similarity

 Hamming Distance (For Categorical or Binary Data)
❖ Measures the number of positions at which two strings
differ.
❖ Hamming distance is a metric for comparing two binary data
strings.
❖ While comparing two binary strings of equal length, Hamming
distance is the number of bit positions in which the two bits
are different.
❖ The Hamming distance between two strings, a and b is
denoted as d(a,b).
❖ In order to calculate the Hamming distance between two
strings, and, we perform their XOR operation, (a⊕ b), and then
count the total number of 1s in the resultant string.
❖ Suppose there are two strings 11011001 and 10011101.
❖ 11011001 ⊕ 10011101 = 01000100. Since, this contains two 1s,
the Hamming distance, d(11011001, 10011101) = 2.
KNN Algorithm:
K-Nearest Neighbors (KNN) is a simple, non-parametric,
and instance-based learning algorithm used for
classification and regression tasks.
Classification: Assigns a class label based on the
majority of its K nearest neighbors.
Regression: Predicts the value based on the average of
K nearest neighbors.
KNN Algorithm Steps:
1. Select the number of neighbors k.
2. Calculate the distance between the query point and all
other points in the dataset using a chosen distance
metric.
3. Sort the distances in ascending order and select the top
k-nearest neighbors.
4. For classification: Assign the query point the class of the
majority of its neighbors.
5. For regression: Predict the value of the query point as the
average of the k-nearest neighbors.
MNIST
MNIST dataset, which is a set of 70,000 small images of
digits handwritten by high school students and
employees of the US Census Bureau.
Each image is labeled with the digit it represents.
This set has been studied so much that it is often called
the “Hello World” of Machine Learning: whenever people
come up with a new classification algorithm, they are
curious to see how it will perform on MNIST
The following code fetches the MNIST dataset
There are 70,000 images, and each image has 784 features.
This is because each image is 28×28 pixels, and each feature
simply represents one pixel’s intensity, from 0 (white) to 255
(black).
Let’s take a peek at one digit from the dataset.
All you need to do is grab an instance’s feature vector, reshape
it to a 28×28 array, and display it using Matplotlib’s imshow()
function.
Decision Trees
Decision Trees are versatile Machine Learning algorithms that
can perform both classification and regression tasks, and even
multi output tasks.
They are very powerful algorithms, capable of fitting complex
datasets.
One of the many qualities of Decision Trees is that they require
very little data preparation.
In particular, they don’t require feature scaling or centering at
all.
 Support Vector Machine
A Support Vector Machine (SVM) is a very
powerful and versatile Machine Learning model,
capable of performing linear or nonlinear
classification, regression, and even outlier
detection.
It is one of the most popular models in Machine
Learning, and any one interested in Machine
Learning should have it in their toolbox.
SVMs are particularly well suited for classification
of complex but small- or medium-sized datasets.
 Support Vector Machine - Types
1. Linear SVM Classification
a. Hard Margin Classification
b. Soft Margin Classification
2. Non Linear SVM Classification
a. Polynomial Kernel
b. Gaussian RBF Kernel
3. SVM Regression(Support Vector Regressor)
 Linear SVM Classification
Mathematical Representation:

where w and b are parameters determined during

training.
 Linear SVM Classification
1. Used when the data is linearly separable (i.e., it can
be separated by a straight line in 2D or a
hyperplane in higher dimensions).
2. Finds the optimal hyperplane that maximizes the
margin between two classes.
3. Decision boundary is a straight line or a flat
hyperplane.
4. Example: Classifying emails into "spam" or "not
spam" based on word counts.
Linear SVM Classification
Linear SVM Classification