The

K-Nearest Neighbors (KNN) model is a simple yet powerful supervised

machine learning algorithm used for both classification and
regression tasks. It is a "lazy" and non-parametric algorithm, meaning it
doesn't build a specific model during a formal training phase but rather
stores the entire dataset and performs computations only when making a
prediction.

How the KNN Model Works

The core idea behind the KNN model is that similar data points exist in
close proximity to one another. When a new, unclassified data point is
introduced, the algorithm follows these steps to determine its class or
value:

1. Choose the number of neighbors (K): The user selects a positive

integer,

𝐾
, which defines how many nearest neighbors the algorithm will consider.

2. Calculate distance: The algorithm calculates the distance

(similarity) between the new data point and all existing points in the
training dataset using a specified distance metric, most commonly
the Euclidean distance.

3. Find the K nearest neighbors: The data points are sorted by

distance in ascending order, and the top

𝐾
nearest points are selected.

4. Make a prediction:

o For Classification: The new data point is assigned to the

class that appears most frequently among its K neighbors (a
majority vote). It's common to choose an odd value for

K
𝐾
to avoid ties.

o For Regression: The new data point's value is predicted as

the average (mean) of the values of its K neighbors.

Key Concepts

 Lazy Learner: Unlike other models that learn a function from the
training data, KNN defers all computation until the prediction phase.

 Non-Parametric: It makes no assumptions about the underlying

data distribution, making it flexible for various data types.

 Distance Metrics: The choice of how to measure "distance" is

crucial. Besides Euclidean distance, other metrics like Manhattan or
Minkowski distances are used depending on the data's nature.

 Curse of Dimensionality: KNN's performance can degrade in high-

dimensional spaces because the concept of proximity becomes less
reliable, as all points tend to be nearly equidistant.

Common Applications

KNN is a versatile algorithm used in many real-world scenarios:

 Recommendation Systems: Suggesting products or movies to

users based on the preferences of similar users.

 Image Recognition: Classifying images or handwritten digits by

comparing pixel patterns.

 Medical Diagnosis: Assisting in predicting disease likelihood by

comparing a patient's data to similar historical cases.

 Anomaly Detection: Identifying unusual data points or outliers in a

dataset.

For practical implementation, data scientists often use libraries like scikit-
learn in Python, which provide optimized functions for the KNN algorithm.

Proximity measures in machine learning are metrics that quantify

the similarity or dissimilarity (distance) between data points, acting
as the foundation for algorithms like clustering, nearest neighbor
classification, and anomaly detection by grouping similar items or finding
outliers. These mathematical functions, such as Euclidean
distance or cosine similarity, calculate how alike or different vectors
representing data points are in a feature space, allowing models to
understand patterns and make informed decisions.
Key Concepts:

 Similarity vs. Dissimilarity: Proximity can be measured as how

alike (similarity, e.g., higher score = more alike) or how different
(dissimilarity/distance, e.g., lower score = more alike) two objects
are.

 Data Representation: Data points are often represented as

vectors in a multi-dimensional space, and proximity measures
calculate their relationship within this space.

 Core Function: They provide a quantitative score (a number) to

describe the relationship between any two data points.

Common Proximity Measures:

 Euclidean Distance: The straight-line distance between two

points, common for numerical data.

 Cosine Similarity/Distance: Measures the angle between vectors,

useful for text data (document similarity).

 Manhattan Distance: Sum of absolute differences in coordinates

(city-block distance).

 Jaccard Index/Distance: For binary data, measuring overlap.

Applications in ML:

 Clustering (e.g., K-Means): Groups similar data points together,

using proximity to define cluster centers and assignments.

 Nearest Neighbor (KNN): Classifies new data based on the

majority class of its closest neighbors.

 Anomaly Detection: Identifies outliers as points far from their

neighbors.

Measures of Distance

Last Updated : 08 Sep, 2025

Measures of distance are mathematical functions used to quantify how

similar or dissimilar two objects are based on their features. These
measures are critical for clustering, classification and information retrieval
because they help determine relationships among data points. The choice
of distance depends on the nature of the data and the application domain.

 Used to quantify similarity/dissimilarity between objects.

 Smaller distance = higher similarity, larger distance = higher

dissimilarity.

 Important for clustering algorithms (e.g., K-means, hierarchical

clustering).

 Choice of distance is context-dependent (numerical, categorical or

text data).

Let's see few types of distances.

1. Euclidean Distance

Euclidean Distance

Euclidean distance is considered the traditional metric for problems with

geometry. It can be simply explained as the ordinary distance between
two points. It is one of the most used algorithms in the cluster analysis.

Formula:

√∑
n
d (x , y)= ❑¿ ¿
i =1

 Best for: Continuous numerical data (when features are

normalized).
  Example: Distance between two cities on a 2D map.

2. Manhattan Distance

Manhattan Distance determines the absolute difference among the pair of

the coordinates. Suppose we have two points P and Q to determine the
distance between these points we simply have to calculate the
perpendicular distance of the points from X-Axis and Y-Axis. In a plane with
P at coordinate (x1, y1) and Q at (x2, y2). Manhattan distance between P
and Q = |x1 – x2| + |y1 – y2|

Formula:
n
d (x , y)=∑ ❑ ∣ x i− y i ∣
i=1

 Best for: High-dimensional data or when diagonal movement has

no meaning.

 Example: Distance between blocks in a city (taxicab geometry).

3. Jaccard Index

Jaccard Index

The Jaccard distance is set-based distance that compares dissimilarity by

looking at the ratio of unique to common elements.

Formula:
 ∣ A ∩ B∣
d ( A , B)=1−
∣ A ∪ B∣
 Best for: Binary or categorical data, especially sets.

 Example: Comparing similarity of shopping carts or tag sets.

4. Minkowski distance

Minkowski distance is a generalized distance measure that includes both

Euclidean and Manhattan distances as special cases, controlled by a
parameter p.

Formula:
n 1
d (x , y)=( ∑ ❑∣ x i− y i ∣ p) p
i=1

 Best for: Flexible distance calculations where p is tuned.

 Example: For p=1, it becomes Manhattan; for p=2, it becomes

Euclidean.

5. Cosine Similarity / Cosine Distance

Measures the cosine distance of the angle between two vectors, focusing
on orientation rather than magnitude. Commonly converted to distance as
1−similarity.

Formula(Similarity):
x⋅y
Cosine (x , y )=
∣∣ x ∣∣∣∣ y ∣ ∣

Formula(Distance):
x⋅ y
d (x , y)=1−
∣∣ x ∣ ∣∣∣ y ∣∣
 Best for: Text mining, NLP, recommendation systems.

 Example: Measuring similarity between two documents regardless

of their length.

6. Hamming Distance

The number of positions where two strings (of equal length) differ.
Commonly used for error detection and sequence comparison.

Formula:
n
d (x , y)=∑ ❑[ xi ≠ y i ]
i=1
where [ [ xi ≠ y i ]=1 if symbols differ, else 0.

 Best for: Binary strings, DNA sequences, error correction.

 Example: Hamming distance between “karolin” and “kathrin” = 3.

Non-metric similarity functions in ML measure how alike items are without

strictly adhering to metric space rules (like triangle inequality), excelling
with categorical, text, or high-dimensional data where geometric distance
fails, using techniques like Cosine Similarity (text), Jaccard
Index (sets/binary), or deep learning (Siamese Networks) to learn
complex patterns, crucial for tasks like image retrieval, recommendation
systems, and NLP. They focus on capturing relative likeness or patterns
rather than absolute distance, often outperforming traditional metrics in
complex, real-world scenarios.

Key Types & Examples:

 Set-Based/Binary:

o Jaccard Similarity: Measures overlap between sets

(intersection/union) for binary/categorical data, good for
document similarity.

o Hamming Distance: Counts differing bits/features in equal-

length binary strings (often used for similarity/dissimilarity).

 Vector-Based (Non-Metric Aspect):

o Cosine Similarity: Measures angle between vectors (0-1),

excellent for sparse text data (TF-IDF) where magnitude isn't
key, effectively a non-metric approach in high dimensions.

o Correlation (Pearson/Spearman): Measures

linear/monotonic relationship between features, common in
recommender systems.

 Information-Theoretic/Distributional:

o KL Divergence/Jensen-Shannon: Compares probability

distributions, useful for comparing data distributions,
notes arXiv.

o Compression-Based (Normalized Compression Distance

- NCD): Similarity based on how much better two items
compress together vs. separately.

 Learned/Neural Network-Based:
 o Siamese Networks: Train two identical neural nets to learn
embeddings where similar items are close, dissimilar are far,
using loss functions like contrastive loss.

o Metric Learning (Non-Metric Focus): Neural nets learn

non-metric functions to model complex relationships (e.g.,
visual similarity in images) better than Euclidean distance.

Why Use Them?

 Handle Non-Numeric Data: Essential for text, categories, or

graphs where Euclidean distance makes little sense.

 High-Dimensional Spaces: Metrics can fail in high dimensions

(curse of dimensionality); non-metrics like cosine handle sparsity
better.

 Capture Complex Patterns: Neural methods learn nuanced

similarities (e.g., facial expressions) that simple metrics miss.

Applications:

 Text Mining: Cosine similarity for document clustering/search.

 Image Retrieval: Siamese networks learning visual similarity.

 Recommender Systems: Correlation/Cosine for user/item

similarity.

 Bioinformatics: Sequence similarity.

AI Mode

All

Images

Videos
Shopping

Short videos

Forums

More

Tools

AI Overview

Proximity between binary patterns in ML quantifies how alike or different

sequences of 0s and 1s are, crucial for clustering, classification, and
pattern recognition, using metrics like Hamming Distance (counts
mismatches) and Jaccard Index (shared '1's over total '1's), focusing on
shared characteristics rather than absent ones for sparse data. These
measures are fundamental for tasks like image texture analysis (Local
Binary Patterns) and anomaly detection, helping algorithms group similar
items or identify outliers.

This video provides an introduction to the concept of proximity measures:

57s

Binod Suman Academy

YouTube • 2019 M04 9

Key Proximity Measures for Binary Data:

 Hamming Distance: Counts the number of positions where two

binary vectors differ. Good for patterns of the same length, like error
detection codes.

o Example: 1010 vs 0011 has a Hamming Distance of 3

(positions 1, 2, 4 differ).

 Jaccard Index (or Coefficient): Measures similarity as the ratio of

shared '1's (intersection) to the total '1's (union). Ignores 0-0
matches (common absences), focusing on shared features.

o Formula: |A ∩ B| / |A ∪ B|.

 Cosine Similarity: Calculates the angle between binary vectors,

useful for sparse data, though more commonly associated with
numeric vectors.

 Simple Matching Coefficient (SMC): Counts total matches (1-1

and 0-0) divided by total attributes.

 Phi Coefficient (φ): A correlation measure for binary data,

sensitive to both 1-1 and 0-0 matches.

Why it Matters:
  Clustering: Groups data points (e.g., customers, symptoms) with
similar binary profiles.

 Classification: Nearest Neighbor algorithms rely on proximity to

assign new data to existing classes.

 Pattern Recognition: Identifies recurring patterns or anomalies in

binary-coded data, like features in images (e.g., Local Binary
Patterns for texture).

 Data Types: Choice depends on data; for sparse data (many 0s),
measures ignoring 0-0 matches (like Jaccard) are often better than
those treating them equally

Supervised Machine Learning Classification

In supervised machine learning, algorithms learn from labeled data. After

understanding the data, the algorithm determines which label should be
given to new data by associating patterns to the unlabeled new data.

Supervised learning can be divided into two categories: classification

and regression.

What Is Classification?

Classification predicts the category the data belongs to. Some examples
of classification include spam detection, churn prediction, sentiment
analysis, dog breed detection and so on.

What Is Regression?

Regression predicts a numerical value based on previously observed data.

Some examples of regression include house price prediction, stock price
prediction, height-weight prediction and so on.

Classification and Regression in Machine Learning. | Video: Quantopian

Dive DeeperThe Top 10 Machine Learning Algorithms Every Beginner

Should Know

5 Types of Classification Algorithms for Machine Learning

Classification is a technique for determining which class the dependent

belongs to based on one or more independent variables.

What Is a Classifier?
A classifier is a type of machine learning algorithm that assigns a label to
a data input. Classifier algorithms use labeled data and statistical
methods to produce predictions about data input classifications.

Classification is used for predicting discrete responses.

1. Logistic Regression

Logistic regression is kind of like linear regression, but is used when the
dependent variable is not a number but something else (e.g., a “yes/no”
response). It’s called regression but performs classification based on the
regression and it classifies the dependent variable into either of the
classes.
Logistic regression is used for prediction of output which is binary, as
stated above. For example, if a credit card company builds a model to
decide whether or not to issue a credit card to a customer, it will model for
whether the customer is going to “default” or “not default” on their card.

Linear Regression

Firstly, linear regression is performed on the relationship between

variables to get the model. The threshold for the classification line is
assumed to be at 0.5.

Logistic Sigmoid Function

Logistic function is applied to the regression to get the probabilities of it

belonging in either class.

It gives the log of the probability of the event occurring to the log of the
probability of it not occurring. In the end, it classifies the variable based
on the higher probability of either class.
2. K-Nearest Neighbors (K-NN)

K-NN algorithm is one of the simplest classification algorithms and it is

used to identify the data points that are separated into several classes to
predict the classification of a new sample point. K-NN is a non-
parametric, lazy learning algorithm. It classifies new cases based on a
similarity measure (i.e., distance functions).
K-NN works well with a small number of input variables (p), but struggles
when the number of inputs is very large.

Find out who's hiring.

See all Data + Analytics jobs at top tech companies & startups

View Jobs
3. Support Vector Machine (SVM)

Support vector is used for both regression and classification. It is based on

the concept of decision planes that define decision boundaries. A decision
plane (hyperplane) is one that separates between a set of objects having
different class memberships.

It performs classification by finding the hyperplane that maximizes the

margin between the two classes with the help of support vectors.
The learning of the hyperplane in SVM is done by transforming the
problem using some linear algebra (i.e., the example above is a linear
kernel which has a linear separability between each variable).

For higher dimensional data, other kernels are used as points and cannot
be classified easily. They are specified in the next section.

Kernel SVM

Kernel SVM takes in a kernel function in the SVM algorithm and transforms
it into the required form that maps data on a higher dimension which is
separable.

Types of kernel functions:

Type of kernel functions

1. Linear SVM is the one we discussed earlier.

2. In polynomial kernel, the degree of the polynomial should be

specified. It allows for curved lines in the input space.

3. In the radial basis function (RBF) kernel, it is used for non-linearly

separable variables. For distance, metric squared Euclidean distance
 is used. Using a typical value of the parameter can lead to
overfitting our data. It is used by default in sklearn.

4. Sigmoid kernel, similar to logistic regression is used for binary

classification.

Kernel trick uses the kernel function to transform data into a higher
dimensional feature space and makes it possible to perform the linear
separation for classification.

Radial Basis Function (RBF) Kernel

The RBF kernel SVM decision region is actually also a linear decision
region. What RBF kernel SVM actually does is create non-linear
combinations of features to uplift the samples onto a higher-dimensional
feature space where a linear decision boundary can be used to separate
classes.
So, the rule of thumb is: use linear SVMs for linear problems, and
nonlinear kernels such as the RBF kernel for non-linear problems.

4. Naive Bayes

The naive Bayes classifier is based on Bayes’ theorem with the

independence assumptions between predictors (i.e., it assumes the
presence of a feature in a class is unrelated to any other feature). Even if
these features depend on each other, or upon the existence of the other
features, all of these properties independently. Thus, the name naive
Bayes.
Based on naive Bayes, Gaussian naive Bayes is used for classification
based on the binomial (normal) distribution of data.

 P(class|data) is the posterior probability of class(target)

given predictor(attribute). The probability of a data point having
 either class, given the data point. This is the value that we are
looking to calculate.

 P(class) is the prior probability of class.

 P(data|class) is the likelihood, which is the probability

of predictor given class.

 P(data) is the prior probability of predictor or marginal likelihood.

Naive Bayes Steps

1. Calculate Prior Probability

P(class) = Number of data points in the class/Total no. of observations

P(yellow) = 10/17

P(green) = 7/17

2. Calculate Marginal Likelihood

P(data) = Number of data points similar to observation/Total no. of

observations

P(?) = 4/17

The value is present in checking both the probabilities.

3. Calculate Likelihood

P(data/class) = Number of similar observations to the class/Total no. of

points in the class.
P(?/yellow) = 1/7

P(?/green) = 3/10

4. Posterior Probability for Each Class

5. Classification

The higher probability, the class belongs to that category as from above
75% probability the point belongs to class green.

Multinomial, Bernoulli naive Bayes are the other models used in

calculating probabilities. Thus, a naive Bayes model is easy to build, with
no complicated iterative parameter estimation, which makes it particularly
useful for very large datasets.

5. Decision Tree Classification

Decision tree builds classification or regression models in the form of a

tree structure. It breaks down a dataset into smaller and smaller subsets
while at the same time an associated decision tree is incrementally
developed. The final result is a tree with decision nodes and leaf nodes. It
follows Iterative Dichotomiser 3 (ID3) algorithm structure for determining
the split.

Entropy and information gain are used to construct a decision tree.

Entropy

Entropy is the degree or amount of uncertainty in the randomness of

elements. In other words, it is a measure of impurity.
Intuitively, it tells us about the predictability of a certain event. Entropy
calculates the homogeneity of a sample. If the sample is completely
homogeneous the entropy is zero, and if the sample is equally divided it
has an entropy of one.

Information Gain

Information gain measures the relative change in entropy with respect to

the independent attribute. It tries to estimate the information contained
by each attribute. Constructing a decision tree is all about finding the
attribute that returns the highest information gain (i.e., the most
homogeneous branches).

Where Gain(T, X) is the information gain by applying

feature X. Entropy(T) is the entropy of the entire set, while the second
term calculates the entropy after applying the feature X.

Information gain ranks attributes for filtering at a given node in the tree.
The ranking is based on the highest information gain entropy in each split.

The disadvantage of a decision tree model is overfitting, as it tries to fit

the model by going deeper in the training set and thereby reducing test
accuracy.
Overfitting in decision trees can be minimized by pruning nodes.

Ensemble Methods for Classification

An ensemble model is a team of models. Technically, ensemble models

comprise several supervised learning models that are individually trained
and the results merged in various ways to achieve the final prediction.
This result has higher predictive power than the results of any of its
constituting learning algorithms independently.

1. Random Forest Classification

Random forest classifier is an ensemble algorithm based on bagging i.e

bootstrap aggregation. Ensemble methods combines more than one
algorithm of the same or different kind for classifying objects (i.e., an
ensemble of SVM, naive Bayes or decision trees, for example.)
The general idea is that a combination of learning models increases the
overall result selected.

Deep decision trees may suffer from overfitting, but random forests
prevent overfitting by creating trees on random subsets. The main reason
is that it takes the average of all the predictions, which cancels out the
biases.

Random forest adds additional randomness to the model while growing

the trees. Instead of searching for the most important feature while
splitting a node, it searches for the best feature among a random subset
of features. This results in a wide diversity that generally results in a
better model.

2. Gradient Boosting Classification

Gradient boosting classifier is a boosting ensemble method. Boosting is a

way to combine (ensemble) weak learners, primarily to reduce prediction
bias. Instead of creating a pool of predictors, as in bagging, boosting
produces a cascade of them, where each output is the input for the
following learner. Typically, in a bagging algorithm trees are grown in
parallel to get the average prediction across all trees, where each tree is
built on a sample of original data. Gradient boosting, on the other hand,
takes a sequential approach to obtaining predictions instead of
parallelizing the tree building process. In gradient boosting, each decision
tree predicts the error of the previous decision tree —
thereby boosting (improving) the error (gradient).

Working of Gradient Boosting

 1. Initialize predictions with a simple decision tree.

2. Calculate residual (actual-prediction) value.

3. Build another shallow decision tree that predicts residual based on

all the independent values.

4. Update the original prediction with the new prediction multiplied by

learning rate.

5. Repeat steps two through four for a certain number of iterations (the
number of iterations will be the number of trees).

Check out this post: Gradient Boosting From Scratch

Metrics to Measure Classification Model Performance

1. Confusion Matrix

A confusion matrix is a table that is often used to describe the

performance of a classification model on a set of test data for which the
true values are known. It is a table with four different combinations of
predicted and actual values in the case for a binary classifier.
The confusion matrix for a multi-class classification problem can help you
determine mistake patterns.

For a binary classifier:

A true positive is an outcome where the model correctly predicts
the positive class. Similarly, a true negative is an outcome where the
model correctly predicts the negative class.

False Positive and False Negative

The terms false positive and false negative are used in determining how
well the model is predicting with respect to classification. A false
positive is an outcome where the model incorrectly predicts
the positive class. And a false negative is an outcome where the
model incorrectly predicts the negative class. The more values in main
diagonal, the better the model, whereas the other diagonal gives the
worst result for classification.

False Positive

False positive (type I error) — when you reject a true null hypothesis.

This is an example in which the model mistakenly predicted the positive

class. For example, the model inferred that a particular email message
was spam (the positive class), but that email message was actually not
spam. It’s like a warning sign that the mistake should be rectified as it’s
not much of a serious concern compared to false negative.

False Negative

False negative (type II error) — when you accept a false null hypothesis.
This is an example in which the model mistakenly predicted the negative
class. For example, the model inferred that a particular email message
was not spam (the negative class), but that email message actually was
spam. It’s like a danger sign that the mistake should be rectified early as
it’s more serious than a false positive.

Accuracy, Precision, Recall and F-1 Score

From the confusion matrix, we can infer accuracy, precision, recall and F-1
score.

Accuracy

Accuracy is the fraction of predictions our model got right.

Accuracy can also be written as

Accuracy alone doesn’t tell the full story when working with a class-
imbalanced data set, where there is a significant disparity between the
number of positive and negative labels. Precision and recall are better
metrics for evaluating class-imbalanced problems.

Precision
Out of all the classes, precision is how much we predicted correctly.

Precision should be as high as possible.

Recall

Out of all the positive classes, recall is how much we predicted correctly. It
is also called sensitivity or true positive rate (TPR).
Recall should be as high as possible.

F-1 Score

It is often convenient to combine precision and recall into a single metric

called the F-1 score, particularly if you need a simple way to compare two
classifiers. The F-1 score is the harmonic mean of precision and recall.
The regular mean treats all values equally, while the harmonic mean gives
much more weight to low values thereby punishing the extreme values
more. As a result, the classifier will only get a high F-1 score if both recall
and precision are high.

2. Receiver Operator Curve (ROC) and Area Under the Curve (AUC)

ROC curve is an important classification evaluation metric. It tells us how

well the model has accurately predicted. The ROC curve shows the
sensitivity of the classifier by plotting the rate of true positives to the rate
of false positives. If the classifier is outstanding, the true positive rate will
increase, and the area under the curve will be close to one. If the classifier
is similar to random guessing, the true positive rate will increase linearly
with the false positive rate. The better the AUC measure, the better the
model.
3. Cumulative Accuracy Profile Curve

The CAP of a model represents the cumulative number of positive

outcomes along the y-axis versus the corresponding cumulative number
of a classifying parameters along the x-axis. The CAP is distinct from the
receiver operating characteristic (ROC), which plots the true-positive rate
against the false-positive rate. CAP curve is rarely used as compared to
ROC curve.
Consider a model that predicts whether a customer will purchase a
product. If a customer is selected at random, there is a 50 percent chance
they will buy the product. The cumulative number elements for which the
customer buys would rise linearly toward a maximum value corresponding
to the total number of customers. This distribution is called the “random”
CAP. Its the blue line in the above diagram. A perfect prediction, on the
other hand, determines exactly which customer will buy the product, such
that the maximum customer buying the property will be reached with a
minimum number of customer selection among the elements. This
produces a steep line on the CAP curve that stays flat once the maximum
is reached, which is the “perfect” CAP. It’s also called the “ideal” line and
is the grey line in the figure above.

In the end, a model should predict where it maximizes the correct

predictions and gets closer to a perfect model line.
K-Nearest Neighbor(KNN) Algorithm

Last Updated : 16 Dec, 2025

K-Nearest Neighbors (KNN) is a supervised machine learning algorithm

generally used for classification but can also be used for regression tasks.
It works by finding the "k" closest data points (neighbors) to a given input
and makes a predictions based on the majority class (for classification) or
the average value (for regression). Since KNN makes no assumptions
about the underlying data distribution it makes it a non-parametric and
instance-based learning method.

K-Nearest Neighbors is also called as a lazy learner algorithm because it

does not learn from the training set immediately instead it stores the
entire dataset and performs computations only at the time of
classification.

For example, consider the following table of data points containing two
features:

The new point is classified as Category 2 because most of its closest

neighbors are blue squares. KNN assigns the category based on the
majority of nearby points. The image shows how KNN predicts the
category of a new data point based on its closest neighbours.

 The red diamonds represent Category 1 and the blue squares

represent Category 2.

 The new data point checks its closest neighbors (circled points).

 Since the majority of its closest neighbors are blue squares

(Category 2) KNN predicts the new data point belongs to Category
2.
 K
NN Algorithm working visualization

KNN works by using proximity and majority voting to make predictions.

What is 'K' in K Nearest Neighbour?

In the k-Nearest Neighbours algorithm k is just a number that tells the

algorithm how many nearby points or neighbors to look at when it makes
a decision.

Example: Imagine you're deciding which fruit it is based on its shape and
size. You compare it to fruits you already know.

 If k = 3, the algorithm looks at the 3 closest fruits to the new one.

 If 2 of those 3 fruits are apples and 1 is a banana, the algorithm

says the new fruit is an apple because most of its neighbors are
apples.

How to choose the value of k for KNN Algorithm?

 The value of k in KNN decides how many neighbors the algorithm

looks at when making a prediction.

 Choosing the right k is important for good results.

 If the data has lots of noise or outliers, using a larger k can make
the predictions more stable.
  But if k is too large the model may become too simple and miss
important patterns and this is called underfitting.

 So k should be picked carefully based on the data.

Statistical Methods for Selecting k

 Cross-Validation: Cross-Validation is a good way to find the best

value of k is by using k-fold cross-validation. This means dividing the
dataset into k parts. The model is trained on some of these parts
and tested on the remaining ones. This process is repeated for each
part. The k value that gives the highest average accuracy during
these tests is usually the best one to use.

 Elbow Method: In Elbow Method we draw a graph showing the

error rate or accuracy for different k values. As k increases the error
usually drops at first. But after a certain point error stops decreasing
quickly. The point where the curve changes direction and looks like
an "elbow" is usually the best choice for k.

 Odd Values for k: It’s a good idea to use an odd number for k
especially in classification problems. This helps avoid ties when
deciding which class is the most common among the neighbors.

Distance Metrics Used in KNN Algorithm

KNN uses distance metrics to identify nearest neighbor, these neighbors

are used for classification and regression task. To identify nearest neighbor
we use below distance metrics:

1. Euclidean Distance

Euclidean distance is defined as the straight-line distance between two

points in a plane or space. You can think of it like the shortest path you
would walk if you were to go directly from one point to another.

√∑
d
distance (x , X i)= ❑¿ ¿
j=1

2. Manhattan Distance

This is the total distance you would travel if you could only move along
horizontal and vertical lines like a grid or city streets. It’s also called
"taxicab distance" because a taxi can only drive along the grid-like streets
of a city.
n
d (x , y)=∑ ∣ x i− y i ∣
i=1

3. Minkowski Distance
Minkowski distance is like a family of distances, which includes both
Euclidean and Manhattan distances as special cases.
n 1
d (x , y)=( ∑ ❑(xi − y i) )
p p

i=1

From the formula above, when p=2, it becomes the same as the Euclidean
distance formula and when p=1, it turns into the Manhattan distance
formula. Minkowski distance is essentially a flexible formula that can
represent either Euclidean or Manhattan distance depending on the value
of p.

Working of KNN algorithm

Thе K-Nearest Neighbors (KNN) algorithm operates on the principle of

similarity where it predicts the label or value of a new data point by
considering the labels or values of its K nearest neighbors in the training
dataset.

Step 1: Selecting the optimal value of K

 K represents the number of nearest neighbors that needs to be

considered while making prediction.

Step 2: Calculating distance

 To measure the similarity between target and training data points

Euclidean distance is widely used. Distance is calculated between
data points in the dataset and target point.

Step 3: Finding Nearest Neighbors

  The k data points with the smallest distances to the target point are
nearest neighbors.

Step 4: Voting for Classification or Taking Average for Regression

 When you want to classify a data point into a category like spam or
not spam, the KNN algorithm looks at the K closest points in the
dataset. These closest points are called neighbors. The algorithm
then looks at which category the neighbors belong to and picks the
one that appears the most. This is called majority voting.

 In regression, the algorithm still looks for the K closest points. But
instead of voting for a class in classification, it takes the average of
the values of those K neighbors. This average is the predicted value
for the new point for the algorithm.

It shows how a test point is classified based on its nearest neighbors. As

the test point moves the algorithm identifies the closest 'k' data points i.e.
5 in this case and assigns test point the majority class label that is grey
label class here.

Implementing KNN from Scratch in Python

1. Importing Libraries

Counter is used to count the occurrences of elements in a list or iterable.

In KNN after finding the k nearest neighbor labels Counter helps count
how many times each label appears.

import numpy as np

from collections import Counter

2. Defining the Euclidean Distance Function

euclidean_distance is to calculate euclidean distance between points.

def euclidean_distance(point1, point2):

return [Link]([Link](([Link](point1) - [Link](point2))**2))

3. KNN Prediction Function

 [Link] saves how far each training point is from the

test point, along with its label.

 [Link] is used to sorts the list so the nearest points come

first.

 k_nearest_labels picks the labels of the k closest points.

  Uses Counter to find which label appears most among those k labels
that becomes the prediction.

def knn_predict(training_data, training_labels, test_point, k):

distances = []

for i in range(len(training_data)):

dist = euclidean_distance(test_point, training_data[i])

[Link]((dist, training_labels[i]))

[Link](key=lambda x: x[0])

k_nearest_labels = [label for _, label in distances[:k]]

return Counter(k_nearest_labels).most_common(1)[0][0]

4. Training Data, Labels and Test Point

training_data = [[1, 2], [2, 3], [3, 4], [6, 7], [7, 8]]

training_labels = ['A', 'A', 'A', 'B', 'B']

test_point = [4, 5]

k=3

5. Prediction

prediction = knn_predict(training_data, training_labels, test_point, k)

print(prediction)

Output:

The algorithm calculates the distances of the test point [4, 5] to all
training points selects the 3 closest points as k = 3 and determines their
labels. Since the majority of the closest points are labelled 'A' the test
point is classified as 'A'.

In machine learning we can also use Scikit Learn python library which has
in built functions to perform KNN machine learning model and for that you
refer to Implementation of KNN classifier using Sklearn.

Applications of KNN

 Recommendation Systems: Suggests items like movies or

products by finding users with similar preferences.

 Spam Detection: Identifies spam emails by comparing new emails

to known spam and non-spam examples.
  Customer Segmentation: Groups customers by comparing their
shopping behavior to others.

 Speech Recognition: Matches spoken words to known patterns to

convert them into text.

Advantages of KNN

 Simple to use: Easy to understand and implement.

 No training step: No need to train as it just stores the data and

uses it during prediction.

 Few parameters: Only needs to set the number of neighbors (k)

and a distance method.

 Versatile: Works for both classification and regression problems.

Disadvantages of KNN

 Slow with large data: Needs to compare every point during

prediction.

 Struggles with many features: Accuracy drops when data has

too many features.

 Can Overfit: It can overfit especially when the data is high-

dimensional or not clean.

r-Nearest neighbors

Last Updated : 12 Jul, 2025

r-Nearest neighbors are a modified version of the k-nearest neighbors. The

issue with k-nearest neighbors is the choice of k. With a smaller k, the
classifier would be more sensitive to outliers. If the value of k is large,
then the classifier would be including many points from other classes. It is
from this logic that we get the r near neighbors algorithm.

Intuition:

Consider the following data, as the training set.

The green color points belong to class 0 and the red color points belong to
class 1. Consider the white point P as the query point whose

If we take the radius of the circle as 2.2 units and if a circle is drawn using
the point P as the center of the circle, the plot would be as follows
As the number of points in the circle belonging to class 1 (5 points) is
greater than the number of points belonging to class 0 (2 points)

Radius Nearest Neighbors: A technique for finding data points in close

proximity within a specified radius.

Radius Nearest Neighbors is a method used in machine learning to identify

data points that are in close proximity to a given point within a specified
radius. This technique is particularly useful in various applications, such as
clustering, classification, and anomaly detection. By analyzing the
relationships between data points, Radius Nearest Neighbors can help
uncover patterns and trends within the data, enabling more accurate
predictions and insights.

One of the main challenges in implementing Radius Nearest Neighbors is

the computational complexity involved in searching for nearest neighbors,
especially in high-dimensional spaces. Several approaches have been
proposed to address this issue, including tree-based methods, sorting-
based methods, and grid-based methods. Each of these methods has its
own advantages and drawbacks, with some offering faster query times
while others require less memory or computational resources.

Recent research in the field has focused on improving the efficiency and
accuracy of Radius Nearest Neighbors algorithms. For example, a paper by
Chen and Güttel proposes a sorting-based method that significantly
improves over brute force and tree-based methods in terms of index and
query time, while reliably returning exact results and requiring no
parameter tuning. Another paper by Kleinbort et al. investigates the
computational bottleneck in sampling-based motion planning and
suggests that motion-planning algorithms could significantly benefit from
efficient and specifically-tailored nearest-neighbor data structures.

Practical applications of Radius Nearest Neighbors can be found in various

domains. In astronomy, the GriSPy Python package developed by Chalela
et al. enables fast fixed-radius nearest-neighbor lookup for large datasets,
with support for different distance metrics and query types. In robotics,
collision detection and motion planning algorithms can benefit from
efficient nearest-neighbor search techniques, as demonstrated by
Kleinbort et al. In materials science, the solid-angle based nearest-
neighbor algorithm (SANN) proposed by van Meel et al. offers a simple
and computationally efficient method for identifying nearest neighbors in
3D images.

A company case study that highlights the use of Radius Nearest Neighbors
is the development of the radius-optimized Locality Sensitive Hashing
(roLSH) technique by Jafari et al. This technique leverages sampling
methods and neural networks to efficiently find neighboring points in
projected spaces, resulting in improved performance over existing state-
of-the-art LSH techniques.

In conclusion, Radius Nearest Neighbors is a valuable technique for

identifying relationships and patterns within data, with applications across
various domains. By continuing to develop more efficient and accurate
algorithms, researchers can help unlock the full potential of this method
and enable its broader adoption in real-world applications.

What is Radius Nearest Neighbors?

Radius Nearest Neighbors is a technique used in machine learning to

identify data points that are in close proximity to a given point within a
specified radius. It is useful in various applications, such as clustering,
classification, and anomaly detection, and helps uncover patterns and
trends within the data for more accurate predictions and insights.

How does Radius Nearest Neighbors work?

Radius Nearest Neighbors works by calculating the distance between a

given data point and all other data points in the dataset. It then identifies
the data points that are within a specified radius of the given point. The
distance can be calculated using various metrics, such as Euclidean
distance, Manhattan distance, or cosine similarity.

What are the challenges in implementing Radius Nearest

Neighbors?
One of the main challenges in implementing Radius Nearest Neighbors is
the computational complexity involved in searching for nearest neighbors,
especially in high-dimensional spaces. This can lead to slow query times
and high memory usage. Several approaches have been proposed to
address this issue, including tree-based methods, sorting-based methods,
and grid-based methods, each with its own advantages and drawbacks.

What are some recent advancements in Radius Nearest Neighbors

research?

Recent research in the field has focused on improving the efficiency and
accuracy of Radius Nearest Neighbors algorithms. For example, a paper by
Chen and Güttel proposes a sorting-based method that significantly
improves over brute force and tree-based methods in terms of index and
query time, while reliably returning exact results and requiring no
parameter tuning. Another paper by Kleinbort et al. investigates the
computational bottleneck in sampling-based motion planning and
suggests that motion-planning algorithms could significantly benefit from
efficient and specifically-tailored nearest-neighbor data structures.

What are some practical applications of Radius Nearest

Neighbors?

Practical applications of Radius Nearest Neighbors can be found in various

domains, such as astronomy, robotics, and materials science. In
astronomy, the GriSPy Python package enables fast fixed-radius nearest-
neighbor lookup for large datasets. In robotics, collision detection and
motion planning algorithms can benefit from efficient nearest-neighbor
search techniques. In materials science, the solid-angle based nearest-
neighbor algorithm (SANN) offers a simple and computationally efficient
method for identifying nearest neighbors in 3D images.

How is Radius Nearest Neighbors used in industry?

A company case study that highlights the use of Radius Nearest Neighbors
is the development of the radius-optimized Locality Sensitive Hashing
(roLSH) technique by Jafari et al. This technique leverages sampling
methods and neural networks to efficiently find neighboring points in
projected spaces, resulting in improved performance over existing state-
of-the-art LSH techniques.

How can I implement Radius Nearest Neighbors in Python?

Python libraries such as scikit-learn provide implementations of Radius

Nearest Neighbors algorithms. The `RadiusNeighborsClassifier` and
`RadiusNeighborsRegressor` classes in scikit-learn can be used for
classification and regression tasks, respectively. Additionally, the GriSPy
Python package can be used for fast fixed-radius nearest-neighbor lookup
in large datasets.

KNN (K-Nearest Neighbors) regression is

a non-parametric, instance-based machine learning algorithm used to

predict continuous values. It operates on the principle that data points
close to each other in the feature space are likely to have similar
outcomes.

How KNN Regression Works

When predicting the value for a new data point, the KNN algorithm follows
these main steps:

1. Select the number of neighbors (K): A hyperparameter,

𝐾
, is chosen to determine how many nearby points to consider. The choice
of

𝐾
is crucial and often determined using methods like cross-validation.

2. Calculate distances: The distance (similarity) between the new

data point and all existing training data points is calculated using a
distance metric, most commonly the Euclidean distance. Other
metrics like Manhattan or Minkowski distances can also be used.

3. Identify the K nearest neighbors: The algorithm selects the

𝐾
points in the training set with the smallest distances to the new point.

4. Predict the target value: The final predicted value for the new
data point is the average (mean) of the continuous target values of
its
 K

𝐾
nearest neighbors. (For classification, it would be the majority class).

Key Characteristics

 Lazy Learning: KNN is a lazy learner because it doesn't build a

specific model during a training phase; all computation occurs
during the prediction phase, as it stores the entire training dataset
in memory.

 Non-Parametric: It makes no assumptions about the underlying

data distribution, making it flexible for various types of data and
non-linear relationships.

 Feature Scaling: The performance of KNN is highly sensitive to the

scale of input features because it relies on distance calculations. It is
essential to normalize or standardize the data so that all features
contribute equally to the distance metrics.

Advantages and Disadvantages

Advantages Disadvantages

Computationally expensive for large datasets

Simple to understand
due to the need to store all data and calculate
and implement.
distances for every prediction.

Adapts easily when new Sensitive to noise and outliers, which can skew
training data is added. the average value in regression tasks.

Few hyperparameters to
tune (mainly

Suffers from the "curse of dimensionality,"

K meaning its performance degrades with a high

𝐾
number of features.

and the distance

metric).

Classifier performance in machine learning measures how well a model

categorizes data, using metrics like Accuracy, Precision, Recall, and
F1 Score, derived from the Confusion Matrix (True Positives, False
Positives, etc.) to assess correctness, particularly crucial for imbalanced
datasets where simple accuracy can mislead. Evaluating these metrics on
test data helps select the best model, with advanced tools like ROC curves
offering deeper insights, but choosing the right metric depends on the
specific problem's needs (e.g., minimizing false alarms vs. catching all
positive cases).

Key Performance Metrics

 Confusion Matrix: A table showing True Positives (TP), True

Negatives (TN), False Positives (FP), and False Negatives (FN).

 Accuracy: (TP+TN) / (Total) – Overall correctness, but misleading

with imbalanced data.

 Precision: TP / (TP+FP) – Of all predicted positives, how many were

actually positive? (Focuses on minimizing false positives).

 Recall (Sensitivity): TP / (TP+FN) – Of all actual positives, how

many did the model find? (Focuses on minimizing false negatives).

 F1 Score: 2 * (Precision * Recall) / (Precision + Recall) – Harmonic

mean of Precision & Recall, balancing both.

 AUC-ROC (Area Under the Receiver Operating Characteristic

Curve): Plots Recall vs. False Positive Rate (1-Specificity) to show
model performance across different thresholds, ideal for binary
classification.

How Performance is Evaluated

1. Training: Model learns patterns from training data.

2. Validation/Tuning: Hyperparameters are adjusted using validation

data to prevent overfitting.

3. Testing: Final model performance is assessed on unseen test data

using the metrics above to gauge real-world effectiveness.

Why Different Metrics Matter

 Imbalanced Datasets: If 95% of emails are not spam, a model

predicting "not spam" always gets 95% accuracy but is useless;
Precision/Recall/F1 are better.

 Problem Context: In medical diagnosis (high cost of missed

disease), high Recall is crucial; in spam filtering (high cost of
blocking good emails), high Precision is k
 Regression in machine learning

Last Updated : 16 Dec, 2025

Regression in machine learning is a supervised learning technique

used to predict continuous numerical values by learning
relationships between input variables (features) and an output
variable (target). It helps understand how changes in one or more
factors influence a measurable outcome and is widely used in
forecasting, risk analysis, decision-making and trend estimation.

 Works with real-valued output variables

 Helps to identify strengths and the type of relationships

 Supports both simple and complex predictive models.

 Used for tasks like price prediction, trend forecasting and risk
scoring.

Types of Regression

Regression can be classified into different types based on the

number of predictor variables and the nature of the relationship
between variables:

1. Simple Linear Regression: A statistical technique that models

the relationship between a single independent variable and a
continuous dependent variable by fitting a straight line (best-fit line)
that minimizes the sum of squared errors. It assumes a constant
rate of change, meaning the output changes proportionally
whenever the input changes.

 Application: Estimating house price from only its size

 Advantage: Highly interpretable due to its simple mathematical

structure

 Disadvantage: Cannot capture curved or complex data patterns

2. Multiple Linear Regression: An extension of simple linear

regression that models the linear relationship between multiple
 input variables and a continuous target. The model assigns weights
(coefficients) to each feature, quantifying how strongly each
variable affects the outcome while holding others constant.

 Application: Predicting house prices using multiple factors like size,

location, age and number of rooms

 Advantage: Captures the combined influence of many factors

simultaneously

 Disadvantage: Performance drops in the presence of

multicollinearity (features highly correlated with each other)

3. Polynomial Regression: A form of regression where the

relationship between the independent variable(s) and dependent
variable is modelled as an nth-degree polynomial. It transforms
input features into higher-order terms (e.g., x², x³), enabling the
model to represent curvature and complex non-linear patterns in the
data.

 Application: Modelling curved growth trends like population

increase or temperature variation

 Advantage: Effectively captures non-linear relationships without

switching to non-linear algorithms

 Disadvantage: Higher-degree polynomials may lead to overfitting

and unstable predictions

4. Ridge & Lasso Regression: Regularization techniques built on

linear regression that add penalty terms to the loss function to
prevent large coefficients.

Ridge adds an L2 penalty, shrinking coefficients smoothly. Lasso

adds an L1 penalty, which can shrink some coefficients to zero,
performing feature selection. Both help stabilize models when
dealing with many features or noisy data.

 Application: Used in high-dimensional datasets like marketing

attribution or gene expression data

 Advantage: Controls overfitting and improves generalization,

especially with many predictors

 Disadvantage: Penalty terms make model interpretation less

straightforward
 5. Support Vector Regression (SVR): A regression method based
on Support Vector Machines, which tries to fit the best possible line
within a defined margin (epsilon-tube) around the data. Rather than
minimizing prediction error for every point, SVR focuses on points
outside this margin, using kernel functions to model both linear and
non-linear relationships.

 Application: Predicting continuous outcomes such as stock values

or energy consumption

 Advantage: Works well with high-dimensional, complex datasets

and non-linear patterns

 Disadvantage: Computationally intensive and requires careful

tuning of kernels and parameters

6. Decision Tree Regression: A non-linear regression technique

that splits the data into branches based on feature thresholds. Each
internal node represents a decision question and leaf nodes
represent predicted continuous values. It learns patterns by
recursively partitioning the data to minimize prediction errors.

 Application: Predicting customer spending behavior based on

demographic and financial features

 Advantage: Easy to visualize and understand decision logic

 Disadvantage: Easily overfits, especially when the tree becomes

deep and complex

7. Random Forest Regression: An ensemble technique that

builds multiple decision trees on different subsets of data and
averages their predictions. This reduces the overfitting tendency of
single trees and improves accuracy through diversity (bagging).
Each tree captures a slightly different aspect of the data.

 Application: Sales forecasting, demand planning, churn prediction

 Advantage: High accuracy and robust performance even on noisy

datasets

 Disadvantage: Acts as a black-box model, making interpretation

difficult due to many trees

Regression Evaluation Metrics

Evaluation in machine learning measures the performance of a

model. Here are some popular evaluation metrics for regression:
 Mean Absolute Error (MAE): The average absolute difference
between the predicted and actual values of the target variable.

 Mean Squared Error (MSE): The average squared difference

between the predicted and actual values of the target variable.

 Root Mean Squared Error (RMSE): Square root of the mean

squared error.

 Huber Loss: A hybrid loss function that transitions from MAE to

MSE for larger errors, providing balance between robustness and
MSE’s sensitivity to outliers.

 R2 – Score: Higher values indicate better fit ranging from 0 to 1.