Classification Summary

Algorithm Category / Classification

C4.5 (Decision Machine Learning → Supervised Learning → Classification →
Tree Algorithm) Decision Tree Induction
Polynomial Machine Learning / Statistics → Supervised Learning → Regression
Regression → Linear Model (extended with polynomial terms)
Multilayer Machine Learning / Deep Learning → Supervised Learning →
Perceptron (MLP) Artificial Neural Network (Feedforward)
Batch Gradient Machine Learning / Numerical Optimization → Optimization
Descent Algorithm (used to train models, not a model itself)
Genetic Computational Intelligence → Evolutionary Computation →
Programming Population-Based Metaheuristic
(GP)
Support Vector Machine Learning → Supervised Learning → Regression → Kernel-
Regression (SVR) Based Method (extension of SVM)
Gaussian Mixture Machine Learning / Statistics → Unsupervised Learning →
Model (GMM) Probabilistic Clustering / Density Estimation
One-Class SVM Machine Learning → Unsupervised / Semi-Supervised Learning →
Anomaly/Novelty Detection → Kernel-Based Method
N-grams Natural Language Processing / Computational Linguistics → Statistical
Language Modeling → Text Representation Technique
Hierarchical Machine Learning / Statistics → Unsupervised Learning → Clustering
Clustering → Connectivity-Based Method
Deep Q-Network Machine Learning → Reinforcement Learning → Value-Based Method
(DQN) + Deep Learning (Convolutional/Feedforward Neural Network as
function approximator)
Bayesian Networks Machine Learning / Statistics → Probabilistic Graphical Model →
Directed Acyclic Graph (DAG)-Based Model
T5 (Text-to-Text Deep Learning → Natural Language Processing → Transformer
Transfer Architecture (Encoder-Decoder, Sequence-to-Sequence Model)
Transformer)
ESRGAN Deep Learning → Computer Vision → Generative Adversarial
(Enhanced Super- Network (GAN) → Image Super-Resolution
Resolution
Generative
Adversarial
Network)
FCN (Fully Deep Learning → Computer Vision → Convolutional Neural Network
Convolutional → Semantic Segmentation
Network)
1. C4.5 (Decision Tree Algorithm)
Classification: Machine Learning → Supervised Learning → Classification → Decision Tree Induction
Overview
C4.5 is a decision tree algorithm developed by Ross Quinlan as a successor to ID3. It builds a tree-
structured classifier by recursively splitting the dataset on the attribute that yields the highest Gain Ratio,
and it natively supports both categorical and continuous attributes, missing values, and tree pruning to
avoid overfitting.
How It Works
• Computes the entropy (impurity) of the target class at each node.
• Calculates Information Gain for every candidate attribute, then normalizes it into Gain Ratio (Gain
divided by Split Information) to remove ID3's bias toward attributes with many values.
• Selects the attribute with the highest Gain Ratio as the splitting criterion for that node.
• Handles continuous attributes by finding an optimal threshold that splits the data into two ranges.
• Handles missing attribute values by probabilistically distributing instances across branches.
• Grows the tree until nodes are pure or a stopping criterion is met, then applies post-pruning (error-
based pruning) by converting subtrees into leaves when this reduces estimated error, producing a
simpler, more generalizable tree.
• Can convert the final tree into a set of if-then rules for interpretability.
Example:
Data set
We are going to create a decision table for the following dataset. It informs about decision making
factors to play tennis at outside for previous 14 days. The dataset might be familiar from the ID3 post.
The difference is that temperature and humidity columns have continuous values instead of nominal
ones.

Day Outlook Temp. Humidity Wind Decision

1 Sunny 85 85 Weak No

2 Sunny 80 90 Strong No

3 Overcast 83 78 Weak Yes

4 Rain 70 96 Weak Yes

5 Rain 68 80 Weak Yes

6 Rain 65 70 Strong No

7 Overcast 64 65 Strong Yes

8 Sunny 72 95 Weak No
 9 Sunny 69 70 Weak Yes

10 Rain 75 80 Weak Yes

11 Sunny 75 70 Strong Yes

12 Overcast 72 90 Strong Yes

13 Overcast 81 75 Weak Yes

14 Rain 71 80 Strong No
We will do what we have done in ID3 example. Firstly, we need to calculate global entropy. There are
14 examples; 9 instances refer to yes decision, and 5 instances refer to no decision.
Entropy(Decision) = ∑ – p(I) . log2p(I) = – p(Yes) . log2p(Yes) – p(No) . log2p(No) = – (9/14) .
log2(9/14) – (5/14) . log2(5/14) = 0.940
In ID3 algorithm, we’ve calculated gains for each attribute. Here, we need to calculate gain ratios
instead of gains.
GainRatio(A) = Gain(A) / SplitInfo(A)
SplitInfo(A) = -∑ |Dj|/|D| x log2|Dj|/|D|
Wind Attribute
Wind is a nominal attribute. Its possible values are weak and strong.
Gain(Decision, Wind) = Entropy(Decision) – ∑ ( p(Decision|Wind) . Entropy(Decision|Wind) )
Gain(Decision, Wind) = Entropy(Decision) – [ p(Decision|Wind=Weak) . Entropy(Decision|
Wind=Weak) ] + [ p(Decision|Wind=Strong) . Entropy(Decision|Wind=Strong) ]
There are 8 weak wind instances. 2 of them are concluded as no, 6 of them are concluded as yes.
Entropy(Decision|Wind=Weak) = – p(No) . log2p(No) – p(Yes) . log2p(Yes) = – (2/8) . log2(2/8)
– (6/8) . log2(6/8) = 0.811
Entropy(Decision|Wind=Strong) = – (3/6) . log2(3/6) – (3/6) . log2(3/6) = 1
Gain(Decision, Wind) = 0.940 – (8/14).(0.811) – (6/14).(1) = 0.940 – 0.463 – 0.428 = 0.049
There are 8 decisions for weak wind, and 6 decisions for strong wind.
SplitInfo(Decision, Wind) = -(8/14).log2(8/14) – (6/14).log2(6/14) = 0.461 + 0.524 = 0.985
GainRatio(Decision, Wind) = Gain(Decision, Wind) / SplitInfo(Decision, Wind) = 0.049 / 0.985 =
0.049
Outlook Attribute
Outlook is a nominal attribute, too. Its possible values are sunny, overcast and rain.
Gain(Decision, Outlook) = Entropy(Decision) – ∑ ( p(Decision|Outlook) . Entropy(Decision|
Outlook) ) =
Gain(Decision, Outlook) = Entropy(Decision) – p(Decision|Outlook=Sunny) . Entropy(Decision|
Outlook=Sunny) – p(Decision|Outlook=Overcast) . Entropy(Decision|Outlook=Overcast)
– p(Decision|Outlook=Rain) . Entropy(Decision|Outlook=Rain)
There are 5 sunny instances. 3 of them are concluded as no, 2 of them are concluded as yes.
Entropy(Decision|Outlook=Sunny) = – p(No) . log2p(No) – p(Yes) . log2p(Yes) = -(3/5).log2(3/5) –
(2/5).log2(2/5) = 0.441 + 0.528 = 0.970
Entropy(Decision|Outlook=Overcast) = – p(No) . log2p(No) – p(Yes) . log2p(Yes) = -(0/4).log2(0/4) –
(4/4).log2(4/4) = 0
Entropy(Decision|Outlook=Rain) = – p(No) . log2p(No) – p(Yes) . log2p(Yes) = -(2/5).log2(2/5) –
(3/5).log2(3/5) = 0.528 + 0.441 = 0.970
Gain(Decision, Outlook) = 0.940 – (5/14).(0.970) – (4/14).(0) – (5/14).(0.970) – (5/14).(0.970) = 0.246
There are 5 instances for sunny, 4 instances for overcast and 5 instances for rain
SplitInfo(Decision, Outlook) = -(5/14).log2(5/14) -(4/14).log2(4/14) -(5/14).log2(5/14) = 1.577
GainRatio(Decision, Outlook) = Gain(Decision, Outlook)/SplitInfo(Decision, Outlook) = 0.246/1.577
= 0.155
Humidity Attribute
As an exception, humidity is a continuous attribute. We need to convert continuous values to nominal
ones. C4.5 proposes to perform binary split based on a threshold value. Threshold should be a value
which offers maximum gain for that attribute. Let’s focus on humidity attribute. Firstly, we need to
sort humidity values smallest to largest.

Day Humidity Decision

7 65 Yes

6 70 No

9 70 Yes

11 70 Yes

13 75 Yes

3 78 Yes

5 80 Yes

10 80 Yes

14 80 No

1 85 No

2 90 No

12 90 Yes
 8 95 No

4 96 Yes
Now, we need to iterate on all humidity values and seperate dataset into two parts as instances less
than or equal to current value, and instances greater than the current value. We would calculate the
gain or gain ratio for every step. The value which maximizes the gain would be the threshold.
Check 65 as a threshold for humidity
Entropy(Decision|Humidity<=65) = – p(No) . log2p(No) – p(Yes) . log2p(Yes) = -(0/1).log2(0/1)
– (1/1).log2(1/1) = 0
Entropy(Decision|Humidity>65) = -(5/13).log2(5/13) – (8/13).log2(8/13) =0.530 + 0.431 = 0.961
Gain(Decision, Humidity<> 65) = 0.940 – (1/14).0 – (13/14).(0.961) = 0.048
* The statement above refers to that what would branch of decision tree be for less than or equal to
65, and greater than 65. It does not refer to that humidity is not equal to 65!
SplitInfo(Decision, Humidity<> 65) = -(1/14).log2(1/14) -(13/14).log2(13/14) = 0.371
GainRatio(Decision, Humidity<> 65) = 0.126
Check 70 as a threshold for humidity
Entropy(Decision|Humidity<=70) = – (1/4).log2(1/4) – (3/4).log2(3/4) = 0.811
Entropy(Decision|Humidity>70) = – (4/10).log2(4/10) – (6/10).log2(6/10) = 0.970
Gain(Decision, Humidity<> 70) = 0.940 – (4/14).(0.811) – (10/14).(0.970) = 0.940 – 0.231 – 0.692 =
0.014
SplitInfo(Decision, Humidity<> 70) = -(4/14).log2(4/14) -(10/14).log2(10/14) = 0.863
GainRatio(Decision, Humidity<> 70) = 0.016
Check 75 as a threshold for humidity
Entropy(Decision|Humidity<=75) = – (1/5).log2(1/5) – (4/5).log2(4/5) = 0.721
Entropy(Decision|Humidity>75) = – (4/9).log2(4/9) – (5/9).log2(5/9) = 0.991
Gain(Decision, Humidity<> 75) = 0.940 – (5/14).(0.721) – (9/14).(0.991) = 0.940 – 0.2575 – 0.637 =
0.045
SplitInfo(Decision, Humidity<> 75) = -(5/14).log2(4/14) -(9/14).log2(10/14) = 0.940
GainRatio(Decision, Humidity<> 75) = 0.047
I think calculation demonstrations are enough. Now, I skip the calculations and write only results.
Gain(Decision, Humidity <> 78) =0.090, GainRatio(Decision, Humidity <> 78) =0.090
Gain(Decision, Humidity <> 80) = 0.101, GainRatio(Decision, Humidity <> 80) = 0.107
Gain(Decision, Humidity <> 85) = 0.024, GainRatio(Decision, Humidity <> 85) = 0.027
Gain(Decision, Humidity <> 90) = 0.010, GainRatio(Decision, Humidity <> 90) = 0.016
Gain(Decision, Humidity <> 95) = 0.048, GainRatio(Decision, Humidity <> 95) = 0.128
Here, I ignore the value 96 as threshold because humidity cannot be greater than this value.
As seen, gain maximizes when threshold is equal to 80 for humidity. This means that we need to
compare other nominal attributes and comparison of humidity to 80 to create a branch in our tree.
Temperature feature is continuous as well. When I apply binary split to temperature for all possible
split points, the following decision rule maximizes for both gain and gain ratio.
Gain(Decision, Temperature <> 83) = 0.113, GainRatio(Decision, Temperature<> 83) = 0.305
Let’s summarize calculated gain and gain ratios. Outlook attribute comes with both maximized gain
and gain ratio. This means that we need to put outlook decision in root of decision tree.

Attribute Gain GainRatio

Wind 0.049 0.049

Outlook 0.246 0.155

Humidity <> 80 0.101 0.107

Temperature <> 83 0.113 0.305

If we will use gain metric, then outlook will be the root node because it has the highest gain value. On
the other hand, if we use gain ratio metric, then temperature will be the root node because it has the
highest gain ratio value. I prefer to use gain here similar to ID3. As a homework, please try to build a
C4.5 decision tree based on gain ratio metric.
After then, we would apply similar steps just like as ID3 and create following decision tree. Outlook is
put into root node. Now, we should look decisions for different outlook types.
Outlook = Sunny
We’ve split humidity for greater than 80, and less than or equal to 80. Surprisingly, decisions would be
no if humidity is greater than 80 when outlook is sunny. Similarly, decision would be yes if humidity
is less than or equal to 80 for sunny outlook.

Day Outlook Temp. Hum. > 80 Wind Decision

1 Sunny 85 Yes Weak No

2 Sunny 80 Yes Strong No

8 Sunny 72 Yes Weak No

9 Sunny 69 No Weak Yes

11 Sunny 75 No Strong Yes

Outlook = Overcast
If outlook is overcast, then no matter temperature, humidity or wind are, decision will always be yes.

Day Outlook Temp. Hum. > 80 Wind Decision

3 Overcast 83 No Weak Yes

7 Overcast 64 No Strong Yes

 12 Overcast 72 Yes Strong Yes

13 Overcast 81 No Weak Yes

Outlook = Rain
We’ve just filtered rain outlook instances. As seen, decision would be yes when wind is weak, and it
would be no if wind is strong.

Day Outlook Temp. Hum. > 80 Wind Decision

4 Rain 70 Yes Weak Yes

5 Rain 68 No Weak Yes

6 Rain 65 No Strong No

10 Rain 75 No Weak Yes

14 Rain 71 No Strong No
Final form of decision table is demonstrated below.

Decision tree generated by C4.5

Key Characteristics
• Splitting criterion: Gain Ratio (an improvement over Information Gain used in ID3).
• Supports both discrete and continuous attributes and missing values.
• Uses pruning (pre- and post-pruning) to control overfitting and tree size.
Advantages
• Highly interpretable — the resulting tree/rules can be read and explained by non-experts.
• Handles mixed data types (categorical and numerical) without heavy preprocessing.
Limitations
• Can still overfit on noisy data if pruning is not tuned properly.
• Greedy, top-down splitting may miss globally optimal trees.
• Small changes in data can produce very different tree structures (instability).
• Less accurate than ensemble methods (e.g., Random Forest, Gradient Boosting) on complex
datasets.
10 Real-Time Applications
# Application How It Is Used In Real Time
1 Medical diagnosis Classifying patients as high/low risk for diseases (e.g.,
systems diabetes, heart disease) using interpretable decision rules
that doctors can verify.
2 Credit risk and loan Banks use C4.5-based trees to decide loan approval by
approval splitting applicants on income, credit history, and debt
ratio.
3 Customer churn Telecom and subscription companies classify customers
prediction likely to cancel service based on usage patterns and billing
history.
4 Fraud detection Insurance and banking systems flag suspicious
claims/transactions using rule-based decision trees derived
from C4.5.
5 Manufacturing quality Classifying defective vs. non-defective products on
control production lines based on sensor readings.
6 Spam email filtering Building interpretable rules that separate spam from
legitimate email using word-frequency and header
features.
7 Student performance Educational institutions predict pass/fail or at-risk students
prediction from attendance, grades, and engagement data.
8 Agricultural crop Classifying soil type or crop disease from sensor and
classification image-derived features for precision farming.
9 Retail market basket Segmenting customers into buying-behavior classes to
segmentation target promotions.
10 Network intrusion Building interpretable rule sets that flag anomalous
detection network traffic patterns as potential intrusions.
2. Polynomial Regression
Classification: Machine Learning / Statistics → Supervised Learning → Regression → Linear Model
(extended with polynomial terms)
Overview
Polynomial Regression models the relationship between an independent variable x and a dependent
variable y as an nth-degree polynomial. It extends simple linear regression by adding powers of the input
feature (x, x^2, x^3, ...), allowing it to fit curved, non-linear relationships while still being a linear model
in terms of its coefficients.
How It Works
• Transforms the original feature x into a feature set {x, x^2, x^3, ..., x^n}.
• Fits a linear model y = b0 + b1*x + b2*x^2 + ... + bn*x^n to these expanded features using ordinary
least squares.
• Estimates coefficients by minimizing the sum of squared residuals between predicted and actual
values.
• Chooses the polynomial degree n via validation techniques (cross-validation) to balance bias and
variance.
• Applies regularization (Ridge/Lasso) when the degree is high, to prevent overfitting from the added
flexibility.
Example:
A polynomial regression model is used when the relationship between the input variable x and the
output variable y is nonlinear, but can be approximated by a polynomial.
Example dataset
Hours Studied Test Score (y)
(x)
1 52
2 58
3 67
4 80
5 88
A scatter plot of these points might show a curved trend rather than a straight line.
Polynomial regression model
For a quadratic (degree 2) model:
2
y=β 0 + β 1 x + β 2 x
Suppose the fitted equation is:
^y =45+ 4 x+ 1.2 x 2Prediction example
For a student who studies 4 hours:
^y =45+ 4(4)+1.2(42 )¿ 45+ 16+19.2=80.2 So the predicted test score is 80.2.
Python example (scikit-learn)
import numpy as np
from [Link] import PolynomialFeatures
from sklearn.linear_model import LinearRegression
# Data
X = [Link]([[1], [2], [3], [4], [5]])
y = [Link]([52, 58, 67, 80, 88])
# Create polynomial features (degree = 2)
poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)
# Train model
model = LinearRegression()
[Link](X_poly, y)
# Predict for 4 hours
prediction = [Link]([Link]([[4]]))
print(prediction)
When to use polynomial regression
Use polynomial regression when:
 The relationship between variables is curved rather than linear.
 A straight-line regression underfits the data.
 You want to model trends such as growth, decay, or U-shaped/inverted U-shaped relationships.
Key Characteristics
• Still a linear regression model in parameter space (coefficients enter linearly), even though the fitted
curve is non-linear in x.
• Degree of polynomial controls model flexibility/complexity.
• Prone to overfitting at high degrees, and to poor extrapolation outside the training range.
• Can be extended to multiple variables (multivariate polynomial regression).
Advantages
• Captures non-linear trends that plain linear regression cannot.
• Simple to implement using standard linear regression machinery after feature transformation.
• Interpretable coefficients and well-understood statistical theory (confidence intervals, p-values).
Limitations
• High-degree polynomials overfit and oscillate wildly, especially near data boundaries (Runge's
phenomenon).
• Sensitive to outliers.
• Extrapolation beyond the observed data range is unreliable.
• Choosing the right degree requires careful validation.

10 Real-Time Applications
# Application How It Is Used In Real Time
1 Growth curve modeling Modeling population, bacterial, or tumor growth trends
 that follow curved trajectories over time.
2 Stock price trend fitting Fitting short-term non-linear trends in financial time series
for trend analysis (not standalone forecasting).
3 Epidemiology curve Modeling the rise and fall of disease case counts during an
fitting outbreak.
4 Physics and engineering Modeling non-linear relationships such as stress-strain
calibration curves or sensor calibration curves.
5 Agricultural yield Relating fertilizer dosage to crop yield, which typically
prediction follows a curved (diminishing-returns) pattern.
6 Real estate price Capturing non-linear effects of variables like house age or
estimation size on price.
7 Pharmacokinetics Modeling drug concentration in the bloodstream over time,
which often follows a curved decay/absorption pattern.
8 Climate and Fitting seasonal temperature curves or long-term warming
temperature modeling trend curvature.
9 Manufacturing process Relating a process parameter (e.g., temperature) to product
optimization quality where the relationship is curved.
10 Sports performance Modeling how athlete performance changes non-linearly
analytics with age or training load.
3. Multilayer Perceptron (MLP)
Classification: Machine Learning / Deep Learning → Supervised Learning → Artificial Neural
Network (Feedforward)
Overview
Multi-layer Perceptron (MLP) is a supervised learning algorithm that learns a function by training on
a dataset, where is the number of dimensions for input and is the number of dimensions for output.
Given a set of features and a target , it can learn a non-linear function approximator for either
classification or regression. It is different from logistic regression, in that between the input and the
output layer, there can be one or more non-linear layers, called hidden layers. Figure 1 shows a one
hidden layer MLP with scalar output.

Figure 1 : One hidden layer MLP.

The leftmost layer, known as the input layer, consists of a set of neurons representing the input features.
Each neuron in the hidden layer transforms the values from the previous layer with a weighted linear
summation , followed by a non-linear activation function - like the hyperbolic tan function. The output
layer receives the values from the last hidden layer and transforms them into output values.
The module contains the public attributes coefs_ and intercepts_. coefs_ is a list of weight matrices,
where weight matrix at index represents the weights between layer and layer . intercepts_ is a list of
bias vectors, where the vector at index represents the bias values added to layer .
How It Works
• Input features are passed through an input layer to one or more hidden layers of neurons.
• Each neuron computes a weighted sum of its inputs plus a bias, then applies a non-linear activation
function (ReLU, sigmoid, tanh).
• The forward pass produces an output prediction (classification or regression).
• A loss function (e.g., cross-entropy, MSE) measures the error between prediction and ground truth.
• Backpropagation computes gradients of the loss with respect to every weight using the chain rule.
• An optimizer (e.g., gradient descent, Adam) updates weights to reduce the loss over many training
epochs.
• Regularization techniques (dropout, weight decay, early stopping) are used to prevent overfitting.
Consider a simple student pass/fail prediction problem.
Input Features
 x₁ = Hours Studied
 x₂ = Attendance (%)
Output
 y = Pass (1) or Fail (0)
 Training Data
Hours Studied Attendance Result (y)
(x₁) (x₂)
2 50 0 (Fail)
3 60 0 (Fail)
5 75 1 (Pass)
6 85 1 (Pass)
8 95 1 (Pass)
MLP Architecture
Input Layer Hidden Layer Output Layer
x₁ (Hours) ───► h₁
\ /
X ───► Pass/Fail
/ \
x₂ (Attendance)► h₂
 Input layer: 2 neurons (Hours Studied, Attendance)
 Hidden layer: 2 neurons (h₁, h₂)
 Output layer: 1 neuron (Pass/Fail)
Example Calculation
Suppose a new student has:
 Hours Studied = 6
 Attendance = 80%
Step 1: Input Layer
x=[6 , 80]Step 2: Hidden Layer
Assume the weights are:

[
W=
]
0.2 0.1
0.4 0.3
Bias:
b=[0.5 , 0.2]Calculate the hidden neurons:
For h₁:
z 1=( 6 ×0.2)+(80 × 0.4)+0.5¿ 1.2+32+0.5=33.7For h₂:
z 2=( 6 ×0.1)+(80 × 0.3)+ 0.2¿ 0.6+ 24+0.2=24.8Apply the ReLU activation function:
ReLU (x )=max ⁡(0 , x)So,
h1=33.7 , h2=24.8Step 3: Output Layer
Assume:
 Output weights = [0.6, 0.4]
 Output bias = 0.1
Compute:
z=(33.7 ×0.6)+(24.8 ×0.4)+0.1¿ 20.22+9.92+0.1=30.24Apply the Sigmoid activation function:
1
Sigmoid (z)= −z Sigmoid (30.24) ≈ 1.0Final Prediction
1+ e
Since the output is approximately 1, the MLP predicts:
Result = Pass (1)

Key Characteristics
• Fully connected layers; depth and width are key hyperparameters.
• Requires non-linear activation functions to model complex, non-linear relationships (a universal
approximator).
Advantages
• Can approximate complex non-linear functions given enough neurons/layers.
• Flexible — applicable to classification, regression, and representation learning.
• Scales well with more data and compute.
Limitations
• Requires large amounts of data and computation for good generalization.
• Prone to overfitting without regularization.
10 Real-Time Applications
# Application How It Is Used In Real Time
1 Handwritten Classic use in OCR systems (e.g., early MNIST digit
digit/character classifiers) to recognize handwritten numerals.
recognition
2 Credit scoring Banks use MLPs to predict the probability of loan default
from applicant financial features.
3 Medical diagnosis Predicting disease presence/severity from structured
support clinical measurements.
4 Speech emotion Classifying emotional state from extracted audio features.
recognition
5 Stock market direction Predicting up/down movement of asset prices from
prediction engineered financial indicators.
6 Customer Predicting which customers are likely to leave a service
churn/retention based on usage data.
modeling
7 Predictive maintenance Predicting equipment failure from sensor readings
(vibration, temperature) in industrial IoT.
8 Recommendation Learning user-item interaction patterns as part of hybrid
systems recommender pipelines.
9 Energy load forecasting Predicting electricity demand from historical consumption
and weather features.
10 Image feature Acting as a classifier head on top of extracted image
classification features in computer vision pipelines.
4. Batch Gradient Descent
Classification: Machine Learning / Numerical Optimization → Optimization Algorithm (used to train
models, not a model itself)
Overview
Batch Gradient Descent is an iterative first-order optimization algorithm used to minimize a loss/cost
function by updating model parameters in the direction of the negative gradient, computed using the
entire training dataset at every iteration.
How It Works
• Initializes model parameters (weights) randomly or with a chosen scheme.
• Computes the loss over the ENTIRE training dataset for the current parameters.
• Computes the gradient of the loss function with respect to each parameter using all training
examples.
• Updates parameters: theta = theta - learning_rate * gradient.
• Repeats the process for a fixed number of epochs or until convergence (loss change below a
threshold).
• Uses the full dataset per update, producing a smooth, stable descent path toward the minimum (for
convex problems, the global minimum).
Example:
Batch Gradient Descent – Simple Example
Suppose we want to predict a student's marks based on hours studied.
Training Data
Hours Studied Marks (y)
(x)
1 2
2 4
3 6
We use the linear regression model:
^y =wx+b Step 1: Initialize Parameters
Assume:
 Weight (w ) = 0
 Bias (b ) = 0
 Learning Rate (α ) = 0.1
Step 2: Predict Outputs
Using ^y =0 x +0 :
x Actual y Predicted ŷ
1 2 0
2 4 0
3 6 0
Step 3: Calculate Loss
The model computes the error using all three training examples.
Errors:
 Example 1: 0−2=−2
 Example 2: 0−4=−4
 Example 3: 0−6=−6
Step 4: Compute Gradient (Using Entire Dataset)
Since this is Batch Gradient Descent, all three examples are used together to calculate the gradient.
 Weight gradient:
∂J
=¿ ¿ Bias gradient:
∂w
∂ J −2−4−6
= =−4 Step 5: Update Parameters
∂b 3
Update rule:
∂J ∂ J Substitute the values:
w=w−α b=b−α
∂w ∂b
w=0−0.1(−9.33)=0.933b=0−0.1(−4)=0.4 Updated model:
^y =0.933 x +0.4 Step 6: Repeat
Using the updated values:
 w=0.933
 b=0.4
The algorithm again:
1. Predicts outputs for all training examples.
2. Computes the loss using the entire dataset.
3. Calculates new gradients.
4. Updates w and b .
This process continues until the loss becomes very small or the maximum number of epochs is
reached.
Key Point
In this example, the dataset contains 3 training samples. During every iteration, Batch Gradient
Descent uses all 3 samples together before updating the weights. This is why it is called Batch
Gradient Descent.
Key Characteristics
• Uses the full dataset for every parameter update (unlike Stochastic or Mini-batch Gradient Descent).
• Produces stable, low-variance gradient estimates and smooth convergence.
• Computationally expensive per iteration for large datasets since it requires a full pass over the data.
• Learning rate is the key hyperparameter controlling step size and convergence stability.
Advantages
• Stable and deterministic convergence for convex loss functions.
• Guaranteed convergence to global minimum for convex functions and to a local minimum for non-
convex ones.
Limitations
• Very slow and memory-intensive on large datasets since it requires the whole dataset per update.
• Cannot be used easily for online/streaming learning.
• May get stuck in local minima/saddle points for non-convex functions (e.g., deep networks).ts.

10 Real-Time Applications
# Application How It Is Used In Real Time
1 Linear/logistic Fitting regression and classification model coefficients on
regression training small-to-medium structured datasets.
2 Econometric model Estimating parameters of economic models where the full
fitting dataset fits in memory.
3 Small-scale neural Training compact models where full-batch updates are
network training computationally feasible.
4 Portfolio optimization Optimizing asset allocation weights to minimize risk/cost
functions over historical return data.
5 Control systems Optimizing controller parameters (e.g., PID) against a
parameter tuning defined cost function.
6 Signal processing filter Optimizing filter coefficients to minimize
design reconstruction/prediction error.
7 Recommender system Training matrix factorization models on datasets small
baseline training enough for full-batch updates.
8 Academic/research Used in research settings to validate optimization theory
prototyping and convergence behavior before scaling to mini-batch
methods.
9 Energy system Fitting parameters of load/demand prediction models over
modeling full historical datasets.
10 A/B testing and Fitting statistical models used in marketing analytics to
statistical modeling minimize prediction error over the complete experiment
dataset.
5. Genetic Programming (GP)
Classification: Computational Intelligence → Evolutionary Computation → Population-Based
Metaheuristic
Overview
Genetic Programming is an evolutionary computation technique that evolves computer programs
(typically represented as tree structures) to solve a problem, using biologically inspired operators —
selection, crossover, and mutation — applied over successive generations to optimize a fitness function.
How It Works
• Initializes a population of random candidate programs/expressions, usually represented as syntax
trees.
• Evaluates each individual's fitness by running it against the target problem (e.g., how well it fits
data or solves a task).
• Selects fitter individuals (e.g., via tournament or roulette-wheel selection) to become parents.
• Applies crossover — swapping subtrees between two parent programs to create offspring.
• Applies mutation — randomly altering a node/subtree to maintain diversity.
• Replaces the old population with the new generation and repeats the evaluate-select-reproduce
cycle.
• Terminates when a maximum number of generations is reached or a satisfactory fitness/solution is
found.
Example:
Suppose the goal is to find a mathematical expression that predicts student marks from the number of
hours studied.
Training Data
Hours Studied Marks (y)
(x)
1 2
2 4
3 6
4 8
1. Initialize Population
Generate a random population of candidate programs (mathematical expressions).
For example:
 Program A: y=x +1
 Program B: y=2 x
2
 Program C: y=x
 Program D: y=x +3
2. Evaluate Fitness
Test each program using the training data and calculate its error.
Example:
Program Expressio Fitness
n
A y=x +1 Medium
B y=2 x Excellent (Error = 0)
2
C y=x Poor
D y=x +3 Poor
Program B has the highest fitness because its predictions exactly match the actual values.
3. Selection
Choose the best-performing programs as parents.
Example:
 Parent 1: y=2 x
 Parent 2: y=x +3
These programs are selected because they have better fitness than the others.
4. Crossover
Exchange parts of the parent programs to create new offspring.
Example:
Parent 1:
2x
Parent 2:
x +3
Offspring:
2 x+3
The offspring combines characteristics of both parents.
5. Mutation
Randomly modify part of a program to introduce diversity.
Example:
Before mutation:
2 x+3
After mutation:
2 x+ 2
Mutation helps explore new solutions and prevents the population from becoming too similar.
6. Replace Population
Replace weaker programs with the newly generated offspring.
New population:
 y=2 x
 y=2 x +2
 y=x +1
2
 y=x
The next generation is expected to have better overall fitness.
7. Repeat Until Termination
Repeat the process:
 Evaluate fitness
 Select parents
 Perform crossover
 Apply mutation
 Create a new generation
The algorithm stops when:
 The maximum number of generations is reached, or
 A satisfactory solution is found.
Final Result
After several generations, Genetic Programming evolves the best expression:
y=2 x
This expression correctly predicts the student marks.
Key Characteristics
 • Solutions are entire programs/expressions (variable-length tree structures), not fixed-length vectors
as in standard Genetic Algorithms.
• Relies on stochastic search rather than gradient information.
Advantages
• Can discover interpretable closed-form solutions (symbolic regression) rather than black-box
models.
• Doesn't require gradient information — works on non-differentiable, discontinuous problems.
• Naturally explores diverse solution structures, useful for open-ended design problems.
Limitations
• Computationally expensive due to large populations and many generations.
• Risk of 'bloat' — programs growing unnecessarily large without fitness improvement.
• No convergence guarantees; results can vary between runs.
10 Real-Time Applications
# Application How It Is Used In Real Time
1 Symbolic regression for Automatically deriving mathematical formulas that fit
scientific discovery experimental physics or biology data.
2 Automated circuit Evolving analog/digital circuit topologies that meet
design performance specifications.
3 Financial trading Evolving trading rules/indicators that maximize returns on
strategy evolution historical market data.
4 Robot Evolving control programs that let robots walk, navigate,
controller/behavior or manipulate objects.
evolution
5 Antenna and hardware Evolving antenna geometries for spacecraft that meet strict
design (NASA) performance constraints.
6 Feature construction Evolving new composite features from raw variables to
for machine learning improve downstream model accuracy.
7 Game-playing agent Evolving strategies/heuristics for board games or real-time
strategy evolution strategy game AI.
8 Image processing filter Evolving pixel-level operators for tasks such as edge
design detection or noise removal.
9 Supply chain and Evolving heuristic scheduling rules for job-shop or
scheduling optimization logistics optimization problems.
10 Bioinformatics pattern Evolving classifiers/rules that detect gene expression
discovery patterns associated with disease.
6. Support Vector Regression (SVR)
Classification: Machine Learning → Supervised Learning → Regression → Kernel-Based Method
(extension of SVM)
Overview
Support Vector Regression applies the principles of Support Vector Machines to regression problems.
Instead of fitting a line that minimizes squared error everywhere, SVR fits a function within an epsilon-
insensitive margin (tube), only penalizing predictions that fall outside this tube, and uses kernel
functions to model non-linear relationships.
How It Works
• Defines an epsilon-insensitive loss function: errors smaller than epsilon are ignored (no penalty).
• Seeks a function f(x) = w·phi(x) + b that is as flat as possible while keeping most training points
within the epsilon-tube.
• Introduces slack variables to allow some points to fall outside the tube, controlled by regularization
parameter C.
• Uses the kernel trick (linear, polynomial, RBF) to map inputs into a higher-dimensional space,
enabling non-linear regression without explicit transformation.
• Solves a convex quadratic optimization problem to find support vectors — the subset of training
points that define the regression function.
• Predicts new values using only the support vectors and the learned kernel-based function.
Support Vector Regression (SVR) – Example
Suppose we want to predict house prices based on the size of the house.
Training Data
House Size ([Link].) Price ($1000)
1000 200
1200 240
1500 300
1800 360
2000 400
Assume the SVR model uses:
 Kernel = Linear
 Epsilon (ε ) = 20
 Regularization parameter (C ) = 100
Step 1: Define the Epsilon Tube
The epsilon value creates a margin around the regression function.
If the predicted price is 300, then the acceptable range is:
300 ± 20
which means any actual price between 280 and 320 is considered accurate enough and receives no
penalty.
Step 2: Train the Model
Suppose the regression function learned by SVR is:
f (x)=0.2 x where:
 x = House size
 f (x)= Predicted house price
Step 3: Check Predictions
House Size Actual Price Predicted Price Within ε-Tube?
 1000 200 200 Yes
1200 240 240 Yes
1500 300 300 Yes
1800 360 360 Yes
2000 400 400 Yes
All predictions lie inside the epsilon tube, so no loss is incurred.
Step 4: Example Outside the Tube
Suppose another house has:
 Size = 1700 [Link].
 Actual Price = 390
Prediction:
f (1700)=0.2 ×1700=340Difference:
390−340=50 Since the error (50) is greater than epsilon (20), the model applies a penalty and adjusts
the regression function during training.
Step 5: Support Vectors
Only the data points that lie on or outside the epsilon tube become support vectors.
Example:
 (1700, 390) → Support Vector
 (1500, 300) → Not a support vector (inside the tube)
Final Prediction
For a new house of 1600 [Link].:
f (1600)=0.2 ×1600=320Key Characteristics
• Key hyperparameters: kernel type, epsilon (tube width), C (regularization/penalty), and kernel-
specific parameters (e.g., gamma for RBF).
• Sparse solution — only support vectors influence predictions.
Advantages
• Robust to outliers within the epsilon margin.
• Effective in high-dimensional and non-linear regression tasks via kernels.
• Global optimum guaranteed due to convex formulation.
Limitations
• Computationally expensive on very large datasets (training scales poorly).
• Sensitive to feature scaling.
10 Real-Time Applications
# Application How It Is Used In Real Time
1 Stock price and financial Predicting asset prices or volatility from historical and technical
forecasting indicator features.
2 Electricity load Predicting short-term power demand from weather and
forecasting historical consumption data.
3 Weather and rainfall Modeling non-linear relationships between meteorological
prediction variables and rainfall/temperature.
4 Real estate price Estimating property values from features like location, size, and
prediction amenities.
5 Air quality index Forecasting pollutant concentration levels from sensor and
prediction meteorological data.
6 Biomedical signal analysis Predicting physiological parameters (e.g., blood glucose levels)
 from sensor time-series data.
7 Chemical process Predicting product yield/quality from process parameters in
modeling chemical engineering.
8 Traffic flow prediction Forecasting vehicle traffic volume/speed for intelligent
transportation systems.
9 Water quality/demand Predicting water consumption or contamination levels from
forecasting environmental data.
10 Remaining useful life Predicting time-to-failure of machinery components from
(RUL) estimation sensor degradation data.
7. Gaussian Mixture Model (GMM)
Classification: Machine Learning / Statistics → Unsupervised Learning → Probabilistic Clustering /
Density Estimation
Overview
A Gaussian Mixture Model represents a probability distribution as a weighted sum of multiple Gaussian
(normal) distributions. It is used for soft clustering and density estimation, assigning each data point a
probability of belonging to each of several underlying Gaussian components rather than a single hard
cluster label.

How It Works
• Assumes the data is generated from a mixture of K Gaussian distributions, each with its own mean,
covariance, and mixing weight.
• Initializes the parameters of each Gaussian component (often via K-Means for a good starting
point).
• E-step (Expectation): computes the posterior probability (responsibility) that each data point
belongs to each Gaussian component, given current parameters.
• M-step (Maximization): updates each component's mean, covariance, and mixing weight using the
responsibilities computed in the E-step.
• Repeats the Expectation-Maximization (EM) cycle until the log-likelihood converges.
• Produces soft cluster assignments (probabilities) rather than hard labels, and can also estimate the
overall data density.
Key Characteristics
• Trained via the Expectation-Maximization (EM) algorithm.
• Number of components K is a key hyperparameter (chosen via BIC/AIC or domain knowledge).
• Provides soft, probabilistic cluster membership, unlike hard-clustering methods like K-Means.
• Can model clusters of different shapes and sizes via full covariance matrices.
Advantages
• Captures uncertainty in cluster assignment through soft probabilities.
• More flexible than K-Means — can model elliptical, correlated clusters.
• Provides a generative probabilistic model that can also be used for density estimation and anomaly
detection.
Limitations
• Sensitive to initialization; may converge to a local optimum.
• Requires specifying the number of components in advance.
• Assumes data within each cluster is Gaussian-distributed, which may not hold in practice.
• Computationally more expensive than K-Means.
10 Real-Time Applications
# Application How It Is Used In Real Time
1 Customer segmentation Grouping customers into overlapping behavioral
segments with probabilistic membership for targeted
marketing.
2 Speaker Modeling voice feature distributions (e.g., MFCCs) for
identification/verification speaker recognition systems.
3 Anomaly/fraud detection Modeling the density of normal transaction behavior and
flagging low-probability points as anomalies.
4 Image segmentation Segmenting an image into regions by modeling pixel
color/intensity distributions as a Gaussian mixture.
5 Background subtraction Modeling pixel intensity over time as a mixture of
in video surveillance Gaussians to separate moving foreground objects from
static background.
6 Financial market regime Identifying different market 'regimes' (e.g.,
detection bull/bear/volatile) as separate Gaussian components in
returns data.
7 Bioinformatics gene Grouping genes/samples with similar expression profiles
expression clustering under probabilistic cluster membership.
8 Handwriting and gesture Modeling variations in stroke/gesture features as mixtures
recognition of Gaussians for classification.
9 Astronomy — Modeling distributions of astronomical object features to
star/galaxy classification classify or cluster celestial bodies.
10 Recommender systems Modeling latent user-preference distributions as mixtures
(latent taste modeling) to generate personalized recommendations.
8. One-Class SVM
Classification: Machine Learning → Unsupervised / Semi-Supervised Learning → Anomaly/Novelty
Detection → Kernel-Based Method
Overview
One-Class SVM is a variant of Support Vector Machines designed for novelty and outlier detection
when only 'normal' class data is available. It learns a decision boundary that encloses the region of
normal data in feature space, so that new points falling outside this boundary are flagged as anomalies.
How It Works
• Maps input data into a high-dimensional feature space using a kernel function (commonly RBF).
• Finds a hyperplane (or hypersphere, in the related SVDD formulation) that separates the mapped
data points from the origin with maximum margin.
• Introduces a parameter nu that controls the trade-off between the fraction of outliers allowed in
training and the fraction of support vectors used.
• Solves a convex quadratic optimization problem to determine the decision boundary using only
'normal' training examples.
• Classifies new points as +1 (normal/inlier) if they fall within the learned region, or -1
(anomaly/outlier) if they fall outside it.
• Uses only support vectors (a subset of training points) to define the final decision function.
Key Characteristics
• Trained using only one class of data (normal/non-anomalous examples).
• Key hyperparameters: kernel type/parameters and nu (upper bound on outlier fraction).
• Produces a binary decision (inlier vs. outlier), not a probability score by default.
• Effective in high-dimensional feature spaces via the kernel trick.
Advantages
• Does not require labeled anomaly examples — only 'normal' data is needed for training.
• Effective for high-dimensional, non-linear novelty detection tasks.
• Well-founded theoretical basis (maximum margin, convex optimization).
Limitations
• Sensitive to choice of kernel and hyperparameters (nu, gamma).
• Scales poorly to very large datasets (like standard SVMs).
• Performance can degrade in very high-dimensional or noisy feature spaces.
• Does not naturally provide interpretable anomaly scores/probabilities.
10 Real-Time Applications
# Application How It Is Used In Real Time
1 Network intrusion Learning the profile of normal network traffic and flagging
detection deviations as potential cyberattacks.
2 Credit card fraud Modeling typical spending behavior and flagging unusual
detection transactions as potential fraud.
3 Industrial equipment Learning normal sensor signal patterns (vibration,
fault detection temperature) and detecting early signs of machine failure.
4 Medical anomaly Detecting abnormal patterns in medical imaging or
detection physiological signals (e.g., ECG) that differ from healthy
 baselines.
5 Manufacturing defect Flagging products whose sensor/image features deviate
detection from the normal production profile.
6 Cybersecurity — Modeling normal software/system behavior and flagging
malware detection processes that deviate as potentially malicious.
7 Environmental Detecting unusual pollution/sensor readings that deviate
monitoring from typical environmental baselines.
8 Video surveillance Identifying unusual events/movements in video feeds that
anomaly detection differ from normal activity patterns.
9 Server/system Detecting unusual system metrics (CPU, memory, latency)
performance that signal outages or attacks.
monitoring
10 Insurance claim Flagging claims that deviate significantly from typical
anomaly detection claim patterns for further investigation.
9. N-grams
Classification: Natural Language Processing / Computational Linguistics → Statistical Language
Modeling → Text Representation Technique
Overview
An N-gram is a contiguous sequence of N items (typically words or characters) extracted from a text. N-
gram models estimate the probability of a word given the previous N-1 words, forming the basis of
classical statistical language models and a common feature representation technique in NLP.
How It Works
• Tokenizes text into words or characters.
• Slides a window of size N over the token sequence to extract overlapping N-grams (unigrams N=1,
bigrams N=2, trigrams N=3, etc.).
• Counts the frequency of each N-gram in a training corpus.
• Estimates conditional probabilities P(word_i | word_i-N+1, ..., word_i-1) using these frequency
counts (Maximum Likelihood Estimation).
• Applies smoothing techniques (Laplace, Kneser-Ney) to handle unseen N-grams and avoid zero
probabilities.
• Uses these probabilities to predict the next word, score sentence likelihood, or generate a feature
vector for downstream classifiers.
N-grams – Example
Suppose the input sentence is:
"I love machine learning"
Step 1: Tokenization
Split the sentence into words:
[I, love, machine, learning]
Step 2: Generate N-grams
Unigrams (N = 1)
Each individual word is a unigram.
I
love
machine
learning
Bigrams (N = 2)
Take two consecutive words.
I love
love machine
machine learning
Trigrams (N = 3)
Take three consecutive words.
I love machine
love machine learning
Step 3: Count Frequencies
Suppose the following sentences are in the training corpus:
1. I love machine learning
2. I love deep learning
3. I love machine vision
The bigram frequencies are:
 Bigram Frequency
I love 3
love machine 2
machine 1
learning
love deep 1
deep learning 1
machine vision 1
Step 4: Calculate Probability
Suppose we want to predict the next word after:
I love
Using Maximum Likelihood Estimation (MLE):
Count(I love machine)
P(machine ∣I love)=
Count(I love)
Here,
 Count(I love machine) = 2
 Count(I love) = 3
Therefore,
2
P(machine ∣I love)= =0.67
3
Similarly,
1
P(deep ∣ I love)= =0.33
3
Since 0.67 > 0.33, the model predicts "machine" as the next word.
Step 5: Smoothing
Suppose the sentence is:
I love artificial
If "I love artificial" never appears in the training data, its probability becomes 0.
Using smoothing techniques (such as Laplace smoothing), a small probability is assigned instead of
zero so the model can still make predictions.
Final Prediction
Input:
I love
Possible next words:
Next Word Probability
machine 0.67
deep 0.33
Predicted next word: machine

Key Characteristics
• N is a key hyperparameter — larger N captures more context but suffers from data sparsity.
• Purely statistical/count-based, not a learned neural representation.
• Basis of classical language models, text classification (bag-of-N-grams), and information retrieval
features.
• Requires smoothing to handle sparse/unseen sequences.

Advantages
 • Simple, fast, and interpretable.
• Requires no complex training — just counting frequencies.
• Effective baseline feature representation for many text classification tasks.
Limitations
• Suffers from data sparsity as N increases (combinatorial explosion of possible N-grams).
• Cannot capture long-range dependencies beyond the window size N.
• Ignores semantic meaning — treats different words as entirely unrelated (no notion of synonymy).
• Largely superseded by neural language models (e.g., Transformers) for capturing context.
10 Real-Time Applications
# Application How It Is Used In Real Time
1 Predictive Powering keyboard word suggestion and autocomplete
text/autocomplete features on mobile devices.
2 Spelling and grammar Identifying unlikely word sequences to suggest corrections
correction in word processors.
3 Spam email Using bag-of-N-grams features to distinguish spam from
classification legitimate email content.
4 Machine translation Serving as the backbone language model component in
(statistical MT) classical statistical machine translation systems.
5 Speech recognition Providing a language model that scores the likelihood of
candidate word sequences from acoustic model outputs.
6 Plagiarism detection Comparing N-gram overlap between documents to detect
copied or paraphrased text.
7 Sentiment analysis Using N-gram features (e.g., 'not good' as a bigram) as
input to sentiment classifiers.
8 Search engine query Suggesting likely next query terms based on common N-
suggestion gram sequences from search logs.
9 Authorship Comparing character/word N-gram frequency profiles to
attribution/forensic identify likely authors of a text.
linguistics
10 DNA/genomic sequence Applying N-gram (k-mer) analysis to identify recurring
analysis patterns in genetic sequences.
10. Hierarchical Clustering
Classification: Machine Learning / Statistics → Unsupervised Learning → Clustering → Connectivity-
Based Method
Overview
Hierarchical Clustering builds a tree-like hierarchy of nested clusters, either by progressively merging
smaller clusters into larger ones (agglomerative, bottom-up) or by splitting a large cluster into smaller
ones (divisive, top-down). The result is typically visualized as a dendrogram, from which any number of
clusters can be obtained by cutting at a chosen level.
How It Works
• Agglomerative approach: starts with each data point as its own cluster.
• Computes pairwise distances/similarities between all clusters using a chosen distance metric
(Euclidean, Manhattan, cosine).
• Merges the two closest clusters at each step, based on a linkage criterion (single, complete, average,
or Ward's method).
• Repeats the merge process, recording each merge and the distance at which it occurred, until all
points form a single cluster.
• Builds a dendrogram representing the nested merge history.
• Obtains a specific number of clusters by 'cutting' the dendrogram at an appropriate height/distance
threshold.
• (Divisive approach performs the reverse: starts with all points in one cluster and recursively splits
them.)
Example of Hierarchical Clustering (Agglomerative Method)
Suppose we have 6 students grouped based on their marks in Mathematics and Science.
Studen Mathematics Science
t
A 40 45
B 42 47
C 43 46
D 80 82
E 81 84
F 83 81

Step 1: Initial Clusters

Initially, each student is considered as an individual cluster.
{A} {B} {C} {D} {E} {F}
Step 2: Find the Closest Pair
Using Euclidean distance, we calculate distances between all pairs.
Some distances are:
 Distance(A, B) ≈ 2.83
 Distance(B, C) ≈ 1.41
 Distance(D, F) ≈ 3.16
 Distance(D, E) ≈ 2.24
The smallest distance is between B and C.
Merge them:
{A} {B,C} {D} {E} {F}
Step 3: Merge Again
Now compute distances between the new clusters.
The cluster {B,C} is closest to A.
Merge:
{A,B,C} {D} {E} {F}
Step 4: Continue Merging
Among the remaining clusters:
 D and E are very close.
 Merge them.
{A,B,C} {D,E} {F}
Step 5: Merge Again
Cluster {D,E} is closest to F.
Merge:
{A,B,C} {D,E,F}
Step 6: Final Merge
Finally, the two large clusters are merged.
{A,B,C,D,E,F}
Choosing the Number of Clusters
If we cut the dendrogram before the final merge, we obtain:
Cluster 1 = {A, B, C}

Cluster 2 = {D, E, F}
These represent two natural groups:
 Cluster 1: Students with lower marks.
 Cluster 2: Students with higher marks.

Key Characteristics
• Does not require specifying the number of clusters in advance (unlike K-Means).
• Produces a full hierarchy (dendrogram), useful for exploring data at multiple granularities.
• Choice of linkage method significantly affects cluster shape and quality.
• Computationally expensive: typically O(n^2 log n) or worse, limiting scalability to large datasets.
Advantages
• No need to pre-specify the number of clusters.
• Produces an interpretable dendrogram showing nested cluster relationships.
• Can capture non-convex or nested cluster structures depending on linkage method.
Limitations
• Computationally expensive and memory-intensive for large datasets.
• Merge/split decisions are irrevocable (greedy) — cannot correct earlier mistakes.
• Sensitive to noise and outliers, especially with single linkage (chaining effect).
• Choice of distance metric and linkage method strongly affects results.
10 Real-Time Applications
# Application How It Is Used In Real Time
1 Gene expression and Clustering genes or species by expression/similarity
phylogenetic analysis profiles to build evolutionary trees (phylogenetics).
2 Customer segmentation Building a hierarchy of customer groups at multiple levels
of granularity for marketing strategy.
3 Document/text Organizing large document collections into nested topic
clustering hierarchies for browsing and retrieval.
4 Social network Identifying nested community structures among users
community detection based on interaction patterns.
5 Image segmentation Grouping pixels/regions hierarchically based on
color/texture similarity for object segmentation.
6 Market basket/product Grouping products into nested category hierarchies based
categorization on co-purchase patterns.
7 Anomaly detection in Detecting nodes/sensors with atypical behavior that don't
sensor networks fit naturally into the emerging hierarchy.
8 Recommendation Constructing hierarchical taxonomies of items/content for
system taxonomy structured recommendations.
building
9 Epidemiological Grouping disease cases by geographic/temporal/genetic
outbreak clustering similarity to trace outbreak clusters.
10 Financial portfolio/asset Grouping stocks/assets by correlation structure to build
clustering diversified, hierarchically-clustered portfolios.
11. Deep Q-Network (DQN)
Classification: Machine Learning → Reinforcement Learning → Value-Based Method + Deep
Learning (Convolutional/Feedforward Neural Network as function approximator)
Overview
Deep Q-Network combines Q-Learning, a classical reinforcement learning algorithm, with deep neural
networks to approximate the action-value function Q(s,a) for high-dimensional state spaces (such as raw
pixel images), enabling an agent to learn optimal policies directly from sensory input, famously
demonstrated on Atari games.

How It Works
• An agent observes the current state s (e.g., a game screen) and uses a neural network to estimate
Q(s,a) for every possible action a.
• Selects an action using an exploration strategy (commonly epsilon-greedy: mostly exploit the best-
known action, sometimes explore randomly).
• Executes the action, observes the reward r and next state s', and stores the transition (s, a, r, s') in a
replay buffer (experience replay).
• Samples random mini-batches from the replay buffer to break correlation between consecutive
experiences and stabilize training.
• Computes the target Q-value using the Bellman equation: target = r + gamma * max_a' Q_target(s',
a'), where Q_target comes from a separate, periodically-updated target network for stability.
• Updates the main Q-network's weights by minimizing the loss (typically MSE or Huber loss)
between predicted Q(s,a) and the target.
• Repeats over many episodes, gradually improving the policy as Q-value estimates converge.
Key Characteristics
• Core innovations: experience replay and a separate target network, both critical for training
stability.
• Uses deep neural networks (often CNNs for image-based states) as function approximators for Q-
values.
• Off-policy learning — learns the optimal policy while following an exploratory behavior policy.
• Extensions include Double DQN, Dueling DQN, and Prioritized Experience Replay, which improve
stability and performance.
Advantages
• Can learn directly from high-dimensional raw sensory input (e.g., pixels) without hand-crafted
features.
 • Achieves human-level or superhuman performance in various sequential decision-making tasks.
• Experience replay improves sample efficiency and training stability.
Limitations
• Requires large amounts of interaction/training data (sample inefficiency).
• Training can be unstable and sensitive to hyperparameters.
• Struggles with continuous action spaces (designed for discrete actions).
• Can overestimate Q-values, partially addressed by Double DQN.
10 Real-Time Applications
# Application How It Is Used In Real Time
1 Game playing AI Learning to play Atari and other video games directly
from pixel input, achieving superhuman performance in
several titles.
2 Robotics control Learning discrete control policies for robotic manipulation
and navigation tasks.
3 Autonomous vehicle Learning high-level discrete decisions (lane change,
decision-making stop/go) in simulated driving environments.
4 Traffic signal control Learning adaptive traffic light timing policies to reduce
optimization congestion.
5 Resource allocation in Learning policies for dynamic bandwidth/resource
networks allocation in telecommunications networks.
6 Algorithmic trading Learning discrete buy/hold/sell trading policies based on
strategy learning market state representations.
7 Energy grid Learning policies for demand response and energy storage
management dispatch decisions.
8 Data center cooling Learning control policies to reduce energy consumption in
optimization cooling systems (as demonstrated in real data-center
deployments).
9 Personalized Modeling recommendation as a reinforcement learning
recommendation as problem to optimize long-term user engagement.
sequential decisions
10 Warehouse/inventory Learning restocking and dispatch policies to optimize
management inventory levels and fulfillment efficiency.
12. Bayesian Networks
Classification: Machine Learning / Statistics → Probabilistic Graphical Model → Directed Acyclic
Graph (DAG)-Based Model
Overview
A Bayesian Network is a probabilistic graphical model that represents a set of random variables and
their conditional dependencies using a directed acyclic graph (DAG). Each node represents a variable,
edges represent direct probabilistic dependencies, and each node has a conditional probability table
(CPT) defining its distribution given its parents.
How It Works
• Defines a set of random variables as nodes in a directed acyclic graph.
• Draws directed edges between nodes to represent direct causal or probabilistic dependencies.
• Assigns a conditional probability table (CPT) to each node, specifying P(node | parents(node)).
• Uses the chain rule of probability factored according to the graph structure to compute the joint
probability distribution: P(X1,...,Xn) = product of P(Xi | parents(Xi)).
• Performs inference — computing the probability of unobserved variables given observed evidence
— using algorithms like variable elimination, belief propagation, or approximate methods (MCMC
sampling).
• Can learn both the graph structure and the CPT parameters from data (structure learning and
parameter learning), or these can be specified by domain experts.
Bayesian Networks – Example
Suppose we want to predict whether a person has the Flu based on symptoms.
Variables
 Flu (F) – Yes/No
 Fever (Fe) – Yes/No
 Cough (C) – Yes/No
Bayesian Network Structure
Flu
/ \
Fever Cough
 Flu is the parent node.
 Fever and Cough depend on whether the person has the flu.
Step 1: Define Conditional Probability Tables (CPT)
Probability of Flu:
Flu Probability
Ye 0.10
s
No 0.90
Probability of Fever given Flu:
Flu Fever=Yes Fever=No
Ye 0.90 0.10
s
No 0.20 0.80
Probability of Cough given Flu:
Flu Cough=Yes Cough=No
Yes 0.80 0.20
No 0.30 0.70
Step 2: Compute Joint Probability
Suppose we want to find the probability that a person:
 Has Flu = Yes
 Has Fever = Yes
 Has Cough = Yes
Using the chain rule:
P(Flu , Fever ,Cough)=P( Flu)× P( Fever ∣ Flu)× P(Cough ∣ Flu)
Substitute the values:
¿ 0.10 × 0.90× 0.80¿ 0.072
So,
P(Flu , Fever ,Cough)=0.072
This means there is a 7.2% probability that a randomly selected person has flu, fever, and cough
simultaneously.
Step 3: Inference
Suppose a patient has:
 Fever = Yes
 Cough = Yes
The Bayesian Network uses the CPTs to infer the probability that the patient has Flu.
Since both symptoms are highly dependent on Flu, the probability of Flu increases compared to the
initial probability (10%).

Key Characteristics
• Structure: Directed Acyclic Graph (DAG) encoding conditional independence assumptions.
• Supports both forward reasoning (cause to effect) and backward/diagnostic reasoning (effect to
cause).
• Can be learned from data or built from expert/domain knowledge.
• Exact inference is NP-hard in general; approximate inference methods are used for large networks.
Advantages
• Explicitly models uncertainty and probabilistic dependencies in an interpretable graphical structure.
• Naturally handles missing data and supports both diagnostic and predictive reasoning.
• Combines expert knowledge with data-driven learning.
Limitations
• Exact inference becomes computationally intractable for large, densely connected networks.
• Structure learning from data is a challenging combinatorial problem.
• Requires careful specification of conditional probability tables, especially with limited data.
• Assumes a valid causal/dependency structure, which may be difficult to determine correctly.
10 Real-Time Applications
# Application How It Is Used In Real Time
1 Medical diagnosis Modeling probabilistic relationships between symptoms,
systems test results, and diseases to support diagnostic decision-
making.
2 Risk assessment in Modeling dependencies between economic factors and
finance default/credit risk for risk management.
3 Fault diagnosis in Identifying probable causes of equipment failure from
engineering systems observed sensor readings/symptoms.
4 Spam filtering (Naive Modeling probabilistic dependence between
Bayes networks) words/features and spam classification.
5 Genetics and Modeling gene regulatory networks and inferring gene
bioinformatics interactions from expression data.
6 Natural disaster/risk Modeling dependencies between environmental factors to
prediction predict flood, earthquake, or wildfire risk.
7 Autonomous vehicle Combining uncertain sensor readings (lidar, camera, radar)
sensor fusion to infer the true state of the environment.
8 Legal/forensic Modeling evidentiary reasoning and probability of
reasoning guilt/innocence based on interdependent evidence.
9 Cybersecurity threat Modeling dependencies between system vulnerabilities
modeling and attack likelihood for risk prioritization.
10 Predictive maintenance Modeling dependencies between sensor readings and
in industry component degradation to predict failures.
13. T5 (Text-to-Text Transfer Transformer)
Classification: Deep Learning → Natural Language Processing → Transformer Architecture
(Encoder-Decoder, Sequence-to-Sequence Model)
Overview
T5 (Text-to-Text Transfer Transformer), developed by Google, is a Transformer-based encoder-decoder
model that reframes every NLP task — translation, summarization, classification, question answering —
as a unified text-to-text problem, where both input and output are always text strings. It is pretrained on
a large text corpus using a 'span corruption' objective and then fine-tuned on downstream tasks.
How It Works
• Casts every NLP task into a common 'text-to-text' format by prefixing inputs with a task descriptor
(e.g., 'translate English to German: ...', 'summarize: ...').
• Uses a Transformer encoder-decoder architecture: the encoder processes the input text using self-
attention, and the decoder generates output text autoregressively using self-attention plus cross-
attention to the encoder's output.
• Pretrains on a large unlabeled text corpus (the C4 dataset) using a denoising objective where
random spans of text are masked and the model learns to reconstruct them.
• Fine-tunes the pretrained model on specific downstream tasks (translation, summarization, question
answering, classification) using labeled task-specific data.
• Generates output tokens one at a time during inference, conditioning each new token on previously
generated tokens and the encoded input.
• Comes in multiple sizes (Small, Base, Large, 3B, 11B), trading off capacity and computational cost.
T5 (Text-to-Text Transfer Transformer) – Example
Suppose we want to use T5 for different NLP tasks. T5 converts every task into a text-to-text format by
adding a task prefix.
Example 1: Translation
Input
translate English to French: Good morning
Output
Bonjour
Example 2: Summarization
Input
summarize: Artificial Intelligence is transforming healthcare by helping doctors diagnose diseases faster
and more accurately.
Output
AI helps doctors diagnose diseases more accurately.
Example 3: Question Answering
Input
question: What is the capital of France?
context: France is a country in Europe. Its capital is Paris.
Output
Paris
Example 4: Text Classification
Suppose we want to classify the sentiment of a movie review.
Input
sentiment: The movie was amazing and the acting was excellent.
Output
positive
How T5 Processes the Input
Step 1: Add Task Prefix
For translation:
translate English to French: Good morning
Step 2: Encoder
The encoder reads and understands the entire input sentence using self-attention.
Step 3: Decoder
The decoder generates the output one word at a time.
For example:
 Step 1 → "Bon"
 Step 2 → "Bonjour"
Final output:
Bonjour
Pretraining Example
Original sentence:
The cat is sleeping on the sofa.
After masking a span:
The <extra_id_0> on the sofa.
Target output:
<extra_id_0> cat is sleeping
During pretraining, T5 learns to reconstruct the missing text.
Final Result
Task Input Output
Translation translate English to French: Good Bonjour
morning
Summarization summarize: AI is transforming AI helps doctors diagnose diseases more
healthcare... accurately.
Question question + context Paris
Answering
Sentiment sentiment: The movie was amazing. positive
Analysis

Key Characteristics
• Unified text-to-text framework — a single model/training approach handles many different NLP
tasks.
• Encoder-decoder Transformer architecture (unlike encoder-only BERT or decoder-only GPT).
• Pretrained with a span-corruption (denoising) self-supervised objective on massive text data.
• Highly transferable via fine-tuning to numerous downstream NLP tasks.
Advantages
 • Single, unified framework simplifies applying one architecture across many diverse NLP tasks.
• Strong transfer learning performance due to large-scale pretraining.
• Flexible — easily adapted to new tasks by reformatting them as text-to-text problems.
Limitations
• Computationally expensive to pretrain and fine-tune, especially larger variants.
• Autoregressive decoding makes inference slower compared to non-generative classifiers.
• Like other large language models, can produce factually incorrect or biased outputs (hallucination).
• Requires significant infrastructure (GPUs/TPUs) for training and deployment at scale.
10 Real-Time Applications
# Application How It Is Used In Real Time
1 Document/article Generating concise summaries of news articles, research
summarization papers, or long reports.
2 Machine translation Translating text between languages by framing translation
as a text-to-text generation task.
3 Question answering Generating direct textual answers to questions posed over
systems a passage or knowledge base.
4 Text classification and Classifying text (e.g., sentiment, topic) by generating the
sentiment analysis class label as text output.
5 Chatbots and Powering dialogue systems that generate contextually
conversational AI relevant text responses.
6 Grammar correction Rewriting or correcting grammatically incorrect sentences
and text rewriting into fluent text.
7 Code generation and Fine-tuned variants generate code snippets or
documentation documentation from natural language descriptions.
8 Search engine query Reformulating or expanding search queries to improve
understanding retrieval relevance.
9 Content paraphrasing Generating paraphrased versions of text for content
variation or data augmentation.
10 Customer support Generating automated responses to customer
automation queries/tickets based on historical support text.
14. ESRGAN (Enhanced Super-Resolution Generative Adversarial Network)
Classification: Deep Learning → Computer Vision → Generative Adversarial Network (GAN) →
Image Super-Resolution
Overview
ESRGAN is a deep learning model that upscales low-resolution images into high-resolution ones with
realistic, perceptually convincing detail. It improves upon the original SRGAN by introducing Residual-
in-Residual Dense Blocks (RRDB), removing batch normalization, and using a relativistic discriminator,
producing sharper and more natural textures.

How It Works
• Uses a generator network built from Residual-in-Residual Dense Blocks (RRDB) — deep residual
blocks with dense connections — to progressively extract and upscale image features without batch
normalization artifacts.
• Upsamples the low-resolution input through the generator to produce a high-resolution output
image.
• Uses a discriminator network trained to distinguish between real high-resolution images and the
generator's super-resolved outputs.
• Employs a Relativistic average GAN (RaGAN) discriminator, which predicts the probability that a
real image is relatively more realistic than a fake one (rather than an absolute real/fake judgment),
improving training stability and detail sharpness.
• Trains the generator using a combination of losses: pixel-wise loss (L1), perceptual loss (comparing
deep VGG feature representations rather than raw pixels), and adversarial loss from the
discriminator.
• Iteratively trains generator and discriminator in an adversarial min-max game until the generator
produces photorealistic high-resolution images.
Key Characteristics
• Core architecture: Residual-in-Residual Dense Blocks (RRDB) without batch normalization.
• Uses perceptual loss (VGG feature-space loss) in addition to pixel loss for more realistic textures.
• Uses a relativistic discriminator instead of a standard binary real/fake discriminator.
• Belongs to the GAN family — trained via adversarial generator-discriminator competition.
Advantages
• Produces sharper, more photorealistic textures than pixel-loss-only super-resolution methods.
• Removing batch normalization reduces artifacts and improves generalization across image types.
• Perceptual + adversarial loss combination captures fine detail that plain MSE-based models blur
out.
Limitations
• GAN training can be unstable and prone to mode collapse or artifacts if not carefully tuned.
 • Computationally intensive, requiring significant GPU resources for training.
• Can occasionally hallucinate unrealistic details not present in the original low-resolution image.
• Requires large, high-quality paired training datasets for best results.
10 Real-Time Applications
# Application How It Is Used In Real Time
1 Old photo and film Upscaling and enhancing degraded historical photographs
restoration or film footage to modern resolution.
2 Medical imaging Enhancing resolution of MRI/CT/X-ray scans to reveal
enhancement finer diagnostic detail.
3 Satellite and aerial Improving resolution of satellite images for better analysis
imagery enhancement in agriculture, urban planning, and defense.
4 Video streaming Upscaling lower-resolution video streams to higher
upscaling resolution displays in real time or near-real time.
5 Surveillance footage Enhancing low-resolution CCTV footage to improve facial
enhancement or license-plate recognition in forensic investigations.
6 E-commerce product Upscaling product photos to high resolution for better
image enhancement online catalog presentation.
7 Gaming texture Enhancing low-resolution game textures/assets to higher
upscaling quality for remastered titles.
8 Print media and Upscaling low-resolution digital images for high-quality
publishing print reproduction.
9 Facial recognition Enhancing low-resolution facial images before feeding
preprocessing them into recognition pipelines.
10 Art and animation Enhancing resolution of digital art, anime, or animation
upscaling frames for remastering or restoration projects.
15. FCN (Fully Convolutional Network)
Classification: Deep Learning → Computer Vision → Convolutional Neural Network → Semantic
Segmentation
Overview
A Fully Convolutional Network is a deep learning architecture designed for pixel-wise prediction tasks,
most notably semantic segmentation. It replaces the fully connected layers of a traditional CNN
classifier with convolutional layers, allowing the network to take input images of arbitrary size and
produce spatially corresponding output maps (e.g., a class label per pixel).

How It Works
• Takes a standard CNN classification backbone (e.g., VGG, ResNet) and converts its fully connected
layers into equivalent 1x1 convolutional layers, preserving spatial information.
• Passes the input image through convolutional and pooling layers, progressively downsampling it
while extracting increasingly abstract features.
• Produces a coarse, low-resolution heatmap of class scores at the end of the downsampling path.
• Uses transposed convolutions (deconvolution/upsampling layers) to upsample the coarse feature
maps back to the original input resolution.
• Employs skip connections that combine (add) feature maps from earlier, higher-resolution layers
with the upsampled output, recovering fine spatial detail lost during downsampling (as in FCN-8s,
FCN-16s architectures).
• Outputs a dense, pixel-wise class probability map the same size as the input image, assigning a class
label to every pixel.
Key Characteristics
• Fully convolutional — no dense/fully-connected layers, enabling variable input image sizes.
• Uses an encoder (downsampling) - decoder (upsampling) structure with skip connections.
• Trained end-to-end with a per-pixel loss function (e.g., cross-entropy per pixel).
• Foundational architecture that inspired later segmentation models like U-Net and SegNet.
Advantages
• Can process input images of any size due to the absence of fixed-size fully connected layers.
• End-to-end trainable for dense pixel-level prediction tasks.
• Skip connections help recover fine spatial detail, improving segmentation boundary accuracy.
Limitations
• Coarse upsampling can still lose fine detail compared to more advanced architectures (e.g., U-Net).
• Computationally intensive, especially for high-resolution images.
 • Requires large amounts of pixel-level annotated training data, which is expensive to produce.
• May struggle with small or thin objects in the segmentation mask.
10 Real-Time Applications
# Application How It Is Used In Real Time
1 Autonomous driving Segmenting road scenes into classes like road, pedestrian,
scene understanding vehicle, and sky for self-driving car perception.
2 Medical image Segmenting organs, tumors, or lesions in MRI/CT scans
segmentation for diagnosis and treatment planning.
3 Satellite/aerial land Segmenting satellite imagery into land-use categories
cover classification (urban, forest, water) for environmental monitoring.
4 Agricultural crop/weed Segmenting drone/field imagery to distinguish crops from
segmentation weeds for precision agriculture.
5 Video surveillance Segmenting people/objects in video frames for tracking
object segmentation and behavior analysis.
6 Robotics scene Providing pixel-level scene segmentation to help robots
understanding navigate and interact with their environment.
7 Fashion and retail Segmenting clothing items in photos for virtual try-on and
image parsing e-commerce tagging systems.
8 Document and text Segmenting scanned documents into text, image, and table
region segmentation regions for OCR pipelines.
9 Industrial defect Segmenting defective regions on manufactured parts from
segmentation inspection camera images.
10 Environmental and Segmenting flood, wildfire, or deforestation extent from
disaster monitoring aerial/satellite imagery for disaster response.