Machine Learning and

GenAI

CA208E
 Unit-2
Supervised and Unsupervised Learning Techniques
Contents of Unit-2
Supervised Learning
• Linear Regression
• Logistic Regression
• Decision Trees
• Evaluation Metrics (Accuracy, Precision, Recall, F1-score)
• Case Studies in Text Classification

Unsupervised Learning
• Clustering
• K-Means
• Hierarchical Clustering
• Dimensionality Reduction: PCA, t-SNE
• Applications in GenAI pipelines (e.g., prompt clustering)
Machine Learning
In machine learning, regression analysis is a statistical technique that predicts continuous numeric values
based on the relationship between independent and dependent variables. The main goal of regression
analysis is to plot a line or curve that best fit the data and to estimate how one variable affects another. In
machine learning, regression is a type of supervised learning. The key objective of regression-based tasks
is to predict output labels or responses, which are continuous numeric values, for the given input data.
The output will be based on what the model has learned in the training phase.

Linear regression is a type of supervised machine-learning algorithm that learns from the labelled
datasets and maps the data points with most optimized linear functions which can be used for prediction
on new datasets. It assumes that there is a linear relationship between the input and output, meaning the
output changes at a constant rate as the input changes. This relationship is represented by a straight line.

For example, we want to predict a student's exam score based on how many hours they studied. We
observe that as students study more hours, their scores go up. In the example of predicting exam scores
based on hours studied. Here,

• Independent variable (input): Hours studied because it's the factor we control or observe.
• Dependent variable (output): Exam score because it depends on how many hours were studied.
 Simple linear regression
• One dependent variable (interval or ratio)

Variations of Linear Regression

• One independent variable (interval or ratio or dichotomous)
Multiple linear regression
• One dependent variable (interval or ratio)
• Two or more independent variables (interval or ratio or dichotomous)
Logistic regression
• One dependent variable (binary)
• Two or more independent variable(s) (interval or ratio or dichotomous)
Ordinal regression
• One dependent variable (ordinal)
• One or more independent variable(s) (nominal or dichotomous)
Multinomial regression
• One dependent variable (nominal)
• One or more independent variable(s) (interval or ratio or dichotomous)
Discriminant analysis
• One dependent variable (nominal)
• One or more independent variable(s) (interval or ratio)
Terminologies Used In Regression Analysis

• Independent Variables − These variables are used to predict the value of the dependent
variable. These are also called predictors. In dataset, these are represented as features.
• Dependent Variables − These are the variables whose values we want to predict. These are
the main factors in regression analysis. In dataset, these are represented as target variables
• Regression line − It is a straight line or curve that a regressor plots to fit the data points best.
• Overfitting and underfitting − Overfitting is when the regression model works well with the
training dataset but not with the testing dataset. It's also referred to as the problem of high
variance. Underfitting is when the model doesn't work well with training datasets. It's also
referred to as the problem of high bias.
• Outliers − These are data points that don't fit the pattern of the rest of the data. They are the
extremely high or extremely low values in the data set.
• Multicollinearity − multicollinearity occurs when independent variables (features) have
dependency among them.
A regression line can be a Positive Linear Relationship or a Negative Linear Relationship.
If the dependent variable expands on the Y-axis and the independent variable progress on X-axis, then
such a relationship is termed a Positive linear relationship (below LHS figure). If the
dependent variable decreases on the Y-axis while the independent variable increases on the X-axis, we
refer to this relationship as a negative linear relationship (below RHS figure).
How Does Linear Regression Work?

Linear regression works by modelling the relationship between two variables, x (independent
variable) and y (dependent variable), using a straight line. The independent variable, x, is represented
on the horizontal axis, while the dependent variable, y, is plotted on the vertical axis. The goal is to
find a line that best fits the data points and explains the relationship between the variables. For simple
linear regression (with one independent variable), the best-fit line is represented by the equation
𝑦=a𝑥+𝑏
Where:
• y is the predicted value (dependent variable)
• x is the input (independent variable)
• a is the slope of the line (how much y changes when x changes)
• b is the intercept (the value of y when x = 0)

The best-fit line will be the one that optimizes the values of a (slope) and b (intercept) so that the
predicted y values are as close as possible to the actual data points.
To find the value of b, a using the following equation:

(σ 𝑦) (σ 𝑥 2 ) −(σ 𝑥) (σ 𝑥𝑦 1
b= 2 2 or b = (σ 𝑦 − 𝑎 σ 𝑥)
𝑛 σ 𝑥 −(σ 𝑥) 𝑛

𝑛 σ 𝑥𝑦 − (σ 𝑥)(σ 𝑦)
a=
𝑛 σ 𝑥 2 −. (σ 𝑥)2
Example 1: Predict the glucose level given the age.

SUBJECT AGE X GLUCOSE LEVEL Y

1 43 99
2 21 65
3 25 79
4 42 75
5 57 87
6 59 81
7 55 ?
Step 1: Make a chart of your data, filling in the columns in the same way as you would fill in the
chart if you were finding the Pearson’s Correlation Coefficient

SUBJECT AGE X GLUCOSE LEVEL Y XY X2 Y2

1 43 99 4257 1849 9801

2 21 65 1365 441 4225
3 25 79 1975 625 6241
4 42 75 3150 1764 5625
5 57 87 4959 3249 7569
6 59 81 4779 3481 6561
Σ 247 486 20485 11409 40022
Step 2: Use the following equations to find m and b.

a b

a b

a b

Step 3: Insert the values into the equation.

y’ = a𝑥+𝑏
y’ = 65.14 + (0.385225 * x)

Step 4: Prediction – the value of y for the given value of x = 55

y’ = 65.14 +(0.385225 ∗55)
y’ =86.327
Now, to calculate the Mean Squared Error (MSE), follow these steps:
Compute the predicted values 𝑦 ′ for each 𝑥 in your dataset. For each subject’s age 𝑥, calculate the
predicted glucose level 𝑦 ′

For 𝑥 = 43:
𝑦 ′ = 0.385 43 + 65.14 = 81.67
For 𝑥 = 21:
𝑦 ′ = 0.385 21 + 65.14 = 73.23
For 𝑥 = 25:
𝑦 ′ = 0.385 25 + 65.14 = 74.75
For 𝑥 = 42:
𝑦 ′ = 0.385 42 + 65.14 = 81.29
For 𝑥 = 57:
𝑦 ′ = 0.385 57 + 65.14 = 87.13
For 𝑥 = 59:
𝑦 ′ = 0.385 59 + 65.14 = 87.90
Find the error (residual) for each subject
Error = 𝑦 − 𝑦 ′
Squared Error = ቀ𝑦 − 𝑦 ′ )2
Example (for Subject 1, 𝑥 = 43, actual 𝑦 = 99):
Error = 99 − 81.67 = 17.33
Squared Error = ൫17.33)2 = 299.8

Compute Mean Squared Error (MSE)

∑(y−y′)2
𝑛

where 𝑛=6 (since we used 6 data points for regression training).

Residuals and squared errors
For each point: residual 𝑟 = 𝑦 − 𝑦 ′ ,squared error 𝑟 2 .
1. 𝑟1 = 99 − 81.67 = 𝟏𝟕. 3
𝑟12 = ൫17.3)2 = 𝟐𝟗𝟗
2. 𝑟2 = 65 − 73.23 = −𝟖. 𝟐
𝑟22 = ൫−8.2)2 = 𝟔8
3. 𝑟3 = 79 − 74.75 = 𝟒. 𝟐
𝑟32 = (4.2) = 𝟏8
4. 𝑟4 = 75 − 81.29 = −𝟔. 𝟑
𝑟42 = ൫−6.3)2 = 𝟒𝟎
5. 𝑟5 = 87 − 87.13 = −𝟎. 𝟏
𝑟52 = ൫−0.1)2 = 𝟎. 𝟎𝟏
6. 𝑟6 = 81 − 87.90 = −𝟔. 9
𝑟62 = ൫−6.9)2 = 𝟒𝟕
Sum of squared errors (add them stepwise) = 299+68+18+40+0.01+47 = 432

෍ ∑(y−y′)2 = 432
𝑖=1

Mean Squared Error (MSE)

(As an exact rational using the decimal representation above:

432
MSE = ≈ 72
6
Root MSE (RMSE),
RMSE = MSE ≈ 8.48

Conclusion: “On average, predictions are about ±9 glucose units away from actual values.”
 Advantages of Linear Regression Disadvantages of Linear Regression

On the other hand in linear regression technique

Linear Regression is simple to implement and
outliers can have huge effects on the regression and
easier to interpret the output coefficients.
boundaries are linear in this technique.

When you know the relationship between the Diversely, linear regression assumes a linear
independent and dependent variable have a linear relationship between dependent and independent
relationship, this algorithm is the best to use variables. That means it assumes that there is a
because of it’s less complexity compared to other straight-line relationship between them. It assumes
algorithms. independence between attributes.

But then linear regression also looks at a

Linear Regression is susceptible to over-fitting but relationship between the mean of the dependent
it can be avoided using some dimensionality variables and the independent variables. Just as the
reduction techniques, regularization (L1 and L2) mean is not a complete description of a single
techniques and cross-validation. variable, linear regression is not a complete
description of relationships among variables.
 # Step 1: Import necessary libraries
# pandas → to handle data
# numpy → for mathematical operations
Python Code for Linear Regression # sklearn → for machine learning (Linear Regression, train-test split, metrics)

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from [Link] import mean_squared_error, r2_score

# Step 2: Load the dataset

# Example: dataset with "Age" as input (X) and "Glucose Level" as output (y)
data = pd.read_csv("[Link]")

X = data[["Age"]] # Independent variable (input feature)

y = data["Glucose"] # Dependent variable (output/target)

# Step 3: Split the dataset into training and testing sets

# train_test_split → divides data into training part (to learn) and testing part (to check accuracy)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
 # Step 4: Create a Linear Regression model
model = LinearRegression()

Python Code for Linear Regression # Step 5: Train (fit) the model on training data
# Model will learn the best line (y = aX + b) that fits the data
[Link](X_train, y_train)

# Step 6: Make predictions on test data

y_pred = [Link](X_test)

# Step 7: Evaluate the model

# Mean Squared Error (MSE) → average error
# R2 Score → how well the model fits (closer to 1 = better)
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print("Mean Squared Error:", mse)
print("R2 Score:", r2)

# Step 8: Predict for a new value

# Example: predict Glucose Level for Age = 55
new_age = [Link]([[55]])
predicted_glucose = [Link](new_age)
print("Predicted Glucose for Age 55:", predicted_glucose[0])
Multiple linear regression in machine learning is a supervised algorithm that models the
relationship between a dependent variable and multiple independent variables. It deals with more than
two features (one dependent variable and more than one independent variables).

In machine learning, multiple linear regression (MLR) is a statistical technique that is used to predict
the outcome of a dependent variable based on the values of multiple independent variables. The
multiple linear regression algorithm is trained on data to learn a relationship (known as a regression
line) that best fits the data. This relation describes how various factors affect the result. This relation is
used to forecast the value of dependent variable based on the values of independent variables.

The multiple linear regression of 2 variables x1 and x2 is as:

y=f(x1, x2)
y= ao+a1x1+a2x2
In general, for given ‘n’ variables as
y=f(x1, x2, …., xn)

y= ao+a1x1+a2x2, ..., anxn + e

Suppose we have the following dataset with one response variable y and two predictor variables X1 and X2:

Use the following steps to fit a multiple linear regression model to this dataset.
Step 1: Calculate x12, x22, x1y, x2y and x1x2.
Step 2: Calculate Regression Sums.
Step 3: Calculate b0, b1, and b2.
Step 5: Place b0, b1, and b2 in the estimated linear regression equation.
Step 1: Calculate x12, x22, x1y, x2y and x1x2.
Step 2: Calculate Regression Sums.

Next, make the following regression sum calculations:

• Σx12 = Σx12 – (Σx1)2 / n = 38,767 – (555)2 / 8 = 263.875
• Σx22 = Σx22 – (Σx2)2 / n = 2,823 – (145)2 / 8 = 194.875
• Σx1y = Σx1y – (Σx1Σy) / n = 101,895 – (555*1,452) / 8 = 1,162.5
• Σx2y = Σx2y – (Σx2Σy) / n = 25,364 – (145*1,452) / 8 = -953.5
• Σx1x2 = Σx1x2 – (Σx1Σx2) / n = 9,859 – (555*145) / 8 = -200.375

Step 3: Calculate b0, b1, and b2.

The formula to calculate b0 is: y – b1x1 – b2x2

The formula to calculate b1 is: [(Σx22)(Σx1y) – (Σx1x2)(Σx2y)] / [(Σx12) (Σx22) – (Σx1x2)2]
The formula to calculate b2 is: [(Σx12)(Σx2y) – (Σx1x2)(Σx1y)] / [(Σx12) (Σx22) – (Σx1x2)2]
b1 = [(194.875)(1162.5) – (-200.375)(-953.5)] / [(263.875) (194.875) – (-200.375)2] = 3.148
b2 = [(263.875)(-953.5) – (-200.375)(1152.5)] / [(263.875) (194.875) – (-200.375)2] = -1.656

b0 = 181.5 – 3.148(69.375) – (-1.656)(18.125) = -6.867

Step 5: Place b0, b1, and b2 in the estimated linear regression equation.

The estimated linear regression equation is: ŷ = b0 + b1*x1 + b2*x2

ŷ = -6.867 + 3.148x1 – 1.656x2

Here, interpret this estimated linear regression equation

b0 = -6.867. When both predictor variables are equal to zero, the mean value for y is -6.867.
b1 = 3.148. A one unit increase in x1 is associated with a 3.148 unit increase in y, on average,
assuming x2 is held constant.
b2 = -1.656. A one unit increase in x2 is associated with a 1.656 unit decrease in y, on average,
assuming x1 is held constant.
Toyota Aygo 1000 790 99
Mitsubishi Space Star 1200 1160 95
Skoda Citigo 1000 929 95
Fiat 500 900 865 90
Mini Cooper 1500 1140 105
VW Up! 1000 929 105
Skoda Fabia 1400 1109 90
Mercedes
Car A-Class
Model 1500
Volume Weight 1365 CO2 92
Ford Fiesta 1500 1112 98
Audi A1 1600 1150 99

Example: predict the CO2 emission of a car where the weight is 2300kg, and the volume is
1300cm3
Challenges of Multiple Linear Regression

Challenge Description
High correlation between independent variables, leading to unstable model
Multicollinearity
coefficients and difficulty in interpreting the impact of individual variables.
The model fits the training data too closely, leading to poor performance on
Overfitting
new, unseen data.
The model fails to capture the underlying patterns in the data, resulting in
Underfitting
poor performance on both training and test data.
Multiple linear regression assumes a linear relationship between the
Non-linearity independent and dependent variables. Non-linear relationships can lead to
inaccurate predictions.
Outliers can significantly impact the model's performance, especially in
Outliers
small datasets.
Missing Data Missing data can lead to biased and inaccurate results.
Difference Between Simple and Multiple Linear Regression

Feature Simple Linear Regression Multiple Linear Regression

Independent Variables One Two or more
Model Equation y = w1x + w0 y=w0+w1x1+w2x2+ ... +wpxp
Complexity Less complex More complex due to multiple variables

Predicting house prices based Predicting sales based on advertising

on square footage, predicting expenditure, price, and competitor activity,
Real-world Applications
sales based on advertising predicting student performance based on
expenditure study hours, attendance, and IQ

More complex to interpret due to multiple

Model Interpretation Easier to interpret coefficients
variables
Logistic Regression is a supervised machine learning algorithm used for classification problems.
Unlike linear regression which predicts continuous values it predicts the probability that an input
belongs to a specific class. It is used for binary classification where the output can be one of two
possible categories such as Yes/No, True/False or 0/1. It uses sigmoid function to convert inputs into a
probability value between 0 and 1
Sigmoid Function: Line equation y=ax+b or p=ax+b where p is the probability of “success”. But it will come
out less than 0 or greater than 1, which is impossible for a probability. Instead of modeling 𝑝 directly, we
model the odds of success: The ratio of success to failure.
𝑝
Odds =
1−𝑝
To predict the odds of success, we take logs on odds formula
𝑝(𝑥)
log1−𝑝(𝑥) = 𝑦
𝑒 𝑎𝑥+𝑏
𝑝 𝑥 =
𝑝(𝑥)
log1−𝑝(𝑥) = 𝑎𝑥 + 𝑏 1 + 𝑒 𝑎𝑥+𝑏

Exponentiating both sides, we have 1

𝑝(𝑥) 𝑝 𝑥 =
𝑙𝑛 𝑎𝑥+𝑏 1 + 𝑒 −(𝑎𝑥+𝑏)
𝑒 1−𝑝(𝑥) =𝑒
Which is the sigmoid function
Since 𝑒 ln(𝑥) = 𝑥, 𝑡ℎ𝑒𝑟𝑒𝑓𝑜𝑟𝑒 𝑒𝑞𝑢𝑎𝑡𝑖𝑜𝑛 𝑤𝑜𝑢𝑙𝑑 𝑏𝑒
𝑝(𝑥)
= 𝑒 𝑎𝑥+𝑏
1 − 𝑝(𝑥)
𝑝(𝑥) 𝑧
Let z = 𝑒 𝑎𝑥+𝑏 𝑡ℎ𝑒𝑛 1−𝑝(𝑥) = 𝑧 means p(x) = z(1-p(x)) implies p(x) = 1+𝑧
replace z with 𝑒 𝑎𝑥+𝑏
Logistic/Sigmoid Function

The sigmoid function is a mathematical function for mapping predicted values to probabilities. It can map
any real value into another value within 0 and 1. When the result of the sigmoid function is greater than
0.5, we classify the label as class 1 or positive class; if it’s less than 0.5, we can classify it as a negative
class or 0.
1
S(Z) = −𝑧
1+𝑒
Suppose a dataset of students entrance marks for JEE and based on the historic data of those who are
selected or not selected in IIT. Based on Logistic Regression, the values of learnt parameters are b=1 and
a=8. Assume marks of x=60. Computer the resultant class. Here ax+b = 8*60+1 = 481
1
P(x) = 1
P(x) = 1+𝑒 −481 = 0.44
1+𝑒 −𝑧
It has been observed that 0.44<0.5, therefore the candidate with marks=60 won’t be selected.
 Cust Income Lot_size Ownership
1 60.0 18.4 Owner
2 64.8 21.6 Owner
3 84.0 17.6 Non-Owner
4 59.4 16.0 Non-Owner
5 108.0 17.6 Owner
6 75.0 19.6 Non-Owner

Example: Construct a Logistic Regression model with the values of learnt parameters b0 = -25.9482,
b1 = 0.1109 and b2 = 0.9638, where b1 and b2 are for income and lot_size variables respectively. Using
Logistic Regression model with probability cutoff 0.75. Classify the given 6 custimers as “owner” or
“non-owner”. Present the results in classification matrix also.
Types of Logistic Regression: Logistic regression can be classified into three main types based on
the nature of the dependent variable:

1. Binomial Logistic Regression: This type is used when the dependent variable has only two
possible categories. Examples include Yes/No, Pass/Fail or 0/1. It is the most common form of
logistic regression and is used for binary classification problems.

2. Multinomial Logistic Regression: This is used when the dependent variable has three or more
possible categories that are not ordered. For example, classifying animals into categories like
"cat," "dog" or "sheep." It extends the binary logistic regression to handle multiple classes.

3. Ordinal Logistic Regression: This type applies when the dependent variable has three or more
categories with a natural order or ranking. Examples include ratings like "low," "medium" and
"high." It takes the order of the categories into account when modeling.
Assumptions of Logistic Regression

1. Independent observations: Each data point is assumed to be independent of the others means
there should be no correlation or dependence between the input samples.

2. Binary dependent variables: It takes the assumption that the dependent variable must be binary,
means it can take only two values.

3. Linearity relationship between independent variables and log odds: The model assumes a linear
relationship between the independent variables and the log odds of the dependent variable which
means the predictors affect the log odds in a linear way.

4. No outliers: The dataset should not contain extreme outliers as they can distort the estimation of
the logistic regression coefficients.

5. Large sample size: It requires a sufficiently large sample size to produce reliable and stable
results.
Advantages of Logistic Regression Models
• One of the simplest machine learning algorithms and easy to implement
• The predicted parameters (trained weights) give inference about the importance of each feature
• Can be updated easily to reflect new data, unlike decision trees or support vector machines
• Outputs are well-calibrated probabilities along with classification results
• Logistic Regression is less prone to overfitting
• Logistic Regression proves to be very efficient when the dataset has features that are linearly separable
• Can easily be extended to multi-class classification using a softmax classifier, this is known as
Multinomial Logistic Regression

Disadvantages of Logistic Regression Models

• On high dimensional datasets, this may lead to the model being over-fit on the training set
• Non linear problems can’t be solved with logistic regression since it has a linear decision surface
• Difficult to capture complex relationships
• Repetition of information could lead to wrong training of parameters (weights) during minimizing the
cost function
• It is sensitive to outliers
• It requires a large dataset and also sufficient training examples for all the categories it needs to identify
 Linear Regression Logistic Regression
Linear regression predicts the categorical dependent Logistic regression predicts the categorical dependent
variable using a given set of independent variables. variable using a given set of independent variables.
The outputs produced must be of a continuous value, The outputs must be categorical values such as 0 or 1,
such as price and age. True or False.
Logistic regression is primarily used to solve The cross-entropy method is used to estimate
classification tasks. accuracy.
We use the best-fit line to help us easily predict The relationship between the dependent and
outputs. independent variables DOES NOT need to be linear.
In Linear regression, the relationship between the The relationship between the dependent and
dependent and independent variables must be linear. independent variables DOES NOT need to be linear.
We use the best-fit line to help us easily predict We use the S-curve (Sigmoid) to help us classify
outputs. predicted outputs.
The mean squared error method is used for the The mean squared error method is used to estimate
estimation of accuracy. accuracy.
There is a possibility of collinearity between the There should not be any collinearity between the
independent variables. independent variables.
 Linear Regression Logistic Regression
Logistic Regression is a supervised classification
Linear Regression is a supervised regression model.
model.
Equation of logistic regression
Equation of linear regression:
𝑒 𝑎0+𝑎1𝑥1+𝑎2𝑥2+⋯+𝑎𝑖𝑥𝑖
𝑦 𝑥 = 𝑎0 + 𝑎1 𝑥1 + 𝑎2 𝑥2 + ⋯ + 𝑎𝑖 𝑥𝑖 𝑦 𝑥 =
1 + 𝑒 𝑎0+𝑎1𝑥1+𝑎2𝑥2+⋯+𝑎𝑖𝑥𝑖
Here,
Here,
y = response variable
y = response variable
xi = ith predictor variable
xi = ith predictor variable
ai= average effect on y as xi increases by 1
ai = average effect on y as xi increases by 1
In Linear Regression, we predict the value by an integer
In Logistic Regression, we predict the value by 1 or 0.
number.
Activation function is used to convert a linear
No activation function is used.
regression equation to the logistic regression equation
No threshold value is needed. A threshold value is added.
We calculate Root Mean Square Error(RMSE) to
We use precision to predict the next weight value.
predict the next weight value.
Straight line Sigmoid function (S-shaped curve)
Decision tree learning is one of the most successful techniques for supervised classification
learning. DT takes input as objects or situations described by a set of attributes and returns a decision.
The input attribute can be discrete or continuous as well as output also. Learning a discrete valued
function is called classification and learning a continuous valued function is called regression
learning. It can be applied to both regression & classification.
Example: What to do this Weekend?
This tree consists of the following components:
• Questions/conditions are Nodes.
• Yes/No options represent Edges.
• End actions are Leafs of the tree.
Decision Trees are a type of
Supervised Machine Learning. A
decision tree is a tree where each –

• Node - a feature(attribute)
• Branch - a decision(rule)
• Leaf - an outcome(categorical or
continuous)

Algorithms to build decision trees

are Iterative Dichotomiser 3 (ID3),
Classification and Regression Trees
(CART), C4.5, Random Forest.
ID3 Algorithm: It is called the ID3 (Iterative Dichotomiser 3) algorithm by J. R.
Quinlan. The algorithm uses Entropy and Information Gain to build the tree. It is a
classification algorithm that follows a greedy approach by selecting a best attribute
that yields maximum Information Gain(IG) or minimum Entropy(H). Entropy is a
measure of the amount of uncertainty in the dataset S. Information Gain IG(A) tells
us how much uncertainty in S was reduced after splitting set S on attribute A.
The steps in ID3 algorithm are as follows:
1. Calculate entropy for dataset.
2. For each attribute/feature.
2.1. Calculate entropy for all its categorical values.
2.2. Calculate information gain for the feature.
3. Find the feature with maximum information gain.
4. Repeat it until we get the desired tree.
How to calculate Entropy and Information Gain:

Calculate Entropy: For each attribute, the algorithm calculates the entropy of the
dataset based on the values of that attribute. Entropy is computed using the formula:

Calculate the Information Gain for each attribute. Information Gain is a measure
of the effectiveness of an attribute in classifying the data. It is computed as the
difference between the entropy of the original dataset and the weighted average of
the entropies of the subsets created by splitting the data based on that attribute.
Select Best Attribute: The algorithm selects the attribute with the highest
Information Gain as the attribute to split the dataset at that node. This process is
repeated recursively for each subset until a stopping condition is met, such as
reaching a certain depth or having subsets that are pure (i.e., all instances belong to
the same class).

By using entropy and information gain, the ID3 algorithm intelligently chooses how
to split the data at each node of the decision tree, resulting in a tree structure that
effectively classifies instances based on the given attributes.
Consider the weather dataset as given below:

Day Outlook Temp. Humidity Wind Decision

1 Sunny Hot High Weak No
2 Sunny Hot High Strong No
3 Overcast Hot High Weak Yes
4 Rain Mild High Weak Yes
5 Rain Cool Normal Weak Yes
6 Rain Cool Normal Strong No
7 Overcast Cool Normal Strong Yes
8 Sunny Mild High Weak No
9 Sunny Cool Normal Weak Yes
10 Rain Mild Normal Weak Yes
11 Sunny Mild Normal Strong Yes
12 Overcast Mild High Strong Yes
13 Overcast Hot Normal Weak Yes
14 Rain Mild High Strong No
Complete entropy of dataset is:
H(S) = - p(yes) * log2(p(yes)) - p(no) * log2(p(no))
= - (9/14) * log2(9/14) - (5/14) * log2(5/14)
= - (-0.41) - (-0.53)
= 0.94
For each attribute of the dataset:
Categorical values of outlook attribute - sunny, overcast and rain
H(Outlook=sunny) = -(2/5)*log(2/5)-(3/5)*log(3/5) =0.97
H(Outlook=rain) = -(3/5)*log(3/5)-(2/5)*log(2/5) =0.97
H(Outlook=overcast) = -(4/4)*log(4/4)-0 = 0

Information Gain (Outlook) = H(S) - p(sunny) * H(Outlook=sunny) - p(rain) * H(Outlook=rain) -

p(overcast) * H(Outlook=overcast)

= 0.94 - (5/14)*0.97 - (5/14)*0.97 - (4/14)*0

= 0.94 - 0.35 - 0.35 - 0 = 0.24
Categorical values of temperature attribute - hot, mild, cool
H(Temperature=hot) = -(2/4)*log(2/4)-(2/4)*log(2/4) = 1
H(Temperature=cool) = -(3/4)*log(3/4)-(1/4)*log(1/4) = 0.82
H(Temperature=mild) = -(4/6)*log(4/6)-(2/6)*log(2/6) = 0.92

Information Gain (Temp) = H(S) - p(hot)*H(Temperature=hot) - p(mild) * H(Temperature=mild) -

p(cool) * H(Temperature=cool)
= 0.94 - (4/14)*1 + (6/14) * 0.9179 + (4/14) * 0.811
= 0.94– 0.29 – 0.4 – 0.24 = 0.01

Categorical values of humidity attribute- high, normal

H(Humidity=high) = -(3/7)*log(3/7)-(4/7)*log(4/7) = 0.98
H(Humidity=normal) = -(6/7)*log(6/7)-(1/7)*log(1/7) = 0.59

Information Gain(Humidity) = H(S) - p(high)*H(Humidity=high) - p(normal)*H(Humidity=normal)

= 0.94 - (7/14) * 0.983 + (7/14) * 0.591
= 0.94 – 0.79 = 0.15
Categorical values of windy attribute - weak, strong
H(Wind=weak) = -(6/8)*log(6/8)-(2/8)*log(2/8) = 0.811
H(Wind=strong) = -(3/6)*log(3/6)-(3/6)*log(3/6) = 1

Information Gain(Windy) = H(S) - p(weak) * H(Wind=weak) - p(strong) * H(Wind=strong)

= 0.94 - 0.89
= 0.04

Here, the attribute with maximum information gain is Outlook. So, the decision tree built so far –
Here, when Outlook = = overcast, it is of pure class(Yes). Now, we have to repeat
same procedure for the data with rows consist of Outlook value as Sunny and then
for Outlook value as Rain. Now, finding the best attribute for splitting the data
with Outlook=Sunny values{Dataset rows = [1, 2, 8, 9, 11]}.

Day Outlook Temp. Humidity Wind Decision

1 Sunny Hot High Weak No
2 Sunny Hot High Strong No
8 Sunny Mild High Weak No
9 Sunny Cool Normal Weak Yes
11 Sunny Mild Normal Strong Yes
Complete entropy of Sunny is -
H(S) = - p(yes) * log2(p(yes)) - p(no) * log2(p(no))
= - (2/5) * log2(2/5) - (3/5) * log2(3/5) = 0.97
Categorical values of temperature attribute in terms of sunny - hot, mild, cool
H(Sunny, Temperature=hot) = -0-(2/2)*log(2/2) = 0
H(Sunny, Temperature=cool) = -(1)*log(1)- 0 = 0
H(Sunny, Temperature=mild) = -(1/2)*log(1/2)-(1/2)*log(1/2) = 1

Information Gain (Stemp) = H(Sunny) - (2/5)*0 - (1/5)*0 - (2/5)*1

= 0.97 - 0.4 = 0.57

Categorical values of humidity attribute in terms of sunny - high, normal

H(Sunny, Humidity=high) = - 0 - (3/3)*log(3/3) = 0
H(Sunny, Humidity=normal) = -(2/2)*log(2/2)-0 = 0

Information Gain(Shumid) = H(Sunny) - p(Sunny, high)*H(Sunny, Humidity=high) + p(Sunny,

normal)*H(Sunny, Humidity=normal)
= 0.97 - (3/5)*0 + (2/5)*0
= 0.97
 Categorical values of windy attribute in terms of sunny - weak, strong
H(Sunny, Wind=weak) = -(1/3)*log(1/3)-(2/3)*log(2/3) = 0.92
H(Sunny, Wind=strong) = -(1/2)*log(1/2)-(1/2)*log(1/2) = 1

Information Gain(Swind) = H(Sunny) - p(Sunny, weak)*H(Sunny, Wind=weak) + p(Sunny,

strong)*H(Sunny, Wind=strong)

= 0.97 - (3/5)*0.92 + (2/5)*1

= 0.97 - 0.95 = 0.02

Here, the attribute with maximum

information gain is Humidity. So, the
decision tree built so far -
Here, when Outlook = Sunny and Humidity = High, it is a pure class of category "no". And When
Outlook = Sunny and Humidity = Normal, it is again a pure class of category "yes". Therefore, we
don't need to do further calculations. Now, finding the best attribute for splitting the data with
Outlook=Sunny values{Dataset rows = [4, 5, 6, 10, 14]}.

Day Outlook Temp. Wind Decision

4 Rain Mild Weak Yes
5 Rain Cool Weak Yes
6 Rain Cool Strong No
10 Rain Mild Weak Yes
14 Rain Mild Strong No

Complete entropy of Rain is -

H(S) = - p(yes) * log2(p(yes)) - p(no) * log2(p(no))
= - (3/5) * log(3/5) - (2/5) * log(2/5)
= 0.97
Categorical values of temperature attribute in terms of rain - mild, cool
H(Rain, Temperature=cool) = -(1/2)*log(1/2)- (1/2)*log(1/2) = 1
H(Rain, Temperature=mild) = -(2/3)*log(2/3)-(1/3)*log(1/3) = 0.918

Information Gain = H(Rain) - p(Rain, mild) * H(Rain, Temperature=mild) - p(Rain, cool) *

H(Rain, Temperature=cool)

=0.97 - (2/5)*1 - (3/5)*0.918

= 0.97 - 0.95
= 0.02
Categorical values windy attribute in terms of rain - weak, strong
H(Wind=weak) = -(3/3)*log(3/3)-0 = 0
H(Wind=strong) = 0-(2/2)*log(2/2) = 0

Information Gain = H(Rain) - p(Rain, weak)*H(Rain, Wind=weak) - p(Rain, strong)*H(Rain,

Wind=strong)
= 0.97 - (3/5)*0 + (2/5)*0
= 0.97 – 0
= 0.97
Here, the attribute with maximum information gain is Wind. So, the decision tree built so far -

Here, when Outlook = Rain and Wind = Strong, it is a pure class of category "no". And When Outlook =
Rain and Wind = Weak, it is again a pure class of category "yes".

And this is our final desired tree for the given dataset.
 Height Hair Eyes Attractive?
Small Blonde Brown No
Tall Dark Brown No
Tall Blonde Blue Yes
Tall Dark Blue No
Small Dark Blue No
Tall Red Blue Yes
Tall Blonde Brown No
Small Blonde Blue Yes

Example: The above dataset (adapted from Quinlan) shows some attributes of people and whether
they are considered attractive. Use the ID3 algorithm to build a decision tree that classifies which
attributes make a person attractive.
You are required to show the following steps:
a. Calculate the entropy of the dataset 𝑆.
b. Calculate the information gain for each attribute (Height, Hair, Eyes).
Advantages and Disadvantages of ID3 Algorithm
Advantages

• Inexpensive to construct
• Extremely fast at classifying unknown records Easy to interpret for small-sized trees.
• Robust to noise (especially when methods to avoid over-fitting are employed).
• Can easily handle redundant or irrelevant attributes (unless the attributes are interacting).
Disadvantages

• The space of possible decision trees is exponentially large. Greedy approaches are often
unable to find the best tree.
• Does not take into account interactions between attributes.
• Each decision boundary involves only a single attribute.
Advantages of Decision Tree Algorithm

• Easy to Interpret and Visualize: Decision trees provide a clear, hierarchical structure that is
simple for humans to understand and visualize, especially for smaller trees.
• Handles Both Data Types: They can work with both categorical (like color) and numerical
(like age) data, often without requiring extensive data transformation.
• Little Data Preparation: They require less data cleaning and normalization compared to other
algorithms.
• Non-Linearity: Decision trees can model complex, non-linear relationships in the data.

• Handles Missing Values: Some decision tree algorithms can handle missing values in the data
without requiring significant imputation.
Disadvantages of Decision Tree Algorithm
• Overfitting: A major issue, decision trees can grow too complex and memorize the training
data, leading to poor performance on new, unseen data.

• Instability: Even small changes in the training data can lead to a significantly different tree
structure and predictions.

• Bias Towards Dominant Classes/Features: They can be biased towards features with many
distinct values or dominant classes, potentially ignoring other important features.

• Computational Cost: As trees get deeper to accommodate complex data, they can become
computationally expensive to build and prune.

• Difficulty with Complex Interactions: Decision trees may struggle to capture highly
complex interactions between features.
Evaluation Metrics in Machine Learning

When building machine learning models, it’s important to understand how well they perform.
Evaluation metrics help us to measure the effectiveness of our models. Whether we are solving a
classification problem, predicting continuous values or clustering data, selecting the right evaluation
metric allows us to assess how well the model meets our goals. In this article, we will see commonly
used evaluation metrics and discuss how to choose the right metric for our model.
Confusion matrix is a simple table used to measure how well a classification model is performing. It
compares the predictions made by the model with the actual results and shows where the model was
right or wrong. This helps you understand where the model is making mistakes so you can improve it. It
breaks down the predictions into four categories:
• True Positive (TP): The model correctly predicted a positive outcome i.e the actual outcome
was positive.
• True Negative (TN): The model correctly predicted a negative outcome i.e the actual outcome
was negative.
• False Positive (FP): The model incorrectly predicted a positive outcome i.e the actual outcome
was negative. It is also known as a Type I error.
• False Negative (FN): The model incorrectly predicted a negative outcome i.e the actual
outcome was positive. It is also known as a Type II error.
Accuracy is a fundamental metric used for evaluating the performance of a classification model. It tells
us the proportion of correct predictions made by the model out of all predictions.

NumberofCorrectPredictions (TP + TN)

Accuracy = TotalNumberofPredictions
= (TP + TN + FP + FN)

While accuracy provides a quick snapshot, it can be misleading in cases of imbalanced datasets. For
example, in a dataset with 90% class A and 10% class B, a model predicting only class A will still achieve
90% accuracy but it will fail to identify any class B instances.
Accuracy is good but it gives a False Positive sense of achieving high accuracy. The problem arises due to
the possibility of misclassification of minor class samples being very high.
Precision measures how many of the positive predictions made by the model are actually correct. It's
useful when the cost of false positives is high such as in medical diagnoses where predicting a disease
when it’s not present can have serious consequences.
TP
Precision =
TP + FP
Where:
TP = True Positives
FP = False Positives
It helps ensure that when the model predicts a positive outcome, it’s likely to be correct.
Recall or Sensitivity measures how many of the actual positive cases were correctly identified by the
model. It is important when missing a positive case (false negative) is more costly than false positives.
TP
Recall =
TP + FN
Where:
FN = False Negatives

In scenarios where catching all positive cases is important (like disease detection), recall is a key metric.

F1 Score is the harmonic mean of precision and recall. It is useful when we need a balance between
precision and recall as it combines both into a single number. A high F1 score means the model performs
well on both metrics. Its range is [0,1].
Lower recall and higher precision gives us great accuracy but then it misses a large number of instances.
More the F1 score better will be performance. It can be expressed mathematically in this way:
Precision × Recall
F1 Score = 2 ×
Precision + Recall
Scenario: A medical diagnosis system classifies 1,000 patients.
• True Positives (TP): 120 correctly diagnosed with the disease.
• False Positives (FP): 20 patients incorrectly diagnosed with the disease.
• True Negatives (TN): 800 correctly diagnosed as not having the disease.
• False Negatives (FN): 60 patients incorrectly diagnosed as not having the disease.

Formulas:
• Accuracy: (TP + TN) / (TP + TN + FP + FN)
• Precision: TP / (TP + FP)
• Recall (Sensitivity): TP / (TP + FN)
• F1-Score: 2 * (Precision * Recall) / (Precision + Recall)
 Clustering or cluster analysis is a machine learning
technique, which groups the unlabeled dataset. It can be
defined as "A way of grouping the data points into different
clusters, consisting of similar data points. The objects with the
possible similarities remain in a group that has less or no
similarities with another group.“
It is an unsupervised learning method, hence no
supervision is provided to the algorithm, and it deals with the
unlabeled dataset.

After applying this clustering technique, each cluster or group is provided with a cluster-ID. ML
system can use this id to simplify the processing of large and complex datasets.
K-Means Clustering is an unsupervised
machine learning algorithm that helps group
data points into clusters based on their inherent
similarity. It groups the objects based on
attributes or features into k number of group,
where k is a positive integer number. The
grouping is done by minimizing the squares of
distances between data and the corresponding
cluster centroid. Initially, we specify how many
clusters we will made, when we put it in
category. We calculate its mean, that is why we
called k-mean clustering
Steps in k-mean clustering algorithm

• Step-1: Select the value of K, to decide the number of clusters to be formed.

• Step-2: Select random K points which will act as centroids.
• Step-3: Assign each data point, based on their distance from the randomly selected points
(Centroid), to the nearest/closest centroid which will form the predefined clusters.
Euclidean Distance, ED = (𝑿𝟐 − 𝑿𝟏 )𝟐 +(𝒀𝟐 − 𝒀𝟏 )𝟐

Manhattan Distance, MD = [|x2 - x1| + |y2 - y1|]

• Step-4: place a new centroid of each cluster.
• Step-5: Repeat step no.3, which reassign each datapoint to the new closest centroid of each
cluster.
• Step-6: If any reassignment occurs, then go to step-4 else go to Step 7.
• Step-7: FINISH
Example: Apply k-mean clustering for the following data sets for two clusters. Tabulate all the
assignments: [Note: Given k=2]

Hint: Initial centroid:

 Distance To
Data Points Cluster New Cluster
K1 K2

(185,72) 0 K1

(170,56) 0 K2

(168,60)

(179,68)

(182,72)

(188,77)
Prob. 1: Assume the following eight points (with (x, y) representing locations): A1(2, 10), A2(2,
5), A3(8, 4), A4(5, 8), A5(7, 5), A6(6, 4), A7(1, 2), A8(4, 9) Assume the initial cluster centroids
are: A1(2, 10), A4(5, 8) and A7(1, 2). The distance function between two points a = (x1, y1) and b
= (x2, y2) is defined as-
Ρ(a, b) = |x2 – x1| + |y2 – y1|
Use K-Means Algorithm to find the three cluster centers after the second iteration.

Prob. 2: Given are the points A = (1,2), B = (2,2), C = (2, 1), D = (-1, 4), E = (-2, -1), F = (-1,-1).
Starting from initial clusters Cluster1 = {A} which contains only the point A and Cluster2 = {D}
which contains only the point D, run the K-means clustering algorithm and report the final
clusters. Draw the points on a 2-D grid and check if the clusters make sense.
K-Means Clustering Algorithm-

Iteration-01:
• We calculate the distance of each point from each of the center of the three clusters.
• The distance is calculated by using the given distance function.

The following illustration shows the calculation of distance between point A1(2, 10) and each of the
center of the three clusters-
Calculating Distance Between A1(2, 10) and C1(2, 10)-
Ρ(A1, C1) = |x2 – x1| + |y2 – y1|= |2 – 2| + |10 – 10| = 0
Calculating Distance Between A1(2, 10) and C2(5, 8)-
Ρ(A1, C2) = |x2 – x1| + |y2 – y1| = |5 – 2| + |8 – 10| = 3 + 2 = 5
Calculating Distance Between A1(2, 10) and C3(1, 2)-
Ρ(A1, C3) = |x2 – x1| + |y2 – y1| = |1 – 2| + |2 – 10| = 1 + 8 = 9

In the similar manner, we calculate the distance of other points from each of the center of the three
clusters. Next, We draw a table showing all the results. Using the table, we decide which point
belongs to which cluster. The given point belongs to that cluster whose center is nearest to it.
New clusters are-
Cluster-01:
• A1(2, 10)
Cluster-02:
• A3(8, 4)
• A4(5, 8)
• A5(7, 5)
• A6(6, 4)
• A8(4, 9)
Cluster-03:
• A2(2, 5)
• A7(1, 2)
Now, we re-compute the new cluster clusters.
• The new cluster center is computed by taking mean of all the points contained in that cluster.
For Cluster-01: We have only one point A1(2, 10) in Cluster-01. So, cluster center remains the same.
For Cluster-02:
Center of Cluster-02 = ((8 + 5 + 7 + 6 + 4)/5, (4 + 8 + 5 + 4 + 9)/5) = (6, 6)
For Cluster-03:
Center of Cluster-03 = ((2 + 1)/2, (5 + 2)/2) = (1.5, 3.5)
This is completion of Iteration-01!!
Iteration-02:

We calculate the distance of each point from each of the center of the three clusters. The following
illustration shows the calculation of distance between point A1(2, 10) and each of the center of the
three clusters-
Calculating Distance Between A1(2, 10) and C1(2, 10)-
Ρ(A1, C1) = |x2 – x1| + |y2 – y1| = |2 – 2| + |10 – 10| = 0
Calculating Distance Between A1(2, 10) and C2(6, 6)-
Ρ(A1, C2) = |x2 – x1| + |y2 – y1| = |6 – 2| + |6 – 10| = 4 + 4 = 8
Calculating Distance Between A1(2, 10) and C3(1.5, 3.5)-
Ρ(A1, C3) = |x2 – x1| + |y2 – y1| = |1.5 – 2| + |3.5 – 10| = 0.5 + 6.5 = 7

In the similar manner, we calculate the distance of other points from each of the center of the three
clusters.
Next,
• We draw a table showing all the results.
• Using the table, we decide which point belongs to which cluster.
• The given point belongs to that cluster whose center is nearest to it.
New clusters are-

Cluster-01:

• A1(2, 10)
• A8(4, 9)

Cluster-02:

• A3(8, 4)
• A4(5, 8)
• A5(7, 5)
• A6(6, 4)

Cluster-03:

• A2(2, 5)
• A7(1, 2)
Advantages of K-Means Clustering Algorithm

• It is very easy to understand and implement.

• If we have large number of variables then, K-means would be faster than Hierarchical
clustering.
• On re-computation of centroids, an instance can change the cluster.
• Tighter clusters are formed with K-means as compared to Hierarchical clustering.

Disadvantages of K-Means Clustering Algorithm

• It is a bit difficult to predict the number of clusters i.e. the value of k.

• Output is strongly impacted by initial inputs like number of clusters (value of k).
• Order of data will have strong impact on the final output.
• It is very sensitive to rescaling. If we will rescale our data by means of normalization or
standardization, then the output will completely change.
Hierarchical clustering is an unsupervised learning
technique used to group similar data points into clusters by
building a hierarchy (tree-like structure). Unlike flat clustering
like k-means hierarchical clustering does not require specifying
the number of clusters in advance.
The algorithm builds clusters step by step either by
progressively merging smaller clusters or by splitting a large
cluster into smaller ones. The process is often visualized using
a dendrogram, which helps to understand data similarity.
Dendrogram
A dendrogram is like a family tree for clusters. It shows how • At the bottom of the dendrogram the points P, Q,
R, S and T are all separate.
individual data points or groups of data merge together. The
• As we move up, the closest points are merged into
bottom shows each data point as its own group and as we move a single group.
up, similar groups are combined. The lower the merge point, the • The lines connecting the points show how they
more similar the groups are. It helps us see how things are are progressively merged based on similarity.
grouped step by step. • The height at which they are connected shows
how similar the points are to each other; the
shorter the line the more similar they are
Types of Hierarchical Clustering

1. Agglomerative Clustering
2. Divisive clustering

Agglomerative Clustering: It is also known as the

bottom-up approach or hierarchical agglomerative
clustering (HAC). Bottom-up algorithms treat each
data as a singleton cluster at the outset and then
successively agglomerate pairs of clusters until all
clusters have been merged into a single cluster that
contains all data.
Hierarchical Divisive clustering: Divisive
clustering is also known as a top-down approach.
Top-down clustering requires a method for splitting
a cluster that contains the whole data and proceeds
by splitting clusters recursively until individual data
have been split into singleton clusters.
PCA (Principal Component Analysis) is a dimensionality reduction technique used in data analysis and
machine learning. It helps you to reduce the number of features in a dataset while keeping the most
important information.

How Principal Component Analysis Works?

PCA uses linear algebra to transform data into new features called principal components. It finds these
by calculating eigenvectors (directions) and eigenvalues (importance) from the covariance matrix.
PCA selects the top components with the highest eigenvalues and projects the data onto them simplify
the dataset.
In PCA method, the transformation is design in such
way that the data set be represented by a reduced
number of “effective” features and yet retains most of
the intrinsic information contained in the data; in other
words, the data set undergoes a dimensionality
reduction. Transforming the variables to a new set of
variables, are known as the principal components.

Steps for PCA

1. Calculate mean of every feature
2. Calculate the Covariance Matrix

3. Calculate the Eigen Values and Eigen Vectors

4. Choose Principal Components
5. Project data onto principal components
Steps for PCA

Mean (μ): Average of a set of data

Variance (σ²): The average of the squared

differences from the mean

Covariance is a measure of how changes in

one variable are associated with changes in a
second variable. It’s similar to variance, but
where variance tells you how a single variable
varies, Covariance tells you how two variables
vary together.
Advantages of PCA

1. Removes Correlated Features: find out the correlation among the features (correlated variables).
Finding correlation manually in thousands of features is nearly impossible, frustrating and time-
consuming. PCA does this for you efficiently. After implementing the PCA on your dataset, all
the Principal Components are independent of one another. There is no correlation among them.
2. Improves Algorithm Performance: PCA is a very common way to speed up your Machine
Learning algorithm by getting rid of correlated variables which don't contribute in any decision
making. The training time of the algorithms reduces significantly with less number of features.
3. Reduces Overfitting: Overfitting mainly occurs when there are too many variables in the dataset.
So, PCA helps in overcoming the overfitting issue by reducing the number of features.
4. Improves Visualization: It is very hard to visualize and understand the data in high dimensions.
PCA transforms a high dimensional data to low dimensional data (2 dimension) so that it can be
visualized easily.
Disadvantages of PCA

• Independent variables become less interpretable: After implementing PCA on the dataset, your
original features will turn into Principal Components. Principal Components are the linear
combination of your original features. Principal Components are not as readable and
interpretable as original features.
• Data standardization is must before PCA: You must standardize your data before implementing
PCA, otherwise PCA will not be able to find the optimal Principal Components. All the
categorical features are required to be converted into numerical features before PCA can be
applied.
• Information Loss: Although Principal Components try to cover maximum variance among the
features in a dataset, if we don't select the number of Principal Components with care, it may
miss some information as compared to the original list of features. PCA assumes that the
principal components are orthogonal.
T-distributed Stochastic Neighbour Embedding
(t-SNE) is an unsupervised non-linear
dimensionality reduction technique used for
exploring high dimensional data. It is used for
visualizing high-dimensional data in a lower-
dimensional space mainly in 2D or 3D. Unlike
linear methods such as Principal Component
Analysis (PCA), t-SNE focus on preserving the
local structure and pattern of the data.

t-SNE is a dimensionality reduction algorithm

which allows us to visualise high-dimensional data
in a 2D plot. It does it by modelling each high-
dimensional object by a two- or three-dimensional
point in such a way that similar objects are
modelled by nearby points and dissimilar objects
are modelled by distant points. But, it
is computationally expensive and sensitive to
hyperparameters, meaning the plot you get at the
end depends on the perplexity you set.
How t-SNE works?

1. Measure pairwise similarities: First, t-SNE calculates how similar each pair of cells is to each other.
It does this by looking at the “distance” between them, often using a method like Gaussian (normal)
distribution. The idea is that if two cells have very similar gene expression profiles, they should have
a high similarity score, and if they’re far apart, the similarity should be low.
2. Create probabilities: These similarities are turned into probabilities (think of it like a “likelihood”
that two cells are close neighbours). The closer two points are, the higher the probability that they
are neighbours.
3. Map to lower dimensions: Now, t-SNE creates a new 2D space and tries to position the data points
there. The goal is to place points so that similar cells in the original space are still close together in
the new space, and dissimilar cells are far apart.
4. Optimize the layout: This is where the “stochastic” part comes in. t-SNE uses a technique called
gradient descent, which is a way of adjusting the positions of the points in the lower-dimensional
space step by step, trying to make the distribution of similarities in the lower space match the
original distribution as closely as possible.
Advantages of t-SNE

• Great for Visualization: t-SNE is particularly used to convert complex high-dimensional data
into 2D or 3D for visualization making patterns and clusters easy to observe.

• Preserve Local Structure: Unlike linear techniques like PCA t-SNE focus on maintaining the
local relationships between data points meaning similar data points remain close in the lower-
dimensional space.

• Non-Linear Capability: It captures non-linear dependencies in the data which makes it suitable
for complex datasets where linear methods fail.

• Cluster Separation: Helps in clearly visualizing clusters and class separability in datasets like
MNIST making it easier for interpretation and exploration.
Disadvantages of t-SNE

• Computationally Intensive: t-SNE is slower and more computationally expensive compared to

linear methods especially on large datasets.

• Non-deterministic Output: The output can vary with each run due to its randomness unless a
fixed random_state is used.

• Not Scalable for Large Datasets: It struggles with very large datasets (e.g., millions of points)
unless optimized or approximated versions are used.

• Not Good for Downstream Tasks: t-SNE is mainly for visualization and is not suitable for
dimensionality reduction when feeding data into other ML algorithms.

• No Global Structure Preservation: It may distort global distances and structures in the data
focusing more on preserving local neighbourhoods.
 Feature PCA t-SNE
Type of Dimensionality Reduction Linear dimensionality reduction technique Non-linear dimensionality reduction technique

Structure Preservation Preserves global structure of data Preserves local structure (clusters) of data

Works well for global patterns but may not capture local One of the best techniques for visualizing local
Effectiveness
clusters effectively clusters

Involves hyperparameters such as learning rate and

Hyperparameters Has fewer hyperparameters
number of steps

Sensitivity to Outliers Sensitive to outliers More robust to outliers

Deterministic algorithm i.e it produces the same result Non-deterministic algorithm as results may vary
Algorithm Type
every time due to randomness

Transforms data into a new coordinate system to maximize Minimizes the distance between points in a
Transformation Method
variance Gaussian probability distribution

Allows control over variance preservation using Preserves distances rather than variance and is
Variance Preservation Control
eigenvalues controlled by hyperparameters

Computational Efficiency Computationally efficient especially for large datasets Computationally expensive for large datasets

Primarily designed for data visualization and

Primary Use Can be used for dimensionality reduction and visualization
exploratory analysis

Data Separability Works well for linearly separable datasets Better suited for non-linearly separable datasets

Sensitivity to Data Ordering Can be sensitive to the ordering of data points Less sensitive to data ordering
What is Prompt clustering?
Prompt clustering is a technique in AI and prompt engineering that involves grouping similar prompts
together based on their content, structure, or intended purpose. This method is used to organize, analyze,
and optimize large sets of prompts, improving efficiency in prompt management and AI system
performance.

Understanding Prompt clustering

Prompt clustering leverages similarity measures and grouping algorithms to identify patterns and
relationships among different prompts. It helps in understanding the diversity of prompts used in an AI
system and can lead to insights for prompt optimization and standardization.
Key aspects of Prompt clustering include:
1. Similarity Analysis: Identifying commonalities between different prompts.
2. Grouping: Categorizing prompts into clusters based on defined criteria.
3. Pattern Recognition: Discovering recurring themes or structures in prompt sets.
4. Efficiency Optimization: Streamlining prompt libraries and reducing redundancy.
5. Insight Generation: Providing a bird's-eye view of prompt usage and effectiveness.
Methods of Prompt clustering

1. Semantic Clustering: Grouping prompts based on their meaning or intent.

2. Structural Clustering: Categorizing prompts by their syntactic structure or format.
3. Task-based Clustering: Grouping prompts that serve similar purposes or tasks.
4. Performance-based Clustering: Clustering prompts based on their effectiveness or output
quality.
5. Hierarchical Clustering: Creating nested clusters of prompts at different levels of similarity.
6. K-means Clustering: Using the k-means algorithm to group prompts into a predefined number
of clusters.
7. Topic Modeling: Applying techniques like LDA (Latent Dirichlet Allocation) to identify themes
in prompt sets.
Example of Prompt clustering

Consider a set of customer service prompts:

1. "How can I reset my password?"
2. "What's the process for changing my account password?"
3. "Tell me about your return policy."
4. "What's your policy on product returns?"
5. "When will my order be delivered?"
6. "What's the estimated shipping time for my purchase?"
Clustering might result in:
• Cluster A (Password Reset): Prompts 1 and 2
• Cluster B (Return Policy): Prompts 3 and 4
• Cluster C (Order Shipping): Prompts 5 and 6
This clustering helps in organizing and potentially optimizing these prompts for better AI responses.
The End