Types of Classification

Classification is a supervised machine learning method where the model

tries to predict the correct label of a given input data.
Classification teaches a machine to sort things into categories. It
learns by looking at examples with labels (like emails marked “spam”
or “not spam”).
After learning, it can decide which category new items belong to, like
identifying if a new email is spam or not.
For example a classification model might be trained on dataset of
images labeled as either dogs or cats and it can be used to predict
the class of new and unseen images as dogs or cats based on their
features such as color, texture and shape.

Prof. Rafiqul Islam Classification March 10, 2025 2 / 17

Types of Classification

There are different types of classification problems depending on how

many categories (or classes) we are working with and how they are
organized. There are two main classification types in machine learning:
Binary Classification: is the simplest kind of classification and the
goal is to sort the data into two distinct categories.
Multiclass Classification: Here, instead of just two categories, the
data needs to be sorted into more than two categories. The model
picks the one that best matches the input.
Basically, machine looks at the features in the image (like shape, color, or
texture) and chooses which object the picture is most likely to be based on
the training it received.

Prof. Rafiqul Islam Classification March 10, 2025 3 / 17

Classification Algorithms
Linear classifier models create a linear decision boundary between classes.
They are simple and computationally efficient.
Logistic Regression.
Support Vector Machine with linear kernel.
Non-linear Classifiers: Non-linear models create a non-linear decision
boundary between classes. They can capture more complex relationships
between input features and target variable.
K-Nearest Neighbours
Kernel SVM
Naive Bayes
Decision Tree Classification
Ensemble learning classifiers:
Random Forests,
AdaBoost,
Bagging Classifier,
Prof. Rafiqul Islam Classification March 10, 2025 4 / 17
Logistic Regression

Logistic Regression is a popular algorithm used for binary classification

problems, where the target variable is categorical with two classes.
It models the probability of the target variable given the input
features and predicts the class with the highest probability.
Logistic regression is a type of generalized linear model, where the
target variable follows a Bernoulli distribution.
The model consists of a linear function of the input features, which is
transformed using the logistic function to produce a probability value
between 0 and 1.

Prof. Rafiqul Islam Classification March 10, 2025 5 / 17

Logistic Regression Derivation

Logistic regression models the probability that a given input belongs

to a particular class.
The logistic function (sigmoid function) is defined as:

1
σ(z) = (1)
1 + e −z
The decision boundary is determined by:
1
P(y = 1|x) = (2)
1+ e −(β0 +β1 x)
The hypothesis function for logistic regression is:
1
hθ (x) = (3)
1 + e −θT x

Prof. Rafiqul Islam Classification March 10, 2025 6 / 17

Numerical Example of Logistic Regression

Consider a dataset with a single feature x and binary output y .

Given β0 = −3, β1 = 0.8, compute the probability for x = 4:
1
P(y = 1|x = 4) = (4)
1+ e −(−3+0.8×4)
Solving:
1
P(y = 1|x = 4) = ≈ 0.55 (5)
1 + e −0.2
Interpretation: The model predicts a 55% probability that y = 1 for
x = 4.

Prof. Rafiqul Islam Classification March 10, 2025 7 / 17

Problem Statement
Objective: Predict whether a loan application will be approved (1) or not
(0) based on applicant income.
Given the dataset:

Income (X) Loan Approved (Y’)

15 0
20 0
25 0
30 0
35 0
40 1
45 1
50 1
55 1
60 1

Prof. Rafiqul Islam Classification March 10, 2025 8 / 17

Step-by-Step Calculation of B0 and B1

1 Compute the mean values:

15 + 20 + 25 + 30 + 35 + 40 + 45 + 50 + 55 + 60
X̄ = = 37.5
10
0+0+0+0+0+1+1+1+1+1
Y¯ ′ = = 0.5
10
Compute (Xi − X̄ )2 and (Xi − X̄ )(Yi′ − Y¯ ′ ):
P P
2

Prof. Rafiqul Islam Classification March 10, 2025 9 / 17

Step-by-Step Calculation of B0 and B1

(Xi − X̄ )2 (Xi − X̄ )(Yi′ − Y¯ ′ )

506.25 -11.25
306.25 -7.5
156.25 -3.75
56.25 -1.25
6.25 -0.25
6.25 0.25
56.25 1.25
156.25 3.75
306.25 7.5
506.25 11.25

Prof. Rafiqul Islam Classification March 10, 2025 10 / 17

Step-by-Step Calculation of B0 and B1

1 Compute B1 and B0 :

(Xi − X̄ )(Yi′ − Y¯ ′ )
P
22.5
B1 = = = 0.0109
(Xi − X̄ )2
P
2060
 ¯′ 
Y
B0 = log − B1 X̄
1 − Y¯ ′
 
0.5
B0 = log − (0.0109 × 37.5) = −0.409
0.5
2 Final logistic regression equation:
1
P(Y ′ = 1|X ) =
1+ e −(−0.409+0.0109X )

Prof. Rafiqul Islam Classification March 10, 2025 11 / 17

New Predictions

Given new income values of 32 and 52, predict the probability of loan
approval:
1
P(Y ′ = 1|X = 32) = −(−0.409+0.0109×32)
1+e
1
P(Y ′ = 1|X = 32) = −(−0.0618)
≈ 0.485
1+e
Since the probability is 0.485, the model predicts that the loan is more
likely to be not approved.
1
P(Y ′ = 1|X = 52) =
1 + e −(−0.409+0.0109×52)
1
P(Y ′ = 1|X = 52) = ≈ 0.539
1 + e 0.157
Since the probability is 0.539, the model predicts that the loan is more
likely to be approved.

Prof. Rafiqul Islam Classification March 10, 2025 12 / 17

Naive Bayes Classification

Naive Bayes is a probabilistic classifier based on Bayes’ theorem.

The algorithm assumes that the features are independent of each
other, which is why it is called ”naive.”
It calculates the probability of a sample belonging to a particular class
based on the probabilities of its features.
Assumes features are conditionally independent given the class.
Formula:
P(X |C )P(C )
P(C |X ) =
P(X )
Where:
P(C |X ) is the posterior probability of class C given data X .
P(X |C ) is the likelihood of data given class.
P(C ) is the prior probability of class.
P(X ) is the evidence (normalizing constant).

Prof. Rafiqul Islam Classification March 10, 2025 13 / 17

Step-by-Step Calculation for Naive Bayes
Given Data:
Word (Feature) Spam (Count) Not Spam (Count)
Free 5 1
Offer 4 2
Click 3 2
Money 6 1
Total emails in dataset: 10 (5 spam, 5 not spam).
Prior Probabilities:
5 5
P(Spam) = = 0.5, P(NotSpam) = = 0.5
10 10
Likelihood Calculation for ”Free” and ”Offer”:
5 1
P(Free|Spam) = , P(Free|NotSpam) =
18 6
4 2
P(Offer |Spam) = , P(Offer |NotSpam) =
18 6
Prof. Rafiqul Islam Classification March 10, 2025 14 / 17
Computing Posterior Probabilities

Using Bayes’ Theorem:

Compute the numerator for each class:

P(Spam|X ) ∝ P(X |Spam)P(Spam)

P(NotSpam|X ) ∝ P(X |NotSpam)P(NotSpam)

Calculate for ”Free” and ”Offer”:
(5/18) × (4/18) × 0.5
P(Spam|X ) =
P(X )

(1/6) × (2/6) × 0.5

P(NotSpam|X ) =
P(X )

Prof. Rafiqul Islam Classification March 10, 2025 15 / 17

Final Probability Calculation

Step 1: Compute the Probability Values

Compute the probability of ”Spam”:

(5/18) × (4/18) × 0.5

P(Spam|X ) = = 0.526
((5/18) × (4/18) × 0.5) + ((1/6) × (2/6) × 0.5)

Compute the probability of ”Not Spam”:

(1/6) × (2/6) × 0.5

P(NotSpam|X ) = = 0.
((5/18) × (4/18) × 0.5) + ((1/6) × (2/6) × 0.5)

Prof. Rafiqul Islam Classification March 10, 2025 16 / 17

Final Decision

Compare P(Spam|X ) and P(NotSpam|X ).

Since P(Spam|X ) = 0.5263 is greater than P(NotSpam|X ) = 0.4737,
the email is classified as Spam.
Conclusion: Since spam probability is higher, the model classifies the
email as Spam.

Prof. Rafiqul Islam Classification March 10, 2025 17 / 17