Introduction to Bayesian Classification

Bayesian Classification is a statistical classification technique based on probability theory.

It predicts the probability that a data item belongs to a particular class.

It is one of the most important supervised learning techniques used in machine learning and
artificial intelligence.

The method is based on Bayes' theorem.

. What is Classification?

Classification is a process of:

• Placing data into predefined categories or classes.

• Predicting the class label of new data.

Examples

Input Class

Email containing “Win money” Spam

Student marks = 85% Pass

Disease
Patient symptoms
type

Key Features of Bayes Classifier

1. Probabilistic Model: Based on conditional probability using Bayes' Theorem.

2. Supervised Learning Approach: Requires labeled training data.

3. Classifies Using Prior and Likelihood: Combines prior class probabilities with data
likelihood.

4. Model Interpretability: Easy to interpret due to mathematical formulation.

5. Scalability: Efficient for larger datasets.

6. Handles Continuous/Discrete Features: Works with both types depending on

implementation.
What is Bayesian Classification?

Bayesian classification uses probability to predict class membership.

It answers:

“What is the probability that this data belongs to a certain class?”

The classifier calculates probabilities for all classes and chooses the class with the highest
probability.

Bayes Theorem
Bayes’ Theorem is a fundamental theorem in probability and machine learning that describes
how to update the probability of an event when given new evidence. It is used as the basis of
Bayes Classification.
𝑃(𝐵 ∣ 𝐴) ⋅ 𝑃(𝐴)
𝑃(𝐴 ∣ 𝐵) =
𝑃(𝐵)

Where:

• 𝑃(𝐶 ∣ 𝑋): Posterior probability of class C given data X

• 𝑃(𝑋 ∣ 𝐶): Likelihood of data X given class C

• 𝑃(𝐶): Prior probability of class C

• 𝑃(𝑋): Marginal probability of data X

Example:

Suppose:

𝐶= Spam email

𝑋= Email contains the word “free”

Then: 𝑃(𝐶 ∣ 𝑋)
means: Probability that the email is spam if it contains the word “free”

Bayes Classification Workflow

Terminologies
 • Prior: Initial belief before evidence (e.g., % of spam emails). It is an Input to the
Classifier. Example:

• 40 out of 100 emails are spam

40
• 𝑃(𝑆𝑝𝑎𝑚) = 100 = 0.4

• Likelihood: Evidence corresponding to a particular given class (e.g., frequency of “free”

in spam). It is an Input to the Classifier.

Example:

• Among spam emails, 30 contain the word “free”

30
• 𝑃(𝐹𝑟𝑒𝑒 ∣ 𝑆𝑝𝑎𝑚) = 40 = 0.75

• Posterior: Updated belief after evidence. It is an Output from the Classifier. This is the
main value used for prediction.

• Evidence: Overall probability of observed features.

The above image demonstrates that Likelihood, Data, and Prior probabilities are used as Input to
the model. Bayes Theorem is used as the mathematical principle. The resultant is Posterior
Distribution.
Steps Involved in Classification

1. Collect Data: From training set, extract features X and target C.

2. Estimate Priors: Calculate P(C) for each class.

3. Estimate Likelihoods: For every feature X, compute P(X∣C).

4. Calculate Posterior: Use Bayes’ Theorem to get P(C∣X).

5. Predict Class: Assign the class with the highest posterior probability. It calculates the
posterior probability for each class and assigns the class with the highest probability.

The most common Bayesian classifier is Naive Bayes.

It assumes:

All features are independent of each other.

This assumption is called the naive assumption.

Example of Independence Assumption

In spam detection:

• Presence of word “free”

• Presence of word “winner”

Naive Bayes assumes these words are independent.

Applications of Bayes Classification

• Stock Market Prediction: Analyzes time-varying relationships and Financial indicators.

• Fraud Detection and Credit Risk Modelling: Analyzes Fraud probability based on
transaction patterns, contextual data, timestamps, etc.

• Medical Diagnosis: Predicts the chances or probability of occurrence of a disease based

on medical history.

• Email Spam Detection and Phishing: Classifies emails and messages as "Spam" or
"Not Spam" based on previous probabilities.

Advantages of Bayes Classifier

 1. Handles small data well.

2. Can incorporate domain knowledge.

3. Probabilistic output allows threshold tuning.

4. Can model non-linear decision boundaries.

5. Adaptable to both discrete and continuous data.

Disadvantages of Bayes Classifier

1. Computationally expensive for high-dimensional data.

2. Complex to model feature dependencies.

3. Requires large data to estimate joint distributions.

4. Not scalable for many features or classes.

5. Poor performance if assumptions are violated.

What is Naive Bayes?

Naive Bayes is a simple machine learning algorithm used for classification.

It predicts which category a sentence or data belongs to based on probability.

Example:

• Spam or Not Spam

• Sports or Not Sports

• Positive or Negative review

It is called Naive because it assumes that all words are independent of each other.

Example:
In the sentence:

“Very close game”

Naive Bayes assumes:

• “Very” is independent
• “close” is independent

• “game” is independent

Even though in real life words may be related

Steps of Naive Bayes:

Step 1: Find Prior Probability

Count how many sentences belong to each class.

Step 2: Count Words

Count how many times each word appears in each class.

Step 3: Calculate Likelihood

Find probability of each word in each class.

Step 4: Multiply Probabilities

Multiply probabilities of all words.

Step 5: Choose Bigger Probability
Class with higher probability is the answer.

Advantages of Naive Bayes

• Simple and fast

• Works well for text classification

• Easy to implement

• Good for spam filtering and NLP tasks

Disadvantages

• Assumes words are independent

• Less accurate for complex relationships

• Needs enough training data

Naive Bayes Example

Question:

Suppose you are building a classifier that predicts whether a message is Positive or Negative.

Training data:

Text Label

“Good movie” Positive

“Bad acting” Negative

“Good story” Positive

“Bad movie” Negative

Classify the sentence:

“Good movie”

using the Naive Bayes algorithm.

Step 1: Find Prior Probabilities

Total sentences = 4

Positive = 2

Negative = 2
2 1
𝑃(𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒) = =
4 2
2 1
𝑃(𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒) = =
4 2

Step 2: Count Words

Positive Class

Sentences:

• Good movie

• Good story

Word counts:

Word Count

Good 2

Movie 1

Story 1

Total Positive words = 4

Negative Class

Sentences:

• Bad acting

• Bad movie

Word counts:
Word Count

Bad 2

Acting 1

Movie 1

Total Negative words = 4

Step 3: Vocabulary Size

Unique words:
• Good

• movie

• story

• Bad

• acting

Total vocabulary:

𝑉=5

We use Laplace smoothing:

𝑐𝑜𝑢𝑛𝑡(𝑤𝑜𝑟𝑑) + 1
𝑃(𝑤𝑜𝑟𝑑 ∣ 𝑐𝑙𝑎𝑠𝑠) =
𝑡𝑜𝑡𝑎𝑙 𝑤𝑜𝑟𝑑𝑠 + 𝑉
The formula means:
“Find probability of a word appearing in a specific class.”

Meaning of Each Part

Symbol Meaning

How many times the word appears in that

𝑐𝑜𝑢𝑛𝑡(𝑤𝑜𝑟𝑑)
class

+1 Added to avoid zero probability

Symbol Meaning

total words Total words in that class

𝑉 Total unique words (vocabulary size)

Important:
Why Do We Add +1 ?

Suppose a word never appeared in training data.

Without +1:

𝑃(𝑤𝑜𝑟𝑑 ∣ 𝑐𝑙𝑎𝑠𝑠) = 0

Then total multiplication becomes zero.

That is a problem.
So we use Laplace smoothing by adding 1.

Step 4: Calculate Probability for Test Sentence

Sentence:

“Good movie”

Words:

• Good
• movie

For Positive
2+1 3
𝑃(𝐺𝑜𝑜𝑑 ∣ 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒) = =
4+5 9
1+1 2
𝑃(𝑚𝑜𝑣𝑖𝑒 ∣ 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒) = =
9 9

Now multiply with prior probability:

 1 3 2
𝑃(𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 ∣ 𝑋) ∝ × ×
2 9 9
6
= = 0.037
162

For Negative
0+1 1
𝑃(𝐺𝑜𝑜𝑑 ∣ 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒) = =
4+5 9
1+1 2
𝑃(𝑚𝑜𝑣𝑖𝑒 ∣ 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒) = =
9 9

Now multiply:
1 1 2
𝑃(𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒 ∣ 𝑋) ∝ × ×
2 9 9
2
= = 0.012
162

Final Decision

Since:

0.037 > 0.012

The sentence:

“Good movie”

is classified as: Positive