Module -4

Naive Bayes: Introduction to Naive Bayes Classifier-Bayes' Theorem and

Conditional Probability-Gaussian, Multinomial, and Bernoulli Naive
Bayes. Bayesian Belief Network-EM algorithm.
Gaussian Distribution in Naïve Bayes

What It Means
When the feature values are continuous numbers (like temperature,
heart rate, blood pressure, etc.),we can’t count frequencies like
“yes/no” or “word present/absent.”
So, the Gaussian Naïve Bayes algorithm assumes that these numeric
features follow a normal (bell-shaped) distribution, also known as the
Gaussian distribution.
Gaussian Distribution Curve

Gaussian distribution curve is a

symmetrical, bell-shaped graph that
plots the probability of a continuous
random variable

It is defined by
• its mean (𝜇), which is the peak of
the curve,
• its standard deviation (𝜎), which
determines the curve's width.
The standard deviations are used to subdivide the area under the normal
curve. Each subdivided section defines the percentage of data, which falls
into the specific region of a graph.

Analysis :
A smaller standard deviation results in a narrower and taller bell curve,
indicating that data points are clustered closely around the mean.

Conversely,

A larger standard deviation leads to a wider and shorter bell curve,

suggesting that data points are more spread out from the mean.

The Empirical Rule, also known as the 68-95-99.7 rule, quantifies the
proportion of data falling within certain intervals around the mean in a
normal distribution. It provides a quick way to estimate the spread of data
without performing detailed calculations.
The formula for the Gaussian distribution is:
Simple Intuition
Example
 Compare probabilities

Prediction → Disease = Yes

Because 0.115 > 0.0044.
Calculate the probability density function of normal distribution using the following data. x = 2,
μ = 3 and σ= 4.

• Variable (x) = 2
• Mean = 3
• Standard Deviation = 4

f(2, 3, 4) = 0.09666703
Properties
Symmetry:The normal distribution is symmetric around its mean. This means the left side of the
distribution mirrors the right side.

Mean, Median, and Mode:In a normal distribution, the mean, median, and mode are all equal and
located at the center of the distribution.

Bell-shaped Curve: The curve is bell-shaped, indicating that most of the observations cluster around
the central peak, and the probabilities for values further away from the mean taper off equally in both
directions.

Standard Deviation:The spread of the distribution is determined by the standard deviation. About
68% of the data falls within one standard deviation of the mean, 95% within two standard deviations,
and 99.7% within three standard deviations
Multinomial Distribution

• The multinomial distribution is a generalization of the binomial

distribution. While the binomial distribution models experiments
with only two possible outcomes (such as success or failure)
• The multinomial distribution extends this concept to experiments
where there are more than two possible outcomes.
Multinomial Naive Bayes(MNB)
• Multinomial Naive Bayes is one of the variation of Naive
Bayes algorithm which is ideal for discrete data and is typically used in
text classification problems.
• It models the frequency of words as counts and assumes each feature
or word is multinomially distributed.
• MNB is widely used for tasks like classifying documents based on
word frequencies like in spam email detection.
• During training, the model estimates the probabilities of each feature
(word) occurring in each class, and these probabilities are used to
predict the class of new documents.
How Does Multinomial Naive Bayes Work?

In Multinomial Naive bayes

• The word "Naive" means that the method assumes all features like
words in a sentence are independent from each other.
• "Multinomial" refers to how many times a word appears or how often
a category occurs.
• It works by using word counts to classify text. The main idea is that it
assumes each word in a message or feature is independent of each
others.
• This means the presence of one word doesn't affect the presence of
another word which makes the model easy to use.
The model looks at how many times each word appears in messages from different
categories (like "spam" or "not spam").
For example if the word "free" appears often in spam messages that will be used to
help predict whether a new message is spam or not.
To calculate the probability of a message belonging to a certain category Multinomial
Naive Bayes uses the multinomial distribution:
To estimate how likely each word is in a particular class like "spam"
or "not spam" we use a method called Maximum Likelihood
Estimation (MLE). This helps finding probabilities based on actual
counts from our data. The formula is:
Slove the below binary classification problem using multinomial
Naive Bayes
Vocabulary size 𝑉 = 10
Not Spam Class:
0+1 1
𝑃 buy ∣ Not Spam = =
6 + 10 16

1+1 2
𝑃 now ∣ Not Spam = =
6 + 10 16
Test Message: "buy now” is classified as Spam as per Multinomial
Naive Bayes classifier.
Multinomial Naïve Bayes Classifier
• The Multinomial Naïve Bayes Classifier is a probabilistic machine
learning model used primarily for text classification problems such as
spam detection, sentiment analysis, and document categorization.

• It is based on Bayes’ Theorem and assumes that the features

(variables) are conditionally independent given the class label —
hence the term “Naïve.”

• The word “Multinomial” refers to the multinomial distribution,

which models the probability of different outcomes in a fixed number
of trials where each trial can result in more than two outcomes.
 The multinomial distribution is applicable to experiments that satisfy the following
conditions:

1. Repeated Trials:
The experiment consists of a fixed number of repeated and identical trials.
Example: Rolling a die five times instead of just once.

2. Independence:
Each trial is independent of the others.
Example: When rolling two dice, the outcome of one die does not affect the outcome of the other.

3. Constant Probabilities:
The probability of each possible outcome remains the same for every trial.
!
Example: For a fair six-sided die, the probability of obtaining any particular face (1 through 6) is on
"
every roll.

4. Mutually Exclusive Outcomes:

Each trial results in one specific outcome among the possible categories.
Example: Rolling a die yields exactly one number (1 to 6) on each trial.
Multinomial Naïve Bayes Classifier

• The word “Multinomial” refers to the multinomial distribution, which models

the probability of different outcomes in a fixed number of trials where each trial
can result in more than two outcomes.

• Multinomial Naive Bayes classifier assumes that the features (e.g., words in text
data) follow a multinomial distribution given the class label. In this case, the
probability of observing a particular word in a document is modeled using a
multinomial distribution

• During training, the model estimates the probabilities of each feature (word)
occurring in each class, and these probabilities are used to predict the class of new
documents.
Multinomial in the context of text classification:

•Each document is viewed as a collection of words (features).

•The frequency (count) of each word is used to predict the category

(e.g., sports, politics, entertainment).

•Since words can appear multiple times and there are more than two
possible words, the multinomial distribution is appropriate.
V → The size of the vocabulary, i.e., the total number of unique words across all
documents.
Example: If the dataset contains 2,000 unique words, then 𝑉 = 2000.

α (Alpha — the smoothing parameter) → Used to avoid zero probability for words that
may not appear in the training data of a class. Commonly set to 1 (called Laplace
smoothing).
Bernoulli Naive Bayes Classifier
• Bernoulli Naive Bayes Classifier — one of the simplest and powerful
algorithms for binary (yes/no or 0/1) classification problems,
especially with text data such as spam detection or sentiment
analysis- used to determine whether a certain word is present in a
document or not.

When to Use Bernoulli Naive Bayes

• Data features are binary (e.g., word presence/absence, yes/no
questions).
Text classification (spam detection, sentiment analysis).
What is Bernoulli Naive Bayes?
The Bernoulli Naive Bayes (BNB) classifier is a variant of the Naive Bayes
algorithm that is specifically designed for binary/boolean features — i.e., features
that can take only two values:
1 (feature present) or 0 (feature absent).

It assumes:
•Each feature is independent of others (the “naive” assumption).
•Each feature follows a Bernoulli distribution, i.e., it can take only two outcomes.
This makes it well-suited for situations where data represents the presence or
absence of certain characteristics.
Bernoulli Distribution
The Bernoulli distribution models a random variable that has only two possible outcomes —
Success (1) or Failure (0).
P(xi∣y) – conditional probability
i is the event
xi The value of feature 𝑖; it can be either 0 or 1 (binary)
y The class label (e.g., Spam or Not Spam).
Step-by-Step with BNB

Data Preparation: Begin with a set of binary data. Each row signifies a data sample while columns represent
features.
Step:1 Calculate the Prior Probabilities of each class
 Step:2 - Calculate Conditional Probabilities for each feature
Compute the conditional probabilities for each feature based on its presence (1) or
absence (0):

For Feature 1:
• P(Feature1=1∣Class0) = 3 /6 =0.5
• P(Feature1=0∣Class0) = 3 /6 =0.5

• P(Feature1=0∣Class1)= 2/4 =0.5

• P(Feature1=1∣Class1)= 2/4 =0.5

For Feature 2:
• P(Feature2=1∣Class0) = 3 /6 =0.5
• P (Feature2=1∣Class1)= 2/4 =0.5

• P(Feature2=0∣Class0) = 3 /6 =0.5
• P (Feature2=0∣Class1)= 2/4 =0.5
Step:3 – Calculate Posterior Probability / Prediction
Prediction: Using the trained model, classify new samples.
Calculate the likelihood of each class for the given features and compute the unnormalized posterior
probability.
For Class 0:
P(Class 0 | Feature 1=1, Feature 2=0 )
= P(Feature 1=1 | Class 0) × P(Feature2=0 | Class 0)
= 0.5 × 0.5= 0.25
•Unnormalized Posterior: P(Class 0) × Likelihood = 0.6 × 0.25 = 0.15

For Class 1:
P (Class 1 | Feature 1=1, Feature 2=0)
= P (Feature 1=1 | Class 1) × P (Feature 2=0 | Class 1)
= 0.5 × 0.5= 0.25
•Unnormalized Posterior: P (Class 1) × Likelihood = 0.4 × 0.25 = 0.10
Slove the below binary classification problem using Bernoulli
Naive Bayes
Calculate prior probability:
Both classes have equal prior probability
𝑃 𝐶 = 0.5
since we have two Spam and two Not Spam
messages.
We have:
•Two classes: Spam and Not Spam
•10 binary word features (presence/absence of a word)
•Each message is represented as a binary vector.
To avoid 0 probabilities, add 1 to every count and increase the denominator by the
total number of possible outcomes (for binary features, add 2):

Why Laplace Smoothing Works

• Laplace Smoothing “pretends” every word (or feature) has appeared at least once in
every class.
• This prevents any word absence from making the entire class probability vanish during
prediction.
• Laplace smoothing is used to handle zero probabilities in Naive Bayes.
• It ensures that each word or feature has a small non-zero probability in every class.
• It makes predictions more stable and realistic, especially for small datasets.
Compute Posterior Probability for Each Class
 d= buy now

Compute Posterior Probability for Each Class

6. Final Classification (Compare Both Classes)
Difference between different types of naïve based classifier
 Bayesian Belief Networks (BBNs)

A Bayesian Belief Network (BBN) is a probabilistic graphical model

that represents a set of variables and their conditional dependencies
using a directed acyclic graph (DAG).

It combines:
• Graph theory → to represent dependencies between variables
• Probability theory → to represent uncertainty
Bayesian Belief Networks (BBNs) Bayesian Belief Networks are valuable tools for understanding
and solving problems involving uncertain events.
 Components of a Bayesian Network

1. Nodes (Vertices)
Each node represents a random variable.
Example: Weather, Traffic, Accident, etc.
2. Edges (Directed Arrows)
Each arrow shows a causal or conditional dependency between variables.
Example: Weather → Traffic means Traffic depends on Weather.
3. Conditional Probability Table (CPT)
Each node has a CPT that quantifies the effect of the parent nodes.
•If a node has no parent → it’s represented by prior probability.
•If it has parents → it’s represented by conditional probability given those parents.
Bayesian Network example
In the diagram, A, B, C, and D
are random variables represented
by the nodes of the network
graph.
•If we are considering node B,
which is connected with node A
by a directed arrow, then node A
is called the parent of Node B.
•Node C is independent of node A.
 Conditional Probability Distribution in a Bayesian Network

Each node 𝑋! in a Bayesian Network is associated with a

Conditional Probability Distribution (CPD) — often represented
as:

Each node in the Bayesian network has condition probability

distribution P(Xi | Parent(Xi) ), which determines the effect of the
parent on that node.
Mathematical Representation
•In the above figure, An alarm 'A' - a node, installed in a
house of a person ‘Alex', which rings upon two
probabilities i.e burglary 'B' and fire 'F’,

•burglary 'B' and fire 'F are - parent nodes of the alarm
node.

•The alarm is the parent node of two probabilities P1

calls and P2 calls

•Upon the instance of burglary and fire, 'P1' and 'P2' call
person ‘Alex', respectively.

•But, there are few drawbacks in this case, as sometimes

'P1' may forget to call the person ‘Alex', even after
hearing the alarm, as he has a tendency to forget things,
quick.

•Similarly, 'P2', sometimes fails to call the person ‘Alex',

as he is only able to hear the alarm, from a certain
distance.
Question : Find the probability that 'P1' is true (P1 has called ‘Alex'), 'P2' is true
(P2 has called ‘Alex') when the alarm 'A' rang, but no burglary 'B' and fire 'F' has
occurred.

P(P1= T, P2 = T, A = T, B = F, F = F). [ where- P1, P2 & A are 'true' events and '~B' & '~F' are 'false' events]
=> P ( P1, P2, A, ~B, ~F)
Calculating Conditional Probability of Events in a Bayesian Network
Step1: If a node has no parent → prior probability
Burglary 'B' -
P (B=T) = 0.001 ('B' is true i.e burglary has occurred)
P (B=F) = (1- 0.001)
= 0.999 ('B' is false i.e burglary has not occurred)

Fire 'F' -
P (F=T) = 0.002 ('F' is true i.e fire has occurred)
P (F=F) = (1-0. .002). = 0.998 ('F' is false i.e fire has not occurred)
Step-2. (Calculating Conditional Probability of Alarm)

P(A=F) = 1- P(A=T)
= 1- 0.95 = 0.05
•The person 'P1' node can be 'true' or 'false' (i.e may have
called the person ‘Alex' or not) .

•It has a parent node, the alarm 'A', which can be 'true' or
'false' (i.e may have rung or may not have rung ,upon
burglary 'B' or fire 'F').

•The person 'P2' node can be 'true' or false' (i.e may have
called the person ‘Alex' or not).

•It has a parent node, the alarm 'A', which can be 'true' or
'false' (i.e may have rung or may not have rung, upon
burglary 'B' or fire 'F').
Question : Find the probability that 'P1' is true (P1 has called ‘Alex'), 'P2' is true
(P2 has called ‘Alex') when the alarm 'A' rang, but no burglary 'B' and fire 'F'
has occurred.
P(P1= T, P2 = T, A = T, B = F, F = F). [ where- P1, P2 & A are 'true' events and
'~B' & '~F' are 'false' events]
=> P ( P1, P2, A, ~B, ~F)

Solution:

P ( P1, P2, A, ~B, ~F)

= P (P1|A) * P (P2 | A) * P (A | ~B~F) * P (~B) * P (~F)
= 0.95 * 0.80 * 0.001 * 0.999 * 0.998
= 0.00075
 Example
Consider a simple BBN with three variables: Rain (R), Sprinkler (S),
and Wet Grass (W). The network encodes the following relationships:

This efficient representation helps the BBN model complex scenarios while
maintaining computational feasibility.
Example: Wet Grass Bayesian Network
Variables:
• 𝑅: Rain
• 𝑆: Sprinkler
• 𝑊: Wet Grass
Relationships: Rain → Sprinkler → WetGrass
 EM (Expectation-Maximization) algorithm
The Expectation-Maximization (EM) algorithm is a
powerful iterative optimization technique used to
estimate unknown parameters in probabilistic
models, particularly when the data is incomplete,
noisy or contains hidden (latent) variables.
 Steps in EM – Algorithm
It works in two steps:
• E-step (Expectation Step): Using the current parameter
estimates, the algorithm calculates the expected values of the
missing or hidden variables. Essentially, it assigns probabilities or
"responsibilities" to different hidden outcomes given the
observed data.
• M-step (Maximization Step): With these updated expectations
from the E-step, the algorithm then re-estimates the model
parameters by maximizing the expected log-likelihood. This
improves how well the model explains the observed data.
Steps in EM – Algorithm
• The EM algorithm works in two repeating phases — the Expectation
step (E-step) and the Maximization step (M-step) — to estimate
parameters when part of the data is hidden or missing.
1. Initialization: The algorithm starts with initial parameter values and assumes the observed
data comes from a specific model.
2. E-Step (Expectation Step):
•Find the missing or hidden data based on the current parameters.
•Calculate the posterior probability of each latent variable based on the observed data.
•Compute the log-likelihood of the observed data using the current parameter estimates.
3. M-Step (Maximization Step):
•Update the model parameters by maximize the log-likelihood.
•The better the model the higher this value.
4. Convergence:
•Check if the model parameters are stable and converging.
•If the changes in log-likelihood or parameters are below a set threshold, stop. If not repeat the
E-step and M-step until convergence is reached
Key Terms in Expectation–Maximization (EM)
Algorithm
1. Latent Variables
2. Likelihood
3. Log-Likelihood
4. Maximum Likelihood Estimation (MLE)
5. Posterior Probability
6. Convergence
latent variable

A latent variable is not directly observed, but influences the observed

data, and must be inferred from the data.
In the EM algorithm, latent variables are the reason why we cannot
directly compute the likelihood.

We see the number of heads → but we don’t know which coin

produced it.
The coin identity (A or B) is the latent variable
Convergence
The EM algorithm runs iteratively until it converges.

Convergence occurs when:

• Parameter updates become extremely small, OR
• The log-likelihood stops increasing significantly.

At this point, the algorithm has reached a stable solution, usually a

local maximum of the likelihood function.
Posterior Probability
Maximum Likelihood Estimation (MLE)
• Maximum Likelihood Estimation (MLE) is a method used to find the
best value of a parameter by choosing the value that makes the
observed data most likely.

In simple words:
• MLE finds the parameter that best explains the data you observed.