Step-by-Step Explanation of Naive Bayes Model Training with Simple Examples
Step 1: Estimate Prior Probabilities 𝑷(𝑪)
What is it?
• The prior probability is the probability of each class before observing the features (words) in the
document. It is based on the relative frequency of each class in the training data.
Formula:
Number of documents in class C
𝑃(𝐶) =
Total number of documents
Example: Imagine we have a dataset of 6 messages:
• 3 Spam messages: "Buy now", "Hurry up", "Exclusive offer"
• 3 Not Spam messages: "How are you?", "Let's meet", "Have a good day"
Total number of messages = 6.
• Number of Spam messages = 3.
• Number of Not Spam messages = 3.
3
Prior probability of Spam: 𝑃(Spam) = 6 = 0.5
3
Prior probability of Not Spam: 𝑃(Not Spam) = 6 = 0.5
Interpretation: Before analyzing any words in the messages, we know that there’s a 50% chance that a
randomly chosen message is Spam or Not Spam, because both categories are equally represented.
Step 2: Estimate Likelihoods 𝑷(𝒙𝒊 ∣ 𝑪)
What is it?
• Likelihood is the probability of each feature (word) occurring given a class (Spam or Not Spam).
• For Multinomial Naive Bayes, the likelihood is calculated using the frequency of words in each class.
• For Bernoulli Naive Bayes, the likelihood is calculated based on the presence or absence of words in each
class.
Example: Let's use the same dataset and compute the likelihood for the word "Buy" in the Spam class.
• Spam Messages: "Buy now", "Hurry up", "Exclusive offer"
• Word “Buy” appears in 1 Spam message ("Buy now").
Total words in Spam messages = 3 ("Buy", "now", "Hurry", "up", "Exclusive", "offer").
For Multinomial Naive Bayes:
• The likelihood of "Buy" in Spam is calculated as:
Frequency of "Buy" in Spam 1
𝑃(Buy ∣ Spam) = = = 0.167
Total words in Spam 6
0
For Not Spam messages, if "Buy" appears in 0 messages: 𝑃(Buy ∣ Not Spam) = 6 = 0
For Bernoulli Naive Bayes:
• The likelihood is calculated based on the presence (1) or absence (0) of the word "Buy" in each class.
• If the word "Buy" is present in 1 Spam message and 0 Not Spam messages, we would have:
𝑃(Buy ∣ Spam) = 1, 𝑃(Buy ∣ Not Spam) = 0
Interpretation:
• Multinomial: We calculate the frequency of the word "Buy" in each class.
• Bernoulli: We only care about whether the word "Buy" appears in each message.
Step 3: Calculate Posterior Probability for Each Class
• The posterior probability is the probability of a class given the features (words) in the document. It is
calculated using Bayes’ Theorem.
• We multiply the prior probability of a class with the likelihoods of each feature (word) in the document,
and then select the class with the highest posterior probability.
𝑛
Formula: 𝑃(𝐶 ∣ 𝑋) = 𝑃(𝐶) × ∏𝑖=1 𝑃( 𝑥𝑖 ∣ 𝐶)
Where: 𝑃(𝐶 ∣ 𝑋): Posterior probability of class 𝐶given the words 𝑋.
• 𝑃(𝐶): Prior probability of class 𝐶.
• 𝑃(𝑥𝑖 ∣ 𝐶): Likelihood of feature 𝑥𝑖 given class 𝐶.
Example: Consider the message “Buy now”. We want to calculate the posterior probability of this message
being Spam or Not Spam.
1. Prior Probability: 𝑃(Spam) = 0.5. 𝑃(Not Spam) = 0.5
2. Likelihoods:
For Spam: 𝑃(Buy ∣ Spam) = 0.167 𝑃(now ∣ Spam) = 0.167
For Not Spam: 𝑃(Buy ∣ Not Spam) = 0 𝑃(now ∣ Not Spam) = 0.167
3. Posterior Probability:
For Spam: 𝑃(Spam ∣ "Buy now") = 𝑃(Spam) × 𝑃(Buy ∣ Spam) × 𝑃(now ∣ Spam)
𝑃(Spam ∣ "Buy now") = 0.5 × 0.167 × 0.167 = 0.0139
For Not Spam: 𝑃(Not Spam ∣ "Buy now") = 𝑃(Not Spam) × 𝑃(Buy ∣ Not Spam) × 𝑃(now ∣ Not Spam)
𝑃(Not Spam ∣ "Buy now") = 0.5 × 0 × 0.167 = 0
4. Class Selection: Since P(Spam | "Buy now") is higher, the message is classified as Spam.
Summary of Steps:
o Prior Probabilities: Estimate the probability of each class based on how many documents belong to that class.
o Likelihoods: Calculate the probability of each word occurring given the class. For Multinomial Naive Bayes,
use word frequencies. For Bernoulli, use word presence/absence.
o Posterior Probability: Multiply the prior probability by the likelihoods for each word and select the class with
the highest posterior probability.
Question 1: Classifying Tweets Based on Sentiment Using Naive Bayes
Given the following four tweets, we will apply the Naive Bayes classification algorithm to categorize them into
Positive or Negative based on the sentiment expressed:
• Tweet 1: "I love the new features on the phone, it’s amazing!"
• Tweet 2: "The phone is slow, and the battery drains too quickly. Disappointing."
• Tweet 3: "Great camera and performance, but the software could be better."
• Tweet 4: "Terrible experience, the phone keeps freezing and crashing."
Task Breakdown:
1. Convert the tweets into a Bag-of-Words (BoW) model.
2. Calculate the prior probabilities for Positive and Negative based on the given tweets.
3. Calculate the likelihoods of each word given the class.
4. Classify each tweet as Positive or Negative using Naive Bayes classification.
5. Explain the classification results.
Step 1: Convert the Tweets into a Bag-of-Words (BoW) Model
We define the vocabulary as follows: [phone, features, amazing, slow, battery, drains, disappointing,
camera, performance, software, terrible, freezing, crashing]
The next step is to convert each tweet into a vector representation based on the vocabulary. We count the
frequency of each word in the tweet. Here's the BoW representation:
performa
software
amazing
crashing
disappoi
freezing
features
camera
terrible
battery
drains
phone
Tweet
nting
slow
nce
Tweet 1: 1 1 1 0 0 0 0 0 0 0 0 0 0
Tweet 2: 1 0 0 1 1 1 1 0 0 0 0 0 0
Tweet 3: 0 0 0 0 0 0 0 1 1 1 0 0 0
Tweet 4: 1 0 0 0 0 0 0 0 0 0 1 1 1
Step 2: Calculate Prior Probabilities
The prior probability is the proportion of each class (Positive or Negative) in the dataset.
Class Distribution:
• Positive Tweets: Tweet 1, Tweet 3 (2 out of 4 tweets are Positive)
• Negative Tweets: Tweet 2, Tweet 4 (2 out of 4 tweets are Negative)
𝟐 𝟐
Prior Probabilities: 𝑷(Positive) = 𝟒 = 𝟎. 𝟓. 𝑷(Negative) = 𝟒 = 𝟎. 𝟓
Step 3: Calculate Likelihoods
We calculate the likelihood for each word in each class (Positive and Negative). This is done by counting the
occurrences of each word in the respective classes and dividing by the total number of words in each class.
For Positive Class (Tweet 1 and Tweet 3): Total words in Positive class = 6 (Tweet 1: "phone, features,
amazing", Tweet 3: "camera, performance, software")
Word Count in Positive Total Positive Likelihood P(word | Positive)
Words
phone 2 6 2/6 = 0.333
features 1 6 1/6 = 0.167
amazing 1 6 1/6 = 0.167
slow 0 6 0/6 = 0
battery 0 6 0/6 = 0
drains 0 6 0/6 = 0
disappointing 0 6 0/6 = 0
camera 1 6 1/6 = 0.167
performance 1 6 1/6 = 0.167
software 1 6 1/6 = 0.167
terrible 0 6 0/6 = 0
freezing 0 6 0/6 = 0
crashing 0 6 0/6 = 0
For Negative Class (Tweet 2 and Tweet 4): Total words in Negative class = 9 (Tweet 2: "phone, slow, battery,
drains, disappointing", Tweet 4: "phone, terrible, freezing, crashing")
Word Count in Negative Total Negative Words Likelihood P(word | Negative)
phone 2 9 2/9 = 0.222
features 0 9 0/9 = 0
amazing 0 9 0/9 = 0
slow 1 9 1/9 = 0.111
battery 1 9 1/9 = 0.111
drains 1 9 1/9 = 0.111
disappointing 1 9 1/9 = 0.111
camera 0 9 0/9 = 0
performance 0 9 0/9 = 0
software 0 9 0/9 = 0
terrible 1 9 1/9 = 0.111
freezing 1 9 1/9 = 0.111
crashing 1 9 1/9 = 0.111
Step 4: Calculate Posterior Probability for Each Tweet
Now, we compute the posterior probability for each tweet being Positive or Negative. We multiply the prior
probability by the likelihoods for each word in the tweet, and then select the class with the highest posterior
probability.
For Tweet 1 ("I love the new features on the phone, it’s amazing!"):
Posterior Probability for Positive:
P(Positive ∣ Tweet 1) = P(Positive) × P(phone ∣ Positive) × P(features ∣ Positive) × P(amazing ∣ Positive)
P(Positive ∣ Tweet 1) = 0.5 × 0.333 × 0.167 × 0.167 = 0.0015
Posterior Probability for Negative:
P(Negative ∣ Tweet 1) = P(Negative) × P(phone ∣ Negative) × P(features ∣ Negative) × P(amazing ∣ Negative)
P(Negative ∣ Tweet 1) = 0.5 × 0.222 × 0 × 0 = 0
Result: Tweet 1 is classified as Positive.
For Tweet 2 ("The phone is slow, and the battery drains too quickly. Disappointing."):
Posterior Probability for Positive:
P(Positive ∣ Tweet 2) = P(Positive ) × P(phone ∣ Positive ) × P(slow ∣ Positive) × P(battery
∣ Positive) × P(drains ∣ Positive )
P(Positive ∣ Tweet 2) = 0.5 × 0.333 × 0 × 0 × 0 = 0
Posterior Probability for Negative:
P(Negative ∣ Tweet 2) = P(Negative) × P(phone ∣ Negative) × P(slow ∣ Negative) × P(battery
∣ Negative) × P(drains ∣ Negative)
P(Negative ∣ Tweet 2) = 0.5 × 0.222 × 0.111 × 0.111 × 0.111 = 0.0002
Result: Tweet 2 is classified as Negative.
Step 5: Summary of Classification Results
Tweet Classified As
Tweet 1: "I love the new features on the phone, it’s amazing!" Positive
Tweet 2: "The phone is slow, and the battery drains too quickly. Disappointing." Negative
Tweet 3: "Great camera and performance, but the software could be better." Positive
Tweet 4: "Terrible experience, the phone keeps freezing and crashing." Negative
Explanation:
• Tweet 1 is classified as Positive because words like "amazing", "features", and "phone" are
more likely to appear in positive reviews.
• Tweet 2 is classified as Negative because words like "slow", "battery", and "disappointing" are
more likely to appear in negative reviews.