Module 3-Naive Bayes

Naïve Bayes
Naïve Bayes is a machine learning algorithm used for classification tasks, especially in
text classification (e.g., spam detection, sentiment analysis).
It is based on Bayes’ theorem and assumes that features (words in a document) are
independent of each other. This assumption is called “naïve” because, in reality, words in a
sentence are related—but this simplification makes the model faster and effective.

How Naïve Bayes Works

1. Learning from Data (Training Phase)
o The algorithm looks at labeled training data (e.g., emails labeled as spam or
not spam).
o It calculates how often each word appears in each class (spam or not spam).

o It also calculates the overall probability of each class occurring.

2. Making Predictions (Testing Phase)

o When given a new document, it looks at the words inside.

o It calculates the probability that the document belongs to each class.

o It assigns the document to the class with the highest probability.

Bayes' Theorem in Naïve Bayes

The algorithm is based on Bayes’ rule, which states:

Where:
 P(c|d) → Probability of class c given document d (what we want to find).
 P(d|c) → Probability of document d appearing in class c.
 P(c) → Probability of class c occurring in general (prior probability).
 P(d) → Probability of document d occurring (same for all classes, so we can ignore
it).
The model picks the class c that maximizes P(c|d).
Module 3-Naive Bayes

Example: Spam Detection

Email Text Class

1 "Win a free iPhone today!" Spam

2 "Meeting at 3 PM today" Not Spam

3 "Congratulations! You won a prize" Spam

4 "Lunch at 12?" Not Spam

 The model learns that words like "win", "free", "prize" are common in spam.
 When a new email arrives, it checks which words match each category.
 If an email contains many spam-like words, it is classified as spam.

Types of Naïve Bayes Classifiers

1. Multinomial Naïve Bayes – Used for text classification (word frequencies).
2. Bernoulli Naïve Bayes – Used for binary features (e.g., word presence or absence).
3. Gaussian Naïve Bayes – Used when features follow a normal distribution (e.g.,
continuous numerical data).

Advantages of Naïve Bayes

✅ Fast & efficient – Works well with large datasets.
✅ Simple & easy to understand – Based on basic probability rules.
✅ Works well with small data – Can still perform well with limited training examples.
✅ Performs well in text classification – Used in spam filtering, sentiment analysis, etc.

Limitations of Naïve Bayes

❌ Strong independence assumption – It assumes words are independent, which is not
always true.
❌ Struggles with complex relationships – Doesn’t handle dependent features well.
❌ Zero probability problem – If a word never appeared in training data, it may assign a
probability of zero, making classification difficult (solved using smoothing techniques like
Laplace Smoothing).

Where is Naïve Bayes Used?

Module 3-Naive Bayes

 Spam detection (Gmail filters)

 Sentiment analysis (Positive/Negative reviews)
 Document classification (News categorization)
 Medical diagnosis (Predicting diseases based on symptoms)

Naïve Bayes is a powerful yet simple classifier that works well for text classification and
other applications. It assumes independence between features, which is not always true, but
it still performs surprisingly well in many real-world tasks.

4.1 Naive Bayes Classifiers:

image illustrates how the Naïve Bayes classifier works using the bag-of-words approach.
How It Works:
1. Original Text:
o The left side shows a movie review written as a normal paragraph.

o The review contains sentences like "I love this movie!" and "I would
recommend it."
2. Bag-of-Words Representation:
o The middle part represents the text as a "bag" of words, where word order is
ignored.
o Instead of remembering full sentences, we just keep track of which words
appear.
3. Word Frequency Count:
o The right side shows a list of words with how many times they appear.
Module 3-Naive Bayes

o For example, "it" appears 6 times, "I" appears 5 times, and words like
"whimsical", "sweet", and "satirical" appear once.
Why Use This Approach?
 The classifier doesn’t care about the order of words, just how often they appear.
 It helps the algorithm focus on important words (like "whimsical", "adventure",
"humor") rather than sentence structure.
 This makes it easier to calculate probabilities and classify the text correctly.
In short, Naïve Bayes uses word frequency to guess which category (like positive or negative
review) the text belongs to!
Naïve Bayes is a probabilistic classifier, meaning it assigns a probability to each possible
class for a given document and chooses the one with the highest probability.

This means:
 Given a document d, the classifier calculates the probability P(c | d) for each possible
class c in the set of all classes C.
 It selects the class ĉ (the one with the hat symbol) that has the highest probability.
 The argmax operation means "find the class that gives the highest value of P(c | d)."
In simple terms, the Naïve Bayes classifier looks at a document, calculates how likely it is to
belong to each class, and assigns it to the most probable class.
Module 3-Naive Bayes
Module 3-Naive Bayes
Module 3-Naive Bayes
Module 3-Naive Bayes

Training the Naive Bayes Classifier

Naive Bayes is a classifier that uses probabilities to predict the category of a document. To
train the model, we need to learn two main probabilities:
1. P(c) - The probability of a class (class prior probability)
2. P(wᵢ | c) - The probability of a word appearing in a class (word likelihood
probability)
1 Learning P(c) – The Class Prior Probability
This tells us how common a particular class (category) is in our dataset.
Module 3-Naive Bayes

• Formula:
Module 3-Naive Bayes

The Problem with Maximum Likelihood Estimation

(MLE)
We are trying to estimate the probability of a word appearing in a given class using MLE.
The formula:

The Problem
If the word "fantastic" does not appear in any positive document in the training data, then:
 count("fantastic",positive)=0count("fantastic",positive)=0
This makes the numerator zero.

 As a result, the probability P("fantastic"∣positive)P("fantastic"∣positive) becomes 0.

Why Is This a Problem?

In Naive Bayes, probabilities are multiplied together to classify a document. If even one
probability is 0, the entire product becomes 0, which means the classifier will fail to make a
proper [Link] example:
 If you're classifying a positive review that contains the word "fantastic," the classifier
will assign a probability of 0 to the positive class
because P("fantastic"∣positive)=0P("fantastic"∣positive)=0, even if the rest of the
words strongly indicate positivity.
Why Does This Happen?
This happens because maximum likelihood estimation (MLE) relies only on the training
data. If a word is missing from the training data for a specific class, MLE assumes the
probability is zero, even though the word might still be relevant to that class.
How Can We Fix This?
Module 3-Naive Bayes

To solve this problem, we use smoothing techniques like Laplace smoothing. These
techniques add a small value (e.g., 1) to all word counts, ensuring that no probability is ever
zero. Let me know if you'd like me to explain smoothing in detail!,
What Does the Formula Do?
The formula calculates the smoothed probability of a word wi appearing in a document,
given that the document belongs to a specific class cc. It ensures that no probability is ever
zero, even if the word wi does not appear in the training data for class cc.

Numerator: count(wi,c)+1count(wi,c)+1
count(wi,c): This is the number of times the word wiwi appears in documents of
class cc.
o The "+1" is added to ensure that even if count(wi,c)=0count(wi,c)=0, the
probability will not be zero. This is the "add-one" part of Laplace smoothing.

Denominator:
o This is the total number of word occurrences in all documents of class cc.

∣V∣: This is the size of the vocabulary (the total number of unique words across
all classes). Adding ∣V∣accounts for the "+1" added to the numerator for every
o

word.
Module 3-Naive Bayes

What If a Word in the Test Data Was Never Seen in Training?

If a word appears in the test document but was never seen in any training document, we
can’t calculate its probability because:
 It’s not in our vocabulary.
 We don’t have a way to estimate its likelihood.

 Solution:
✅ We ignore unknown words—just remove them from the test document and do not
include them in the probability calculations.
 This is better than assigning them zero probability, which could cause classification
errors.

What About Stop Words?

Stop words are very common words like "the," "is," "a," "and" that appear frequently in
almost all documents but don’t carry much meaning.
Some systems:
 Remove stop words from training and test documents.
 Use a predefined stop word list (e.g., the top 100 most common words).
However, removing stop words doesn’t usually improve performance in text
classification. Most modern classifiers just keep all words.
Module 3-Naive Bayes

 The vocabulary V must include all unique words from all classes for fair probability
calculations.
 Unknown words in test data (not in training) are simply ignored.
 Stop words can be removed, but it usually doesn’t help classification performance.
Module 3-Naive Bayes
Module 3-Naive Bayes

4.4 Optimizing for Sentiment Analysis

What is Binary Naive Bayes?
It’s a method used to classify text (like deciding if a review is positive or negative).
Normally, when we analyze text, we count how many times each word appears.
For example, if we have a review:
➡️"The movie was great, great, great!"

A normal method would count "great" three times.

What’s Different in Binary Naive Bayes?
Instead of counting words multiple times, we only check if a word is present or not.
 If a word appears, we mark it as 1.
Module 3-Naive Bayes

 If it doesn’t appear, we mark it as 0.

For the same sentence:
➡️"The movie was great, great, great!"
Binary Naive Bayes will count "great" only once, no matter how many times it appears.

Why Do This?
Some studies found that just knowing a word is present is often more useful than
knowing its exact count. For example, in sentiment analysis:
 If a review has words like "great", "amazing", or "bad", it’s easy to guess if it’s
positive or negative.
 The exact number of times these words appear doesn’t matter as much.
So, Binary Naive Bayes ignores repeated words and focuses only on which words appear in
a document.

We are trying to classify documents as positive (+) or negative (-) using a

method called Binary Naive Bayes.
Normally, when analyzing text, we count how many times each word appears.
But in Binary Naive Bayes, we only check if a word appears or not (not how
many times).
For example:
Module 3-Naive Bayes

 The word "great" appears in 3 positive documents and 1 negative

document → (3,1)
 The word "boxing" appears in 1 negative document and 0 positive
documents → (0,1)
 The word "film" appears in 1 positive document and 0 negative
documents → (1,0)
 Binary Naive Bayes ignores word frequency and only records if a
word is present (1) or not (0).
 Words like "great" appear mostly in positive (+) documents, so they
indicate positive sentiment.
 Words like "pathetic" and "worst" only appear in negative (-)
documents, so they indicate negativity.

What is the Problem?

 Sometimes, a single word can have different meanings depending on whether it is

negated (not, no, never, didn’t).
 For example:
✅ "I like this movie." → Positive
❌ "I don’t like this movie." → Negative
 Even though both sentences have "like", the second one is negative because of
"don’t" (a negation word).
 A computer might not understand this difference unless we help it!

The Simple Trick to Fix This

 To help the computer understand negation, we add "NOT_" before words that
come after negation words (until the next punctuation mark like a comma or period).
 Example 1:
 ✅ "I don’t like this movie."
🔹 After adding "NOT_": "I don’t NOT_like NOT_this NOT_movie."
 Now, instead of just "like," the computer sees "NOT_like," which appears more in
negative reviews.

Example 2:
 ✅ "You should not ignore this film."
🔹 After adding "NOT_": "You should not NOT_ignore NOT_this NOT_film."
 Now, "NOT_ignore" appears in positive reviews, so the computer knows it's a good
thing!

Why Does This Work?

Module 3-Naive Bayes

 "NOT_like" appears in negative sentences, so the computer learns it’s negative.

 "NOT_ignore" appears in positive sentences, so the computer learns it’s positive.
This is a simple trick that helps the computer understand the meaning of sentences
better!

When we teach a computer to understand if a sentence is positive or negative,

we usually show it lots of examples (positive and negative reviews).
But sometimes, we don’t have enough examples, so the computer doesn’t
learn well.

Solution: Use a Sentiment Lexicon

A sentiment lexicon is just a list of words that are already labeled as positive
or negative.
For example:
✅ Positive words: beautiful, great, happy, confident, love
❌ Negative words: awful, bad, hate, ugly, terrible
Instead of learning from scratch, we can help the computer by using this list.

How Does This Help?

Instead of counting each word separately, we use just two features:
1. “This sentence contains a positive word” → ✅ (Yes or No)
2. “This sentence contains a negative word” → ❌ (Yes or No)
For example:
 Sentence: "This movie is beautiful and amazing!"
o Contains "beautiful" (positive word) ✅
o So the computer thinks this is positive.
 Sentence: "This movie is awful and boring!"
o Contains "awful" (negative word) ❌
o So the computer thinks this is negative.
Module 3-Naive Bayes

When Do We Use This Trick?

 If we have lots of training data, the computer learns from that.
 If we have very little training data, using a sentiment lexicon helps the
computer guess better.

4.5 Naive Bayes for other text classification

tasks:
What is Spam Detection?
Spam detection is the process of finding unwanted emails (spam) and keeping
them out of your inbox.
For example, if you get an email saying:
“You won $1,000,000! Click here to claim your prize!”
This is probably spam because it looks like a scam.

How Does a Computer Detect Spam?

A computer can use Naive Bayes (a type of learning method) to figure out if an
email is spam or not.
Instead of checking every single word, we predefine certain words or
patterns that are common in spam emails.

Examples of Spam Words & Features

Some spam emails use specific words and phrases, like:
 "100% guaranteed" (Scams often promise too much)
 "Millions of dollars" (Fake lottery scams)
 "Urgent reply" (Tries to rush you into clicking)
 "Online pharmaceutical" (Fake medicine sales)
Module 3-Naive Bayes

Some spam emails look different, like:

 The subject line is in ALL CAPS (Spam emails often shout)
 The email contains more images than text (Spam tries to hide words in
images)
 The email has broken HTML code (Poorly made spam emails)
 It says "You can be removed from the list" (Tricking you into clicking)

Instead of analyzing every word in an email, we focus on these common spam

signals.
 If an email contains many of these spam features, the computer marks it
as spam.
 If it doesn’t, the email is safe.

Example
Email 1 (Spam)
SUBJECT: “WIN A FREE VACATION NOW!”
“You have won millions of dollars! Click here to claim your prize.”
 🚨 Spam signals:
o Subject is ALL CAPS
o Mentions "millions of dollars"
o Contains urgent language
✔ Marked as spam!

Email 2 (Not Spam)

SUBJECT: “Meeting reminder”
“Hi, don’t forget our meeting tomorrow at 10 AM.”
Module 3-Naive Bayes

 ✅ No spam signals
✔ Marked as safe!
We train the computer by showing it examples of spam and normal emails.
Over time, it learns to recognize spam based on these patterns.

Using Naïve Bayes for Language Detection:

Language detection means figuring out which language a piece of text is
written in.
For example, if we see:
 "Hello, how are you?" → This is English.
 "Hola, cómo estás?" → This is Spanish.
 "Bonjour, comment ça va?" → This is French.
A computer needs a way to detect the language automatically.

How Does a Computer Detect Language?

Instead of looking at whole words, the computer looks at small pieces of
words called n-grams.

What Are N-Grams?

Think of n-grams as tiny letter groups that appear in a sentence.
For example, in English, the word "the" appears a lot.
If we break "the" into small letter groups:
 2-grams (two-letter groups): "th", "he"
 3-grams (three-letter groups): "the"
Each language has its own common letter patterns:
 English has "th", "ing", "he", "er".
 French has "le", "que", "é ".
Module 3-Naive Bayes

 German has "sch", "ung".

The computer counts these letter groups and figures out which language uses
them the most.

Example
If we see this text:
"The weather is nice today."
We break it into n-grams:
 2-grams: "th", "he", "er", "is", "ni", "ce".
 3-grams: "the", "wea", "her", "ice".
These patterns appear more in English than in other languages, so the
computer guesses "English"! ✅

How Does a Real System Work?

A program called [Link] does this automatically.
It looks at thousands of n-grams (letter patterns) and selects the most useful
ones.
Then, it uses those patterns to quickly and accurately detect languages.

Why Not Just Use Full Words?

 Words can be similar across languages ("hotel" exists in English and
French).
 Small texts (like tweets) might not have enough words to analyze.
 N-grams work for languages with different alphabets, like Arabic or
Chinese!

Instead of looking at whole words, computers look at small letter patterns (n-grams).
Each language has its own patterns, so the computer can guess the language correctly!

What is a Language ID System?

Module 3-Naive Bayes

A Language ID system is a computer program that can guess which language

a sentence is written in.
For example:
 "Hello, how are you?" → The system should recognize English.
 "Hola, cómo estás?" → The system should recognize Spanish.
To do this, the computer needs to learn different languages first.

How Does the Computer Learn?

The computer learns by looking at lots of text in different languages.
Researchers give it text from:
1. Wikipedia (which has articles in many languages).
2. News articles (to learn formal language).
But that’s not enough!

Why Add More Types of Text?

Languages are spoken in different ways in different places.
For example:
 English in Nigeria is a little different from English in the USA.
 Spanish in Spain is different from Spanish in Mexico.
To make sure the system learns all versions of a language, researchers also
give it:
✅ Twitter posts (people use slang and short words).
✅ Bible & Quran translations (available in many languages).
✅ Slang words (from websites like Urban Dictionary).
✅ Dialects (like African American English).
This helps the system understand real-world language!
Module 3-Naive Bayes

1. The computer learns languages by reading lots of

text.
2. It learns from Wikipedia, news, and books.
3. It also learns from tweets, slang, and different
dialects.
4. This helps the system recognize any type of language,
even informal ones!

4.6 Naive Bayes as a Language Model:

Naïve Bayes is a classifier, meaning it helps us decide which category a piece
of text belongs to (e.g., positive/negative, spam/not spam).
But if we only focus on individual words (instead of other features like URLs,
phrases, etc.), then Naïve Bayes starts to look like a "language model."

What is a "Language Model"?

A language model predicts how likely a sentence is to appear in a given
language or category.
For example:
 A positive review model might think words like "amazing," "love," and
"great" are common.
 A negative review model might think words like "terrible," "boring,"
and "hate" are common.
So, a language model assigns a probability to each word, which helps it
determine how likely a sentence is to belong to a category.
Module 3-Naive Bayes

 Each category (positive/negative) has its own language model.

 The model assigns a probability to each word in the sentence.
 To classify a sentence, we multiply the probabilities of all words and see
which category it fits better.

 Naïve Bayes can act like a language model when it only considers
words.
 Each category (positive, negative, spam, etc.) has its own word
probability model.
 The model calculates the probability of the entire sentence by
multiplying word probabilities.
 The sentence is classified into the category with the higher probability.

Module 3-Naive Bayes

We are using Naïve Bayes to classify a sentence as either:

✅ Positive (+) (good review)
❌ Negative (-) (bad review)

The probability we calculated is only part of the Naïve Bayes model.

Module 3-Naive Bayes

 This is called the likelihood → It tells us how well the words match
each category.
 BUT, Naïve Bayes also includes another important factor: Prior
Probability.
Example with Priors
Imagine we have these priors:
 P(positive) = 0.3 (only 30% of reviews are positive).
 P(negative) = 0.7 (70% of reviews are negative).
Even if P(sentence | positive) > P(sentence | negative),
the model might still classify it as negative because negative reviews are more
common.

 We first calculate how well the words fit each category → likelihood.
 We also check how common each category is → prior probability.
 The final decision is based on both of these factors.

4.7 Evaluation: Precision, Recall, F-

measure:
When we build a system to automatically classify text (like detecting spam
emails or finding tweets about a specific topic), we need to measure how well it
performs. A good way to do this is by using Precision, Recall, and F-measure
instead of just looking at accuracy.
1)Why Accuracy Can Be Misleading
Imagine you're trying to find something rare, like a lost key in a huge house.
 If you don’t even search and just say, "The key isn’t here" for every
room, you’ll still be correct most of the time—but you’ll never actually
find the key!
 Your accuracy might look high because most rooms really don’t have the
key, but the result is useless because you didn’t find what you were
looking for.
Module 3-Naive Bayes

Example 1: Spam Detection

Suppose you have 1,000,000 emails, but only 1,000 are spam.
A bad system might just say, "No email is spam."
 ✅ It correctly labels 999,000 emails as "not spam" (True Negatives).
 ❌ But it misses all 1,000 spam emails (False Negatives).
Accuracy = 999,000 / 1,000,000 = 99.9% 🎉 (Looks great, right?)
❌ But the system is useless because it never finds spam!

The Problem with Accuracy

Accuracy only looks at how many total answers are correct—but it doesn’t
tell you if the system is actually finding what matters!
When what you’re looking for is rare (like spam emails or pie tweets), a system
can appear to have high accuracy while still missing all the important cases.
2)Confusion Matrix: Understanding System Errors
To evaluate our system properly, we use a confusion matrix. This is a table that
compares:

This table (confusion matrix) helps us check how well a classifier is working
when it's trying to separate two things, like:
 Spam vs. Not Spam emails
 Tweets about pie vs. Tweets not about pie
Module 3-Naive Bayes

The classifier makes predictions, but sometimes it's right and sometimes it's
wrong. This table helps us see where it's making mistakes.

 If you only care about not making mistakes, you want high precision.
 If you want to find as many pie tweets as possible, you want high recall.
 If the dataset is imbalanced (e.g., only a few tweets are about pies),
accuracy is not always a good measure
 What our system predicted (spam or not spam, pie tweet or not pie
tweet)
 What is actually true (gold labels, which are the correct answers made
by humans)
Module 3-Naive Bayes

4. Why These Metrics Matter

✅ If you only focus on Precision, your system might only pick tweets when it is
very sure they are about pie, but it will miss a lot of real ones (low Recall).
✅ If you only focus on Recall, your system might flag almost everything as pie-
related, but many will be wrong (low Precision).
✅ F1-score helps balance both, giving a fair measure of performance.
So, next time you want to evaluate a system that detects rare things—whether
spam emails or Delicious Pie tweets—remember that Precision, Recall, and
F1-score are much better than just looking at accuracy

Understanding Precision and Recall

Now that we know accuracy can be misleading, we need better ways to measure
how well our system is working. That’s where Precision and Recall come in! 🚀
Module 3-Naive Bayes

Why Do We Need Both?

Imagine Two Bad Systems:

❌ System A (Low Precision, High Recall):
 Finds almost everything, but many are wrong (too many False
Positives).
Module 3-Naive Bayes

 Example: Marks tweets about "pizza pie" as Delicious Pie tweets.

❌ System B (High Precision, Low Recall):
 Everything it finds is correct, but it misses a lot (too many False
Negatives).
 Example: Only finds tweets with exact words "Delicious Pie" and ignores
similar ones.
✅ A good system needs BOTH high precision & high recall!
 We want to find many correct results (high recall)
 While avoiding too many wrong ones (high precision).

· Accuracy can be misleading when classes are unbalanced.

· Precision = "Of the things we said were positive, how many were actually
correct?"
· Recall = "Of all the actual positive cases, how many did we find?"
· We need a balance of both to make a useful system!

Understanding F1-Score in Simple Terms

We know that Precision and Recall are both important, but sometimes they pull
in opposite directions:
 If we focus too much on Precision, we might miss many real positives.
 If we focus too much on Recall, we might make too many mistakes
(False Positives).
To balance both Precision and Recall, we use the F-measure (or F-score).
Module 3-Naive Bayes
Module 3-Naive Bayes

Why Do We Use the Harmonic Mean for F1-Score?

We use the harmonic mean instead of the arithmetic mean because it gives
more importance to the lower value of Precision and Recall.

· The harmonic mean pulls the F1-score towards the lower number.
· This makes sure that both Precision and Recall must be high to get a good
score.
Module 3-Naive Bayes

· It prevents overestimating system performance when one metric is much

lower.

Understanding the F-Measure and Harmonic Mean in Simple Words

The F-measure (or F-score) is a way to combine Precision and Recall into
one number. It does this using the harmonic mean instead of the regular
average (arithmetic mean).
Module 3-Naive Bayes
Module 3-Naive Bayes

4.7.1 Evaluating with more than two classes:

Understanding Multi-Class Classification Evaluation
When we evaluate a classification system, we often deal with more than two
categories (e.g., urgent, normal, and spam in the given example). This makes
precision and recall calculations a bit more complex.

Confusion Matrix

A confusion matrix helps us see how well the system is classifying different
categories.
 Each row represents the system's prediction (what the model said).
 Each column represents the actual class (the real category).
 The diagonal values (e.g., 8, 60, 200) are the correct classifications.
 The off-diagonal values (e.g., 10, 5, 3) are misclassifications (wrong
predictions).
For example:
 The system predicted "urgent" 8 times correctly, but mistakenly called 10
normal emails "urgent."
 The system correctly predicted "normal" 60 times, but mistakenly labeled
50 spam emails as "normal."
Module 3-Naive Bayes

 The system correctly predicted "spam" 200 times but misclassified 3

urgent emails as spam.
To evaluate performance, we calculate precision and recall for each category.

2. Precision and Recall for Each Class

Each class (urgent, normal, spam) gets its own precision and recall:
 Precision = How many times the system's prediction was correct for a
class?

 This means the system found 50% of all urgent emails correctly. Each
class has its own precision and recall values.
3. Combining Precision and Recall for All Classes
Since we want a single score to evaluate the system, we combine precision and
recall using:
Macroaveraging
 Calculate precision and recall for each class separately.
Module 3-Naive Bayes

 Take the average of all classes.

 Treats all classes equally, even if some are more frequent than others.
Microaveraging
 Combine all predictions into one large confusion matrix.
 Calculate precision and recall globally for all classes.
 Heavily influenced by larger classes (e.g., "spam" in this case).

 · Microaveraging is good when you want to measure overall

performance (biased towards bigger categories).
 · Macroaveraging is good when you want equal importance for all
categories (even rare ones like "urgent").
 · Use both metrics depending on what matters more: overall
correctness or fairness across all categories.

shows confusion matrices for a three-class classification task (Urgent, Normal, Spam).
They help us understand how well a model classifies emails into these categories. Let’s go
step by step.

(Figure 4.6): Class-Specific Confusion Matrices

This breaks down Figure 4.5 into three separate confusion matrices—one for each class
(Urgent, Normal, Spam). It also introduces microaveraged and macroaveraged precision.
Module 3-Naive Bayes

.
Breaking Down the Three-Class Confusion Matrices
Each small table focuses on one class:
1. Class 1: Urgent
o True Positives (TP): 8

o False Positives (FP): 11 (predicted urgent but actually something else)

o False Negatives (FN): 8 (actually urgent but misclassified)

Module 3-Naive Bayes

 Figure 4.5 shows a combined confusion matrix for all three classes.
 Figure 4.6 separates them into individual confusion matrices for better understanding.
 Macroaveraged precision (0.60) gives equal weight to all classes.
 Microaveraged precision (0.73) gives more weight to the frequent class (spam).
 Microaverage is dominated by spam (most frequent class), while macroaverage
treats all classes equally.
This helps in deciding:
Module 3-Naive Bayes

 If all classes are equally important → Use macroaverage.

 If overall accuracy matters more → Use microaverage.

4.8 Test sets and Cross-validation:

Cross-validation is a method used in machine learning to evaluate model performance
while making the best use of the available data. The goal is to ensure that our model
generalizes well to unseen data.

Figure 4.7: 10-Fold Cross-Validation

This diagram visually represents 10-fold cross-validation, a technique used in machine
learning to evaluate model performance effectively. Let’s break it down in simple terms.
1. What is 10-Fold Cross-Validation?
In 10-fold cross-validation, the dataset is divided into 10 equal parts (folds). The model is
trained and tested 10 times, each time using a different fold as the test set while the
remaining 9 folds are used for training.
Step-by-Step Process:
1. Split the dataset into 10 equal-sized folds.
2. In Iteration 1, use the first fold for testing and the remaining 9 folds for training.
3. In Iteration 2, use the second fold for testing and the other 9 folds for training.
4. Repeat this process 10 times, ensuring that every fold is used as a test set once.
5. Finally, compute the average performance over all 10 test sets to get a more reliable
accuracy.
Module 3-Naive Bayes

2. How to Read the Image?

 The rows represent 10 different training iterations (one per fold).
 The yellow boxes represent training data.
 The blue boxes labeled "Dev" represent the test set for that iteration.
 The test set moves from top to bottom, ensuring each fold is tested once.
Example:
 Iteration 1: The 1st fold is used as the test set, and the remaining 9 are used for
training.
 Iteration 2: The 2nd fold is used as the test set, and the remaining 9 are used for
training.
 This continues until Iteration 10, where the last fold is used for testing.
At the end, every data point has been:
✔ Used in training 9 times
✔ Used in testing exactly once

3. Why Use 10-Fold Cross-Validation?

✅ More Reliable Model Evaluation

 Instead of relying on just one training/testing split, we test multiple times, leading to
more accurate performance estimation.
✅ Better Use of Data

 Instead of setting aside a large chunk of data for testing, every data point is used for
both training and testing at different times.
✅ Reduces Overfitting

 Since the model is trained multiple times on different data, it generalizes better to
unseen data.
✅ Balances Bias and Variance

 Using 9 folds for training reduces bias (underfitting), and testing on 1 fold prevents
overfitting.
4. Practical Considerations
🔹 Avoiding Bias:

 We should not look at the data before training to avoid making biased decisions.
 Instead, we perform cross-validation only on training data and evaluate the final
model on a completely separate test set.
Module 3-Naive Bayes

🔹 Choosing k (number of folds):

 k = 10 is a common choice because it provides a good balance between accuracy and

efficiency.
 If the dataset is very small, we might use k = 5 (5-fold cross-validation).
 If the dataset is very large, we might use k = 20 to get a more stable estimate.
 10-fold cross-validation divides the data into 10 equal parts and runs
training/testing 10 times.
 Every data point is used for training 9 times and for testing once.
 It improves accuracy and reduces overfitting by using all the data effectively.
 It gives a more reliable estimate of model performance than a single train-test split.

4.9 Statistical Significance Testing:

When building machine learning models, we often want to compare the performance of two
systems. How do we know if one model is actually better than another, or if the
difference in performance is just due to chance?
This is where statistical significance testing comes in. It helps us determine whether the
performance difference between two models is real or just a random fluctuation.
1. Why Do We Need Statistical Significance Testing?
Imagine you have two sentiment classification models:
 Model A: A logistic regression classifier
 Model B: A Naïve Bayes classifier
You test both models on the same dataset and find:
 Model A’s accuracy: 80%
 Model B’s accuracy: 76%
 Difference (δ): 80% - 76% = 4% improvement
At first glance, it looks like Model A is better. But can we be sure that this 4%
improvement is meaningful? Or could it be due to random chance in this specific dataset?
To answer this, we need statistical significance testing.
Module 3-Naive Bayes

3. The Problem: Random Variation

 The dataset we test on (let’s call it x) is just one sample of all possible real-world
data.
 Model A might just be lucky on this specific dataset, meaning it performs better by
chance.
 If we test on a different dataset (x'), Model B might perform better instead.
Example of Randomness Affecting Performance
Imagine flipping a coin 10 times:
 You get 6 heads and 4 tails.
 Does this mean the coin is unfair? Not necessarily.
 If you flip it 1000 times, you'll get closer to 50-50, revealing the true fairness of the
coin.
Similarly, in machine learning, we need to ensure that the difference in model performance is
not just a coincidence.
4. How Do We Test If the Difference Is Real?
To prove that Model A is truly better than Model B, we use statistical hypothesis testing:
1. Null Hypothesis (H₀):
o Assumes there is no real difference between Model A and Model B.

o Any observed difference is just due to random chance.

2. Alternative Hypothesis (H₁):

Module 3-Naive Bayes

o Assumes Model A is actually better than Model B.

3. Perform a Statistical Test:

o We use statistical methods (like a t-test or bootstrap testing) to check if the
difference in scores is significant.
o If the test gives a very low probability (p-value < 0.05), we reject H₀ and
conclude that Model A is truly better.
Effect size (δ) tells us how much better one model is compared to another.
✅ Random chance can make one model look better than another on a specific dataset.
✅ Statistical significance testing helps us confirm if the difference is real or just luck.
✅ If the test shows a very low probability of randomness (p-value < 0.05), we conclude the
model is truly better.
Hypothesis Testing and p-Value
When comparing two models (A and B), we want to know if Model A is truly better or if its
better performance is just due to random chance. To answer this, we use hypothesis testing.

1. Hypothesis Testing: What Are We Trying to Prove?

We set up two competing hypotheses:
1. Null Hypothesis (H₀):
o Assumes that Model A is not better than Model B.

o This means the difference in their performance d(x) is 0 or negative.

2. Alternative Hypothesis (H₁):

o Assumes that Model A is truly better than Model B.

o This means the difference d(x) is positive.

👉 Our goal: Try to disprove H₀ and show that Model A is actually better.

2. What Is the p-Value?

The p-value helps us measure how surprising our results are if H₀ is actually true.
🔹 p-value definition:
It is the probability of getting a performance difference as big as (or bigger than) d(x),
assuming H₀ is true.

3. Example to Illustrate p-Value

Module 3-Naive Bayes

Imagine we are testing two students, Alice (A) and Bob (B), in math.
 Alice scores 90/100
 Bob scores 85/100
 Difference (d) = 5 points
Now, we ask: Is Alice actually better, or did she just get lucky with the questions?
 If we test many other students under similar conditions, and most of the time the
difference is still around 5 points or more, then Alice is likely truly better (small p-
value).
 But if the difference is random and varies a lot when repeating the test, the p-value
will be high, meaning Alice’s higher score might just be luck.

4. Why Do We Use the p-Value?

 It prevents us from making false conclusions based on random variations.
 It helps us quantify uncertainty instead of just guessing.
 It ensures that we only claim Model A is better if the evidence is strong enough.

✅ Null hypothesis (H₀): Assumes no real difference between models.

✅ Alternative hypothesis (H₁): Assumes Model A is truly better.
✅ p-value: Measures how likely we would get our observed result if H₀ were true.
✅ Low p-value (< 0.05): We reject H₀ → Model A is very likely better.
✅ High p-value (> 0.05): We cannot be sure Model A is better → More testing needed.

Statistical Significance and Non-Parametric Tests

When we compare two models, we want to be sure that one is truly better than the other, and
not just performing better by random chance. This is where statistical significance testing
helps.

1. What Does a Small p-Value Mean?

The p-value tells us how surprising our results are if the null hypothesis (H₀) were true.
 If p-value < 0.05 (or < 0.01 for stricter tests), we reject H₀ and say Model A is
statistically better than Model B.
 If p-value > 0.05, we cannot be sure Model A is better, as the difference might be
due to chance.
Module 3-Naive Bayes

Example:
 Imagine you flip a coin 10 times and get 9 heads.
 Normally, a fair coin should give about 5 heads.
 The probability (p-value) of getting 9 heads by chance is very low.
 So, you might reject H₀ and conclude the coin is biased.
Similarly, if the p-value is very low in an NLP experiment, we can reject H₀ and conclude
that Model A is significantly better than Model B.

2. Why Don’t We Use t-Tests or ANOVA in NLP?

In statistics, there are parametric tests like t-tests and ANOVA, which assume the data
follows a normal distribution (bell curve). However, in NLP, our data often does not follow
a normal distribution, so we use non-parametric tests instead.

3. What Are Non-Parametric Tests?

Non-parametric tests do not assume any specific shape of the data distribution. They work
by resampling and randomizing the data multiple times to estimate the p-value.
Two Common Non-Parametric Tests in NLP:
1) Approximate Randomization Test
 Instead of assuming a normal distribution, we shuffle and reassign the test results
between Model A and Model B many times.
 This helps us estimate how often we would observe the same difference just by
chance.
 If our original difference is much larger than the shuffled results, we conclude that
Model A is truly better.
2) Bootstrap Test (Paired Testing)
 We create many artificial test sets by randomly selecting test examples with
replacement (some examples may appear multiple times).
 We then compare Model A and Model B on each of these new test sets.
 If Model A consistently performs better, we conclude that it is significantly better.
Why is it called a paired test?
 Each test example exists in both test sets (Model A and Model B), so we can directly
compare their performance on the same data.
Module 3-Naive Bayes

4. Key Takeaways
✅ Small p-value (< 0.05 or < 0.01): Model A is statistically better than Model B.
✅ Non-parametric tests are better for NLP because they don’t assume a normal distribution.
✅ Approximate randomization and bootstrap testing work by shuffling and resampling
test data many times.
✅ Paired testing ensures we compare models on the same data, making results more
reliable.