MODULE 3 VI SEM NLP [BAI601]

MODULE 3
SYLLABUS:
Naive Bayes, Text Classification and Sentiment: Naive Bayes Classifiers, Training the
Naive Bayes Classifier, Worked Example, Optimizing for Sentiment Analysis, Naive
Bayes for Other Text Classification Tasks, Naive Bayes as a Language Model. .
_________________________________________________________________

____________________LECTURE 01______________________

Naive Bayes, Text Classification and Sentiment Analysis

INTRODUCTION:
• Classification, heart of both human and machine intelligence assigning a category to
an input.
• Deciding what letter, word, or image has been presented to our senses, recognizing
faces or voices, sorting mail, assigning grades to homework.
• Naïve Bayes algorithm for text categorization: the task of assigning a label or category
to an entire text or document.
• common text categorization tasks:

1. sentiment analysis, the extraction of sentiment, the positive or negative orientation that

• a writer expresses toward some object.

• A review of a movie, book, or product on the web.

Sentiment Analysis is a Natural Language Processing (NLP) technique used to determine

the emotional tone or opinion expressed in a piece of text. It analyzes whether the
sentiment of the text is positive, negative, or neutral.

It is also called Opinion Mining, because it extracts opinions, attitudes, and emotions
from text data.
Sentence Sentiment
This mobile phone is excellent Positive
The service is very poor Negative
Neutral
The product is okay

+ ... any characters and richly applied satire, and some great plot twists

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 1

MODULE 3 VI SEM NLP [BAI601]
- It was pathetic. The worst part about it was the boxing scenes ...
+ ... awesome caramel sauce and sweet toasty almonds. I love this place!
- .. awful pizza and ridiculously overpriced
Words like great, richly, awesome, and pathetic, and awful and ridiculously are very
informative cues: unification is based on the functional specifications of the verb, which
predicts the overall sentence structure.
Sentiment analysis helps organizations understand public opinion, customer feedback,
and user emotions from large amounts of textual data.
2. Spam detection:
Spam detection is the process of identifying unwanted or unsolicited messages such as
spam emails or spam SMS. Machine learning techniques like the Naive Bayes algorithm
are widely used to classify messages as spam or ham (not spam).
Naive Bayes works well for spam detection because it efficiently handles large text
datasets and calculates the probability that a message belongs to a particular class.
Message Class
Win money now Spam
Free lottery ticket Spam
Meeting tomorrow Ham
Project discussion today Ham

Binary classification task of assigning an email to one of the two classes spam or not-
spam.
3. Assigning a library subject category or topic label to a text
Various sets of subject categories exist.
Deciding whether a research paper concerns epidemiology, embryology, etc..is an
important component of information retrieval.
Text Classification is the process of automatically assigning predefined categories or
labels to text documents based on their content.
In libraries or digital archives, documents such as books, articles, or reports are
categorized into subjects like:
Science, Technology, History, Literature, Mathematics.
Text Document Assigned Category
“Neural networks and deep learning methods” Artificial Intelligence
“Shakespeare's literary works” Literature
“World War II historical events” History

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 2

MODULE 3 VI SEM NLP [BAI601]

Naive Bayes Classifiers:

The intuition of the classifier is shown in Fig. 1
We represent a text document as if it were a bag of words, that is, an unordered set of
words with their position ignored, keeping only their frequency in the document.

Instead of representing the word order in all the phrases like “I love this movie” and “I
would recommend it”, we simply note that the word I occurred 5 times in the entire
excerpt, the word it 6 times, the words love, recommend, and movie once, and so on.
• Naive Bayes is a probabilistic classifier.
• For a document d, out of all classes c € C the classifier returns the class 𝐶̂ which has
the maximum posterior probability given the document.

•
• Use Bayes’ rule to break down any conditional probability P(x|y) into three other
probabilities:

We can then substitute Eq.2 into Eq.1 to get Eq.3

Since P(d) doesn't change for each class, we can conveniently simplify Eq. 3 by dropping
the denominator.

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 3

MODULE 3 VI SEM NLP [BAI601]

We call Naive Bayes a generative model, Eq. 4 can be read as class is sampled from P(c),
and then the words are generated by sampling from P(d|c) and a document is generated.
Eq. 4 states, we compute the most probable class 𝑪̂ given some document d by choosing
the class which has the highest product of two probabilities: the prior probability of the
class P(c) and the likelihood of the document P(d|c):

we can represent a document d as a set of features f1, f2, ..... ,fn:

Eq. 6 is still too hard to compute directly: without some simplifying assumptions,
estimating the probability of every possible combination of features (for example, every
possible set of words and positions) would require huge numbers of parameters and
impossibly large training sets.
Naive Bayes classifiers therefore make two simplifying assumptions.
The first is the bag-of-words assumption, that the features f1, f2, ... ,fn only encode
word identity and not position.
The second is commonly called the naive Bayes assumption, the conditional
independence assumption that the probabilities P(fi|c) are independent given the class c.
Therefore, P(f1, f2, .... ,fn|c) = P(f1|c).P(f2|c) ..... P(fn|c) (7)
The final equation for the class chosen by a naive Bayes classifier is:

To apply the naive Bayes classifier to text, we will use each word in the documents as a
feature, as suggested above, and we consider each of the words in the document by
walking an index through every word position in the document:

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 4

MODULE 3 VI SEM NLP [BAI601]

Naive Bayes calculations, like calculations for language modelling, are done in log space,
to avoid underflow and increase speed. Thus Eq. 9 is generally instead expressed as,

Eq. 10 computes the predicted class as a linear function of input features. Classifiers
that use a linear combination of the inputs to make a classification decision -like naive
Bayes and also logistic regression are called linear classifiers. How can we learn the
probabilities P(c) and P(fi|c)?
To learn class priori P(c): What percentage of the documents in our training set are in
each class c. Let Nc be the number of documents in our training data with class c. Ndoc
be the total number of documents. Then,

____________________LECTURE 02______________________

To learn the probability P(fi|c):

We will assume a feature is just the existence of a word in the document's bag of words,
and so we'll want P(wi|c), we compute as the fraction of times the word wi appears
among all words in all documents of topic c.
Concatenate all documents with category c into one big "category c" text.
Then we use the frequency of wi in this concatenated document to give a maximum
likelihood estimate of the probability:

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 5

MODULE 3 VI SEM NLP [BAI601]
Here the vocabulary V total number of unique words in your vocabulary in all classes,
not just the words in one class c.

Issues with training:

1. Zero Probability problem with maximum likelihood training:
major issue occurs when a word in the test data never appears in the training data for
a particular class. This leads to the Zero Probability Problem. If a word does not appear
in the training data for a class, its probability becomes zero.
P(w∣c)=0
Since Naive Bayes multiplies probabilities of all words in the document, one zero
probability makes the entire probability zero.
P(c∣d)=P(w1∣c)×P(w2∣c)×P(w3∣c)
If any term is zero:
P(c∣d)=0
Thus, the classifier may incorrectly reject a class.
Sentence Class
I love this movie Positive
This movie is amazing Positive
I hate this movie Negative

Vocabulary:
I, love, this, movie, is, amazing, hate
Test Sentence
"This movie is fantastic“
Word "fantastic" never appeared in the training data.
P(fantastic∣Positive)=0
P(fantastic∣Negative)=0
When computing probability:
P(Positive∣Sentence)=P(this∣Positive)×P(movie∣Positive)×P(fantastic∣Positive)
Since:
P(fantastic∣Positive)=0
Final probability becomes:
P(Positive∣Sentence)=0

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 6

MODULE 3 VI SEM NLP [BAI601]
This creates the Zero Probability Problem.
To solve the zero probability problem, Laplace smoothing (Add-One Smoothing) is
used
Formula:

Where, +1 ensures no probability becomes zero,

Vocabulary size(V) = number of unique words
Example with Laplace Smoothing
If fantastic never appeared in the training data:
Without smoothing:
P(fantastic∣Positive)=0
With smoothing:

Now the probability becomes small but not zero, allowing the classifier to continue
calculation.
2. Words that occur in our test data but are not in our vocabulary:
During testing, a document may contain new words that were not present in the training
vocabulary. These are called unknown words or out-of-vocabulary (OOV) words.
Handling such words properly is important to avoid errors in probability calculations. The
vocabulary is the set of all distinct words found in the training data.
Sentence Class
I love this movie Positive
This movie is amazing Positive
I hate this movie Negative

Vocabulary created from training data:

V = {love, movie, amazing, hate, this, is, I}
This vocabulary is used to compute probabilities.
Example test sentence:"This movie is fantastic“

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 7

MODULE 3 VI SEM NLP [BAI601]
The word "fantastic" is not present in the training vocabulary.

Therefore: P(fantastic∣class)

cannot be computed using the normal probability formula. This may cause zero
probability problems, leading to incorrect classification.

Approaches to Handle Unknown Words:

1. Ignore Unknown Words: The classifier simply ignores words not present in the
vocabulary during probability calculations.

Exmple:Test sentence:"This movie is fantastic“

Word fantastic is ignored, and classification is based on the remaining known words.

Laplace Smoothing:

Laplace smoothing assigns a small probability to unseen words.

Formula:

Even if the word count is zero, the probability becomes non-zero, allowing classification

3. Unknown Token (<UNK>)

During training, rare words are replaced with a special token <UNK> representing
unknown words.

Example vocabulary:

love, movie, amazing, hate, <UNK>

When a new word appears in testing, it is treated as <UNK>

NAÏVE BAYES ALGORITHM USING ADD-1 SMOOTHING

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 8

MODULE 3 VI SEM NLP [BAI601]

_______________LECTURE 03______________________

Solution:

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 9

MODULE 3 VI SEM NLP [BAI601]
We determine whether Play = Yes or No when Outlook = Sunny.

Step 1: Total Observations

Total records = 14

Class Count
Play = Yes 10
Play = No 4

Step 2: Prior Probabilities

P(Yes)=10/14

P(No)=4/14

Step 3: Conditional Probabilities

Count occurrences of Sunny.

Sunny appears 5 times

Outlook Play
Sunny Yes
Sunny No
Sunny Yes
Sunny No
Sunny Yes

Therefore:

Sunny & Yes = 3

Sunny & No = 2

P(Sunny∣Yes)=3/10

P(Sunny∣No)=2/4

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 10

MODULE 3 VI SEM NLP [BAI601]
Step 4: Compute Posterior Numerators

For Play = Yes

P(Yes∣Sunny)=P(Sunny∣Yes)×P(Yes)

=(3/10)*(10/14)

=30/140=➔ 0.2143

For Play = No

P(No∣Sunny)=P(Sunny∣No)×P(No)=(2/4)*(4/14)=0.1429

Step 5: Normalize the Probabilities

First compute denominator:

0.2143+0.1429=0.3572

Normalized Probability for Yes

P(Yes∣Sunny)=0.2143/0.3572=0.60

Normalized Probability for No

P(No∣Sunny)=0.1429/0.3572=0.4

Step 6: Final Result

Class Probability

Play = Yes 0.60

0.40
Play = No

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 11

MODULE 3 VI SEM NLP [BAI601]

Step 1: Total words in each document

• Doc 1: 12 + 12 + 1 + 1 + 13 = 39
• Doc 2: 17 + 17 + 0 + 0 + 17 = 51
• Doc 3: 14 + 14 + 2 + 2 + 15 = 47

Step 2: Probability of “good” in each document

Step 3: Classify “I am good” using Naïve Bayes

P(D∣sentence)∝P(I∣D)×P(am∣D)×P(good∣D)
For Document 1: (12/39)*(12/39)*(13/39)=0.03159

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 12

MODULE 3 VI SEM NLP [BAI601]

Step 4: Final Decision

Compare probabilities:
• Doc 1 → 0.0316
• Doc 2 → 0.0370 (Highest)
• Doc 3 → 0.0283
Final Answer
• Probability of “good”:
• Doc 1 = 0.333
• Doc 2 = 0.333
• Doc 3 = 0.319
• The sentence “I am good” belongs to → Document 2
EX: 2
Let’s use a sentiment analysis domain with the two classes positive (+) and negative (-
), and take the following miniature training and test documents simplified from actual
movie reviews.

Step 1: P(-)=3/5 and P(+)=2/5

Step 2: The word “with” doesn't occur in the training set, so we drop it completely.
Step 3: The likelihoods from the training set for the remaining three words
"predictable", "no", and “fun” are as follows:

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 13

MODULE 3 VI SEM NLP [BAI601]
Step 4: For the test sentence S = "predictable with no fun", after removing the word
'with', the
P(-)P(S|-)=(3/5)*((2/34)*(2/34)*(1/34))=6.1X10^-5
P(+)P(S|+)=(2/5)*((1/29)*(1/29)*(2/29))=3.2X10^-5
Optimizing for Sentiment Analysis:
Optimizing for sentiment analysis means improving the accuracy, efficiency, and
reliability of the sentiment classification model by enhancing different stages of the
system such as data preprocessing, feature selection, model training, and evaluation.
1. Data Preprocessing:
Proper preprocessing improves the quality of input data.
Tokenization: Splitting text into individual words.
Sentence: "This movie is really good“
Tokens: this, movie, really, good
2. Stop Word Removal:
Remove common words that do not contribute much meaning
Example: this, is, the, a, an
Sentence after removal: movie, really, good
3. Stemming / Lemmatization:
Reduce words to their root form
Word Root Form
loved love
loving love
loves love

Lowercasing:
Convert all words to lowercase.
Example:
GOOD → good
2.2 Feature Extraction:
Text must be converted into numerical features before applying machine learning
algorithms.
1. Bag of Words (BoW): Represents documents using word frequency.
Example:
Sentence:
"good movie good acting"

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 14

MODULE 3 VI SEM NLP [BAI601]
Word Frequency
good 2
movie 1
acting 1

2. TF–IDF
TF-IDF assigns higher weight to important words.
TF−IDF=TF×IDF
Where
TF = Term Frequency
IDF = Inverse Document Frequency
Words that appear frequently in one document but rarely in others get higher
importance.
3. N-grams:
N-grams capture word sequences.
Example: Sentence: "very good movie"
Unigrams
very, good, movie
Bigrams
very good, good movie
This improves sentiment detection.
Optimizing for Sentiment Analysis (Naïve Bayes):
Optimizing sentiment analysis means improving how accurately a model (like Naïve
Bayes) classifies text into positive, negative, or neutral categories.
1. Basic Idea (Before Optimization):
A standard Naïve Bayes classifier uses:
Treats all words equally, Sensitive to noise (e.g., “the”, “is”), Cannot handle negation
well.
2. Key Optimization Techniques:
Text Preprocessing
Improve input quality before training.
Remove stopwords → “the”, “is”, “and”
Apply stemming/lemmatization
“loved”, “loving” → “love”
Convert to lowercase
Example:

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 15

MODULE 3 VI SEM NLP [BAI601]
"I REALLY loved this movie!!!"
→ "love movie“
(ii) Handling Negation
Negation flips sentiment.
Without optimization:
"not good" → classified as positive
With optimization:
Convert to: "not_good“
Now the model learns:
"good" → positive
"not_good" → negative
iv) N-grams (Context Awareness)
Instead of single words (unigrams), use:
Bigrams: "very good", "not bad"
Trigrams: "not at all good"
Example:
"bad" → negative
"not bad" → positive (captured using bigrams)
(v) Laplace Smoothing
Avoid zero probability problem.
If a word never appears in training:
𝑃(𝑤𝑜𝑟𝑑 ∣ 𝑐𝑙𝑎𝑠𝑠) = 0
Use smoothing:
𝑐𝑜𝑢𝑛𝑡 + 1
𝑃=
𝑡𝑜𝑡𝑎𝑙 + 𝑉
This ensures:
No probability becomes zero
Model becomes robust
Example: "not good" → negative sentiment

____________________LECTURE 04______________________

Naive Bayes for other text classification tasks:

a) Spam Detection and Naïve Bayes
Spam detection—deciding whether an email is unsolicited bulk mail—was one of the
earliest applications of naïve Bayes In text classification.

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 16

MODULE 3 VI SEM NLP [BAI601]
Rather than treating all words as individual features, effective systems often use
predefined sets of words or patterns, along with non-linguistic features.
Specific phrases like "one hundred percent guaranteed“
Regex patterns like mentions of ‘millions of dollars’.
Structural properties like HTML with a low text-to-image ratio
Non-linguistic metadata, such as the email’s delivery path
Subject lines written entirely in capital letters
Urgent phrases like "urgent reply“
Keywords such as "online pharmaceutical"

HTML anomalies like unbalanced head tags

Claims such as "you can be removed from the list"
Language Identification :
In contrast, tasks like language identification rely less on words and more on subword
units like character n-grams or even byte n-grams.
These can capture statistical patterns at the start or end of words, especially when spaces
are included as characters.
A well-known system, [Link] (Lui & Baldwin, 2012), starts with all possible n-
grams of lengths 1–4 and uses feature selection to narrow down to the 7,000 most
informative.
Training data for language ID systems often comes from multilingual sources such as
Wikipedia (in 68+ languages), newswire, and social media.
To capture regional and dialectal diversity, additional corpora include:
• Geo-tagged tweets from Anglophone regions like Nigeria or India.
• Translations of the Bible and Quran
• Slang from Urban Dictionary
• Corpora of African American Vernacular English (Blodgett et al., 2016)
These diverse sources help models capture the full range of language use across different
communities and contexts (Jurgens et al., 2017).
Naive Bayes as a Language Model:
• Naive Bayes classifiers can use any sort of feature: dictionaries, URLs, email
addresses, network features, phrases, and so on.
• A naive Bayes model can be viewed as a set of class-specific unigram language
models, in which the model for each class instantiates a unigram language model.
• Assign a probability to each word P(word|c), the model also assigns a probability
to each

Example: Consider a naive Bayes model with the classes positive (+) and negative (-)
and the following model parameters

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 17

MODULE 3 VI SEM NLP [BAI601]

Each of the two columns above instantiates a language model that can assign a
probability to the sentence “I love this fun film”:
P(“I love this fun film”+) = 0.1 * 0.1 * 0.01 * 0.05 * 0.1=5*10^-7
P(“I love this fun film” - ) = 0.2 * 0.001 * 0.01* 0.005 * 0.1=1.0 * 10^-9
The positive model assigns a higher probability to the sentence: P(s|pos) > P(s|neg).

____________________LECTURE 05______________________

Evaluation: Precision, Recall, F-measure:

Text classification evaluation often starts with binary detection tasks.
Example 1: Spam Detection
Goal: Label each text as spam (positive) or not spam (negative).
Need to compare:
• System’s prediction
• Gold label (human-defined correct label)
Example 2: Social Media Monitoring for a Brand
Scenario: CEO of Delicious Pie Company wants to track mentions on social media

• Build a system to detect tweets about Delicious Pie.

• Positive class: Tweets about the company.
• Negative class: All other tweets.
Why we need metrics:
• To evaluate how well a system (e.g., spam detector or pie-tweet detector)
performs.
• Confusion Matrix:

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 18

MODULE 3 VI SEM NLP [BAI601]

• A table that compares system predictions vs. gold (human) labels.

• Each cell represents a type of outcome:
• True Positive (TP): Correctly predicted positives (e.g., actual spam labelled as
spam).
• False Negative (FN): Actual positive incorrectly labelled as negative (e.g., spam
labelled as non-spam).
• Accuracy: Formula: (Correct predictions) / (Total predictions).
• Appears useful but misleading for unbalanced classes.
• Why accuracy can fail: Real-world data is often skewed (e.g., most tweets are not
about pie).
• Example: 1,000,000 tweets → only 100 about pie.
• A naive classifier labels all tweets as "not about pie“
• Result: 99.99% accuracy, but 0 useful results.

Conclusion: Accuracy is not a reliable metric when the positive class is rare.
That’s why, instead of relying on accuracy, we often use two more informative
metrics:
Precision measures the percentage of items labeled as positive by the system that are
actually positive (according to human-annotated “gold” labels).
Precision = true positives/ true positives + false positives
Recall measures the percentage of actual positive items that were correctly identified
by the system.
Recall = true positives/ true positives + false negatives
These metrics address the issue with the “nothing is pie” classifier.
Despite its seemingly excellent 99.99% accuracy, it has a recall of 0 —because it
misses all 100 true positive cases, identifying none.
Its precision is also meaningless, since it detects nothing (since there are no true
positives, and 100 false negatives, the recall is 0/100).
Unlike accuracy, precision and recall focus on true positives, helping us measure how
well the system finds the things it’s actually supposed to detect.
To combine both precision and recall into a single metric, we use the F-measure with
the most common version being the F1 score:
Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 19
MODULE 3 VI SEM NLP [BAI601]

The ß parameter differentially weights the importance of recall and precision, based
perhaps on the needs of an application.
Values of ß > 1 favor recall, while values of ß < 1 favor precision.
When ß = 1, precision and recall are equally balanced; this is the most frequently used
metric, and is called Fβ=1 or just F1:

Evaluating with more than two classes:

For sentiment analysis we generally have 3 classes (positive, negative, neutral) and
even more classes are common for tasks like part-of-speech tagging, word sense
disambiguation, semantic role labeling, emotion detection, and so on.
The naive Bayes algorithm is already a multi-class classification algorithm.
Consider the sample confusion matrix for a hypothetical 3-way one-of email
categorization decision (urgent, normal, spam) shown in Fig.
The matrix shows, for example, that the system mistakenly labeled one spam
document as urgent, and we have shown how to compute a distinct precision and
recall value for each class.

Confusion matrix for a three-class categorization task, showing for each pair of classes
(c1,c2), how many documents from c1 were (in)correctly assigned to c2.

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 20

MODULE 3 VI SEM NLP [BAI601]
In order to derive a single metric that tells us how well the system is doing, we can
combine these values in two ways.
1. In macroaveraging, we compute the performance for each class, and then average
over classes.
2. In microaveraging, we collect the decisions for all classes into a single confusion
matrix, and then compute precision and recall from that table.
Fig. shows the confusion matrix for each class separately, and shows the computation
of microaveraged and macroaveraged precision

As the figure shows, a microaverage is dominated by the more frequent class (in this
case spam), since the counts are pooled.
The macro average better reflects the statistics of the smaller classes, and so is more
appropriate when performance on all the classes is equally important.
Test sets and Cross-validation
Training & Testing for Text Classification:
1. Standard Procedure:
Train the model on the training set.
Use the development set (devset) to tune parameters and choose the best
model.
Evaluate the final model on a separate test set.
2. Issue with Fixed Splits:
Fixed training/dev/test sets may lead to small dev/test sets.
Smaller test sets might not be representative of overall performance.
3. Solution – Cross-Validation (as shown in Fig):
Cross-validation allows use of all data for training and testing.
Process:
Split data into k folds.
For each fold:
Train on k-1 folds, test on the remaining fold.
Repeat k times, average the test errors.
Example: 10-fold cross-validation (train on 90%, test on 10%, repeated 10 times).
4. Limitation of Cross-Validation:

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 21

MODULE 3 VI SEM NLP [BAI601]
All data is used for testing → can't analyze the data in advance (avoiding
"peeking").
Looking at data is important for feature design in NLP systems.
5. Common Compromise:
Split off a fixed test set.
Do 10-fold cross-validation on the training set.

Use test set only for final evaluation

____________________LECTURE 06______________________

Statistical Significance Testing:

When building NLP systems, we often need to compare performance between two
systems (e.g., a new model vs. an existing one).
Simply observing different scores (e.g., accuracy, F1) isn't enough — we need to
know if the difference is statistically significant.
This is where statistical hypothesis testing comes in.
Inspired by Dror et al. (2020) and Berg-Kirkpatrick et al. (2012), these tests help
determine if the observed improvement is real or due to chance.
Example:
Classifier A (e.g., logistic regression) vs. Classifier B (e.g., naive Bayes).
Metric M (e.g., F1-score), tested on dataset x.
Let M(A, x) be the score for A, and δ(x) be the difference in performance between
A and B.

Understanding Effect Size and Significance:

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 22

MODULE 3 VI SEM NLP [BAI601]
We want to know if δ(x) > 0, meaning A (logistic regression) performs better than B
(naive Bayes).
δ(x) is the effect size — larger δ means a bigger performance gap.
But a positive δ alone isn’t enough.
Example: A has 0.04 higher F1 than B — is that meaningful?
Problem: The difference might be due to chance on this specific test set.
What we really want to know:
Would A still outperform B on another test set or under different conditions?
That’s why we need statistical testing, not just raw differences.
Statistical Hypothesis Testing Paradigm
We compare models by setting up two formal hypotheses:

Null hypothesis (H₀): There's no real difference between A and B — any observed
difference is due to chance.
Statistical tests help us decide whether to reject H₀ in favor of H₁ based on the data.
Null Hypothesis and p-value
Null hypothesis (H₀): Assumes δ(x) ≤ 0 — A is not better than B.
We want to see if we can reject H₀ and support H₁ (that A is better).
We imagine δ(x) over many possible test sets.
The p-value measures how likely we are to observe our δ(x), or a larger one, if H₀ were
true.
A low p-value suggests our result is unlikely due to chance, supporting H₁.
Interpreting p-values and Statistical Testing in NLP
The p-value is the probability of observing a performance difference δ(x) (or larger),
assuming A is not better than B (null hypothesis H₀).
If δ(x) is large (e.g., A’s F1 = 0.9 vs. B’s = 0.2), it's unlikely under H₀ → low p-value →
we reject H₀.

The P-value (probability value) is a measure that helps you decide whether to reject the
null hypothesis.
The probability of getting results as extreme as (or more extreme than) the observed
results, assuming the null hypothesis (H₀) is true.
Small P-value → Evidence against null hypothesis
Large P-value → Evidence supports null hypothesis
Compare P-value with significance level α or threshold value
If P ≤ α → Reject H₀
If P > α → Fail to reject H₀
Common α values:
0.05 (most common)

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 23

MODULE 3 VI SEM NLP [BAI601]
0.01 (strict test)
Example:
Suppose:
H₀: A medicine has no effect
You perform an experiment and get:
P-value = 0.03
since, 0.03<0.05
Reject H₀
Conclude: Medicine likely has an effect

Parametric tests are statistical tests that assume your data follows a specific distribution
(usually normal distribution) and depend on parameters like:
Mean (μ), Variance (σ²)
1. t-test
Used to compare means between two groups.
a) One-sample t-test: Compare sample mean with a known value

____________________LECTURE 07______________________

b) Independent t-test: Compare means of two different groups

EX: Marks of Class A vs Class B
2. ANOVA (Analysis of Variance)
Used when comparing more than two groups
Instead of multiple t-tests, ANOVA checks all groups together.

If δ(x) is small, it’s more plausible (believable/possible) under H₀ → higher p-value →

we may fail to reject H₀.
What Counts as “Small”?
Common p-value
thresholds: 0.05 or 0.01

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 24

 MODULE 3 VI SEM NLP [BAI601]
If p < threshold, the result is considered statistically significant (we reject H₀ and conclude
A is likely better than B).
How Do We Compute the p-value in NLP?
NLP avoids parametric tests (like t-tests or ANOVAs) because they assume certain
distributions that often don't apply.
Instead, we use non-parametric tests that rely on sampling methods.
Key Idea:
Simulate many variations of the experiment (e.g., using different test sets x′).
Compute δ(x′) for each → this gives a distribution of δ values.
If the observed δ(x) is in the top 1% (i.e., p-value < 0.01), it's unlikely under H₀ → reject
H₀
Common Non-Parametric Tests in NLP:
1. Approximate Randomization (Noreen, 1989)
2. Bootstrap Test (paired version is most common)
Compares aligned outputs from two systems (e.g., A vs. B on the same inputs xi).
Measures how consistently one system outperforms the other across samples.
The Paired Bootstrap Test :
The bootstrap test is a flexible, non-parametric method that can be applied to any
evaluation metric—like precision, recall, F1, or BLEU(Bilingual Evaluation
Understudy)
BLEU is used to evaluate machine translation and text generation tasks by comparing a
generated text against one or more reference(human) text.
What is bootstrapping?
It involves repeatedly sampling with replacement from an original dataset to create
many "bootstrap samples" or virtual test sets.
The key assumption is that the original sample is representative of the larger population.
Example:
Imagine a small classification task with 10 test documents. Two classifiers, A and B, are
evaluated.
Each document outcome falls into one of four categories:
Both A and B correct
Both incorrect
A correct, B wrong
A wrong, B correct
If A has 70% accuracy and B has 50%, then the performance difference δ(x) = 0.20.
How bootstrap works:
1. Generate a large number (e.g., 100,000) of new test sets by sampling 10 documents with
replacement from the original set.
2. For each virtual test set, recalculate the accuracy difference between A and B.
3. Use the distribution of these differences to estimate a p-value, telling us how likely the
observed δ(x) is under the null hypothesis (that A is not better than B).
This helps determine whether the observed performance difference is statistically
significant or just due to random chance.

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 25

 MODULE 3 VI SEM NLP [BAI601]

Figure: The paired bootstrap test: Examples of b pseudo test sets x(i) being created from
an initial true test set x. Each pseudo test set is created by sampling n = 10 times with
replacement; thus an individual sample is a single cell, a document with its gold label and
the correct or incorrect performance of classifiers A and B.
With the b bootstrap test sets, we now have a sampling distribution to analyze whether A’s
advantage is due to chance. Following Berg-Kirkpatrick et al. (2012), we assume the null
hypothesis (H₀)—that A is not better than B—so the average δ(x) should be zero or
negative.
If our observed δ(x) is much higher, it would be surprising under H₀. To measure
this, we calculate the p-value by checking how often the sampled δ(xᵢ) values exceed the
observed δ(x).

What this formula represents

This is an empirical (simulation-based) way to compute a p-value, commonly used in:
Permutation tests, Bootstrap methods.
b-→ Number of random samples (simulations/permutations)
X-→ observed data
x(i)-→ The i-th simulated dataset (under null hypothesis)
δ(x)-→ Test statistic computed on observed data
(e.g., mean difference, t-statistic)
𝛿(𝑥 (𝑖) )--→ Test statistic computed on simulated data
Indicator Function 1:

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 26

 MODULE 3 VI SEM NLP [BAI601]

We use the notation 1(x) to mean “1 if x is true, and 0 otherwise.” Although the expected
value of δ(X) over many test sets is 0, this isn't true for bootstrapped test sets due to the bias
in the original test set, so we compute the p-value by counting how often δ (x(i)) exceeds the
expected δ(x) by δ(x) or more.

If we have 10,000 test sets and a threshold of 0.01, and in 47 test sets we find δ(x(i)) ≥
2δ(x), the p-value of 0.0047 is smaller than 0.01. This suggests the result is surprising,
allowing us to reject the null hypothesis and conclude A is better than B.

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 27

MODULE 3 VI SEM NLP [BAI601]
____________________LECTURE 08 _____________________

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 28

MODULE 3 VI SEM NLP [BAI601]
The full algorithm for the bootstrap is shown in Fig. It is given a test set x, a number of
samples b, and counts the percentage of the b bootstrap test sets in which δ (x *(i)) > 2δ
(x). This percentage then acts as a one-sided empirical p-value.
Avoiding Harms in Classification (Summary) :
Classifiers can cause harm, including representational harms (e.g., reinforcing
stereotypes).
Example: Sentiment analysis systems rated sentences with African American
names more negatively than identical ones with European American names.
Toxicity classifiers may falsely label non-toxic content as toxic, especially when
it references marginalized groups or dialects (e.g., AAVE), leading to silencing.
Harms can arise from:
a) Biased training data b) Biased labels or resources (e.g., lexicons, embeddings)
c) Model design choices
No universal fix exists, so transparency is key.
A proposed solution: release model cards (Mitchell et al., 2019), which include:
Training algorithms and parameters
Training data sources, motivation, and preprocessing
Evaluation data sources, motivation, and preprocessing
Intended use and users
Model performance across different demographic or other groups and
environmental situation.

Prepared by: Vinod Kumar B, Dept. of AIML, CEC Page 29