CLN 708: Machine Learning for Computational
Linguistics (Lecture 2)
Prof. Olusanya E. Olubusoye
2025-02-17
Bayesian Inference and Naïve Bayes Classification
Learning Objectives
By the end of this lecture, you should be able to:
▶ Understand probability fundamentals for Bayesian inference.
▶ Explain Bayes’ Theorem and how it applies to text
classification.
▶ Implement and evaluate a Naïve Bayes classifier for text
classification.
PART 1: PROBABILITY FUNDAMENTALS FOR
BAYESIAN INFERENCE
Introduction to Probability Theory
Probability Basics:
▶ Probability (P): Measures the likelihood of an event
occurring.
▶ Formula:
Number of favorable outcomes
P(A) =
Total number of outcomes
Example:
▶ Rolling a die: P(3) = 1
6
In a standard deck of 52 playing cards, the distribution of colors
is as follows:
▶ Red Cards (26 total):
▶ Hearts – 13 cards
▶ Diamonds – 13 cards
▶ Black Cards (26 total):
▶ Spades – 13 cards
▶ Clubs – 13 cards
Thus, the deck is evenly split:
▶ 50% Red (Hearts & Diamonds)
▶ 50% Black (Spades & Clubs)
▶ Each suit has the same 13 ranks: Ace, 2-10, Jack, Queen,
and King.
▶ Drawing a red card from a deck: P(Red) = 26
52 = 0.5
Conditional Probability
Definition:
▶ The probability of event A occurring given that event B has
already occurred.
▶ Formula:
P(A ∩ B)
P(A|B) =
P(B)
Example:
▶ P(Spam | “Congratulations!”) = Probability of an email
being spam given that it contains the word “Congratulations!”.
PART 2: BAYES’ THEOREM & TEXT
CLASSIFICATION
Bayes’ Theorem
Formula:
P(B|A) · P(A)
P(A|B) =
P(B)
Meaning:
▶ P(A|B): Posterior probability (probability of A given B)
▶ P(B|A): Likelihood (probability of B given A)
▶ P(A): Prior probability (probability of A before seeing B)
▶ P(B): Evidence (probability of observing B)
Example in Text Classification:
▶ P(Spam|"Congratulations")
▶ Goal: Compute how likely an email is spam given certain
words.
Applying Bayes’ Theorem to Text Classification
Spam Detection Example:
Let’s classify an email as Spam (S) or Not Spam (¬S) based on
words.
Example:
P("Win a prize"|Spam) · P(Spam)
P(Spam|"Win a prize") =
P("Win a prize")
Where:
- P("Win a prize"|Spam) = Probability of the phrase occurring in
spam emails.
- P(Spam) = Prior probability of emails being spam.
- P("Win a prize") = Probability of seeing the phrase in any email.
Limitations of Bayes’ Theorem
Challenges in Text Classification:
▶ Computational Complexity: Computing all word
probabilities is expensive.
▶ Data Sparsity: Many word combinations are unseen in
training data.
▶ Independence Assumption: Words in a sentence are not
truly independent.
Solution? Naïve Bayes Classifier
PART 3: NAÏVE BAYES CLASSIFIER
What is Naïve Bayes?
Definition:
▶ A probabilistic classifier based on Bayes’ Theorem, assuming
independence between features (words).
▶ Formula:
P(X |C ) · P(C )
P(C |X ) =
P(X )
Where:
▶ C = Class (e.g., Spam or Not Spam)
▶ X = Feature set (e.g., words in the email)
▶ Why “Naïve”?
▶ Assumes words are independent, which is simplistic but
effective!
Types of Naïve Bayes Classifiers
Common Variants Used in NLP:
▶ Multinomial Naïve Bayes: Used for text classification (word
counts).
▶ Bernoulli Naïve Bayes: Used when features are binary (word
presence/absence).
▶ Gaussian Naïve Bayes: Used for continuous-valued features.
Implementing Naïve Bayes in R
Step 1: Load Required Libraries
[Link]("tm")
[Link]("e1071")
library(tm) # Text Mining
library(e1071) # Naïve Bayes
Step 2: Load and Preprocess Text Data
text_data <- c("Win a prize now!", "This is not spam",
"Congratulations, you won!", "Meeting at 3 P
labels <- c("Spam", "Not Spam", "Spam", "Not Spam")
# Convert to Corpus
corpus <- Corpus(VectorSource(text_data))
# Text Cleaning: Lowercase, Remove Punctuation & Stopwords
corpus_clean <- tm_map(corpus, content_transformer(tolower)
corpus_clean <- tm_map(corpus_clean, removePunctuation)
corpus_clean <- tm_map(corpus_clean, removeWords, stopwords
# Convert to Document-Term Matrix
dtm <- DocumentTermMatrix(corpus_clean)
# Convert to Data Frame
data <- [Link]([Link](dtm))
data$Label <- labels
Step 3: Train the Naïve Bayes Model
# Split into Training and Test Sets
[Link](123)
train_indices <- sample(1:nrow(data), 3)
train_data <- data[train_indices, ]
test_data <- data[-train_indices, ]
# Train Naïve Bayes Model
model <- naiveBayes(Label ~ ., data=train_data)
# Predict on Test Data
predictions <- predict(model, test_data)
# Show Results
table(predictions, test_data$Label)
Evaluating Naïve Bayes Classifier
Metrics Used:
▶ Accuracy – Overall correctness
▶ Precision – True Positives
True Positives
+ False Positives
▶ Recall – True Positives
True Positives
+ False Negatives
▶ F1-Score – Harmonic mean of Precision & Recall
Confusion Matrix Interpretation:
▶ True Positives (TP): Correct spam detections
▶ False Positives (FP): Legitimate emails misclassified as spam
▶ False Negatives (FN): Spam emails misclassified as
non-spam
Summary & Next Steps
▶ Bayesian Inference helps estimate probabilities for
classification.
▶ Bayes’ Theorem is useful for spam filtering and text
classification.
▶ Naïve Bayes simplifies computations by assuming
independence.
▶ Practical Implementation in R shows how it classifies text
efficiently.
Next Week:
▶ Logistic Regression for Text Classification
Hands-on Activity
Task:
▶ Train a Naïve Bayes model on a movie reviews dataset
(positive/negative).
▶ Evaluate performance with accuracy and confusion matrix.