Module 3: Navie Bayes Classifier
Module-3
Naive Bayes and Sentiment Classification
4.1 Naive Bayes Classifiers
Definition:
A probabilistic classifier based on Bayes’ theorem with a naive assumption of conditional
independence among features. It is called “naive” because it assumes that the features (words in a
document) are conditionally independent given the class label.
It is widely used for text classification tasks such as
o spam detection,
o sentiment analysis, etc.
There are two types of assumptions:
1. The first is the bag of words assumption.
2. The second is commonly called the naive Bayes assumption.
The Bag-of-Words (BoW) assumption treats a document as an unordered collection of words,
Ignoring the grammar and
The order of words but keeping the frequency of each word.
This simplifies the representation of the document and is commonly used in Naive Bayes and
other text classification models.
NLP-6 CSE-DS Prof. Shwetha H C
Module 3: Navie Bayes Classifier
[Link] the working of a Multinomial Naive Bayes Classifier with suitable equations and
assumptions.
step1. Assumption: Text is represented using the bag-of-words model, and word features are
conditionally independent given the class.
step 2. Goal: Predict the class c∈C for a document d.
step 3. Bayes rule:
step 4. Navie Bayes Classifier:
The Naive Bayes classifier is a probabilistic machine learning algorithm based on Bayes’
Theorem.
Note: ^=our estimate of the correct class.
We thus compute the most probable class ˆ c given some document d by choosing the class which has
the highest product of two probabilities: the prior probability of the class P(c) and the likelihood of the
document P(dlc):
Without loss of generalization, we can represent a document d as a set of features
f1; f2;……… fn
NLP-6 CSE-DS Prof. Shwetha H C
Module 3: Navie Bayes Classifier
Thus we assume that the features f1; f2;…….fn only encode word identity and not position.
1.1 Navie Bayes Assumption
The second is commonly called the naive Bayes assumption: this is the conditional independence
assumption that the probabilities P( flc) are independent given the class c and hence can be ‘naively’
multiplied as follows:
The final equation for the class chosen by a naive Bayes classifier is thus:
To apply the naive Bayes classifier to text, we need to consider word positions, by simply walking an
index through every word position in the document:
Naive Bayes calculations, like calculations for language modeling, are done in log space, to avoid
underflow and increase speed.
Classifiers that use a linear combination of the inputs to make a classification decision —like naive Bayes
and also logistic regression—are called linear classifiers.
Advantages
Simple, fast, and efficient.
Works well with large datasets.
Effective for text classification tasks like spam filtering, sentiment analysis, etc.
Limitations
Assumes independence between features — which is rarely true in real text.
Not suitable when feature interactions are important.
NLP-6 CSE-DS Prof. Shwetha H C
Module 3: Navie Bayes Classifier
4.2 Training the Naive Bayes Classifier
How can we learn the probabilities P(c) and P( fi c)?
1. Learn the probabilities P(c)
For the document prior P(c) we ask what percentage of the documents in our training set are in
each class c. The class prior is calculated using Maximum Likelihood Estimate (MLE).
Let Nc :be the be the number of documents in our training data with class c
and Ndoc : be the total number of documents. Then
[Link] the probabilities P( fi c)
we’ll assume a feature is just the existence of a word in the document’s bag of words,
and so we’ll want P(wi c), which we compute as the fraction of times the word wi appears among
all words in all documents of topic c.
We first concatenate all documents with category c into one big “category c” text.
Then we use the frequency of wi in this concatenated document to give a maximum likelihood
estimate of the probability:
Here the vocabulary V consists of the union of all the word types in all classes, not just the words in one class c.
To estimate the likelihood of the word “fantastic” given class positive,
If a word appears zero times in a class, its likelihood becomes 0.
Since Naive Bayes multiplies probabilities, this leads to the entire class probability becoming
zero.
To prevent zero probabilities, apply Laplace Smoothing
Words not present in training vocabulary are ignored during testing i.e. unknown words.
Frequent function words like "the", "is", "a" are stop words.
Usually removed, but many systems today retain them as removal may not improve performance.
NLP-6 CSE-DS Prof. Shwetha H C
Module 3: Navie Bayes Classifier
Train Naïve Bayes Algorithm
TEST NAIVE BAYES(testdoc, logprior, loglikelihood, C, V)
Initialize sum for each class with logprior
For each word in the test document:
o If word ∈ V: add loglikelihood[word, class]
Return class with maximum sum
4.3 Naïve Bayes Example
Let’s walk through an example of training and testing naive Bayes with add-one smoothing.
We’ll use a sentiment analysis domain with the two classes positive (+) and negative (-),
and take the following miniature training and test documents simplified from actual movie
reviews.
NLP-6 CSE-DS Prof. Shwetha H C
Module 3: Navie Bayes Classifier
The prior P(c) for the two classes is computed
The word with doesn’t occur in the training set, so we drop it completely.
The likelihoods from the training set for the remaining three words
o “predictable”,
o “no”,
o “fun”,
For the test sentence S = “predictable with no fun”, after removing the word ‘with’,
the chosen class,
The model thus predicts the class negative for the test sentence.
4.4 Optimizing for Sentiment Analysis
NLP-6 CSE-DS Prof. Shwetha H C
Module 3: Navie Bayes Classifier
4.5 Naive Bayes for other text classification tasks
Naive Bayes is a flexible and efficient classification algorithm that can be applied to a variety of
text classification tasks beyond sentiment analysis.
While it traditionally uses words as features, any property of the input text can be used as a
feature.
Spam detection:
Consider the task of spam detection, deciding if a particular piece of email is an example of spam
(unsolicited bulk email) — and one of the first applications of naive Bayes to text classification
.
A common solution here, rather than using all the words as individual features, is to predefine
likely sets of words or phrases as features, combined these with features that are not purely
linguistic
For example the open-source Spam Assassin tool predefines features like the phrase “one hundred
percent guaranteed”, or the feature mentions millions of dollars, which is a regular expression that
matches suspiciously
More sample SpamAssassin features:
_ Email subject line is all capital letters
_ Contains phrases of urgency like “urgent reply”
NLP-6 CSE-DS Prof. Shwetha H C
Module 3: Navie Bayes Classifier
_ Email subject line contains “online pharmaceutical”
_ HTML has unbalanced ”head” tags
_ Claims you can be removed from the list
Example Tool: SpamAssassin — open-source spam detection tool using rule-based and Naive Bayes-based
detection.
Language ID
Goal: Identify the language of a given text.
Features are not words but byte n-grams (sequences of 1–4 characters including spaces).
Example:
o 2-gram: 'zw'
o 3-gram: 'nya'
o 4-gram: 'thei'
Advantage: Byte n-grams capture orthographic and structural patterns across languages.
Tools like [Link]:
Train on multilingual corpora like Wikipedia, Twitter, Bible translations, and Urban
Dictionary.
Feature selection is used to retain the most informative 7,000 n-gram features.
Why is it advantageous to use non-word features in tasks like language ID?
Non-word features like byte n-grams:
Are language-neutral and can capture morphological or phonetic clues.
They perform well even when words are short or not in the dictionary.
They help distinguish between dialects or closely related languages.
They are effective when labeled data is sparse.
4.6 Naive Bayes as a Language Model
Specifically, a naive Bayes model can be viewed as a set of class-specific unigram language models, in which
the model for each class instantiates a unigram language model .
Since the likelihood features from the naive Bayes model assign a probability to each word P(word/c),
the model also assigns a probability to each sentence:
Thus consider a naive Bayes model with the classes positive (+) and negative (-) and the following model
parameters
NLP-6 CSE-DS Prof. Shwetha H C
Module 3: Navie Bayes Classifier
Each of the two columns above instantiates a language model that can assign a probability to the sentence “I love
this fun film”:
Result:
The positive model assigns a higher probability to the sentence: P(sjpos) > P(sjneg). This is the
positive sentence.
NLP-6 CSE-DS Prof. Shwetha H C