Language Modeling, Smoothing, and the
Noisy Channel Model
Dr. Sambit Praharaj, Assistant Professor (II), KIIT
1 Introduction to Language Modeling
A Language Model (LM) assigns a probability to a sequence of words. Given a
sentence:
W = w1 , w2 , . . . , wn
a language model estimates:
P (W ) = P (w1 , w2 , . . . , wn )
Higher probability implies a more natural or fluent sentence.
Applications
• Speech recognition
• Machine translation
• Spell correction
• Text prediction and autocomplete
2 Chain Rule of Probability
Using the chain rule:
P (w1 , w2 , . . . , wn ) = P (w1 )P (w2 |w1 )P (w3 |w1 , w2 ) · · · P (wn |w1 , . . . , wn−1 )
Problem: Conditioning on all previous words is computationally infeasible.
Solution: Use the Markov assumption ⇒ N-gram models.
3 N-Gram Language Models
An N-gram model assumes the probability of a word depends only on the previous
N − 1 words.
Model Probability Assumption
Unigram P (wi )
Bigram P (wi |wi−1 )
Trigram P (wi |wi−2 , wi−1 )
1
4 Bigram Language Model
4.1 Definition
The bigram approximation is:
P (wi |w1 , . . . , wi−1 ) ≈ P (wi |wi−1 )
4.2 Example Corpus
I love NLP
I love AI
Vocabulary:
V = {I, love, NLP, AI}
4.3 Counts
Bigram Count
(I, love) 2
(love, NLP) 1
(love, AI) 1
4.4 Bigram Probability
2
P (love|I) = =1
2
1
P (N LP |love) =
2
Sentence probability:
P (I love NLP) = P (I|⟨s⟩) × P (love|I) × P (N LP |love)
5 Trigram Language Model
5.1 Definition
P (wi |w1 , . . . , wi−1 ) ≈ P (wi |wi−2 , wi−1 )
5.2 Formula
Count(wi−2 , wi−1 , wi )
P (wi |wi−2 , wi−1 ) =
Count(wi−2 , wi−1 )
5.3 Advantage and Limitation
• Better contextual modeling
• Suffers from data sparsity
2
6 Zero Probability Problem
If an N-gram never appears in training:
Count = 0 ⇒ P = 0
This causes the entire sentence probability to become zero, even if the sentence is
reasonable.
7 Laplace (Add-One) Smoothing
7.1 Idea
Assign a small probability to unseen events by adding 1 to all counts.
7.2 Bigram Laplace Formula
Count(wi−1 , wi ) + 1
P (wi |wi−1 ) =
Count(wi−1 ) + V
7.3 Numerical Example
Given:
• Vocabulary size V = 5
• Count(I) = 2
• Count(I, hate) = 0
0+1 1
P (hate|I) = =
2+5 7
7.4 Effect of Laplace Smoothing
• Avoids zero probability
• Redistributes probability mass
• Over-smooths frequent events
8 Backoff Smoothing
8.1 Core Idea
If a higher-order N-gram is unseen, back off to a lower-order model.
Trigram → Bigram → Unigram
3
8.2 Formal Definition
(
Ptri (wi |wi−2 , wi−1 ) if count > 0
P (wi |wi−2 , wi−1 ) =
αPbi (wi |wi−1 ) otherwise
8.3 Example
Sentence:
I love AI
• Trigram (I, love, AI) unseen
• Back off to P (AI|love)
• If unseen again, use unigram P (AI)
Thus, the sentence gets a non-zero probability.
9 Noisy Channel Model
9.1 Motivation
We observe a noisy sentence O and want to recover the intended sentence S.
Ŝ = arg max P (S|O)
S
Using Bayes’ rule:
Ŝ = arg max P (O|S)P (S)
S
9.2 Components
• P (S): Language model (fluency)
• P (O|S): Channel model (noise)
9.3 Example
Observed sentence:
I sea you
Candidates:
• I see you
• I sea you
• Language model prefers see
• Channel model explains typo: see → sea
Final decision:
arg max P (O|S)P (S)
S
4
10 Key Takeaways
• Language models assign probabilities to word sequences
• N-grams approximate long histories
• Smoothing avoids zero probabilities
• Backoff balances context and reliability
• Noisy channel separates fluency and noise