Module 2

WORD LEVEL ANALYSIS: Unsmoothed N-grams, Evaluating N-grams,

Smoothing, Interpolation and Backoff – Word Classes, Part-of-Speech
Tagging, Rule-based, Stochastic and Transformation-based tagging, Issues in
PoS tagging – Hidden Markov and Maximum Entropy models.

Unsmoothed N-grams
What is an N-gram?
An N-gram is a sequence of words (or tokens) that appear together in a text.
The "N" stands for how many words are in the sequence:

Type Example

Unigram (1) ["I"], ["love"], ["NLP"]

Bigram (2) ["I love"], ["love NLP"]

Trigram (3) ["I love NLP"]

We use N-grams to model how likely one word follows another in a sentence—
a core idea in language modeling.
What Are "Unsmoothed" N-gram Models?
Unsmoothed models are like raw counts. They don't do anything fancy—they
just look at how often N-gram sequences occur in the training text. Here's
how it works:
• Suppose your training text is: "I love NLP. I love learning NLP."
• You count the frequency of each bigram:
o "I love": appears 2 times
o "love NLP": 1 time
o "love learning": 1 time
What's the Problem with This?
Here's the catch: if a word sequence never appears, its probability is 0. For
example:
That’s bad, especially for tasks like spelling correction or auto-complete. It’s
saying something is impossible just because it didn’t appear in the training
data. That’s not true in real life! —which we know isn't true!
Why “Unsmoothed”?

1
“Unsmoothed” means we don’t add any fudge factor or fix to account for rare
or unseen word combinations. Later, we’ll look at smoothing techniques like
Laplace (add-one) or backoff models that try to solve this issue.
• Unsmoothed N-grams only count what's in the training data.
• If an N-gram never occurs, its probability is zero.
• Simple to implement, but not reliable for rare or new word sequences.
Why use Unsmoothed N-grams?
• It’s simple and fast to build.
• Helps understand how often word combinations occur.
• Useful when you have a large dataset that covers most word pairs.
Limitations
• Zero probability for unseen N-grams.
• Not good for small datasets.
• Doesn't handle grammar or context well.
Examples:
• Unigram (1 word): “I”, “like”, “books”
• Bigram (2 words): “I like”, “like books”
• Trigram (3 words): “I like books”
N-grams are helpful in modeling language because they capture word patterns
and help us predict the next word.
What is an N-gram Model?
An N-gram model uses statistics (probabilities) to guess how likely a sequence
of words is.
For example:
If “I like” appears often in your training text, then:
P(like | I) will be high.
We calculate this using:
P(current word | previous word(s))
Why Still Use It?
• Simple to build and understand
• Fast to compute

2
 • Good starting point before using more advanced techniques
Where is It Used?
• Basic spell checkers
• Autocomplete
• Early speech recognition
• As a baseline model in NLP research
Evaluating N-grams
What is "Evaluation" in N-gram Models?
After building an N-gram model (which predicts words based on previous
ones), evaluation is how we check:
“Is this model doing a good job?”
Just like checking how well a student does in a test, we test the model to see:
• Is it good at predicting the next word?
• Does it make natural-sounding sentences?
• Can it help in real tasks like correcting spelling or understanding
speech?
How Do We Evaluate N-gram Models?
We mainly use two methods:
1. Perplexity
2. Accuracy (used in prediction tasks)
1. What is Perplexity?
Simple Meaning:
Perplexity measures how confused the model is when it sees a new sentence.
• Low perplexity = Model is good (less confused, more confident)
• High perplexity = Model is bad (more confused)
Real-life Example:
Imagine this sentence:
"I love chocolate ice cream"
If your bigram model has seen similar sentences in training like:
"I love chocolate"
"chocolate ice cream"

3
Then, it will easily assign high probability to each word, and perplexity will be
low
But if the sentence is:
"Bananas fly underwater"
Your model might get confused because it never saw those word
combinations. It assigns low probabilities → perplexity becomes high
Think of Perplexity like a quiz:
• If you studied the topic, quiz is easy → low perplexity
• If you didn’t, quiz is hard → high perplexity

2. What is Accuracy?
Sometimes we use the N-gram model to predict the next word. In that case,
we check:
"How many times did it guess the correct word?"
Example:
Sentence: "I love __"
Model predicts: “pizza”
Correct answer: “pizza” → Correct
If model predicted “burger” → Wrong
If it got 7 out of 10 right:

4
Accuracy = 7/10 = 70%
This method is useful when the N-gram model is used in chatbots,
autocorrect, next-word suggestions, etc.

Steps to Evaluate N-gram Models

Step 1: Split the dataset
• Training set: To build the model
• Test set: To evaluate the model
We don’t test on training data because we want to see how the model performs
on new, unseen data.
Step 2: Use the model on the test set
• Calculate probabilities of each sentence
• Or, make next-word predictions
Step 3: Calculate Perplexity or Accuracy
• Perplexity for how well the model fits the test sentences
• Accuracy if the task is next-word prediction

5
Smoothing
What Is Smoothing in Language Models?
When we use N-gram models to predict the next word, sometimes we face this
problem:
A valid word (which is in our vocabulary) shows up in a new context we’ve
never seen before in the training data.
Problem:
The model gives zero probability to this new word sequence — even if it’s
perfectly valid.
Example:
Training data:
“I like coffee”
“She drinks tea”
Now in the test sentence:
“I like tea” – this never appeared in training.
So, the bigram model gives P(tea | like) = 0
But that’s not realistic! “I like tea” is a perfectly normal sentence.
Why Smoothing?
To avoid zero probability, we apply a method called smoothing.
Main Idea:
We take a tiny bit of probability from the common word pairs and give it to
the unseen ones, so that everything has some chance of happening.

6
This makes the model more flexible and smarter.
Types of Smoothing Techniques
We’ll explain two basic ones here:
1. Laplace (Add-One) Smoothing
2. Add-k Smoothing
Laplace (Add-One) Smoothing
What It Does:
• Add 1 to every count — even to those that are zero.
• So, if a bigram like “like tea” never appeared, we still give it a count of
1, not 0.
• This way, every word sequence has some probability.

Unigram Case (1 word at a time):

7
Bigram Case (2-word sequences):

What’s the Problem?

Laplace smoothing is too aggressive:
• It gives too much probability to rare or unseen word pairs.
• It hurts accuracy, especially in big datasets where rare words are really
rare.
So, it’s not used in modern N-gram language models, but it’s still useful for
teaching and small tasks (like text classification).

8
Add-k Smoothing (More Gentle)
What It Does:
Instead of adding 1, we add a small fraction like:
• k = 0.5
• k = 0.1
• k = 0.01
This is why it’s called add-k smoothing.

Why Use Add-k?

• It’s less harsh than add-one.
• You can tune k (by testing different values) to get better results.
Problem:
• Even add-k is not great for language modeling.
• The probabilities still don’t match real language behavior well.
Other Smoothing Techniques
• Stupid Backoff: If a trigram is missing, fall back to a bigram; if bigram
is missing, fall back to unigram.
• Kneser-Ney Smoothing: Advanced technique — one of the best for N-
gram language models.
Summary
Smoothing is used in N-gram models to avoid zero probabilities for valid but
unseen word sequences. The simplest method is Laplace (Add-One)
smoothing, where 1 is added to each count. A more flexible version is Add-k
smoothing, where a smaller value is added. Smoothing helps the model handle
rare or new sentences better by distributing some probability mass from
frequent events to unseen ones.
Interpolation and Backoff
The Challenge with N-gram Models
An N-gram model predicts the next word by looking at the previous N−1
words. So a trigram model (N=3) tries to calculate:

9
But what if this specific sequence—say, “ice cold cream”—never appears in
training?
Even though all three words are familiar, their combination is unknown. If we
only rely on raw counts:
• The trigram probability becomes zero
• That’s a problem, especially if the phrase is perfectly valid but rare
So how can we estimate the probability anyway?
Two Solutions: Backoff & Interpolation
Both techniques tap into a hierarchy of language knowledge:
• Trigram: specific sequence
• Bigram: less specific
• Unigram: most general
Let’s go step by step
Backoff: Falling Back Gracefully
Imagine you're trying to guess the next word using three words before it (a
trigram model). But suppose this 3-word pattern doesn’t exist in your training
data. Instead of giving up, you "back off" to a 2-word pattern (bigram) — just
use the last two words. If that’s also missing, you "back off" again and just
use a 1-word pattern (unigram).
Concept
If we can’t find the full trigram, we back off to a simpler model:
• No trigram? → Try bigram
• No bigram? → Try unigram
We only use simpler models when necessary

10
Rather than switching between trigram → bigram → unigram, interpolation
combines all three!
Every prediction uses:
• The trigram probability
• The bigram probability
• The unigram probability
Each with its own weight (λ)

11
Word Classes
What Are Word Classes?
Word classes — also called parts of speech or lexical categories — are like
labels that describe how a word functions in a sentence.
Just like in sports, each player has a role (goalkeeper, striker, midfielder), in
language, each word plays a specific role:
• Some name things (nouns)
• Some show action (verbs)
• Some describe things (adjectives)
• Some connect parts (conjunctions)
Why Do Word Classes Matter?
Understanding word classes helps in:
• Grammar: Knowing how to form correct sentences.
• Language Learning: Helps learners form and understand sentences.
• NLP & AI: Helps computers analyze, process, and understand text (like
ChatGPT does!).
Explanation of Word Classes
Let’s go deeper into each main word class with definitions, examples, and how
they behave.
1. Nouns (NN)
• What they do: Name people, places, things, or ideas
• Examples: student, apple, India, love, algorithm
• Forms: Can be singular or plural
o One student → Many students
• Role: Usually the subject or object of a sentence
o The student is reading.
o I saw the apple.
2. Verbs (VB)
• What they do: Express action or state
• Examples: eat, run, think, become, is, have
• Forms: Change based on tense and subject

12
 o I run → She runs
o They played, She is playing
• Types:
o Main verbs: carry the main meaning (run, teach)
o Auxiliary verbs: help main verbs (is, was, have, do)
▪ She is studying, I have eaten
3. Adjectives (ADJ)
• What they do: Describe nouns
• Examples: tall, blue, interesting, fast
• Position: Often come before a noun
o A blue car
o An interesting book
• Degree:
o Positive: big
o Comparative: bigger
o Superlative: biggest
4. Adverbs (RB/JJ in some tag sets)
• What they do: Describe verbs, adjectives, or other adverbs
• Examples: slowly, very, happily, extremely
• Position: Can move around
o She runs quickly.
o Quickly, she runs.
• Common types:
o Manner: how? (slowly, carefully)
o Time: when? (yesterday, soon)
o Frequency: how often? (always, rarely)
o Degree: to what extent? (very, too)
5. Prepositions (IN)
• What they do: Show the relationship between two words
• Examples: in, on, under, over, between, to, of

13
 • Often show location, direction, or time
o The book is on the table.
o He walked to school.
o We met after lunch.

6. Pronouns (PRP)
• What they do: Replace nouns to avoid repetition
• Examples: I, you, he, she, it, we, they
• Usage:
o Instead of: “John went to John’s car.”
Say: “He went to his car.”
7. Determiners (DET)
• What they do: Introduce nouns and tell us which one, how many, or
whose
• Examples: the, a, an, this, those, my, some
• Usage:
o The apple
o Some students
o My book
8. Conjunctions (CC)
• What they do: Connect words, phrases, or clauses
• Examples: and, but, or, because, although
• Usage:
o I like tea and coffee.
o She was late because of the traffic.
9. Interjections (UH)
• What they do: Show strong emotion or reaction
• Examples: wow, oh, ouch, hey, ugh, yay
• Usage:
o Wow! That’s amazing!
o Ouch! That hurt!

14
Open vs Closed Word Classes
Open Word Classes – They Grow!
These classes can easily add new words as language evolves.

Word Class Can You Add New Words? Examples (Old + New)

Nouns Yes table, smartphone, selfie

Verbs Yes eat, google, scroll

Adjectives Yes nice, woke, aesthetic

Adverbs Yes fast, digitally, remotely

Closed Word Classes – They Stay Fixed

These classes have a limited set of words that don’t change much.

Word Class Can You Add New Words? Examples

Prepositions Rarely in, on, under, to, of

Pronouns No I, he, they, it

Determiners No the, a, some, this

Conjunctions No and, but, because

Auxiliary Verbs No is, have, do, was

These words are fundamental to sentence structure, and are almost universal
in usage.

15
Part-of-Speech Tagging
What Is Part-of-Speech Tagging?
Part-of-Speech (POS) tagging means identifying the role of each word in a
sentence — like whether it is a noun, verb, adjective, etc.
It’s like giving each word a label based on how it is used in that sentence.
Example:
Sentence: I am reading a good book.
• I → Pronoun
• am → Auxiliary Verb
• reading → Verb (present participle)
• good → Adjective
• book → Noun
So, POS tagging is the process of:
• Looking at each word
• Checking the context
• Assigning the correct grammatical tag
Why Is It Tricky?
Many words in English can be used in different ways depending on the
sentence. So the same word may have different tags in different contexts.
Example:
1. The police booked the thief.
o “booked” = Verb (meaning to arrest)
2. I read a book every night.
o “book” = Noun
So, to tag correctly, we must look at the meaning in context — not just the
word alone.
Input and Output of POS Tagging
• Input: A sentence (sequence of words) and a tag set (list of possible
labels)
• Output: The best part-of-speech tag for each word

16
Example:
Sentence: They have eaten all the apples.
Tagged Output:
• They/PRP
• have/VBP
• eaten/VBN
• all/DT
• the/DT
• apples/NNS
What Is a Tag Set?
A tag set is a predefined list of labels (tags) used to identify word classes.
• Basic categories: noun, verb, adjective, adverb, preposition
• Some tag sets add more details:
o Singular/plural
o Tense
o Gender
o Person (1st, 2nd, 3rd)
Examples from the Penn Treebank Tag Set (used in NLP tools)

Tag Meaning Example Word

VB Verb base form eat

VBD Verb past tense ate

VBG Verb gerund / present participle eating

VBN Verb past participle eaten

VBP Verb present (not 3rd person singular) eat

VBZ Verb present (3rd person singular) eats

Look at These Sentences:

1. Zuha eats an apple daily.

17
 o “eats” → VBZ (3rd person present)
2. Aman ate an apple yesterday.
o “ate” → VBD (past tense)
3. They have eaten all the apples.
o “eaten” → VBN (past participle)
4. I like to eat guavas.
o “eat” → VB (base form)
Even though all these sentences use a form of the word eat, the form and
function change, so they receive different tags.
How Is POS Tagging Done?
There are three main methods:
1. Rule-Based Tagging
• Uses a set of grammar rules
• Applies linguistic knowledge
• Example rule: If a word ends in “-ing” and follows “is” → tag as VBG
Advantage: Interpretable
Disadvantage: Lots of manual rules needed
2. Stochastic (Statistical) Tagging
• Uses probabilities from large datasets
• Learns patterns from tagged sentences
• Example: If “book” follows “a”, it’s likely a noun (not a verb)
Advantage: Learns from real data
Disadvantage: May fail if the data is too limited
3. Hybrid Tagging
• Combines rules and statistics
• Uses the strengths of both methods
• Most modern NLP systems use this
Advantage: More accurate and flexible
Tag Set Sizes

18
Tag Set Number of Tags

Penn Treebank 45

C7 164

TESS 200

TOSCA-ICE 270

• English is not morphologically rich, so fewer tags are often better.

• Too many tags → more mistakes (overfitting, confusion)
Final Summary

Concept Simplified Meaning

POS Tagging Labeling each word with its grammatical role

Tag Set Predefined labels like NN, VB, JJ, etc.

Ambiguity Same word can play different roles (e.g., book)

Context Helps determine correct meaning and tag

Methods Rule-based, statistical, hybrid

Examples eat/VB, eats/VBZ, eaten/VBN, etc.

Rule-based
Rule-Based POS Tagger
What is a Rule-Based Tagger?
A rule-based tagger assigns parts of speech (like noun, verb, etc.) to words
using rules written by linguists or programmers.
It works in two stages, like a two-step detective process.
Stage 1: Dictionary Lookup
• First, the tagger looks up each word in a dictionary (also called a
lexicon).
• The dictionary contains:
o The possible tags for each word.
o Some grammatical features (like tense, plural/singular, etc.).

19
Example:
Word: "show"
Possible tags:
• NN → Noun (e.g., a good show)
• VB → Verb (e.g., They show respect)
So after this step, we don’t know yet which tag is correct — just that both
are possible.
Stage 2: Apply Handwritten Rules
Next, the tagger uses a set of human-written rules to choose the correct tag
based on context (the surrounding words).
Example:
Sentence: The show must go on
• “show” has two possible tags: {VB, NN}
• Look at the word before “show” → it's "The"
• “The” is a determiner (DT) – it usually comes before nouns, not verbs.
Rule:
“If a word comes after a determiner, it is probably a noun → remove the
verb tag (VB)”
Result: "show" is tagged as NN (noun).
Types of Rules Used
Rule-based taggers use rules based on:
1. Contextual clues
→ What are the words before and after the current word?
Example:
o If the previous word is “to” and current word ends with "-ing",
then it might be a verb.
2. Morphological clues
→ What is the structure of the word?
Example:
o If a word ends in -ing, and the previous word is a verb like “is”
or “was”, then the word is likely a verb too.

20
 ▪ e.g., She is dancing. → “dancing” = VBG (present
participle)
3. Capitalization
→ If a word starts with a capital letter and isn’t the first word in a
sentence, it might be a proper noun.
How Are the Rules Created?
Creating the rules can be done by:
1. Writing manually using grammar knowledge
(tedious and time-consuming)
2. Learning from correction
o Feed untagged text into the system
o Manually correct the errors
o The system “learns” from this feedback
Famous Rule-Based Taggers
TAGGIT (1971)
• Used 3,300 rules
• Tagged 77% of the Brown corpus correctly (a large English dataset)
ENGTWOL (1995)
• Known for being fast and accurate
• Used for tagging English and other languages
Advantages of Rule-Based Taggers

Advantage Explanation

Accurate and If the rules are well-written, the tagging is very

Deterministic precise

Explainable You can clearly see why a tag was chosen

Unlike ML models, they don’t need huge

No need for big data
training datasets

Disadvantages of Rule-Based Taggers

21
Disadvantage Explanation

Time-consuming to Writing thousands of rules by hand is very hard

build and slow

If language or usage changes, you must rewrite

Hard to maintain
the rules

Not flexible for other A tagger built for English won’t work for Hindi or
languages French without writing a whole new rule set

Stochastic and Transformation-based Tagging

1. Introduction to POS Tagging
• Part-of-Speech (POS) Tagging: The task of labeling each word in a
sentence with its grammatical category (e.g., noun, verb, adjective).
• Example: “The cat sleeps.” → The/DET cat/NOUN sleeps/VERB
• POS tagging is crucial for:
o Syntax parsing
o Information retrieval
o Machine translation
o Question answering systems
There are three main approaches:
1. Rule-based tagging (manual linguistic rules)
2. Stochastic (probabilistic) tagging
3. Transformation-based tagging (hybrid approach)
Here we focus on Stochastic and Transformation-based approaches.
2. Stochastic Tagging
Definition
Stochastic tagging is a probability-based method.
• Instead of fixed rules, it assigns tags based on statistical likelihood
derived from large annotated corpora.
• The main idea: Choose the tag with the highest probability, given
the word and its context.

22
Core Principles
• Unigram Model: Assigns the most frequent tag for a word.
o Example: “book” → most often a noun.
• Bigram Model: Considers the current word + previous tag.
o P(tag | previous tag)
• Trigram Model: Considers the current word + two previous tags.
o P(tag | previous 2 tags)
The most common method is Hidden Markov Model (HMM), where:
• States = POS tags
• Observations = words
• Goal = find the most likely sequence of tags.
Example
Sentence: “The can is on the table.”
• The word “can” could be:
o Noun (“a can of juice”)
o Verb (“I can swim”)
Steps:
1. Calculate probabilities from corpus:
o P(noun | “can”) = 0.65
o P(verb | “can”) = 0.35
2. Consider context: “The can …” → after “The”, a noun is more likely.
3. Final decision: “can” → noun.
Advantages
• Handles ambiguity using probabilities.
• Works well with large annotated data.
• Provides high accuracy with sufficient training.

Disadvantages
• Requires large training corpus.

23
 • Computationally intensive.
• Less interpretable (statistical black-box).
3. Transformation-based Tagging (Brill Tagger)
Definition
Transformation-based tagging (TBL), introduced by Eric Brill (1992), is a
hybrid approach that combines:
• Simplicity of rule-based tagging.
• Accuracy of stochastic tagging.
Also called Brill Tagging.
Core Idea
• Start with a baseline tagging (e.g., assign the most frequent tag for
each word).
• Then apply rules to correct mistakes.
• Rules are automatically learned from training data.
How it Works
1. Initial Tagging: Assign tags using a simple method (unigram or
default tagger).
2. Rule Learning: Identify errors by comparing with the gold-standard
corpus.
3. Generate Rules: Extract patterns like:
o If a word is “can” and previous word is a pronoun → change
tag from noun to verb.
4. Rule Ordering: Rank rules based on how many errors they correct.
5. Apply Rules: Sequentially apply rules until no significant
improvement.
Example
Sentence: “He can swim.”
• Initial tagging: “can” → noun (since noun is the most frequent tag in
corpus).
• Transformation rule:
o If previous word is a pronoun, change “can” from noun → verb.

24
 • Final tagging: “can” → verb.
Advantages
• Rules are readable and interpretable.
• Does not require extremely large corpora.
• Easy to adapt rules to new domains.

Disadvantages
• Rule application can be slow for long texts.
• Does not handle rare/ambiguous cases as well as probabilistic
models.
• Accuracy slightly lower than advanced statistical/neural models.
4. Comparison Between Stochastic and Transformation-based
Tagging

Feature Stochastic Tagging Transformation-based Tagging

Probability &
Basis Rule learning (Brill method)
statistics

Initial Based on most frequent/default

Based on likelihood
Assignment tag

Uses n-gram Uses learned rules (e.g.,

Context
probabilities previous/next word’s tag)

Interpretability Black-box Human-readable rules

Data Needs large

Works with smaller corpora
Requirement annotated corpus

Slower (sequential rule

Speed Faster once trained
application)

Accuracy High with large data Moderate but interpretable

HMM, Maximum
Example Models Brill Tagger
Entropy, CRF

5. Applications
Both methods are used in:
• Part-of-speech tagging

25
 • Named Entity Recognition (NER)
• Chunking and parsing
• Machine translation
• Question answering systems
6. Summary
• Stochastic Tagging: Uses probabilities from large corpora to assign
the most likely tag.
• Transformation-based Tagging: Starts with a baseline, then corrects
errors with learned rules.
• Both approaches laid the foundation for modern Neural Network-
based taggers (e.g., BiLSTM, Transformers like BERT).
Issues in PoS tagging
1. Introduction
• Part-of-Speech (POS) Tagging: The process of assigning grammatical
categories (Noun, Verb, Adjective, etc.) to each word in a sentence.
• Importance: Used in parsing, machine translation, information
extraction, question answering, etc.
• Challenge: Natural language is complex, and words often change
meaning depending on context, making POS tagging a non-trivial
task.
2. Detailed Issues in POS Tagging
1. Ambiguity of Words
• Lexical Ambiguity: Many words belong to multiple parts of speech.
• Correct tag depends on context.
• Examples:
o “He can swim.” → “can” = verb (ability)
o “I bought a can of juice.” → “can” = noun (container)
o “They can the fruits.” → “can” = verb (to preserve food)
• Ambiguity is the biggest challenge in POS tagging.
2. Unknown Words (Out-of-Vocabulary Problem)
• Taggers are trained on annotated corpora.

26
 • New words (slang, technical terms, names, social media words) may
not exist in training data.
• Example:
o Word “selfie” (new in modern usage).
o Word “googled” (verb form of a company name).
• Solution attempts:
o Use morphological analysis (suffix/prefix clues).
o Use neural embeddings (to capture meaning from similarity).
3. Context Sensitivity
• Correct tag often depends on surrounding words.
• Example:
o “I saw her duck.”
▪ “duck” = noun (animal) OR verb (action of bending).
o “They refuse to permit us to obtain the refuse permit.”
▪ First “refuse” = verb; second “refuse” = noun;
▪ First “permit” = verb; second “permit” = noun.
• Context is crucial for correct tagging, but models sometimes fail to
capture it fully.
4. Multi-word Expressions
• Certain word combinations behave as a single unit.
• Example:
o “Look up” = verb (phrasal verb, meaning “search”).
o “by and large” = idiomatic expression (adverb).
• If words are tagged individually, the meaning is lost.
5. Proper Nouns vs. Common Words
• Distinguishing between names and regular words is difficult.
• Example:
o “May is beautiful.” → “May” = Proper noun (month).
o “It may rain today.” → “may” = Modal verb.

27
 • Ambiguity in capitalization and usage causes confusion.
6. Domain Dependence
• Taggers trained on one domain may not work well in another.
• Example:
o Word “operation”:
▪ Medical domain → surgery.
▪ Military domain → planned mission.
▪ Mathematics → mathematical function.
• Domain-specific corpora are required for accuracy.
7. Errors in Training Data
• POS taggers rely on manually annotated corpora.
• If annotations contain human errors, models will learn incorrectly.
• Example: A wrongly tagged word in corpus → repeated tagging
errors in the model.
8. Free Word Order in Languages
• English has relatively fixed word order (SVO: Subject–Verb–Object).
• But many languages (e.g., Hindi, Tamil, Latin, Russian) allow
flexible word order.
• Example (Hindi):

o “राम ने सीता को फूल दिया।” → Subject, object, and verb can

appear in multiple orders.
o POS tagger must rely on morphology, not word position.
9. Morphologically Rich Languages
• Some languages (e.g., Turkish, Finnish, Tamil) use inflections and
suffixes heavily.
• A single word may represent what would be a whole phrase in
English.
• Example (Turkish):
o “evlerinizden” = “from your houses” (root + plural + possessive
+ case ending).

28
 • Such complexity increases tagging difficulty.
10. Code-Switching and Mixed Languages
• In multilingual societies, people often mix two or more languages.
• Example:
o “I will kal tomorrow.” (English + Tamil: “kal” = tomorrow).
• POS taggers trained on one language cannot handle such switches.
11. Ambiguity in Punctuation and Special Symbols
• Punctuation can change meaning.
• Example:
o “Let’s eat, grandma.” (inviting grandma to eat).
o “Let’s eat grandma.” (implies eating grandma).
• Taggers sometimes misinterpret sentence boundaries due to
punctuation.
3. Summary
• POS tagging is not straightforward due to:
1. Word ambiguity (same word = multiple POS)
2. Unknown words (OOV problem)
3. Context sensitivity
4. Multi-word expressions and idioms
5. Confusion with proper nouns vs. common words
6. Domain-specific meanings
7. Errors in annotated corpora
8. Free word order in some languages
9. Morphologically rich languages
10. Code-switching in multilingual contexts
11. Punctuation ambiguities
• Modern Solutions:
o Hybrid methods (rules + probabilities).
o Neural networks (BiLSTM, CRF).

29
 o Transformers (BERT, GPT) → capture deeper context and
semantics.
Hidden Markov and Maximum Entropy models.

1. Hidden Markov Model (HMM)

Definition:
A Hidden Markov Model is a probabilistic model used for sequence labeling
tasks like PoS tagging. It models the sequence of hidden states (PoS tags)
that generate the sequence of observed events (words).
What is it?
A Hidden Markov Model is a probabilistic model used to describe a
sequence of observations (like words in a sentence) where the real state is
hidden but generates observable data.
Think of it like this:
You see the symptoms (observable), but you’re guessing the disease
(hidden).
Where is it used?
• Part-of-Speech tagging
• Speech recognition
• Named Entity Recognition (NER)
How It Works:
1. States = POS tags (hidden): e.g., NN, VB
2. Observations = Words (visible): e.g., “eat”, “fish”
3. Transition Probabilities = How likely a tag follows another (e.g., verb
after noun)
4. Emission Probabilities = How likely a word is used for a tag (e.g., "eat"
as a verb)
Then it uses the Viterbi algorithm to find the most probable path of tags
for your sentence.
Basic Example: POS Tagging
Input (Observation): Time flies like an arrow
We only see words. The POS tags (Noun, Verb, etc.) are hidden states.

30
Goal: Predict the most likely sequence of POS tags that generated the
words.
HMM Assumptions :
1. Markov Assumption:
The current state (POS tag) depends only on the previous state, not
the entire past.
P(tag_n | tag_1,...,tag_n-1) ≈ P(tag_n | tag_n-1)
2. Output Independence Assumption:
The observed word depends only on the current tag, not on other
words or tags.
P(word_n | tag_n, previous...) ≈ P(word_n | tag_n)
Components of an HMM:

Component Meaning

States Hidden things like POS tags (Noun, Verb)

Observations Actual data (words in a sentence)

Transition Prob. Probability of going from one tag to another

Emission Prob. Probability of a word being generated by a tag

Initial Prob. Prob. of starting with a particular tag

2. Maximum Entropy Model (MaxEnt)

What is it?
A Maximum Entropy model is another probabilistic model, but unlike
HMM, it doesn’t make strong independence assumptions.
It uses features and tries to maximize entropy, i.e., be as unbiased as
possible while still fitting the training data.
This just means:
It doesn’t give just one answer — it gives probabilities of possible
outcomes.
It tells you how likely something is, not just what is most likely.

Example:
For the word “flies”, it might say:
• Noun: 0.4
• Verb: 0.6

31
So, it chooses “Verb” as the most probable, but also shows there is some
chance it’s a Noun.
Real-life analogy:
You want to make the fairest prediction possible using all the available
facts (features), without making unnecessary assumptions.
Where is it used?
• POS tagging
• Text classification
• Sentiment analysis
• NER
How it works:
• Takes features of the data (like word, previous word, suffix, etc.)
• Learns weights for each feature
• Uses these weights to compute the probability of a class (like a POS
tag)
“Uses features and tries to maximize entropy”
What is “entropy” in this context?
Entropy means uncertainty or randomness.
"Maximum Entropy" means: Don't assume anything more than the
evidence supports.
If the data does not strongly suggest a pattern, the model stays
neutral.

In other words:
MaxEnt chooses the most fair, unbiased probability distribution that
still matches the training data.
It only makes predictions that are justified by what it has learned —
nothing extra.
Key difference from HMM:

Feature HMM MaxEnt

Assumes dependence
Assumptions No independence assumption
(Markov)

32
 Transitions and
Based on Features and weights
emissions

Less (harder to model

Flexibility More (can use many features)
rich context)

Based on optimization (likelihood

Learning Based on frequencies
maximization)

MaxEnt Formula:

Where:
• y is the class (like a tag)
• x is the observation (word and features)
• f_i(x, y) is the i-th feature function
• w_i is the weight for that feature
• Z(x) is the normalization factor
Summary Table

Hidden Markov Model Maximum Entropy Model

Feature
(HMM) (MaxEnt)

Type of model Generative Discriminative

Assumes words are

Observations Uses features from context
generated from tags

Tag Only previous tag Can consider more context

dependencies matters (Markov) (like previous 2 tags)

Probability Transition and emission

Feature weights
trained on probabilities

POS tagging, classification,

Example use Speech, POS tagging
sentiment analysis

Flexibility Limited High

33
Which one is better?
• HMM is easier to implement, especially when you don’t have many
features.
• MaxEnt (or similar models like Conditional Random Fields - CRFs)
is better when you have rich context and want more accurate
predictions.

34