2-Mark Questions (with Bloom's Taxonomy)

1. Define an N-gram with an example.

Answer: An N-gram is a contiguous sequence of N words from a text, used in language

modeling.

Examples:

• Unigram (N=1): "Natural", "Language", "Processing"

• Bigram (N=2): "Natural Language", "Language Processing"

• Trigram (N=3): "Natural Language Processing"

Used in speech recognition, machine translation, and text prediction.

2. What is the limitation of Unsmoothed N-grams?

Answer: Unsmoothed N-grams face the zero probability problem where unseen N-grams
get zero probability, causing:

• Entire sentence probability becomes zero

• Cannot handle new word combinations

• Poor generalization to test data

• Over-dependence on training corpus

Smoothing techniques are needed to solve this issue.

3. What is perplexity in evaluating N-gram models?

Answer: Perplexity measures how well a language model predicts test data. It quantifies
prediction uncertainty.

Formula: PP(W) = P(w₁w₂...wₙ)^(-1/N)

Key Points:

• Lower perplexity = better model

• Represents average branching factor (choices for next word)

• Perplexity of 100 means choosing among 100 equally likely words

4. Mention advantages of smoothing techniques.

Answer: Smoothing addresses zero probability and improves models:

Advantages:

1. Assigns non-zero probabilities to unseen N-grams

2. Better generalization to test data

3. Redistributes probability from seen to unseen events

4. Reduces overfitting to training data

Common methods: Laplace, Good-Turing, Kneser-Ney smoothing.

5. What is interpolation in N-gram models?

Answer: Interpolation combines probabilities from different order N-grams using weighted
averages.

Formula: P(wₙ|wₙ₋₂wₙ₋₁) = λ₃P(trigram) + λ₂P(bigram) + λ₁P(unigram) where λ₁ + λ₂ + λ₃ = 1

Benefits:

• Uses higher-order models when data sufficient

• Falls back to lower-order for sparse data

• Weights learned from held-out data

6. Define backoff strategy in N-gram modeling.

Answer: Backoff uses lower-order N-grams only when higher-order N-grams are unavailable.

Strategy:

• Use trigram if available

• Else back off to bigram with discount factor

• Else back off to unigram

Formula: P(wₙ|context) = P*(trigram) if seen, else α × P(bigram)

Difference from interpolation: Uses only one estimate at a time.

7. List two major word classes in English.

Answer: Two Major Word Classes:

 1. Nouns: Name persons, places, things, or ideas

• Examples: student, Delhi, knowledge

2. Verbs: Express actions, states, or occurrences

• Examples: run, study, is, become

Other classes: adjectives, adverbs, pronouns, prepositions, conjunctions.

8. What is Part-of-Speech tagging?

Answer: PoS tagging assigns grammatical categories to each word based on definition and
context.

Example:

• Input: "The cat sits on the mat"

• Output: The/DT cat/NN sits/VB on/IN the/DT mat/NN

Significance:

• Resolves word ambiguities

• Essential for parsing, information extraction, machine translation

9. What is rule-based PoS tagging?

Answer: Rule-based tagging uses manually crafted linguistic rules for tag assignment.

Approach:

1. Dictionary lookup assigns all possible tags

2. Contextual rules disambiguate based on neighbors

3. Rules applied sequentially

Example Rule: "Tag as noun if word follows determiner and precedes verb"

Pros: High precision, interpretable | Cons: Labor-intensive, limited coverage

10. Define stochastic tagging.

Answer: Stochastic tagging uses statistical models trained on corpora to assign most likely
tags.

Components:
 • Lexical Probability: P(word|tag)

• Contextual Probability: P(tag|previous tags)

• Goal: Maximize P(tags|words)

Models: HMM, MaxEnt, Neural networks Accuracy: 95-97% on benchmarks

11. What is transformation-based learning in PoS tagging?

Answer: TBL (Brill tagging) learns transformation rules to correct initial tag assignments.

Process:

1. Initial tagging with most frequent tag

2. Learn rules to fix errors

3. Apply rules in order

Example Rule: "Change NN to VB if previous word is 'to'"

Advantage: Combines rule-based interpretability with automatic learning.

12. What is the role of HMM in PoS tagging?

Answer: HMM models PoS tagging where tags are hidden states and words are
observations.

Components:

• Hidden States: PoS tags (to infer)

• Observations: Words (visible)

• Transition Prob: P(tag|previous tag)

• Emission Prob: P(word|tag)

Decoding: Viterbi algorithm finds optimal tag sequence.

13. Mention one issue in PoS tagging.

Answer: Ambiguity: Single word can have multiple grammatical categories.

Types:

• Lexical: "book" = noun or verb

 • Syntactic: "Visiting relatives" - is "visiting" adjective or verb?

Challenges:

• Needs contextual information

• Unknown words lack prior tag information

• Some words have 5-7 possible tags

14. Define Maximum Entropy model for PoS tagging.

Answer: MaxEnt is a discriminative classifier that models P(tag|word, context) with

maximum entropy principle.

Features: Uses word suffixes, capitalization, neighboring words

Formula: P(t|w,c) = (1/Z) exp(Σλᵢfᵢ(t,w,c))

Advantages:

• Better sparse data handling than HMM

• Flexible feature integration

• Higher accuracy (1-2% better)

15. State one difference between HMM and MaxEnt.

Answer: HMM (Generative):

HMM can compute the probability for both observed events (like words that

we see in the input) and hidden events (like POS tags).

• Models joint probability P(words, tags)

• Independence assumptions required

• Limited feature representation

MaxEnt (Discriminative):

• Directly models P(tags|words, context)

• No strict independence needed

• Rich, overlapping features possible

• 1-2% higher accuracy( Key: MaxEnt's flexible features make it more accurate.)
 5-Mark Questions (with Bloom's Taxonomy)
1. Explain the working of unsmoothed N-gram language models with suitable examples.
(BT Level: Understand)

Answer:

N-gram Language Models estimate the probability of a word sequence by decomposing it

into conditional probabilities of individual words given their preceding context.

Working Principle:

The probability of a sentence W = w₁w₂...wₙ is calculated as: P(W) = P(w₁) × P(w₂|w₁) ×

P(w₃|w₁w₂) × ... × P(wₙ|w₁...wₙ₋₁)

Since computing full history is impractical, N-gram models use Markov assumption that a
word depends only on the previous (N-1) words:

Unigram Model (N=1): P(W) = P(w₁) × P(w₂) × P(w₃) × ... × P(wₙ)

• Assumes words are independent

• Example: P("I love NLP") = P(I) × P(love) × P(NLP)

Bigram Model (N=2): P(W) = P(w₁) × P(w₂|w₁) × P(w₃|w₂) × ... × P(wₙ|wₙ₋₁)

• Each word depends on previous word

• Example: P("I love NLP") = P(I) × P(love|I) × P(NLP|love)

Trigram Model (N=3): P(wₙ|wₙ₋₂wₙ₋₁) - word depends on previous two words

Probability Estimation from Corpus:

For bigrams: P(wₙ|wₙ₋₁) = Count(wₙ₋₁wₙ) / Count(wₙ₋₁)

Example Calculation:

Given corpus: "I love NLP. I love AI. I enjoy NLP."

Count statistics:

• Count(I love) = 2

• Count(I enjoy) = 1

• Count(I) = 3

• Count(love NLP) = 1

• Count(love) = 2

Bigram probabilities:
 • P(love|I) = 2/3 = 0.67

• P(enjoy|I) = 1/3 = 0.33

• P(NLP|love) = 1/2 = 0.5

Limitation: Unsmoothed models assign zero probability to unseen N-grams, causing the
entire sentence probability to become zero. This makes them impractical for real
applications without smoothing.

2. How do you evaluate an N-gram model? Explain the use of perplexity. (BT
Level: Apply)
Answer: Evaluation Methods for N-gram Models:

N-gram models are evaluated using two approaches: extrinsic and intrinsic evaluation.

1. Extrinsic Evaluation:

• Embed the language model in an application (speech recognition, machine

translation)

• Measure end-task performance improvement

• Limitation: Time-consuming and application-specific

2. Intrinsic Evaluation:

• Evaluate model quality independent of applications

• Uses metrics like perplexity

Perplexity - Primary Evaluation Metric:

Definition: Perplexity measures how well a probability model predicts a test sample. It
quantifies the model's uncertainty in predicting the next word.

Formula:

PP(W) = P(w₁w₂...wₙ)^(-1/N)

Or equivalently:

PP(W) = ᴺ√(1/P(w₁w₂...wₙ))

Interpretation:

• Lower perplexity indicates better model performance

 • Perplexity represents the weighted average branching factor

• A perplexity of K means the model is as confused as if it had to choose uniformly

among K words at each step

Example Calculation:

Suppose we have a test sentence: "I love NLP"

Bigram probabilities:

• P(I|) = 0.3

• P(love|I) = 0.2

• P(NLP|love) = 0.1

• P(|NLP) = 0.5

P(sentence) = 0.3 × 0.2 × 0.1 × 0.5 = 0.003

For N=4 words: PP = (1/0.003)^(1/4) = (333.33)^0.25 ≈ 4.27

Practical Application:

Comparing models:

• Model A: Perplexity = 150

• Model B: Perplexity = 120

• Model B is better (lower perplexity)

Relationship to Entropy: Perplexity = 2^H(W), where H is entropy

Key Points:

• Training set used to estimate probabilities

• Test set (held-out data) used to compute perplexity

• Lower perplexity = better prediction = better model

• Valid comparison only on same test set

3. Describe various smoothing techniques used in N-gram models. (BT Level:

Understand)

Answer:
Smoothing techniques address the zero probability problem in N-gram models by
redistributing probability mass from seen to unseen events.
Major Smoothing Techniques:

1. Add-One (Laplace) Smoothing:

Adds 1 to all N-gram counts before probability calculation.

Formula: P_Laplace(wₙ|wₙ₋₁) = (Count(wₙ₋₁wₙ) + 1) / (Count(wₙ₋₁) + V)

where V = vocabulary size

Example:

• Count(love NLP) = 0, Count(love) = 2, V = 1000

• P(NLP|love) = (0+1)/(2+1000) = 1/1002 ≈ 0.001

Limitation: Too much probability mass given to unseen events; poor performance on large
vocabularies.

2. Add-k (Additive) Smoothing:

Generalization of Laplace where k < 1.

Formula: P(wₙ|wₙ₋₁) = (Count(wₙ₋₁wₙ) + k) / (Count(wₙ₋₁) + kV)

3. Good-Turing Smoothing:

Uses frequency of frequency information to re-estimate counts.

Principle:

• Count of things seen r times is re-estimated using count of things seen (r+1) times

• Formula: c* = (c+1) × N_{c+1} / N_c

where N_c = number of N-grams occurring c times

4. Kneser-Ney Smoothing:

Most effective smoothing method; uses absolute discounting and considers continuation
probability.

Key Idea:

• Discount fixed amount (d) from seen counts

• Distribute discounted mass based on how likely word appears in novel contexts

Formula: P_KN(wₙ|wₙ₋₁) = max(Count(wₙ₋₁wₙ)-d, 0)/Count(wₙ₋₁) + λ(wₙ₋₁)P_continuation(wₙ)

5. Interpolation:

Combines multiple N-gram models using weighted average (discussed in detail in Q4).

6. Backoff:
Uses lower-order model only when higher-order unavailable (discussed in detail in Q4).

Comparison:

• Laplace: Simple but crude

• Good-Turing: Better redistribution

• Kneser-Ney: State-of-the-art, best performance

• Interpolation/Backoff: Combine different order models

Why Smoothing is Essential:

• Real data is sparse; many valid N-grams never appear in training

• Without smoothing, model assigns zero to test sentences with unseen N-grams

• Smoothing improves generalization and robustness

4. Compare interpolation and backoff methods in language modeling. (BT

Level: Analyze)
Answer:
Both interpolation and backoff are smoothing strategies that leverage lower-order N-gram
models, but they differ fundamentally in approach.

INTERPOLATION:

Definition: Combines probability estimates from all N-gram orders using weighted linear
combination.

Formula (Trigram): P_interp(wₙ|wₙ₋₂wₙ₋₁) = λ₃P(wₙ|wₙ₋₂wₙ₋₁) + λ₂P(wₙ|wₙ₋₁) + λ₁P(wₙ)

where λ₁ + λ₂ + λ₃ = 1 and all λ > 0

Characteristics:

• Always uses all models: Combines trigram, bigram, and unigram

• Weights are fixed: λ values learned from held-out data

• Continuous blending: Even for frequent N-grams, lower-order models contribute

Example: For "P(NLP | love machine)"

• λ₃ × P(NLP|love,machine) + λ₂ × P(NLP|machine) + λ₁ × P(NLP)

 • All three terms computed and combined

BACKOFF:

Definition: Uses lower-order N-gram only when higher-order N-gram is unavailable or

unreliable.

Formula (Katz Backoff): P_backoff(wₙ|wₙ₋₂wₙ₋₁) =

• P*(wₙ|wₙ₋₂wₙ₋₁) if Count(wₙ₋₂wₙ₋₁wₙ) > 0

• α(wₙ₋₂wₙ₋₁) × P_backoff(wₙ|wₙ₋₁) otherwise

Characteristics:

• Conditional usage: Uses one probability estimate at a time

• Discounting: Reduces probability of seen N-grams (P*) to reserve mass for unseen

• Recursive: Backs off through orders until finding evidence

Example: For "P(NLP | love machine)"

• If trigram seen: Use P*(NLP|love,machine) only

• If not: Use α × P_backoff(NLP|machine)

• Recurse if needed to unigram

COMPARISON TABLE:

Aspect Interpolation Backoff

Strategy Always combine all orders Use lower order only if needed

Computation All models computed every time Conditional computation

Weights Fixed λ values Dynamic α based on context

Smoothness Smooth blending Sharp transitions

Complexity Simpler formula More complex with discounting

Performance Generally better Slightly less accurate

Efficiency Less efficient More efficient

When to Use:

• Interpolation: When computational resources available; slightly better performance

• Backoff: When efficiency matters; storage constrained scenarios

Example Scenario:

Training corpus: "I love AI. I love NLP."

For test phrase "I love machine learning":

• Interpolation: Combines P(learning|machine), P(learning|love), P(learning)

• Backoff: Tries P(learning|love,machine) → fails → tries P(learning|machine) → uses

it

Both methods significantly outperform unsmoothed models by handling unseen N-grams

effectively.

5. Discuss the classification of words into word classes with examples. (BT
Level: Understand)
Answer:

Words are classified into word classes or Parts of Speech (PoS) based on their grammatical
function, syntactic behavior, and semantic properties.

MAJOR WORD CLASSES:

1. OPEN WORD CLASSES (Lexical Categories):

New members can be added as language evolves.

a) Nouns (N/NN):

• Name persons, places, things, concepts

• Examples: student, Delhi, artificial intelligence, happiness

• Subtypes:

• Proper nouns: India, Shakespeare

• Common nouns: book, computer

• Abstract nouns: knowledge, freedom

• Collective nouns: team, data

b) Verbs (V/VB):

• Express actions, states, occurrences

• Examples: run, study, is, become, analyze

• Subtypes:

• Action verbs: write, compute

• Linking verbs: is, seems

• Auxiliary verbs: has, will, should

• Modal verbs: can, must, might

c) Adjectives (ADJ/JJ):

• Describe or modify nouns

• Examples: intelligent, large, neural, efficient

• Subtypes:

• Descriptive: beautiful, complex

• Quantitative: many, few

• Demonstrative: this, that

• Comparative: better, larger

d) Adverbs (ADV/RB):

• Modify verbs, adjectives, or other adverbs

• Examples: quickly, very, accurately, often

• Subtypes:

• Manner: carefully, efficiently

• Time: now, yesterday

• Frequency: always, rarely

• Degree: very, extremely

2. CLOSED WORD CLASSES (Functional Categories):

Fixed membership; rarely add new members.

a) Pronouns (PRP):

• Replace nouns
 • Examples: I, you, he, she, it, they, this

• Personal, possessive, reflexive, demonstrative

b) Determiners (DT):

• Specify nouns

• Examples: the, a, an, this, that, some, every

• Articles, demonstratives, quantifiers

c) Prepositions (IN):

• Show relationships between words

• Examples: in, on, at, by, with, from, to

d) Conjunctions (CC):

• Connect words, phrases, clauses

• Examples: and, but, or, because, although

• Coordinating, subordinating, correlative

e) Interjections (UH):

• Express emotions

• Examples: oh, wow, ouch, hey

3. SPECIAL CATEGORIES:

a) Numerals:

• Cardinal: one, two, three

• Ordinal: first, second, third

b) Particles:

• Verb particles: up (in "give up")

• Negative particles: not, n't

CLASSIFICATION CRITERIA:

1. Distributional: Where the word can appear in sentences

2. Morphological: What inflections/affixes it takes

3. Semantic: What meaning category it represents

4. Syntactic: What role it plays in sentence structure

Example Analysis:

Sentence: "The intelligent student quickly solved the complex problem."

• The → Determiner

• intelligent → Adjective (modifies student)

• student → Noun (subject)

• quickly → Adverb (modifies solved)

• solved → Verb (action)

• the → Determiner

• complex → Adjective (modifies problem)

• problem → Noun (object)

Importance in NLP:

• Foundation for syntactic parsing

• Essential for information extraction

• Crucial for machine translation

• Key to semantic analysis

6. What are the different approaches to Part-of-Speech tagging? Explain

briefly. (BT Level: Understand)
Answer:

Part-of-Speech (PoS) Tagging – Overview

• Definition:
Part-of-Speech (PoS) tagging is the process of assigning grammatical categories (like
noun, verb, adjective, etc.) to each word in a sentence based on its definition and
context.

Example:
Sentence: “The cat sat on the mat.”
→ [The/DT, cat/NN, sat/VBD, on/IN, the/DT, mat/NN]

• PoS tagging helps in understanding sentence structure, syntax, and meaning, which
are essential for tasks like parsing, translation, and information extraction.
Main Approaches to PoS Tagging

There are three major approaches to PoS tagging:

1. Rule-Based PoS Tagging

• Concept:
Uses a dictionary (lexicon) and a set of manually written linguistic rules to assign
correct tags to words.

• Working:

1. Each word is first assigned all possible tags from the lexicon.

2. Then, contextual rules are applied to disambiguate and select the correct tag.

▪ Example: If the previous word is an article (“a”, “the”), the next word is
likely a noun.

• Example:
Sentence: “A book”

o “A” → Article

o “Book” → Could be Noun or Verb

o Rule: If word follows an article → assign Noun

→ Output: [(A, Article), (Book, Noun)]

• Features:

o Based on linguistic knowledge.

o Around 1000 rules are usually used.

o High accuracy for restricted domains.

o Limited adaptability and requires manual effort.

2. Stochastic (Statistical) PoS Tagging

• Concept:
Uses probabilities and statistics from a large annotated corpus to assign the most
likely tag to each word.

• Working Approaches:
 1. Word Frequency Approach:
Assigns the most frequent tag seen for a word in the training data.

▪ Example: If “book” appears 10 times — 6 as Noun, 4 as Verb, → tag as

Noun.

2. Tag Sequence Probability (N-gram) Approach:

Uses the probability of a tag given the previous tags (e.g., bigram or trigram
models).

▪ Formula:
P(tag sequence | words) = P(t₁) × P(t₂ | t₁) × P(t₃ | t₂) …

• Features:

o Data-driven, uses models like Hidden Markov Models (HMM).

o Handles ambiguity statistically.

o Requires a large, annotated training corpus.

o Struggles with unseen (out-of-vocabulary) words.

3. Transformation-Based (Brill’s) Tagging

• Concept:
A hybrid approach combining rule-based and stochastic ideas.
It starts with an initial tagging (often based on frequency) and iteratively applies
learned correction rules.

• Working:

1. Assign initial tags using a simple method (like most frequent tag).

2. Apply a series of transformation rules to fix errors.

▪ Example: If a word is tagged as “Verb” but follows a determiner,

change it to “Noun”.

3. Repeat until no significant improvement occurs.

• Features:

o Learns rules automatically (machine learning-based).

o Easy to interpret and debug.

o More accurate than purely rule-based systems.

o Slower training time for large corpora.

Summary Table

Data
Approach Basis Accuracy Example Model
Requirement

High for
Handcrafted No corpus Rule sets (lexical
Rule-Based known
linguistic rules required + contextual)
patterns

High (with
Statistical Requires Hidden Markov
Stochastic sufficient
probabilities annotated corpus Model
data)

Transformation- Combination of
Moderate corpus Very High Brill’s Tagger
Based rules and learning

Conclusion

Each PoS tagging approach has its own strengths:

• Rule-based is interpretable but rigid,

• Stochastic is adaptive but data-dependent, and

• Transformation-based balances both by learning correction rules automatically.

In modern NLP, hybrid models and deep learning taggers are commonly used for better
accuracy and adaptability.

7. Describe rule-based and stochastic PoS tagging techniques with

differences. (BT Level: Analyze)
ANS:Rule-Based PoS Tagging

• Definition:
Rule-based Part-of-Speech (PoS) tagging is one of the earliest techniques that assigns
tags to words based on a set of handcrafted linguistic rules and a lexicon (dictionary).
 • Working Principle:

1. Lexical Lookup: Each word is assigned a list of possible tags from a dictionary.

2. Rule Application: Handwritten disambiguation rules are applied to select the

most appropriate tag based on context.

▪ Example: If a word follows an article (a, an, the), it must be a noun.

3. The rules may use context-patterns or regular expressions compiled into

finite-state automata.

• Architecture:

o Stage 1: Assign all possible tags using a lexicon.

o Stage 2: Apply syntactic or contextual rules to remove unlikely tags.

• Example:
Sentence: “A Book”

o “A” → Article (DT)

o “Book” → Could be Noun or Verb

o Rule: If a word follows an article, it is likely a noun → “Book” → Noun

→ Final tags: [(A, Article), (Book, Noun)]

• Properties:

o Knowledge-driven, manually coded rules (~1000).

o Explicit language modeling and smoothing.

o Accurate for simple, structured text but limited flexibility for unseen words.

Stochastic PoS Tagging

• Definition:
Stochastic PoS tagging uses probability or frequency information derived from a
labeled corpus to assign the most likely tag to each word.

• Working Principle:

o Tags are assigned based on statistical models such as Hidden Markov Models
(HMMs) or N-gram models.

o It predicts tags using:

 1. Word Frequency Approach: Chooses the most frequent tag of a word
from training data.
Example: If “book” appears 10 times — 6 as Noun, 4 as Verb, → assign
Noun.

2. Tag Sequence Probability Approach: Chooses the tag sequence with

the highest probability using previous tags (bigram/trigram approach).

• Example:
Using bigram probabilities:
P(tag sequence | words) = P(tag₁) × P(tag₂ | tag₁) × P(tag₃ | tag₂) …

• Properties:

o Data-driven and corpus-based.

o Requires large annotated training data.

o Automatically handles ambiguity but fails for unseen (out-of-vocabulary)

words.

o Often implemented using HMM or Maximum Entropy models.

Difference Between Rule-Based and Stochastic PoS Tagging

Feature Rule-Based PoS Tagging Stochastic PoS Tagging

Basis Uses handcrafted linguistic rules. Uses probabilistic/statistical models.

Data Requires a large annotated corpus for

No need for large training data.
Requirement training.

Uses probability of tag occurrence or

Disambiguation Uses contextual linguistic rules.
sequence.

Difficult to adapt to new Easily adaptable through retraining on

Adaptability
languages or domains. new data.

High for well-defined rules and

Accuracy Generally higher with sufficient data.
small domains.

Example
If word after “the” → Noun. P(tag
Approach
Feature Rule-Based PoS Tagging Stochastic PoS Tagging

Manual rule creation and

Human Effort Automatic learning from data.
maintenance.

Analytical Conclusion

Rule-based PoS tagging depends on human linguistic knowledge, making it interpretable but
rigid, while stochastic tagging leverages statistical learning, making it more robust and
scalable for large, real-world corpora.
In practice, hybrid models combining both methods (like Transformation-Based Tagging)
often achieve the best performance.

8. What are the main challenges in PoS tagging? How are they typically addressed? (BT
Level: Evaluate)

ANS
Main Challenges in PoS Tagging:

1. Word Class Ambiguity:

Many words belong to multiple classes depending on context.
Example: “Book” (Noun – a book) or “Book” (Verb – to book a ticket).
→ This contextual ambiguity is the primary challenge in PoS tagging.

2. Context Dependency:
The correct tag often depends on surrounding words.
Example: “The police books the culprit” – “books” is a verb, not a plural noun.

3. Unknown or Out-of-Vocabulary Words:

Words absent in the training corpus (like names or new terms) cause tagging failures due to zero
probabilities.

4. Inadmissible Tag Sequences:

Simple frequency-based tagging may generate grammatically impossible tag combinations (e.g.,
Determiner–Verb).

5. Rare or Complex Grammatical Structures:

Uncommon syntactic patterns are poorly represented in training data, making N-gram taggers less
reliable.

6. Tagset Granularity:
Choosing between coarse-grained and fine-grained tagsets (like the 45-tag UPenn TreeBank) affects
disambiguation and complexity.
 7. Corpus & Computational Dependence:
Statistical and transformation-based taggers need large annotated corpora and high computational
resources.

How These Challenges Are Addressed:

1. Rule-Based Tagging:
Uses linguistic rules and lexicons (e.g., If previous word is an article → next is likely a noun).
Good for structured grammar but limited by manual rule creation.

2. Stochastic (Statistical) Tagging:

Employs word and tag sequence probabilities (HMM, N-gram).
Learns from data but struggles with rare or unseen words.

3. Transformation-Based Tagging (Brill Tagger):

Starts with statistical tagging, then iteratively applies learned correction rules.
Handles complex structures more accurately.

4. Smoothing & Backoff Techniques:

Handle data sparsity and unknown words by redistributing probability mass (e.g., Laplace
smoothing).

5. Hybrid & Neural Approaches:

Combine rule-based and probabilistic models or use neural networks (BiLSTM, Transformer) to learn
long-range context.

In summary:
Part-of-Speech tagging is challenged by ambiguity, unknown words, and contextual variation. These issues are
addressed through rule-based, statistical, transformation-based, and hybrid models, supported by smoothing
techniques and large annotated corpora to improve accuracy and generalization

Q9. Explain how Hidden Markov Models are used for Part-of-Speech (PoS) Tagging. (BT
Level: Apply)
A Hidden Markov Model (HMM) is a stochastic (probabilistic) model used in Natural Language Processing to
assign the most likely sequence of part-of-speech (POS) tags to a sequence of words in a sentence.

Concept:

In HMM-based POS tagging:

• The words in a sentence are observations (visible).

• The POS tags are the hidden states.

• The goal is to find the most probable sequence of tags for the given sequence of words.

Components of HMM:

1. States: POS tags (e.g., Noun, Verb, Adjective, etc.)

2. Observations: Words in the sentence.

 3. Transition Probability (A):
P(ti∣ti−1)P(t_i | t_{i-1})P(ti∣ti−1) — Probability of a tag given the previous tag.

4. Emission Probability (B):

P(wi∣ti)P(w_i | t_i)P(wi∣ti) — Probability of a word given a tag.

5. Initial Probability (π):

Probability of starting with a particular tag.

Working:

Given a sequence of words W=w1,w2,…,wnW = w_1, w_2, …, w_nW=w1,w2,…,wn, we want to find the tag
sequence T=t1,t2,…,tnT = t_1, t_2, …, t_nT=t1,t2,…,tn that maximizes:

P(T∣W)=P(W∣T)×P(T)P(T|W) = P(W|T) \times P(T)P(T∣W)=P(W∣T)×P(T)

This is computed efficiently using the Viterbi algorithm, which finds the most likely sequence of tags.

Example:

Sentence: Book that flight

Possible Tags:

• “Book” → Noun / Verb

• “that” → Conjunction / Determiner

• “flight” → Noun

The HMM uses transition and emission probabilities to determine that the most probable tag sequence is:
Verb – Determiner – Noun

Advantages:

• Based on solid probabilistic foundations.

• Performs well (over 90% accuracy).

• Easy to train using tagged corpora.

Limitations:

• Assumes the current state depends only on the previous one (Markov assumption).

• Cannot easily model long-range dependencies.

In short:
HMM-based POS tagging predicts the most probable sequence of tags for words using transition and emission
probabilities, applying the Viterbi algorithm for efficient decoding.

Q10. Write a note on Maximum Entropy approach in PoS tagging. (BT Level: Understand)
ANS:The Maximum Entropy (MaxEnt) approach is a statistical (probabilistic) model used for Part-of-
Speech (PoS) tagging.
It is a log-linear model that predicts the most probable tag for a word based on multiple contextual
features, without making strong assumptions about the data distribution.

Concept:

• The MaxEnt model selects the tag sequence that has the maximum entropy (i.e., the most
uniform distribution) among all possible distributions satisfying the known constraints from
training data.

• It uses various features like:

o Current word

o Surrounding words

o Prefixes, suffixes, capitalization, etc.

o Previous and next tags

The probability of a tag ttt given context xxx is modeled as:

where fi(x,t)f_i(x,t)fi(x,t) are feature functions, wiw_iwi are learned weights, and Z(x)Z(x)Z(x) is a
normalization factor.

Advantages:

• Can handle overlapping and correlated features.

• Does not assume independence between features (unlike HMM).

• Provides probabilistic interpretation and flexibility in feature selection.

• Works well with limited linguistic assumptions.

Limitations:

• Training is computationally expensive.

• Requires large annotated data for accurate weight estimation.

• May struggle with sequential dependencies (later addressed by MaxEnt Markov Models or
CRFs).

Applications:
Used in PoS tagging, named entity recognition, text classification, and sentiment analysis — where
contextual and linguistic features are combined for accurate prediction.

In summary:
The Maximum Entropy approach to PoS tagging is a flexible, feature-based statistical method that
combines different contextual clues to assign the most probable tag to each word while maintaining
maximum uncertainty consistent with the training data.

Q11. Compare and Contrast HMM and Maximum Entropy Models for PoS Tagging (BT
Level: Analyze)
1. Overview:

Both Hidden Markov Model (HMM) and Maximum Entropy (MaxEnt) are statistical approaches used for Part-
of-Speech (PoS) tagging, but they differ in their assumptions, features, and handling of context.

Comparison Table:

Feature Hidden Markov Model (HMM) Maximum Entropy Model (MaxEnt)

Generative (models joint probability ( P(W, Discriminative (models conditional probability (

Model Type
T) )) P(T

Follows Markov assumption – current tag Makes no independence assumptions

Assumption
depends only on the previous tag between features

Input Uses feature functions (word form, suffixes,

Uses transition and emission probabilities
Representation capitalization, neighboring words, etc.)

Finds the most probable tag sequence Computes tag probabilities using a log-linear
Output
using Viterbi algorithm (exponential) model

Handling Rich – considers multiple contextual and lexical

Limited – depends on previous tag(s) only
Context features simultaneously

Training Data Requires annotated corpus to estimate Requires annotated corpus to learn feature
Need probabilities weights via maximum likelihood

- Flexible and feature-rich - Handles

- Simple, mathematically elegant - Easy to
Advantages overlapping, non-independent features -
train - Performs well with small data
Better accuracy in complex structures

- Relies on strong independence - Computationally expensive to train - Needs

Limitations assumptions - Struggles with unseen large, diverse corpus - Does not inherently
words - Limited contextual understanding handle tag sequence dependencies
Feature Hidden Markov Model (HMM) Maximum Entropy Model (MaxEnt)

Example Generalized Iterative Scaling / Gradient

Viterbi Algorithm
Algorithm Descent

In Summary:

• HMM focuses on the probability of generating both words and tags (generative).

• MaxEnt focuses directly on the probability of tags given words (discriminative).

• HMM is simpler but less flexible; MaxEnt is more powerful and accurate for complex, context-rich
tagging tasks.

Q12. Describe the architecture of a Transformation-Based PoS Tagger with an example. (BT
Level: Apply)

Introduction:
A Transformation-Based PoS Tagger (TBT), also known as the Brill Tagger, is a hybrid approach that combines
rule-based and machine learning methods.
It starts with a simple tagging (like the most frequent tag) and iteratively learns correction rules to improve
accuracy.

Architecture of Transformation-Based Tagger:

1. Initialization Stage:

o Each word is assigned an initial tag, usually the most frequent tag from the training data or
based on a lexicon.

2. Learning Stage:

o The tagger compares the initial tags with the correct tags in the annotated training corpus.

o It identifies errors and generates transformation rules (context-based corrections).

o Example of a rule:
“If a word is tagged as NN (noun) but the previous word is ‘to’, change the tag to VB (verb).”

3. Rule Selection:

o Each possible rule is evaluated to determine how many errors it corrects and how many new
errors it introduces.

o The rule with the highest net gain (maximum improvement) is selected.

4. Iteration:

o The selected rule is applied to the corpus.

 o The process repeats until no further improvement is possible or performance stabilizes.

5. Tagging Stage:

o During tagging, the same learned transformation rules are applied sequentially to new,
unseen text.

Example:

Sentence: “The can will rust.”

1. Initial tagging:

o “The/DT can/NN will/NN rust/VB”

2. Rule applied:

o If a word is tagged as NN but preceded by a modal (MD), change tag to VB.

3. After transformation:

o “The/DT can/MD will/VB rust/VB”

Advantages:

• Combines human-readable rules with machine learning.

• High accuracy with fewer rules.

• Easy to debug and interpret.

Limitations:

• Training is computationally intensive.

• Does not output tag probabilities.

In summary:
Transformation-Based PoS Tagging is an error-driven learning approach that starts with simple tagging and
iteratively applies learned correction rules to improve accuracy, effectively blending rule-based and statistical
methods.

Q13. Illustrate with an example how interpolation improves the performance of N-gram
models. (BT Level: Apply)
Concept:

Interpolation is a smoothing technique used in N-gram language models to improve prediction accuracy by
combining probabilities from different N-gram orders (unigram, bigram, trigram, etc.) instead of relying on just
one.
This helps when higher-order N-grams have insufficient data or zero counts.

Advantages:

• Prevents zero probabilities for unseen word combinations.

• Balances between specific context (higher N-gram) and general trends (lower N-gram).

• Improves model robustness and accuracy on unseen data.

In summary:
Interpolation improves N-gram model performance by combining probabilities from multiple N-gram levels,
giving more reliable estimates when data is sparse.
3. Backoff & Interpolation:
Use lower-order N-gram probabilities (like bigram → unigram) when higher-order counts are missing.
Importance:

• Prevents zero-probability errors on unseen data.

• Improves generalization to new sentences.

• Reduces overfitting to the training corpus.

• Ensures robust performance in real-world applications like speech recognition, machine translation,
and text prediction.

In summary:
Smoothing is essential because real-world language is vast and unpredictable. It ensures that unseen words or
combinations receive small but non-zero probabilities, making the model more realistic, reliable, and accurate.

Q15. How do sparsity and ambiguity affect the accuracy of PoS tagging systems? (BT Level:
Analyze)
ANS:

1. Data Sparsity:

Definition:
Data sparsity occurs when many word–tag or tag–sequence combinations are rare or unseen in the training
corpus.

Impact on PoS Tagging:

• Statistical models (like HMMs) rely on word–tag probabilities from training data.

• Unseen words or tag sequences are assigned zero probability, leading to incorrect tagging.

• Reduces the ability of the model to generalize to new text.

Example:
If “tweeted” never appeared in training data, the model cannot assign a proper tag (Verb), even though it’s
grammatically correct.

Mitigation Techniques:

• Smoothing (Add-1, Good-Turing, Backoff) to assign small non-zero probabilities.

• Larger, domain-diverse corpora and morphological analysis to infer tags for unknown words.

2. Ambiguity:

Definition:
Ambiguity arises when a word can belong to multiple parts of speech depending on context.

Impact on PoS Tagging:

• Leads to confusion in selecting the correct tag without sufficient contextual understanding.

• Causes tagging errors in sentences where syntax or semantics are unclear.

Example:

• “Book” → Noun in “I read a book.”

• “Book” → Verb in “I will book a ticket.”

Mitigation Techniques:

• Contextual models like HMM, MaxEnt, or neural taggers (BiLSTM, Transformer) that use surrounding
words to infer meaning.

• Rule-based or hybrid systems to apply linguistic context rules.

Summary:

• Sparsity affects accuracy by limiting the model’s exposure to possible tag combinations.

• Ambiguity affects accuracy by making tag decisions context-dependent.

Both must be handled using probabilistic, rule-based, and neural methods to build robust PoS
taggers.