Levels in NLP

• Levels represent the hierarchical stages of

linguistic analysis required to understand human
language computationally.
 Levels in NLP
• Phonology:
• Deals with the sound structure of language, including how sounds are organized
and pronounced.
• Morphology:
• Focuses on the internal structure of words, breaking them down into meaningful
units (morphemes) and understanding how they combine.
• Lexicon:
• Involves the vocabulary of a language, including the meanings of individual
words.
 Levels in NLP
• Syntax:
• Analyzes the grammatical structure of sentences, examining how words are
combined to form phrases and clauses.
• Semantics:
• Deals with the meaning of words and sentences, including relationships between
words and how meaning is conveyed.
• Pragmatics:
• Considers the context of language use, including how meaning is affected by the
speaker, listener, and situation.
 Levels in NLP

• Discourse:
• Focuses on how sentences are connected to form larger
units of text, such as paragraphs or conversations, and
how meaning is created across these units.
Level Focus Example Key Challenge
Phonetics/ Sounds & pronunciation "cat" = /kæt/ vs. "bat" = Accents, speech
Phonology /bæt/ recognition noise

Morpholog Word structure & "un+happy+ness" → 3 Handling

y formation morphemes agglutinative
languages (e.g.,
Turkish)
Syntax Sentence structure & "The cat sat" (S → NP + Ambiguity: "I saw the
grammar VP) man with glasses"

Semantics Meaning of "Bank" (financial vs. river) Context-dependent

words/sentences word meanings

Pragmatic Context & implied "It's cold here" → Hint to Cultural/subtextual

s meaning close window interpretation
 Complexities at each level of NLP
• Lexical Analysis:
• Tokenizing text (splitting it into words or phrases) is not
always straightforward. Contractions, hyphenated
words, and punctuation can pose challenges
• Example:
• Dealing with "can't" (contraction), "well-being"
(hyphenated), or "!" (punctuation) requires specific rules
or algorithms
 Complexities at each level of NLP
• Syntactic Analysis:
• Determining the grammatical structure of a sentence (subject,
verb, object, etc.) and identifying relationships between words
is crucial. Ambiguity in sentence structure (e.g., "I saw the man
with the telescope") can be difficult to resolve.
• Example:
• Disambiguating the phrase "old men and women" to either "old
(men and women)" or "(old men) and women" requires
understanding grammatical rules and context.
 Complexities at each level of NLP
• Semantic Analysis:
• Understanding the meaning of words and sentences, including
identifying relationships between concepts and handling
ambiguity. Words can have multiple meanings (polysemy), and
context is essential for disambiguation.
• Example:
• Determining the meaning of "bank" (financial institution or
riverbank) requires considering the surrounding words and
context.
 Complexities at each level of NLP
• Discourse Integration:
• Understanding how sentences relate to each other
within a larger text or conversation. This involves
resolving anaphora (referring back to previously
mentioned entities) and cataphora (referring forward to
entities).
• Example:
• Identifying that "he" in a sentence refers to a specific
 Complexities at each level of NLP
• Pragmatic Analysis:
• Understanding the speaker's intent, the context of the situation,
and how language is used in real-world interactions. This
includes interpreting idioms, sarcasm, and other non-literal
language.
• Example:
• Recognizing that the phrase "Break a leg!" is not a literal
request to break someone's leg, but rather a wish for good luck.
 Additional Challenges
• Data Sparsity and Quality:
• NLP models require large amounts of labeled data, but
obtaining high-quality data can be challenging and time-
consuming.
• Context and Understanding:
• Capturing and representing context accurately is a
persistent challenge in NLP
 Additional Challenges
• Computational Complexity:
• Some NLP tasks, like machine translation or large-scale
text analysis, can be computationally expensive.
• Ambiguity and Polysemy:
• As mentioned earlier, words and phrases often have
multiple meanings, requiring context to understand the
intended meaning
 Morphology
• Morpheme: The smallest meaningful unit of language. Cannot
be divided further without losing meaning.
• Free Morpheme: Can stand alone as a word (e.g., dog, run,
happy).
• Bound Morpheme: Must be attached to another morpheme
(e.g., prefixes, suffixes: un-, -ly, -s, -ed).
• Affix: A bound morpheme attached to a base (root/stem).
Types: Prefixes (before), Suffixes (after), Infixes (within - rare in
English), Circumfixes (around - rare in English).
 Derivational Morphology
• Purpose: Creates new words, often with a change in meaning
and/or grammatical category (part of speech).
• Function:
• Changes word class (e.g., verb -> noun: teach (V) + -er ->
teacher (N); adjective -> verb: modern (Adj) + -ize -> modernize
(V)).
• Changes meaning (e.g., do -> undo; kind -> unkind; friend ->
friendship).
• Can be used to form nouns, verbs, adjectives, adverbs from
 Characteristics
• Productivity: Some derivational affixes are highly productive
(e.g., -ness, -able), others are less so (e.g., -th as in warmth).
• Position: Usually applied closer to the root than inflectional
affixes.
• Lexical Meaning: Adds significant semantic content.
• Optional: Not grammatically required; creates new dictionary
entries.
• Can change phonology: May trigger sound changes in the
root/base (e.g., electric -> electricity /k/ -> /s/; produce (V) /u:/ -
 Examples
• happy (Adj) + -ness -> happiness (N) - Category Change
• agree (V) + -able -> agreeable (Adj) - Category Change
• nation (N) + -al -> national (Adj) - Category Change
• national (Adj) + -ize -> nationalize (V) - Category Change
• quick (Adj) + -ly -> quickly (Adv) - Category Change
• un- + do (V) -> undo (V) - Meaning Change (Same Category)
• re- + write (V) -> rewrite (V) - Meaning Change
• friend (N) + -ship -> friendship (N) - Meaning Change (Same
Category)
 Inflectional Morphology
• Purpose: Adds grammatical information to existing
words without changing their core meaning or
grammatical category. Signals grammatical relationships
in a sentence.
 Inflectional Morphology
• Function: Marks features like:
• Nouns: Number (cat -> cat+s), Possessive Case (cat ->
cat+'s).
• Verbs: Tense (walk -> walk+ed), Aspect (walk ->
walk+ing), Agreement (3rd person singular: walk ->
walk+s).
• Adjectives/Adverbs: Degree of Comparison (fast ->
 Characteristics
• Obligatory: Often required by the grammatical context of the
sentence.

• Position: Applied after derivational affixes (to the outermost part

of the word).

• Grammatical Meaning: Adds syntactic information (tense,

number, case, comparison).

• Does NOT change category: cat (N) -> cats (N); walk (V) ->
 Characteristics
• Productivity: Highly productive within their grammatical class
(e.g., almost all verbs take -ed for past tense, with some
irregular exceptions).

• Closed Set: English has a very small, finite set of inflectional

suffixes

• Regular Forms: Usually follow predictable patterns (though

many common words are irregular: go/went, child/children,
 Examples (The "Big 8" English
Inflectional Suffixes)
• -s 3rd person singular present verb (He walk+s).

• -ed Past tense verb (They walk+ed).

• -ing Progressive aspect verb (She is walk+ing).

• -en Past participle verb (Irregular/Alternative: eat/en, take/n;

Regular: walk+ed).
 Examples (The "Big 8" English
Inflectional Suffixes)
• -s Plural noun (cat+s).

• -'s Possessive noun (the cat+'s bowl).

• -er Comparative adjective/adverb (fast+er).

• -est Superlative adjective/adverb (fast+est).

Feature Derivational Inflectional
Morphology Morphology
Primary Purpose Create new words / Add grammatical
Lexemes information
Changes Meaning? Often significantly Usually minimally
(grammatical)
Changes Category? Yes (e.g., V -> N, Adj No
-> Adv)
Position Closer to the root At the very end
(outer layer)
Productivity Variable (some high, High (within its
some low) class)
Feature Derivational Inflectional
Morphology Morphology
Obligatory? No (Optional word Often Yes (Required
formation) by grammar)
Affix Type Prefixes & Suffixes Suffixes Only (in
English)
Creates new entry? Yes (New dictionary No (Forms of
word) existing word)
Examples un+happy, teach+er, cat+s, walk+ed,
modern+ize fast+er, talk+ing
 Introduction to POS Tagging
• Part-of-Speech (POS) Tagging: Assigns grammatical categories
(e.g., noun, verb, adjective) to words in a sentence.

• Importance:

• Foundation for NLP tasks (parsing, machine translation,

sentiment analysis).

• Resolves ambiguity (e.g., "book" can be a noun or verb).

 Rule-Based Taggers
• Concept: Use handcrafted linguistic rules.

• Examples:

• ENGTWOL (1983): Used ~1,100 constraints (e.g., "If a word is

preceded by a determiner, it cannot be a verb").

• Workflow:

• Assign all possible tags to each word (using a dictionary).

• Apply constraints to eliminate invalid tags.

 Rule-Based Taggers
• Strengths:

• High precision with clear rules.

• No training data needed.

• Limitations:

• Labor-intensive to create rules.

• Fails on unknown words/ambiguous contexts.

 Unigram (Baseline) Tagger
• Concept: Assign the most frequent tag for a word (e.g., "bank"
→ NOUN).

• Training: Count tag frequencies in annotated corpora.

• Strengths:

• Simple, fast, and baseline for evaluation.

 Unigram (Baseline) Tagger
• Limitations:

• Ignores context → Fails on ambiguity (e.g., "book" is always

NOUN, even in "book a flight").

• Accuracy: ~90% (low for practical use).

 Hidden Markov Model (HMM)
Taggers
• Concept: Probabilistic model using:

• Decoding: Viterbi algorithm finds the most likely tag sequence.

• Key Work: Church (1988), DeRose (1988).

 Hidden Markov Model (HMM)
Taggers
• Strengths:

• Robust context modeling.

• Handles known words well (~95% accuracy).

• Limitations:

• Struggles with unknown words (no emission data).

• Requires large annotated corpora.

• Assumes Markov property (limited context window).

 Rule-Based + Stochastic Hybrids
• Example: Constraint Grammar (Karlsson, 1990).

• Concept: Combine probabilistic disambiguation with

handcrafted rules.

• Strengths: Better handling of edge cases than pure HMMs.

• Limitations: Still rule-heavy and complex to scale.

 Early Neural Network Taggers
• Concept: Use feedforward networks (1990s).

• Method:

• Input: Word embeddings + context window.

• Output: Probability distribution over tags.

• Limitations (pre-2000):

• Computationally expensive.

• Outperformed by HMMs due to limited data/hardware.

 Evolution to Brill’s Tagger
• Early Approaches:

• Rule-Based Taggers: Handcrafted rules (e.g., "If word ends with

-ing, tag as verb"). Accurate but labor-intensive.

• Stochastic Taggers: Use probabilistic models (e.g., Hidden

Markov Models). Limited by training data and context.
 Evolution to Brill’s Tagger
• Problem: Both struggled with unknown words and context
nuances.

• Brill’s Solution (1992): Combines rules with data-driven

learning.
 Core Idea: Transformation-Based
Learning (TBL)
• Key Insight:
• Start with a crude initial tagger (e.g., assign most frequent tag
to each word).
• Iteratively apply context-sensitive rules to correct errors.
• Analogy: Like a "proofreading" process:
• Make an initial guess.
• Refine using rules (e.g., "Change tag from X to Y if preceding
word is Z").
 How Brill’s Tagger Works
• Training Phase: Learning Optimal Rules
• Initial Tagging: Tag each word with its most frequent POS (e.g.,
"run" = noun).
• Rule Generation:
• Templates: Define rule patterns (e.g., "Change tag A to B if
previous word is C").
• Scoring: Test all template instances; keep rules that maximize
error correction.
 How Brill’s Tagger Works
• Rule Selection:
• Greedily select the best rule, apply it, and repeat until no
significant improvement.
• Tagging Phase: Applying Learned Rules
• Process:
• Run initial tagger.
• Sequentially apply learned rules to update tags.
 Example
• Sentence: "They book flights."
• Initial Tags:
• "They" → Pronoun, "book" → Noun (most frequent tag),
"flights" → Noun.
• Error: "book" should be a verb here.
 Example
• Learned Rule:
• "Change tag from Noun to Verb if next word is a Noun."
• Corrected Tags:
• "book" → Verb (since "flights" is a noun).
 Rule Templates
• Brill defined ~100 templates, including:
• Change tag A to B if:
• Preceding/following word is C.
• Preceding/following tag is D.
• Word has prefix/suffix X.
• Example:
• "Change NN → VB if next word is TO." (e.g., "to run" → "run" is
verb).
 Advantages
• Interpretability: Rules are human-readable.
• Handles Unknown Words: Uses morphology (e.g., suffixes like
*-ly* → adverb).
• Context-Aware: Corrects tags based on surrounding words.
• Efficiency: Faster training than HMMs/neural models.
 Limitations
• Rule Order Sensitivity: Rules apply sequentially; order affects
outcome.
• Greedy Learning: May not find the globally optimal rule set.
• Performance Plateau: Accuracy stagnates after ~200 rules.
• Modern Alternatives: Outperformed by neural models (BiLSTM,
Transformers) in accuracy.
 Summary
• Brill’s Tagger bridges rule-based and statistical methods via
transformation rules.
• Legacy: Pioneered context-sensitive learning; influenced
modern systems (e.g., CRF taggers).
• Relevance Today: Demonstrates how lightweight, interpretable
models can solve complex NLP tasks.
 Key Challenges in Pre-Brill Era
• Unknown Words: Rules/HMMs failed on new vocabulary.
• Ambiguity: Needed context beyond local n-grams.
• Data Sparsity: HMMs required massive annotated corpora.
• Labor Cost: Rule-based systems needed linguistic expertise.
 Why Brill’s Tagger Was Revolutionary
• Solved: Automated rule-learning + context sensitivity.
• Bridge: Combined strengths of:
• Rule-based (interpretability, morphology).
• Stochastic (data-driven learning).
• Legacy: Paved the way for modern methods (CRFs, BiLSTMs,
BERT).