Natural Language Processing — Complete Study Guide

NATURAL LANGUAGE PROCESSING

Complete Study Guide — All Units with Examples

Units Covered

Unit 1: NLP Fundamentals, Probability & Language Models

Unit 2: Regular Expressions, Morphology & Parsing
Unit 3: Semantic Analysis & Discourse Processing
Unit 4: NLG & Machine Translation
Unit 5: Information Retrieval & Lexical Resources

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

UNIT 1: NLP Fundamentals, Probability & Language Models

1.1 Introduction to NLP

Natural Language Processing (NLP) is a branch of Artificial Intelligence concerned with the
interaction between computers and human language — enabling machines to read, understand, and
generate text and speech.

Core Tasks in NLP

NLP tasks are broadly categorized into three linguistic levels:

Syntax (Structure of Language)

Definition: Deals with the grammatical structure of sentences.

• POS Tagging: "The quick brown fox jumps" → The/DT quick/JJ brown/JJ fox/NN
jumps/VBZ

• Parsing: Breaking a sentence into its grammatical components (subject, verb,

object).

• Chunking: Grouping related words: [The quick brown fox] [jumps over] [the lazy dog]

Semantics (Meaning of Language)

Definition: Deals with the meaning of words and sentences.

• Word Sense Disambiguation: "Bank" → river bank OR financial bank? Context

decides.

• Named Entity Recognition: "Sachin Tendulkar scored 100 in Mumbai." → Person,

Location

• Semantic Role Labeling: "John gave Mary a book." → John=Giver, Mary=Receiver,

book=Theme

Pragmatics (Use of Language in Context)

Definition: Deals with how context affects meaning — beyond literal words.

• Coreference: "Ravi met Suresh. He smiled." — Who is "he"?

• Discourse: Understanding how sentences connect to form coherent paragraphs.

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

• Speech Acts: "Can you pass the salt?" — not asking about ability, but requesting
action.

1.2 Issues in NLP

NLP faces several fundamental challenges:

Issue Description Example

Ambiguity Same words/structure with multiple "I saw the man with a telescope" —
meanings who has the telescope?

Variability Many ways to express the same idea "The car is fast" / "The vehicle is
speedy" / "This auto moves quick"

World Knowledge Requires background knowledge "The city council refused the
humans have protesters a permit because they
feared violence" — who feared?

Robustness Real text has errors, slang, "gonna", "ur", "lol", "plz" — informal
abbreviations language

Scalability Handling billions of Web-scale NLP: processing the

words/documents entire internet

Multilinguality Working across 7000+ languages Many Indian languages have

complex morphology

1.3 Applications of NLP

NLP powers many real-world systems:
• Machine Translation: Google Translate — translates between 100+ languages automatically.
• Sentiment Analysis: "This product is amazing!" → Positive. "Worst experience ever." →
Negative.
• Question Answering: Siri, Alexa, ChatGPT — answer natural language questions.
• Information Extraction: Extracting dates, names, events from news articles automatically.
• Text Summarization: Auto-summarizing long research papers into 3-4 sentences.
• Spell Checking: MS Word detecting and correcting 'teh' → 'the'.
• Chatbots: Customer service bots that understand and respond to user queries.

1.4 Role of Machine Learning in NLP

Modern NLP relies heavily on Machine Learning (ML), especially Deep Learning:

ML Approach NLP Application Example System

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

Naive Bayes Text Classification, Spam Email spam filter

Detection

HMM (Hidden Markov POS Tagging, Speech Classical tagger

Models) Recognition

CRF (Conditional Named Entity Recognition Medical entity extractor

Random Field)

RNN / LSTM Language Modeling, Seq2Seq MT

Translation

Transformers (BERT, All NLP tasks ChatGPT, BERT, T5

GPT)

Word2Vec / GloVe Word Embeddings Semantic similarity

1.5 Probability Basics for NLP

Probability theory is the backbone of statistical NLP:

Key Concepts
• Joint Probability: P(A and B) = P(A) × P(B|A). Example: P('hot' and 'dog') in a corpus.
• Conditional Probability: P(B|A) = P(A,B) / P(A). Example: P('dog'|'hot') — given 'hot',
probability next word is 'dog'.
• Bayes' Theorem: P(A|B) = P(B|A) × P(A) / P(B). Used in spam classification, POS tagging.

Bayes' Theorem — Spam Classification Example

Problem: Is email containing the word 'FREE' spam or not?

Given: P(spam) = 0.3, P('FREE'|spam) = 0.8, P('FREE'|not spam) = 0.1

Calculate: P(spam|'FREE') = P('FREE'|spam) × P(spam) / P('FREE')

P('FREE') = 0.8×0.3 + 0.1×0.7 = 0.24 + 0.07 = 0.31

Result: P(spam|'FREE') = 0.8 × 0.3 / 0.31 = 0.77 → 77% chance it's spam!

1.6 Information Theory for NLP

• Entropy: Measures uncertainty/randomness in a probability distribution. H(X) = -Σ P(x) log₂P(x)
• Cross-Entropy: Measures how well a model's distribution matches the true distribution. Used
as a loss function in neural NLP models.

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide
• Perplexity: Standard metric for evaluating language models. Lower is better. PP(W) = 2^H(W).
A model with perplexity 100 is as confused as choosing uniformly from 100 words.

Entropy Example:

Fair coin: H = -(0.5×log₂0.5 + 0.5×log₂0.5) = 1 bit (maximum uncertainty)

Biased coin (P(H)=0.9): H = -(0.9×log₂0.9 + 0.1×log₂0.1) ≈ 0.47 bits (low

uncertainty)

NLP Application: A language model should assign low entropy to likely next
words.

After "The sun rises in the", → "east" should have HIGH probability → LOW
entropy

1.7 N-gram Language Models

N-gram models predict the next word based on the previous N-1 words:

N-gram Type Formula Example

Unigram (N=1) P(wᵢ) = count(wᵢ)/total_words P('cat') = 50/10000 = 0.005

Bigram (N=2) P(wᵢ|wᵢ₋₁) = count(wᵢ₋₁,wᵢ)/count(wᵢ₋₁) P('dog'|'hot') = count('hot

dog')/count('hot')

Trigram (N=3) P(wᵢ|wᵢ₋₂,wᵢ₋₁) P('dog'|'the','hot') — uses 2-word

context

4-gram (N=4) P(wᵢ|wᵢ₋₃,wᵢ₋₂,wᵢ₋₁) Captures longer dependencies

N-gram Example — Corpus: 'I love NLP. I love coding. NLP is fun.'

Bigram Probabilities:

• P('love' | 'I') = count('I love') / count('I') = 2/2 = 1.0

• P('NLP' | 'love') = count('love NLP') / count('love') = 1/2 = 0.5

• P('coding' | 'love') = count('love coding') / count('love') = 1/2 = 0.5

Sentence Probability: P('I love NLP') = P(I) × P(love|I) × P(NLP|love) = 0.2 × 1.0 × 0.5 = 0.1

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

1.8 Smoothing Techniques

Problem: Unseen N-grams get probability 0, making model useless for new text.

Laplace (Add-1) Smoothing

Formula: P(wᵢ|wᵢ₋₁) = (count(wᵢ₋₁,wᵢ) + 1) / (count(wᵢ₋₁) + V)
where V = vocabulary size. Every bigram gets count+1, so no zero probabilities.

Good-Turing Smoothing
Redistributes probability mass from seen N-grams to unseen ones based on frequency-of-frequency
counts.

Kneser-Ney Smoothing
State-of-the-art technique. Uses absolute discounting + back-off. Considers how often a word appears
in different contexts, not just raw frequency.

Laplace Smoothing Example:

Corpus: 'I love NLP' (V=5: I, love, NLP, is, fun)

Without smoothing: P('is' | 'I') = 0/2 = 0 ← PROBLEM!

With Laplace: P('is' | 'I') = (0+1)/(2+5) = 1/7 ≈ 0.143 ← OK!

With Laplace: P('love'| 'I') = (2+1)/(2+5) = 3/7 ≈ 0.429 ← Adjusted

down

1.9 Evaluating Language Models

• Perplexity: PP(W) = P(w₁w₂...wₙ)^(-1/N). Lower perplexity = better model. A perplexity of 50
means the model is as uncertain as uniformly choosing among 50 words.
• BLEU Score: Used for MT. Compares n-gram overlap between machine output and human
references. Range 0–1 (higher is better).
• Held-out Test Set: Always evaluate on data NOT used in training.

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

UNIT 2: Regular Expressions, Morphology & Syntax

2.1 Regular Expressions

Regular expressions (regex) are patterns for matching text strings — fundamental to NLP
preprocessing.

Pattern Meaning Example Match

. Any single character c.t → cat, cut, cot, c3t

* Zero or more of preceding ca*t → ct, cat, caat, caaat

+ One or more of preceding ca+t → cat, caat (NOT ct)

? Zero or one of preceding colou?r → color, colour

[abc] Character class (any of [Cc]at → Cat, cat

a,b,c)

[a-z] Range of characters [a-z]+ → any lowercase word

^ Start of line ^The → 'The' at start

$ End of line end$ → 'end' at line end

\d Any digit [0-9] \d{10} → 10-digit phone number

\w Word character [a-zA-Z0- \w+ → any word

9_]

\s Whitespace \s+ → spaces/tabs/newlines

{n,m} Between n and m \d{2,4} → 2 to 4 digits

repetitions

() Group (ha)+ → ha, haha, hahaha

| Alternation (OR) cat|dog → cat OR dog

Real NLP Regex Examples

# Match email addresses:

Pattern: [\w.+-]+@[\w-]+\.[a-zA-Z]{2,}

Matches: user@[Link], [Link]+tag@[Link]

# Match Indian phone numbers:

Pattern: (\+91|0)?[6-9]\d{9}

Matches: +919876543210, 09876543210, 9876543210

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

# Match dates (DD/MM/YYYY):

Pattern: (0[1-9]|[12]\d|3[01])/(0[1-9]|1[0-2])/\d{4}

Matches: 25/12/2024, 01/01/2025

# Tokenization (split on spaces/punctuation):

Pattern: \w+|[^\w\s]

"Hello, world!" → ["Hello", ",", "world", "!"]

2.2 Finite-State Automata (FSA)

FSA are mathematical models for pattern recognition — the theoretical foundation of regex.

Deterministic FSA (DFA)

At each state, exactly one transition per input symbol. Efficient but may need many states.

Non-deterministic FSA (NFA)

Multiple transitions possible for same input. More compact but requires subset construction for
implementation.

FSA Example: Recognizing 'sheep' or 'sheeeep' (elongated)

States: q0 → q1 → q2 → q3 → q4(accept)

• q0 --s--> q1

• q1 --h--> q2

• q2 --e--> q3

• q3 --e--> q3 (self-loop: one or more 'e')

• q3 --p--> q4 (accept state)

Recognizes: 'sheep', 'sheeep', 'sheeeep', ... (any number of e's ≥ 1)

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

Finite-State Transducers (FST)

FSTs map one string to another — used in morphological analysis. Input: surface form → Output:
lemma + morphological features.
FST Example — English Morphology:

Input: 'walked' → Output: 'walk+PAST'

Input: 'running' → Output: 'run+PRESENT_PARTICIPLE'

Input: 'foxes' → Output: 'fox+PLURAL'

Rule: {e → ε} / _d (delete 'e' before 'd' in past tense)

2.3 Morphological Parsing

Morphology studies the internal structure of words. Morphological parsing breaks words into their
component morphemes.

Morpheme Type Description Example

Root/Stem Core meaning carrier play, run, teach

Prefix Added before root un+happy, re+write, pre+fix

Suffix Added after root teach+er, quick+ly, happi+ness

Inflectional Grammatical function (tense, walk+ed, dog+s, fast+er

number)

Derivational Creates new words/changes teach(V) → teacher(N)

POS

Morphological Parsing Examples

unhappiness → un + happy + ness (prefix + root + suffix)

un=[NEG] happy=[ADJ] ness=[NOUN_FORMING]

walked → walk + ed (root + past_tense)

running → run + ing (root + present_participle)

dogs → dog + s (root + plural)

Hindi Example: padh+taa+hoon → read+HABITUAL+1SG (I read/I am reading)

Tamil Example: paar+tt+aan → see+PAST+3MSG (He saw)

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

2.4 Spelling Error Detection & Correction

Spell checkers identify and fix errors in written text using various techniques:

Types of Spelling Errors

• Non-word errors: "teh" instead of "the" — not a valid dictionary word.
• Real-word errors: "their/there/they're" — valid words used in wrong context.
• Typographic errors: Transposition: 'hte→the', Insertion: 'thhe→the', Deletion: 'th→the'.

Correction Methods
• Edit Distance (Levenshtein): Minimum edit operations (insert, delete, substitute) to convert
one string to another.
• Soundex/Metaphone: Phonetic algorithms — match words that sound similar.
• Language Model Correction: P('the cat' > 'teh cat') — use context to choose best correction.

Edit Distance Example

Distance between 'kitten' and 'sitting':

kitten → sitten (substitute k→s) cost: 1

sitten → sittin (substitute e→i) cost: 1

sittin → sitting (insert g) cost: 1

Total edit distance: 3

Candidates for 'teh':

the → distance 1 (transpose t,h)

ten → distance 1 (substitute e)

tea → distance 2

Best correction: 'the'

2.5 Part-of-Speech (POS) Tagging

POS tagging assigns grammatical labels to each word in a sentence.

POS Tag Full Name Examples

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

NN Noun (singular) dog, city, happiness

NNS Noun (plural) dogs, cities

NNP Proper noun Ravi, Mumbai, Google

VB Verb (base form) run, eat, play

VBD Verb (past tense) ran, ate, played

VBG Verb (gerund/-ing) running, eating

JJ Adjective happy, large, red

RB Adverb quickly, very, well

DT Determiner the, a, an, this

IN Preposition in, on, at, with

CC Coordinating and, but, or

conjunction

PRP Personal pronoun I, he, she, they

POS Tagging Example

Sentence: 'The quick brown fox jumps over the lazy dog.'

The/DT quick/JJ brown/JJ fox/NN jumps/VBZ over/IN the/DT lazy/JJ dog/NN

Another example:

'Ravi quickly ran to the large market in Mumbai.'

Ravi/NNP quickly/RB ran/VBD to/IN the/DT large/JJ market/NN in/IN

Mumbai/NNP

POS Tagging Methods

• Rule-based: Handwritten linguistic rules. E.g., word ending in '-ly' is likely RB (adverb).
• Statistical (HMM): Uses transition probabilities between tags + emission probabilities of words
given tags.
• Neural (BiLSTM-CRF, BERT): State-of-the-art, learns features automatically from context.

2.6 Context-Free Grammar (CFG)

CFG formally defines the syntactic structure of sentences using production rules.

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

CFG Rule Format: A → α (A is a non-terminal, α is a string of terminals/non-terminals)

Simple English CFG:

S → NP VP (Sentence = Noun Phrase + Verb Phrase)

NP → Det N (Noun Phrase = Determiner + Noun)

NP → Det Adj N (NP with adjective)

VP → V NP (Verb Phrase = Verb + Noun Phrase)

VP → V (Intransitive verb)

Det → 'the' | 'a'

N → 'dog' | 'cat' | 'bone'

V → 'chased' | 'ate' | 'saw'

Adj → 'big' | 'small' | 'old'

Parse: 'the dog chased a big cat'

S → NP VP

NP → Det N → 'the' 'dog'

VP → V NP → 'chased' (Det Adj N) → 'a' 'big' 'cat'

2.7 Constituency Parsing

Constituency (phrase-structure) parsing builds a parse tree showing how words group into phrases.

Parse Tree Example: 'The cat sat on the mat'

__________|__________

NP VP

____|____ ______|______

DT NN VBD PP

| | | ____|____

'The' 'cat' 'sat' IN NP

| ____|____

'on' DT NN

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

| |

'the' 'mat'

2.8 Probabilistic Parsing (PCFG)

Probabilistic Context-Free Grammars attach probabilities to rules to resolve ambiguity:

PCFG Rules (probability attached to each rule):

VP → V NP [0.7] (70% of VPs are Verb + NP)

VP → V PP [0.2] (20% are Verb + PP)

VP → V [0.1] (10% are just a Verb)

Resolving: 'I saw the man with a telescope'

Parse 1: [saw [the man with a telescope]] VP→V NP (man has telescope)

Parse 2: [saw [the man]] [with a telescope] VP→V NP PP (I have telescope)

PCFG picks the parse with HIGHEST probability product.

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

UNIT 3: Semantic Analysis & Discourse Processing

3.1 Meaning Representation

Computers need formal languages to represent meaning, not just strings of words.

Representation Description Example

First-Order Logic Predicate logic with ∀x Person(x) → Mortal(x)

quantifiers

Semantic Nets Graph-based meaning dog --ISA--> animal, --HAS--> tail

representation

Frames Slot-filler structures BUYING: buyer=John, item=book,

price=$10

Abstract Meaning Graph representation for AMR graph for 'The boy wants to go'
Repr. sentences

Lambda Calculus Compositional semantics λ[Link](x): function taking x, true if x is a

dog

First-Order Logic Examples

"Every student passed the exam"

FOL: ∀x (Student(x) → Passed(x, exam1))

"Some students failed"

FOL: ∃x (Student(x) ∧ Failed(x))

"John gave Mary a book on Monday"

FOL: Gave(john, mary, book1) ∧ On(monday) ∧ Book(book1)

3.2 Lexical Semantics

Lexical semantics studies the meanings of individual words and their relationships.

Relation Definition Example

Synonymy Same or similar meaning big ≈ large, fast ≈ quick, car ≈ automobile

Antonymy Opposite meaning hot ↔ cold, love ↔ hate, old ↔ new

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

Hyponymy IS-A relationship rose IS-A flower, dog IS-A animal

(specific→general)

Hypernymy IS-A relationship animal HYPERNYM of dog, vehicle

(general→specific) HYPERNYM of car

Meronymy PART-OF relationship wheel PART-OF car, leaf PART-OF tree

Polysemy One word, multiple related bank (financial/river), foot (body part/unit)
meanings

Homonymy Same form, unrelated bark (dog sound/tree covering)

meanings

3.3 Word Sense Disambiguation (WSD)

WSD determines which sense of an ambiguous word is being used in context.

WSD Examples

Word: 'bank'

• Financial sense: "I deposited money at the bank downtown."

• River sense: "The fisherman sat on the bank of the river."

Disambiguation clues: "money", "deposited" → financial sense; "river", "fisherman" → river

sense

Word: 'light'

• Illumination: "Turn off the light before sleeping."

• Weight: "This suitcase is very light."

• Color: "She wore a light blue dress."

WSD Approaches
• Knowledge-based (Lesk Algorithm): Find sense whose dictionary definition has maximum
overlap with the context.
• Supervised ML: Train classifier on sense-tagged corpus (e.g., SemCor). Features: surrounding
words, POS tags.
• Unsupervised (Word Embeddings): Use Word2Vec/BERT — ambiguous words get different
vector representations in different contexts.

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

3.4 Discourse Processing

Discourse analysis studies how sentences connect to form coherent text.

3.4.1 Cohesion
Cohesion refers to the grammatical and lexical devices that link sentences together:

Cohesion Type Definition Example

Reference Using "John came in. He sat down." — He=John

pronouns/determiners to
refer back

Substitution Replacing a phrase with a "Would you like some tea?" "Yes, please." —
shorter one please substitutes "I would like some"

Ellipsis Omitting understood "Can you swim?" "Yes [I can]." — bracketed

elements part omitted

Lexical Cohesion Repeating or using related "Dog... animal... pet..." — semantic field creates
words cohesion

Conjunction Linking with connectives "First... then... finally...", "However, therefore..."

3.4.2 Reference Resolution (Coreference)

Determines which expressions in text refer to the same entity.

Coreference Examples

Example 1:

"The president signed the bill. He then addressed the nation."

Coreference chain: [The president] = [He]

Example 2 (Ambiguous):

"Ravi told Suresh that he should submit the report."

Problem: who does 'he' refer to — Ravi or Suresh?

Likely interpretation: Suresh should submit (pronoun resolution via syntax)

Example 3 (Bridging):

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

"I bought a car. The engine is very powerful."

Bridging: [the engine] is part of [the car] — not same entity, but related

3.4.3 Discourse Coherence & Structure

Coherent text follows logical structure. Rhetorical Structure Theory (RST) analyzes relations between
discourse segments:
• Elaboration: "He is a doctor. He specializes in cardiology."
• Contrast: "The morning was sunny. However, it rained in the evening."
• Cause-Effect: "It rained heavily. Therefore, the match was cancelled."
• Evidence: "The company is doing well. Its profits rose 40% this quarter."
• Concession: "Although he studied hard, he failed the exam."

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

UNIT 4: Natural Language Generation & Machine Translation

4.1 Natural Language Generation (NLG)

NLG is the process of automatically producing readable text from non-linguistic data (databases,
knowledge graphs, etc.).

NLG Application Examples

• Weather Reports: Data: Temp=25°C, Humidity=80%, Wind=NE 15km/h → Text:

"Warm and humid day with a light northeast breeze."

• Financial Summaries: Data: Revenue=$1.2B, +15% YoY → Text: "Company

revenue grew 15% to $1.2 billion this year."

• Medical Reports: Lab results → Natural language diagnostic summaries for doctors.

• Sports Commentary: Match statistics → Real-time commentary generation.

4.2 Architecture of NLG Systems

The classic NLG pipeline has six stages:

Stage Task Example

1. Content What to say? Select relevant Choose to mention: temperature, weather,

Determination information wind

2. Document How to organize? Structure Weather summary → hourly then daily

Planning and order forecast

3. Aggregation Combine related content Merge two sentences about rain into one

4. Lexicalization Choose words 'precipitation' vs 'rain' vs 'shower'

5. Referring How to refer to entities First: 'The storm', then: 'it'

Expressions

6. Surface Produce grammatical text Apply grammar rules, agreement,

Realization punctuation

4.3 Generation Tasks & Representations

Template-Based Generation
Simplest approach — fill slots in predefined templates:
Template: 'The [TEAM1] defeated [TEAM2] by [SCORE] runs in [VENUE].'

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

Data: TEAM1=India, TEAM2=Australia, SCORE=45, VENUE=Chennai

Output: 'The India defeated Australia by 45 runs in Chennai.'

More complex template (conditional):

IF score_diff > 100 THEN: 'dominant victory'

IF score_diff < 10 THEN: 'narrow escape'

Neural NLG (Seq2Seq, Transformers)

Modern NLG uses encoder-decoder neural networks. Input: structured data/prompt → Output: fluent
natural language text.

4.4 Machine Translation (MT)

MT automatically translates text from one language (source) to another (target).

4.5 Problems in Machine Translation

MT faces numerous linguistic challenges:

Problem Description Example

Lexical Ambiguity Words with multiple "bank" in English can mean financial
meanings institution OR river bank - different Hindi
words

Syntactic Ambiguity Multiple parse structures "I saw the man with the telescope" — 2
meanings, 2 translations

Idioms Cannot translate literally "kick the bucket" ≠ बाल्टी लात मारो → must
translate as "die" = "मरना"

Word Order Different sentence English: SVO (I eat rice) | Hindi: SOV (मैं चावल
structures खाता हूँ)

Morphological Rich inflection systems Sanskrit/Tamil/Finnish have complex case

Complexity marking

Cultural Context Culture-specific references "Diwali" may need explanation for non-Indian
audiences

Pronouns Different pronoun systems Hindi has formal/informal 'you': आप vs तुम vs

तू

Named Entities Names should be "Sachin Tendulkar" → सचिन तेंडुलकर (not

transliterated translated)

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

4.6 Characteristics of Indian Languages

Indian languages have unique features that make NLP challenging:

• SOV Word Order: Subject-Object-Verb in Hindi, Tamil, Telugu vs. SVO in English.
• Rich Morphology: Agglutinative languages (Tamil, Kannada) pack multiple meanings into one
word. E.g., Tamil: படித்தேன் (paditthen) = I read (past).
• Pro-drop: Subject can be omitted when clear from context. Hindi: 'खाया' = (I/he/she) ate.
• Postpositions: Prepositions come AFTER the noun. Hindi: 'घर में' (ghar mein) = 'in the house'.
• Gender Agreement: Hindi has masculine/feminine grammatical gender affecting verbs and
adjectives.
• Multiple Scripts: Devanagari, Tamil, Telugu, Kannada, Malayalam, Bengali, Gujarati, etc.
• Code-mixing: Hinglish (Hindi+English): 'Main office ja raha hoon aaj.' — common in social
media.

4.7 MT Approaches

1. Rule-Based MT (RBMT)
Uses handcrafted linguistic rules for analysis, transfer, and generation. Requires large grammar and
bilingual dictionaries.
Rule-Based MT: English → Hindi

Step 1 (Analysis): 'The boy eats food'

Det+Noun+Verb+Noun

Step 2 (Transfer): DET→ (dropped), boy→लड़का, eat→खाना, food→खाना

Step 3 (Generation): लड़का खाना खाता है

(SOV word order applied, verb agreement added)

2. Statistical MT (SMT)
Learns translation probabilities from parallel corpora (aligned sentence pairs in two languages).
Formula: best_translation = argmax_T P(T|S) = argmax_T P(S|T) × P(T)
P(S|T) = translation model (which words/phrases translate to what)
P(T) = language model (fluency of target language)

3. Neural MT (NMT)
State-of-the-art. Uses encoder-decoder with attention mechanism (Transformer). Models like Google
Translate, DeepL use this.
NMT Architecture (Transformer-based):

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

Source: 'The cat sat on the mat.'

[Encoder] — processes source sentence

Creates contextual embeddings for each word

[Attention] — aligns source and target words

[Decoder] — generates target word by word

Target: 'बिल्ली चटाई पर बैठी।'

4.8 Translation Involving Indian Languages

Key challenges and solutions for Indian language MT:
• Parallel Corpora: IIT Bombay Hindi-English corpus, OPUS, FLORES-200.
• Transliteration: Proper names must be script-converted. 'Mumbai' → 'मुंबई'.
• Anusaaraka/Shakti: Early rule-based MT systems for Indian languages.
• Multilingual Models: IndicTrans2 (AI4Bharat) supports 22 Indian languages.

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

UNIT 5: Information Retrieval & Lexical Resources

5.1 Information Retrieval (IR)

IR systems find documents relevant to a user's information need (query) from a large collection.

Core IR Concepts

• Document: Any text unit — web page, article, book chapter, email.

• Query: User's expression of information need. E.g., 'best restaurants Mumbai'

• Relevance: How well a document answers the query.

• Recall: Fraction of all relevant documents that were retrieved. Recall = TP/(TP+FN)

• Precision: Fraction of retrieved documents that are relevant. Precision = TP/(TP+FP)

• F1-Score: Harmonic mean of Precision and Recall: F1 = 2PR/(P+R)

5.2 Design Features of IR Systems

Component Function Example

Crawler/Indexer Collects and indexes Google crawls billions of web pages

documents

Inverted Index Maps terms to documents 'apple' → [doc3, doc7, doc12, ...]
containing them

Query Processor Parses and expands user 'cars' → also search 'car', 'automobile'
query

Ranking Module Orders results by relevance TF-IDF, BM25, PageRank

score

User Interface Presents results to user Google's 10-blue-links interface

Inverted Index Example

Document Collection:

• Doc1: 'The cat sat on the mat'

• Doc2: 'The cat chased the mouse'

• Doc3: 'A dog chased the cat'

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

Inverted Index:

cat → [Doc1, Doc2, Doc3]

chased→ [Doc2, Doc3]

dog → [Doc3]

mat → [Doc1]

mouse → [Doc2]

sat → [Doc1]

Query: 'cat AND dog' → Doc3 (appears in both lists)

Query: 'cat OR mat' → Doc1, Doc2, Doc3

5.3 Classical IR Models

Boolean Model
Documents are either relevant (1) or not (0) based on exact term matching with Boolean operators
(AND, OR, NOT).
Query: 'information AND retrieval NOT database'

Matches: documents containing BOTH 'information' AND 'retrieval'

but NOT containing 'database'

Limitation: No ranking — all matching docs considered equally relevant

Vector Space Model (VSM)

Documents and queries are represented as vectors in a high-dimensional term space. Similarity is
measured by cosine similarity.
TF-IDF Weighting:

TF(t,d) = term frequency of t in document d

IDF(t) = log(N / df(t)) where N=total docs, df=docs containing t

TF-IDF = TF(t,d) × IDF(t)

Cosine Similarity:

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

sim(Q,D) = (Q·D) / (|Q| × |D|)

Example:

Query = [cat:1, dog:1]

Doc1 = [cat:2, dog:0, mat:1] → sim = 2/(1.41×2.24) ≈ 0.63

Doc2 = [cat:1, dog:1, mouse:1] → sim = 2/(1.41×1.73) ≈ 0.82

Doc2 ranked higher!

Probabilistic Model (BM25)

BM25 (Best Match 25) models relevance probabilistically. Standard in modern search engines:
BM25 Formula:

Score(D,Q) = Σ IDF(qᵢ) × [f(qᵢ,D)×(k1+1)] / [f(qᵢ,D)+k1×(1-b+b×|D|/avgdl)]

f(qᵢ,D) = term frequency, k1=1.2 to 2.0, b=0.75

avgdl = average document length in corpus

Advantage: Handles term saturation and document length normalization

5.4 Non-Classical IR Models

Model Key Idea Advantage

Fuzzy IR Partial matching using fuzzy Handles imprecise queries

logic

Neural IR (Dense BERT-based dense vector Understands semantic similarity

Retrieval) matching

Latent Semantic SVD to find latent topics Handles synonymy and polysemy
Analysis (LSA)

Language Model for P(Q|D): probability query Principled probabilistic framework

IR generated from doc

Learning to Rank ML models trained on Combines many features for ranking

(LTR) relevance judgments

5.5 WordNet
WordNet is a large lexical database of English, organizing words by meaning relationships.

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

WordNet Structure
• Synset (Synonym Set): Group of words with same meaning. E.g., {car, automobile, auto,
motorcar}
• Gloss: Definition of each synset. car: 'a motor vehicle with four wheels; usually propelled by an
internal combustion engine'
• Hypernymy/Hyponymy hierarchy: dog → canine → carnivore → mammal → animal →
organism
• Meronymy: car has_part: wheel, engine, door, window, steering_wheel

WordNet Example — Synsets for 'bank'

bank#1: 'a financial institution' (synset: {bank, banking_company,

banking_concern})

Hypernym: financial_institution → institution → organization

bank#2: 'sloping land beside a body of water' (synset: {bank, riverbank})

Hypernym: slope → geological_formation

bank#3 (verb): 'tip laterally' (e.g., an aircraft banks into a turn)

WordNet WSD: context determines which sense/synset applies

5.6 FrameNet
FrameNet is a lexical resource based on Frame Semantics (Charles Fillmore). Each semantic frame
describes an event/situation and its participants.

FrameNet Frame Example: COMMERCE_BUY

Frame: COMMERCE_BUY

• Definition: A Buyer pays money to a Seller in exchange for Goods.

Frame Elements (Roles):

• Buyer: The agent who acquires the goods

• Seller: The entity from whom goods are purchased

• Goods: The item(s) being purchased

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

• Money: The amount paid

• Place: Where the transaction occurs

Lexical Units: buy, purchase, acquire, pay for, get, pick up

"[John]_Buyer bought [a laptop]_Goods from [Dell]_Seller for

[$1200]_Money."

"[She]_Buyer purchased [flowers]_Goods at [the market]_Place."

"[The customer]_Buyer paid [₹500]_Money for [the meal]_Goods."

5.7 Stemmers
Stemmers reduce words to their root/stem form — enabling better matching in IR:

Algorithm Language Example

Porter Stemmer English running→run, studies→studi, happiness→happi

Snowball Stemmer Multiple More aggressive: generalization→general

(English,
French, etc.)

Lovins Stemmer English Removes longest matching suffix

Hindi Stemmer Hindi पढ़ना→पढ़, चलती→चल

(Ramanathan)

Morpheme-based Morphologically handles→handle (not just suffix removal)

rich

Porter Stemmer Rules (simplified):

SSES → SS: caresses → caress

IES → I: ponies → poni

SS → SS: caress → caress

S → : cats → cat

EED → EE: agreed → agree (if stem > 1)

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

ED → : plastered→ plaster

ING → : running → run

NLP Use: 'Computing', 'computation', 'computed' all stem to 'comput'

→ treated as same term in IR index → better recall

5.8 POS Taggers (Overview)

POS taggers are critical preprocessing tools in NLP pipelines. Popular tools:
• NLTK Tagger (Python): Maximum Entropy tagger. Universal and Penn Treebank tagsets.
• Stanford NLP: Log-linear Maximum Entropy tagger with high accuracy on English.
• spaCy: Industrial-strength, fast neural tagger supporting 60+ languages.
• TreeTagger: Rule + probabilistic, supports many languages including German, French.
• Bi-LSTM-CRF: Neural architecture, state-of-the-art for many languages.

5.9 Research Corpora

Annotated corpora are essential for training and evaluating NLP systems:

Corpus Language Type Size/Usage

Penn Treebank English POS + Parse trees 1M words, Wall Street Journal

SemCor English Word Sense ~200K words, WordNet senses

annotated

OntoNotes 5.0 English/ Multi-level 1.7M words

Chinese/Arabic annotation

BNC (British National English General text 100M words, balanced

Corpus)

COCA (Corpus of American Various genres 1B+ words

Contemp. American English
Eng.)

ILCI (Indian Languages) 22 Indian POS + NE tagged Govt. of India project

languages

SemEval Datasets Multiple Task-specific NLP Annual shared tasks

CoNLL Datasets Multiple NER, Parsing Standard NLP benchmarks

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

5.10 Word Embeddings & Word2Vec

Modern NLP uses dense vector representations (embeddings) where similar words have similar
vectors.

Word2Vec (Mikolov et al., 2013) — Key Models

CBOW (Continuous Bag of Words): Predicts center word from surrounding context words.

Context: "The ___ sat on the mat"

CBOW uses: ["The", "sat", "on", "the", "mat"] → predicts "cat"

Skip-gram: Predicts surrounding context words from the center word.

Center: "cat" → predicts: "The", "sat", "on", "the", "mat"

Better for rare words and larger datasets

Amazing property — Vector Arithmetic:

vector('King') - vector('Man') + vector('Woman') ≈ vector('Queen')

vector('Paris') - vector('France') + vector('Germany') ≈ vector('Berlin')

vector('walking') - vector('swimming') + vector('swam') ≈ vector('walked')

NLP Study Guide | All Units Page

Natural Language Processing — Complete Study Guide

Quick Revision Summary

Key formulas and concepts at a glance:

Concept Formula / Key Point

Bigram Probability P(wᵢ|wᵢ₋₁) = count(wᵢ₋₁,wᵢ) / count(wᵢ₋₁)

Laplace Smoothing P(wᵢ|wᵢ₋₁) = (count(wᵢ₋₁,wᵢ)+1) / (count(wᵢ₋₁)+V)

Perplexity PP(W) = P(w₁...wₙ)^(-1/N) — lower is better

TF-IDF TF(t,d) × log(N/df(t)) — high for frequent-in-doc but rare-in-

corpus

Cosine Similarity sim(Q,D) = (Q·D) / (|Q|×|D|) — range: 0 to 1

Precision TP / (TP+FP) — of retrieved, how many relevant?

Recall TP / (TP+FN) — of all relevant, how many retrieved?

F1 Score 2PR/(P+R) — harmonic mean of precision and recall

Edit Distance Min insert+delete+substitute to convert string A to B

PCFG Attach probabilities to CFG rules; choose highest probability

parse

CBOW Context words → predict center word

Skip-gram Center word → predict context words

Bayes Theorem P(A|B) = P(B|A)×P(A) / P(B)

Entropy H(X) = -Σ P(x) log₂P(x)

SMT Formula best_T = argmax P(S|T)×P(T) — translation × language model

— End of NLP Study Guide —

NLP Study Guide | All Units Page