Understood I’ll now expand the notes into a very detailed version with step‑by‑step explanations, extended
derivations, and detailed diagrams. This will look like a complete textbook chapter set, covering every topic in your NLP
syllabus.
NLP Notes – Part 1
(Expanded)
1. Introduction to NLP
1.1 What is NLP?
Natural Language Processing (NLP) is a subfield of Artificial Intelligence (AI) and Computational Linguistics that
deals with the interaction between humans and computers using natural languages (English, Hindi, Chinese, etc.). The
challenge lies in the ambiguity of human language (same word can mean different things in different contexts).
Goal: Create systems that can automatically analyze, understand, and generate human language.
Why Difficult? Because natural languages are ambiguous, context-dependent, and ever-changing.
Example of Ambiguity:
Sentence: "I saw the man with a telescope."
Did I use the telescope?
Or did the man have the telescope?
This ambiguity motivates the need for multi-level linguistic processing.
1.2 Applications of NLP
1. Machine Translation (MT)
Example: Google Translate converts "Bonjour" → "Hello".
Challenge: Idioms and cultural expressions.
2. Speech Recognition
Converting spoken words into text.
Example: Alexa, Siri, Google Assistant.
3. Chatbots & Conversational Agents
Example: ChatGPT, customer service bots.
4. Information Retrieval (IR)
Search engines like Google.
5. Sentiment Analysis
Identify whether text is positive, negative, or neutral.
Example: "This phone is amazing" → Positive.
6. Text Summarization
Extractive (select sentences).
Abstractive (generate new summary).
7. Question Answering (QA)
Example: "Who is the president of India?"
System must extract answer from knowledge base.
8. Spam Filtering
Emails classified as spam or non-spam.
1.3 Levels of Linguistic Processing
Language understanding requires different levels:
1. Morphological Processing
Deals with word structure.
Morpheme = smallest unit of meaning.
Example: "unhappiness" = un- (negation) + happy (root) + -ness (noun form).
2. Syntax (Grammatical Structure)
Focuses on how words form valid sentences.
Example:
"The cat sat on the mat."
"Cat mat the on sat."
Derivation: Syntax is modeled using Context-Free Grammars (CFGs) where rules like: [ S → NP , VP ] define
valid structures.
3. Semantics (Meaning)
Determines the literal meaning of words/sentences.
Example: "John kicked the bucket" → literal: action of kicking a bucket.
4. Pragmatics (Contextual Meaning)
Deals with intended meaning in context.
Example: "Can you pass the salt?" → request, not about ability.
5. Discourse Processing
Looks at how multiple sentences relate.
Example: "John went to the store. He bought apples." ("He" refers to John).
Diagram – NLP Processing Pipeline
Raw Text → Morphology → Syntax → Semantics → Pragmatics → Discourse → Application
2. Tokenization, Stemming, Lemmatization
2.1 Tokenization
Splitting text into tokens (words, subwords, sentences).
Example: "I love NLP!" → ["I", "love", "NLP", "!"]
Types:
Word-level: splits at spaces.
Subword-level: Byte-Pair Encoding (BPE), WordPiece.
Sentence-level: splits by sentence boundaries.
Mathematical Model: Tokenization = function (f: String → List(Tokens)).
2.2 Stemming
Rule-based reduction to word roots.
Example: running, runs, ran → run.
Example: studies, studying → studi (not meaningful).
Algorithm – Porter Stemmer (Simplified):
1. If word ends with 'sses' → replace with 'ss'.
2. If word ends with 'ies' → replace with 'i'.
3. If word ends with 's' → remove 's'.
2.3 Lemmatization
Uses dictionary + grammar rules to find correct lemma.
Example: better → good, cars → car.
More accurate than stemming.
Mathematical Process: [ lemma(w) = argmin_{x ∈ V} distance(w, x) ] where V = vocabulary.
3. N-gram Language Models
3.1 Definition
An n-gram = sequence of n consecutive words.
Unigram (n=1): P(w1)
Bigram (n=2): P(w2|w1)
Trigram (n=3): P(w3|w1,w2)
Probability Chain Rule: [ P(w_1, w_2, …, w_n) = \prod_^n P(w_i | w_1, …, w_) ]
Markov Assumption (for n-grams): [ P(w_i | w_1, …, w_) ≈ P(w_i | w_{i-n+1}, …, w_) ]
3.2 Example
Sentence: "I love NLP"
Unigrams: [I], [love], [NLP]
Bigrams: [I love], [love NLP]
Trigrams: [I love NLP]
Bigram Probability Example: [ P(“I love NLP”) ≈ P(I)·P(love|I)·P(NLP|love) ]
3.3 Smoothing Techniques
Without smoothing → unseen words = 0 probability.
(a) Add-1 (Laplace) Smoothing
[ P(w_i|w_) = \frac{C(w_,w_i) + 1}{C(w_) + V} ] where V = vocabulary size.
(b) Add-k Smoothing
[ P(w_i|w_) = \frac{C(w_,w_i) + k}{C(w_) + kV} ]
(c) Good-Turing Smoothing
Redistributes probability to unseen events. [ P^*(r) = \frac{(r+1)N_{r+1}} ] where (N_r) = number of n-grams with
frequency r.
(d) Kneser-Ney Smoothing
Uses discounted counts + continuation probabilities. Formula: [ P_(w_i|w_) = \frac{max(C(w_,w_i)-d,0)}{C(w_)} +
λ(w_)P_(w_i) ]
Diagram – Bigram Probability Tree
I → love (0.5), eat (0.3), play (0.2)
love → NLP (0.6), you (0.4)
4. Word Representations
4.1 Bag of Words (BoW)
Represents text as unordered set of word counts.
Example:
"I love NLP" → {I:1, love:1, NLP:1}
"NLP I love" → same representation.
Problem: No order, no context.
4.2 Word2Vec
Neural embedding model with two architectures:
(a) Skip-gram Model
Predict context given a word.
Example: Input = "NLP" → Output = [I, love].
Objective Function: [ J(θ) = -\frac{1} \sum_^T \sum_{-c ≤ j ≤ c, j≠0} \log P(w_{t+j} | w_t) ]
(b) CBOW Model
Predict word given context.
Example: Input = [I, __, NLP] → Output = "love".
Softmax Probability: [ P(w_o | w_i) = \frac{exp(v_^T v_)}{\sum_V exp(v_wT v_)} ]
4.3 Word Embeddings
Dense vector representation of words.
Capture semantic similarity.
Example (Vector Arithmetic): [ vector(king) - vector(man) + vector(woman) ≈ vector(queen) ]
4.4 Vector Semantics
Similarity measured by Cosine Similarity: [ cos(A,B) = \frac{A·B}{||A||·||B||} ]
cos(doctor, nurse) ≈ 0.8 (similar)
cos(doctor, banana) ≈ 0.1 (not similar)
NLP Notes – Part 2
(Expanded)
5. Resources for NLP
5.1 Lexicons
Lexicons = dictionaries for NLP.
Store: word → POS, meaning, synonyms, sentiment polarity.
Example: Sentiment Lexicon:
happy (+), sad (–).
5.2 Knowledge Bases
Store facts/entities/relationships.
Example: DBpedia, Wikidata.
5.3 Lexical Knowledge Networks
Words linked by semantic relations.
Example: Word "car" linked to "vehicle" (hypernym), "sedan" (hyponym).
5.4 WordNet
Lexical database of English.
Organizes words into synsets.
Relations:
Synonymy: car ↔ automobile.
Antonymy: hot ↔ cold.
Hypernymy: vehicle → car.
Hyponymy: sedan → car.
Diagram – WordNet Structure
vehicle
├── car
│ ├── sedan
│ └── SUV
└── truck
6. Computational Morphology
6.1 Lemmatization
Already covered → uses dictionary + morphology.
6.2 POS Tagging
Assigns word categories (Noun, Verb, Adj, Adv, etc.).
Example: "The black cat runs fast."
The → DET
black → ADJ
cat → NOUN
runs → VERB
fast → ADV
Methods:
1. Rule-based tagging.
2. Statistical tagging (HMM, CRF).
3. Neural tagging (BiLSTMs, Transformers).
Derivation – HMM POS Tagging: We want most likely tag sequence T for word sequence W. [ T^* = argmax_T P(T|W) ]
By Bayes rule: [ P(T|W) = \frac{P(W|T)P(T)}{P(W)} ] Since P(W) is constant: [ T^* = argmax_T P(W|T)P(T) ]
P(T) = probability of tag sequence (transition probabilities).
P(W|T) = emission probability of words given tags.
6.3 Finite-State Analysis
Finite State Automata (FSA) used for word formation.
Example: cats → cat + s (plural).
Diagram – FSA for plurals
(q0) --cat--> (q1)
(q1) --s--> (q2)
6.4 Noun Phrase Chunking
Chunk = shallow parsing to identify phrases.
Example: "The quick brown fox jumps over the lazy dog."
NP1 = "The quick brown fox"
NP2 = "the lazy dog"
Methods:
Rule-based grammars.
Machine learning with features (POS, context).
Now the notes are expanded like a textbook, with detailed formulas, step-by-step derivations, and diagrams.
Do you want me to add solved examples for each formula (like a worked-out POS tagging with HMM, or actual
probability calculation with bigrams)? That would make it even more exam-ready.