Below are complete, exam-oriented notes for WEEK 13 – NLP: Vector Semantics &
Embeddings (Part-II).
I have covered everything from the slides and explained it in very simple words, keeping
exam answers easy to remember and write.
**Natural Language Processing
Vector Semantics & Embeddings – Part II (Dense Representations)**
1. Sparse vs Dense Representations
Sparse Vectors
● Very long vectors (20,000–50,000 dimensions)
● Mostly contain zeros
● Examples:
○ Bag of Words
○ TF-IDF
○ PPMI
● Problems:
○ Waste memory
○ Slow computation
○ Poor generalization
📌 Example:
❌
If vocabulary = 10,000 words
A document uses only 100 words → 9,900 zeros
Dense Vectors
● Short vectors (50–1000 dimensions)
● Most values are non-zero
✅
● Store important information only
● Faster, compact, efficient
📌 Example:
Instead of storing
[0, 12, 0, 24, 0, 9]
We store only meaningful values
Why Dense Representations Are Better
● Fewer parameters → easier ML training
● Capture similar meanings
● Better handling of synonyms
● Work well in practice
● Ready-made embeddings available
2. Neural Language Modeling
Problem with Traditional Language Models
● Curse of dimensionality
● Cannot generalize well
● Rare word sequences unseen in training
Key Idea
👉 Predict words instead of counting them
● Learn:
○ Dense word vectors C(w)
○ Probability function f
● Similar words → similar vectors
● Smooth probability function
📌 Small change in vector → small change in meaning
How It Works (Simple Flow)
1. Convert words to one-hot vectors
2. Multiply with embedding matrix
3. Get dense word vectors
4. Combine context vectors
5. Neural network predicts next word
6. Output via softmax
3. One-Hot Encoding
Definition
● Each word → vector of size |V|
● Only one position = 1
● All others = 0
📌 Example (Vocabulary = 5):
Word Vector
cat [1 0 0 0 0]
dog [0 1 0 0 0]
❌ Problem:
● Very large
● No meaning encoded
4. Training Neural Language Model
● Initialize random weights
● Input: large text corpus
● Predict one word at a time
● Loss: cross-entropy
● Use backpropagation
● Learn:
○ Embedding matrix E
○ Network parameters
📌 Learned embeddings can be reused for other NLP tasks
5. Word2Vec
What is Word2Vec?
● Technique to learn word embeddings
● Developed by Mikolov (2013)
● Converts words into meaningful vectors
Two Architectures
(A) CBOW – Continuous Bag of Words
● Predicts center word from context
● Faster
● Works well for frequent words
📌 Example:
Context → predicts “sits”
(B) Skip-Gram
● Predicts context words from center word
● Slower
● Better for rare words
📌 Example:
“Sits” → predicts “cat”, “mat”, “on”
Central Idea of Word2Vec
👉
● Words are similar if:
Dot product is high
● Dot product → passed through sigmoid
● Gives probability of word appearing in context
6. Properties of Embeddings
Context Window Size (C)
Small Window (C = 2)
● Focus: Syntax / Grammar
● Captures nearby words
● Example neighbors of “market”:
○ trade, commerce
📌 Syntactic similarity
Large Window (C = 5 or more)
● Focus: Meaning / Topic
● Captures thematic similarity
● Example neighbors of “market”:
○ economy, investment
📌 Semantic similarity
7. Co-occurrence Types
First-Order Co-occurrence (Syntagmatic)
● Words appear together
● Focus: grammar, phrases
📌 Examples:
● wrote → poem
● eat → apple
Second-Order Co-occurrence (Paradigmatic)
● Words share similar neighbors
● Can replace each other
● Focus: meaning
📌 Examples:
● wrote ↔ said
● car ↔ truck
● king ↔ queen
8. Embeddings Capture Relationships
Word embeddings learn analogies:
king − man + woman ≈ queen
Paris − France + Italy ≈ Rome
📌 Shows embeddings store abstract relations, not just similarity
Types of Relationships Captured
● Geography (country–capital)
● Grammar (big → bigger)
● Profession (Einstein → scientist)
● Company–Product (Microsoft → Windows)
9. Other Dense Embedding Models
GloVe
● Developed by Stanford
● Uses global word co-occurrence
● Combines:
○ Count-based + prediction-based methods
fastText
● Developed by Facebook
● Uses sub-word information
● Handles:
○ Misspellings
○ Rare words
○ Out-of-vocabulary words
10. Embeddings & Word Meaning Over Time
● Train embeddings on historical text
● Compare word vectors across time
📌 Example: “mouse”
● 1950 → animal
● 1990 → computer device
➡ Meaning shifts → vector position changes
11. Sentiment Change Over Time
📌 Example: “terrific”
● Earlier: negative (terror)
● Now: positive (excellent)
👉 Embeddings detect sentiment evolution
12. Bias in Word Embeddings
Why Bias Exists
● Trained on human-written text
● Human language contains bias
Types of Bias
● Gender
● Race
● Profession
● Culture
● Politics
📌 Examples:
● nurse → female
● engineer → male
How Bias Is Learned
● Frequent word co-occurrence
● Model cannot distinguish fact vs stereotype
Impact of Bias
● Reinforces stereotypes
● Affects:
○ Search
○ Recommendation
○ Classification systems
13. Implicit Association Test & Embeddings
● Psychological tests show bias
● Same bias found in embeddings
● Embeddings replicate societal bias
📌 Example:
● African-American names ↔ unpleasant words
● European-American names ↔ pleasant words
14. Embeddings as a Historical Tool
● Used by historians & sociologists
● Reveal:
○ Gender roles
○ Cultural attitudes
○ Stereotype evolution
📌 Language reflects society over time
15. Final Summary (Very Important for Exams)
● Embeddings = vector models of meaning
● Can be:
○ Sparse (TF-IDF)
○ Dense (Word2Vec, GloVe, fastText)
● Excellent for:
○ Similarity
○ Analogy
● Capture:
○ Syntax
○ Semantics
❗
○ Relationships
● Also encode cultural bias
○ Must be handled carefully
✅ Exam Tip
If asked:
● “Why embeddings are useful?”
→ Talk about dense vectors, similarity, analogy, generalization
● “Limitations?”
→ Mention bias and stereotypes