NLP Notes
Vector Space Model (VSM) for Information Retrieval
→
1. Vector Space Model (VSM) is an Information Retrieval model used to
represent documents and queries as vectors in multidimensional
space.
2. It helps in finding the similarity between user query and documents.
3. Each document is represented as a vector of terms (words).
4. Every term is given a weight using methods like TF or TF-IDF.
5. User query is also converted into a vector.
6. Similarity between document and query is measured using Cosine
Similarity.
7. Documents with higher similarity score are considered more relevant.
8. VSM is widely used in search engines and text retrieval systems.
Formula
𝐷.𝑄
𝐶𝑜𝑠𝑖𝑛𝑒 𝑆𝑖𝑚𝑖𝑙𝑎𝑟𝑖𝑡𝑦 = |𝐷||𝑄|
Where:
● DDD = Document Vector
● QQQ = Query Vector
Cross-Lingual Information Retrieval (CLIR)
→
1. Cross-Lingual Information Retrieval (CLIR) is an Information Retrieval
technique in which the user searches in one language and retrieves
documents in another language.
2. Example:User enters query in English and gets documents in Hindi or
Marathi.
3. CLIR helps users access information written in different languages.
4. Query and documents are translated into a common language for
matching.
5. Machine Translation techniques are commonly used in CLIR.
6. It supports multilingual search systems and global information access.
7. CLIR reduces language barriers in information retrieval.
8. It is widely used in search engines, digital libraries, and multilingual
websites.
Working of CLIR
● User enters query in source language.
● Query is translated into target language.
● Documents are searched in translated language.
● Relevant documents are retrieved and displayed.
Diagram
User Query (English)
Query Translation
Search in Hindi Documents
Relevant Documents Retrieved
Types of CLIR
Query Translation
Only query is translated.
Document Translation
Documents are translated.
Interlingua Approach
Both converted into common representation.
Named Entity Recognition (NER) and evaluation metrics
→
1. Named Entity Recognition (NER) is an NLP technique used to identify
and classify important entities present in text into predefined
categories.
2. Examples of entities:
● Person Name
● Location
● Organization
● Date
● Time
● Money etc.
3. Example:“Sachin Tendulkar lives in Mumbai.”
● Sachin Tendulkar → Person
● Mumbai → Location
4. NER helps in extracting useful information from unstructured text.
5. It identifies and classifies named entities automatically.
6. Common entity categories are Person, Location, Organization, Date,
and Currency.
7. NER is a part of Information Extraction in NLP.
8. Machine Learning and Deep Learning methods are widely used for
NER.
9. It is used in chatbots, search engines, healthcare, and
recommendation systems.
Working of NER
1. Input text is preprocessed.
2. Important words/entities are identified.
3. Entities are classified into categories.
4. Output is generated with labeled entities.
Diagram
Input Sentence
|
v
"Rohit works at Google in Pune"
|
v
Named Entity Recognition
|
v
Rohit -> Person
Google -> Organization
Pune -> Location
Evaluation Metrics of NER
Evaluation metrics are used to measure the performance of NER systems.
1. Precision
● Measures correctness of identified entities.
● High precision means fewer false positives.
Formula:
𝑇𝑃
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑇𝑃+𝐹𝑃
2. Recall
● Measures how many actual entities are correctly identified.
● High recall means fewer missed entities.
Formula:
𝑇𝑃
𝑅𝑒𝑐𝑎𝑙𝑙 = 𝑇𝑃 + 𝐹𝑁
3. F1-Score
● Harmonic mean of Precision and Recall.
● Most commonly used metric for NER evaluation.
Formula:
2 * 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 * 𝑅𝑒𝑐𝑎𝑙𝑙
𝐹1 = 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙
Markov / Bigram Language Model
→
1. A Markov Language Model is a probabilistic model used in NLP to
predict the next word based on previous words.
In a Bigram Model, the probability of a word depends only on the
immediately previous word.
2. Bigram model is based on the Markov Assumption.
3. It considers only one previous word to predict the next word.
4. Probability of a sentence is calculated using conditional probabilities.
5. It is simpler and faster than higher-order language models.
6. Bigram models are widely used in speech recognition and text
prediction.
7. Accuracy may decrease when context is limited.
Formula
𝑃(𝑤𝑛∣𝑤𝑛 − 1)
Working of Bigram Model
1. Collect text corpus.
2. Count occurrences of word pairs.
3. Calculate transition probabilities.
4. Predict next word using previous word.
Diagram
Previous Word -----> Next Word
I ----------> am
am ----------> learning
learning --------> NLP
Lesk Algorithm for Word Sense Disambiguation.
→
1. Lesk Algorithm is a knowledge-based Word Sense Disambiguation
technique used to determine the correct meaning of a word based on its
context.
2. It works by comparing dictionary definitions (glosses) of words and
finding overlapping words.
3. Example:Sentence: “He sat on the bank of the river.”
● Possible meanings of “bank”:
● Financial institution
● River side
● The algorithm selects “river side” because of overlap with the
word “river”.
4. Lesk Algorithm uses dictionary meanings to identify correct word sense.
5. It compares the gloss of ambiguous word with surrounding context
words.
6. Sense with maximum word overlap is selected.
7. It is one of the earliest WSD algorithms in NLP.
8. No training data is required because it is knowledge-based.
9. Performance depends on quality of dictionary definitions.
Working of Lesk Algorithm
1. Take ambiguous word from sentence.
2. Find all possible meanings from dictionary.
3. Compare gloss of each meaning with context words.
4. Count overlapping words.
5. Select sense with highest overlap.
WordNet / IndoWordNet.
→
1. WordNet is a lexical database used in NLP in which words are grouped
into sets of synonyms called Synsets.
2. It stores semantic relationships between words such as synonym,
antonym, hypernym, and hyponym.
3. IndoWordNet is the Indian language version of WordNet developed for
multiple Indian languages.
4. WordNet organizes words according to their meanings rather than
alphabetical order.
5. Words having similar meanings are grouped into Synsets.
6. It provides semantic relations between words.
7. IndoWordNet supports Indian languages like Hindi, Marathi, Bengali,
Tamil, etc.
8. It helps in Machine Translation, Information Retrieval, and Word Sense
Disambiguation.
9. Widely used in NLP and multilingual language processing systems.
Semantic Relations in WordNet
● Synonym
○ Words with same meaning
● Antonym
○ Opposite meaning words
● Hypernym
○ General category word
● Hyponym
○ Specific type word
IndoWordNet
● Developed for Indian languages.
● Based on expansion approach from Hindi WordNet.
● Maintains semantic relations among Indian language words.
● Useful in multilingual NLP applications.
Applications
● Machine Translation
● Information Retrieval
● Chatbots
● Question Answering Systems
● Word Sense Disambiguation
NLP tools and explain features of NLTK / spaCy / TextBlob /
Gensim
NLP Tools
Some commonly used NLP tools are:
● NLTK
● spaCy
● TextBlob
● Gensim
● Stanford NLP
● OpenNLP
● CoreNLP
These tools help in text processing, tokenization, parsing, sentiment
analysis, and machine learning tasks.
1. NLTK (Natural Language Toolkit)
NLTK is a popular Python library used for Natural Language Processing
and text analysis.
Features of NLTK
● Provides tokenization and stemming techniques.
● Supports POS tagging and parsing.
● Includes WordNet for semantic analysis.
● Contains large text corpora and lexical resources.
● Useful for teaching and research in NLP.
● Easy to use for beginners.
Applications
● Text classification
● Sentiment analysis
● Language processing research
Diagram
Input Text
Tokenization
POS Tagging
Parsing
Processed Output
2. spaCy
spaCy is an industrial-level NLP library designed for fast and efficient text
processing.
Features of spaCy
● Very fast and memory efficient.
● Supports Named Entity Recognition (NER).
● Provides dependency parsing and POS tagging.
● Supports deep learning integration.
● Useful for large-scale NLP applications.
● Pretrained language models are available.
Applications
● Chatbots
● Information extraction
● Real-time NLP systems
3. TextBlob
TextBlob is a simple Python NLP library built on top of NLTK and Pattern.
Features of TextBlob
● Easy to learn and use.
● Supports sentiment analysis.
● Provides noun phrase extraction.
● Performs spelling correction.
● Supports translation and text classification.
● Good for beginners and rapid prototyping.
Example
Sentence:
“I love NLP”
Sentiment:
Positive
4. Gensim
Gensim is an NLP library mainly used for topic modeling and vector space
modeling.
Features of Gensim
● Efficient for large text collections.
● Supports Word2Vec and Doc2Vec models.
● Used for topic modeling with LDA.
● Provides document similarity analysis.
● Memory-efficient processing.
● Widely used in semantic analysis.
Applications
● Topic extraction
● Document similarity
● Recommendation systems
Advantages of NLP Tools
● Reduce manual text processing effort
● Improve accuracy and speed
● Useful for research and industrial applications
Explain Latent Dirichlet Allocation (LDA) for topic modeling
→
1. Latent Dirichlet Allocation (LDA) is an unsupervised machine learning
algorithm used for topic modeling in NLP.
2. It identifies hidden topics from a collection of documents a document
contains multiple topics.
3. Each topic contains a group of related words.
4. LDA automatically discovers topics from large text data.
5. It represents documents as a mixture of topics.
6. Each topic is represented by probability distribution of words.
7. It is widely used in text mining and document classification.
8. LDA is a probabilistic generative model.
9. Commonly used in news analysis, recommendation systems, and
research papers.
Working of LDA
1. Collect document dataset.
2. Assume a fixed number of topics.
3. Assign topics to words probabilistically.
4. Find frequently occurring word patterns.
5. Generate topic-word and document-topic distributions.
Example
Document Collection:
● Sports articles
● Politics articles
● Technology articles
LDA identifies topics like:
● Sports → player, match, score
● Politics → election, government, minister
● Technology → AI, computer, software
Calculate TF-IDF score / Explain TF-IDF
→
1. TF-IDF is a statistical technique used in NLP and Information Retrieval
to measure the importance of a term in a document.
2. TF measures how frequently a term occurs in a document.
3. IDF measures how unique or rare the term is across all documents.
4. Higher TF-IDF score means the term is more important.
Formula
Term Frequency
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑜𝑐𝑐𝑢𝑟𝑟𝑒𝑛𝑐𝑒𝑠 𝑜𝑓 𝑡𝑒𝑟𝑚
𝑇𝐹 = 𝑇𝑜𝑡𝑎𝑙 𝑡𝑒𝑟𝑚𝑠 𝑖𝑛 𝑑𝑜𝑐𝑢𝑚𝑒𝑛𝑡
Inverse Document Frequency
𝑁
𝐼𝐷𝐹 = 𝑙𝑜𝑔 ( 𝐷𝐹 )
TF-IDF
TF-IDF = TF * IDF
Where:
● NNN = Total number of documents
● DFDFDF = Number of documents containing the term
Given Document-Term Matrix
Document 1 Document 2 Document 3
Term 1 10 5 0
Term 2 2 0 8
Term 3 1 3 6
Step 1 : Calculate TF
For Document 1:
Total terms:
10+2+1=13
TF of Term 1:
10
TF= 13
=0.769
Step 2 : Calculate IDF
Total documents:
N=3
Document Frequency of Term 1:
● Appears in Document 1 and Document 2
● Therefore: DF=2
IDF:
3
𝐼𝐷𝐹 = 𝑙𝑜𝑔 ( 2 ) = 0. 176
Step 3 : Calculate TF-IDF
TF-IDF=0.769×0.176
TF-IDF=0.135
The TF-IDF score of Term 1 in Document 1 is: 0.135
Consider the following three documents:
• Dl: “neural networks are powerful”
• D2: deep learning powers neural models”
• D3: “networks and models are important”
1) Compute the TF-IDF score of the term “neural” in all three
documents.
2) Show how document frequency (DF) and inverse
document frequency (IDF) are computed.
3) Interpret the results to determine in which document
“neural” is more significant?
1) Compute TF-IDF Score of “neural”
Step 1 : Calculate Term Frequency (TF)
Formula:
𝐹𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦 𝑜𝑓 𝑡𝑒𝑟𝑚 𝑖𝑛 𝑑𝑜𝑐𝑢𝑚𝑒𝑛𝑡
𝑇𝐹 = 𝑇𝑜𝑡𝑎𝑙 𝑤𝑜𝑟𝑑𝑠 𝑖𝑛 𝑑𝑜𝑐𝑢𝑚𝑒𝑛𝑡
Document D1
Words in D1 = 4
“neural” appears = 1 time
1
𝑇𝐹(𝐷1) = 4
= 0. 25
Document D2
Words in D2 = 5
“neural” appears = 1 time
1
𝑇𝐹(𝐷2) = 5
= 0. 20
Document D3
Words in D3 = 5
“neural” appears = 0 time
TF(D3)=0
2) Compute DF and IDF
Document Frequency (DF)
“neural” appears in:
● D1
● D2
Therefore:
DF=2DF=2DF=2
Total documents:
N=3N=3N=3
Inverse Document Frequency (IDF)
Formula:
𝑁
𝐼𝐷𝐹 = 𝑙𝑜𝑔( 𝐷𝐹 )
Substituting values:
3
𝐼𝐷𝐹 = 𝑙𝑜𝑔( 2 ) = 0. 176
3) Calculate TF-IDF
Formula:
TF-IDF=TF×IDF
For D1
TF-IDF(D1)=0.25×0.176=0.044
For D2
TF-IDF(D2)=0.20×0.176=0.035TF\text{-}IDF(D2)=0.20\times0.176=0.035T
F-IDF(D2)=0.20×0.176=0.035
For D3
TF-IDF(D3)=0×0.176=0
Final TF-IDF Scores
Document TF-IDF Score
D1 0.044
D2 0.035
D3 0
Explain Information Retrieval in NLP.
→
1. Information Retrieval (IR) is the process of searching and retrieving
relevant information or documents from a large collection of data based
on a user query.
2. IR helps users find relevant documents quickly.
3. It matches user queries with stored documents.
4. Documents are indexed for efficient searching.
5. Ranking algorithms are used to show most relevant results first.
6. NLP techniques improve understanding of user queries.
7. IR systems are used in Google Search, chatbots, and recommendation
systems.
Components of IR System
● Document Collection
● Indexing
● Query Processing
● Ranking Mechanism
● Retrieval Module
It is widely used in:
● Search engines
● Digital libraries
● Question answering systems
● NLP applications
Statistical Machine Translation (SMT) vs Explain Rule-Based
Machine Translation (RBMT)
Basis Statistical Machine Rule-Based Machine
Translation (SMT) Translation (RBMT)
Definition Translation is performed using Translation is performed
statistical and probability using predefined linguistic
models. rules.
Working Uses bilingual text corpus and Uses grammar rules and
Principle probabilities. dictionaries.
Dependenc Depends heavily on large Depends on manually
y training datasets. created language rules.
Translation Predicts most probable Converts source language
Method translation. using linguistic rules.
Accuracy Improves with more training Accuracy depends on
data. quality of rules.
Human Less manual rule creation Requires extensive manual
Effort required. rule development.
Flexibility Easily adaptable to new Difficult to adapt because
languages with data. new rules are needed.
Grammar May produce grammatically Better grammatical
Handling incorrect sentences. structure due to language
rules.
Processing Faster after training. Slower because of
Speed complex rule processing.
Example Google Translate (older SYSTRAN
Systems versions)
Advantages Handles large data efficiently Produces linguistically
and learns automatically. meaningful translations.
Limitations Requires huge parallel corpus Rule creation and
data. maintenance are difficult.
Natural Language Generation (NLG) with reference
architecture
→
1. Natural Language Generation (NLG) is a branch of NLP that converts
structured data or machine representation into human-readable natural
language text.
2. It automatically generates meaningful sentences, paragraphs, or
reports from data.
3. Examples:ChatGPT responses, Weather reports, Automated news
generation, Voice assistants
4. NLG transforms data into understandable language.
5. It generates grammatically correct and meaningful text.
6. Used in chatbots, report generation, and virtual assistants.
7. NLG systems use linguistic and AI techniques.
8. It improves human-computer interaction.
9. Widely used in healthcare, finance, and customer support systems.
Structured Data
Content Determination
v
Document Planning
Sentence Planning
Surface Realization
Natural Language Output
Main Components of NLG System
1. Content Determination
● Selects important information from input data.
2. Document Planning
● Organizes information in proper sequence.
3. Sentence Planning
● Chooses words, sentence structure, and grammar.
4. Surface Realization
● Generates final natural language sentences.
5. Output Generation
● Produces readable text for users.