Unit 4

Information Retrieval using NLP

 Contents

• Information Retrieval: Introduction, Vector Space Model

• Named Entity Recognition: NER
• System Building Process, Evaluating NER System Entity Extraction
• Relation Extraction
• Reference Resolution
• Coreference resolution
• Cross Lingual Information Retrieval
 Information Retrieval (IR): Introduction
What is Information Retrieval (IR)?
• Information Retrieval (IR) is the process of obtaining relevant information from a large collection
of unstructured or semi-structured data (e.g., text documents, web pages, or databases).
• It is widely used in search engines, document retrieval systems, and AI-driven knowledge
discovery.

How IR Works
1. Indexing – Data is processed and stored in an optimized structure for retrieval.
2. Query Processing – A user inputs a query, which is interpreted by the system.
3. Matching & Ranking – Relevant documents are retrieved and ranked based on relevance.
4. Presentation of Results – The user receives a ranked list of results.
Applications of IR

• Search engines like Google, Bing

• Library databases (e.g., PubMed, IEEE Xplore)
• E-commerce search (e.g., Amazon product search)
 Vector Space Model (VSM) in Information Retrieval
What is the Vector Space Model (VSM)?

• The Vector Space Model (VSM) is a popular model used in IR where documents and queries are
represented as vectors in an n-dimensional space.
• Each document is treated as a vector of terms (words), and similarity between a query and
documents is measured using techniques like Cosine Similarity.

Steps in VSM Approach

1. Convert documents & queries into vectors (using TF-IDF or Bag of Words).
2. Compute similarity between query and documents (using cosine similarity).
3. Rank documents based on similarity score.
Consider a Small Corpus of Documents
We have the following three documents in our dataset:

D1: "Machine learning is amazing."

D2: "Artificial intelligence and machine learning are related."
D3: "I love learning new things in artificial intelligence.“

And we have a query:

Q: "machine learning intelligence“

Our goal is to find which document is most relevant to the query using VSM and
Cosine Similarity.
Create a Term-Document Matrix
Step 1: Extract Unique Words (Vocabulary)

Term D1 D2 D3 Query
Machine 1 1 0 1
Learning 1 1 1 1
Is 1 0 0 0
Amazing 1 0 0 0
Artificial 0 1 1 1
Intelligence 0 1 1 1
And 0 1 0 0
Are 0 1 0 0
Related 0 1 0 0
I 0 0 1 0
Love 0 0 1 0
New 0 0 1 0
Things 0 0 1 0
Each document and query is now represented as a vector:

•D1: [1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0]
•D2: [1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0]
•D3: [0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1]

•Q: [1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0]
 Compute Cosine Similarity

Compute Dot Products

• Compute Magnitudes

• Compute Cosine Similarities

Ranking the Documents

Based on cosine similarity, we rank the documents for the query "machine learning
intelligence":

Document Cosine Similarity

D2: "Artificial intelligence and machine learning are
0.67
related."
D3: "I love learning new things in artificial
0.57
intelligence."
D1: "Machine learning is amazing." 0.50

D2 is the most relevant document, followed by D3 and D1.

 Vector Space Model + Cosine Similarity
D1:
"Neural network optimization techniques improve deep learning performance in large
scale data environments"
D2:
"Deep learning architectures such as convolutional neural networks are widely used in
medical image analysis"
D3:
"Information retrieval systems apply vector space model and probabilistic ranking
methods“
D4:
"Optimization algorithms including gradient descent improve neural network training
efficiency"
D5:
"Medical diagnosis using machine learning and deep neural networks improves predictive
accuracy"
Q:
"deep neural network optimization for medical diagnosis"
Perform the following steps:
1. Preprocess (lowercase, remove stopwords)
2. Construct full vocabulary
3. Compute:
1. Term Frequency (TF)
2. Document Frequency (DF)
3. IDF using:
𝑁
𝐼𝐷𝐹 𝑡 = log 10 𝑑𝑓
4. Compute TF-IDF vectors
5. Normalize vectors
6. Compute cosine similarity between query and all documents
7. Rank documents
To Sum Up:-

•VSM converts text into vectors and measures similarity mathematically.

•Cosine Similarity is used to rank documents based on their relevance to a query.

•VSM is used in search engines, recommendation systems, and document retrieval.

Named Entity Recognition (NER):

What is Named Entity Recognition (NER)?

Named Entity Recognition (NER) is a NLP technique that identifies and classifies
named entities in text into predefined categories such as:

• Persons (e.g., "Albert Einstein")

• Organizations (e.g., "NASA")
• Locations (e.g., "Himalayas")
• Dates (e.g., "21st June 2023")
• Monetary Values (e.g., "$1000")
 How NER Works?
[Link] – Splitting text into words.
[Link]-of-Speech Tagging – Identifying nouns, verbs, etc.
[Link] Recognition – Assigning labels to detected entities.

Example of NER
• Text:
"Elon Musk, the CEO of Tesla, announced a new AI project in California on
January 10, 2024." Entity Category
Elon Musk PERSON
Tesla ORGANIZATION
NER Output: AI MISCELLANEOUS
California LOCATION
January 10, 2024 DATE
Applications of NER

• News Analytics – Extract key people, places, and dates.

• Chatbots – Identify user intents.
• Healthcare – Extract disease names, drugs, and treatments.
 System Building Process for Named Entity Recognition (NER)
• Named Entity Recognition (NER) is a crucial Natural Language Processing (NLP) task that
extracts and classifies named entities such as persons, locations, organizations, dates, and
more from text.
• The process of building an NER system involves several steps:

Data Collection & Preprocessing

To build an NER system, we first need a dataset containing labelled entities.
Datasets for NER:
• CoNLL-2003 (Widely used, contains PERSON, LOCATION, ORGANIZATION, MISC
categories)
• OntoNotes 5.0 (More diverse dataset with additional entity types)
• Custom dataset (If working on specialized texts-domain specific)
Preprocessing Steps:

•Tokenization → Splitting text into words.

•Lowercasing → Standardizing words (optional).
•Removing special characters & stop words (unless they are meaningful in context).
•Part-of-Speech (POS) Tagging (to improve entity recognition).

Example:
Input Text:
"Elon Musk, CEO of Tesla, announced a new AI project in California on January 10,
2024.“

Preprocessed Tokens:
["Elon", "Musk", ",", "CEO", "of", "Tesla", ",", "announced", "a", "new", "AI",
"project", "in", "California", "on", "January", "10", "2024", "."]
Choosing an NER Model

Several methods exist to develop an NER system:

Approach Description Examples

Uses predefined patterns &
Rule-Based Regular expressions, gazetteers
dictionaries
Uses probabilistic models to Hidden Markov Models (HMMs),
Statistical
identify entities Conditional Random Fields (CRFs)
Uses labeled datasets to train Support Vector Machines (SVMs),
Machine Learning
classifiers CRF-based models
Uses neural networks for higher LSTMs, BiLSTMs, Transformers
Deep Learning
accuracy (BERT, spaCy, Flair)
Example of a Rule-Based Approach:

Using regular expressions to detect dates:

Example of a Deep Learning-Based Approach (Using
spaCy):

Output:-
• Apple Inc. -> ORG
• Steve Jobs -> PERSON
• California -> GPE 1976 -> DATE
Training the NER Model

• If using a machine learning or deep learning approach, we must train the model
with annotated data.
• Feature Selection (for ML models): Word embeddings, capitalization, POS tags,
context.
• Training a Deep Learning Model (like BiLSTM-CRF or BERT-based models).
• Fine-tuning on domain-specific datasets (e.g., Yoga texts, Sanskrit manuscripts,
medical drug discoveries).

Example of Training a Custom Model Using spaCy:

Evaluating the NER System (Entity Extraction Evaluation)

• Once we have built the NER system, we need to evaluate its

performance using different metrics.
• Key Evaluation Metrics for NER
1. Precision
2. Recall
3. F1 Score
 Relation Extraction (RE)
• Relation Extraction (RE) is a Natural Language Processing (NLP) technique used to
identify and classify relationships between entities in text.
• It plays a crucial role in building knowledge graphs, information retrieval, and
question-answering systems.

Types of Relation Extraction

[Link] Learning-based RE: Uses labeled datasets to train models for extracting
relationships.
[Link] Learning-based RE: Identifies patterns in text without labeled data.
[Link]-Supervised & Distant Supervision RE: Uses a small amount of labeled data or
external knowledge bases to learn relationships.
[Link]-based RE: Relies on predefined linguistic patterns and rules.
Example of Relation Extraction
Consider the sentence: 👉 "Albert Einstein was born in Germany."

Entities Identified:
• Albert Einstein (Person)
• Germany (Location)

Extracted Relation:
("Albert Einstein", Born In, "Germany")

Example:
"Google acquired DeepMind in 2014.“

• Entities: Google (Company), DeepMind (Company)

• Extracted Relation: ("Google", Acquired, "DeepMind")
Applications of Relation Extraction

• Knowledge Graph Construction: Extracts relationships for knowledge

graphs like Google’s Knowledge Panel.
• Biomedical NLP: Identifies drug-disease relationships from research
papers.
• Event Extraction: Helps in summarizing news articles by extracting
event relationships.
 Reference Resolution

• Reference resolution is the process of identifying referring expressions (like pronouns, noun
phrases, demonstratives) and linking them to the correct entity mentioned in the discourse.
• It is crucial for understanding context in NLP applications such as chatbots, machine
translation, and summarization.
Example:
Sentence
Riya submitted the paper. She was confident about it.
Here:
She → refers to Riya
it → refers to the paper
The system must correctly link:
(She → Riya)
(it → paper)
Types of Reference Resolution

Coreference Resolution: Determines when two or more expressions

in text refer to the same entity.

"Einstein was a genius. He developed the theory of relativity."

“He" refers to Einstein (Coreference Resolution).

Anaphora Resolution: Identifies pronouns and resolves them to their correct
antecedents.

"John bought a book. The book was about Artificial Intelligence."

"The book" refers to "a book" (Anaphora Resolution).

Cataphora Resolution: Resolves forward references where an entity appears later

in the text

"Before she arrived, Maria had already planned the trip.“

"She" refers to Maria (Cataphora Resolution).

 "Before he could say anything, John was interrupted."
• Resolution: "He" refers to John. (Cataphora Resolution).

"Although he was nervous, John gave a great speech."

• Resolution: "he" refers to John.

Both Anaphora and Cataphora Resolution are essential in NLP for tasks like
pronoun disambiguation, sentiment analysis, and text coherence improvement
in AI-based models
 Coreference Resolution
• Coreference Resolution is the process of identifying and linking multiple
expressions in a text that refer to the same entity.
• This helps in understanding the context of a sentence by associating pronouns
or noun phrases with their corresponding subjects.

Types of Coreference Resolution

[Link] Coreference – Resolving pronouns to their antecedents.

[Link] Coreference – Linking noun phrases that refer to the same entity.
[Link] Coreference – Resolving multiple mentions of the same named entity.
[Link] Anaphora – Understanding implicit relationships between entities.
Examples of Coreference Resolution

Example 1 (Pronominal Coreference):

• 👉 "Elon Musk founded SpaceX. He is also the CEO of Tesla.“

• Resolution: "He" refers to Elon Musk.

Example 2 (Nominal Coreference):

• 👉 "Barack Obama was the 44th U.S. President. The former president was known for his
leadership.“

• Resolution: "The former president" refers to Barack Obama.

Example 3 (Identity Coreference):
• "Amazon acquired Whole Foods. The company made the acquisition for $13.7
billion."
• Resolution: "The company" refers to Amazon.

Example 4 (Bridging Anaphora):

• "I bought a book yesterday. The cover is beautiful."

Resolution: "The cover" is related to the book, though "cover" is not explicitly
mentioned earlier.
 Cross-Lingual Information Retrieval (CLIR)

• Cross-Lingual Information Retrieval (CLIR) is the task of retrieving information

written in one language when queries are issued in another language.
• It is useful for accessing global knowledge without language barriers.

CLIR Working

[Link] Translation: Converting the search query into the target language.
[Link] Translation: Translating indexed documents into the query language.
[Link] Representation: Using a shared representation, such as embeddings,
to match content across languages.
Examples of CLIR

Example 1 (Query Translation Method)

• User Query: "Weather in Paris" (English)
Translation to French: "Météo à Paris"
Retrieved Document: A French weather report about Paris is retrieved and either
presented as is or translated back into English.

Example 2 (Document Translation Method)

• Original Document: A Japanese news article on AI development.
User Query in English: "Latest AI research in Japan.“
Process: The system translates Japanese documents into English and ranks
relevant results.
Example 3 (Interlingual Representation Method using Embeddings)

User Query in Spanish: "Mejorar la productividad“

English Document: "Boosting workplace efficiency“

Matching via Embeddings: The system recognizes "Mejorar la productividad" and

"Boosting workplace efficiency" as similar concepts and retrieves the English
document.
Applications of CLIR
• Multilingual Search Engines (e.g., Google Translate in search).
• Academic Research Retrieval (Accessing research papers in different
languages).
• Legal and Medical Information Access (Providing critical information across
languages).
• Cross-Border Business Intelligence (Understanding foreign market trends).

Both Coreference Resolution and Cross-Lingual Information Retrieval are

crucial for multilingual AI, document understanding, and global knowledge
access! 🌍