CSI 3004 – Text Mining

Faculty Name : Yuvaraj Natarajan

School : SCOPE
Cabin : SJT 613A
1

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 CSI 3004 – Text Mining

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 What is Text Mining ?

Text mining is the science

of transforming raw text
into meaningful insights.

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Why Text Mining ?

v Almost 80 percent of all data in the

world is unstructured, and most of
that is text: emails, tweets, reviews,
medical reports, legal documents,
research articles, and more.

v Understanding this unstructured

information is now critical for industries
ranging from health care and finance to
retail and cyber security.
4

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 What Text Mining Actually Does?

Machines cannot understand language directly; We must

convert text into numerical representations & extract patterns.

Key capabilities to explore (But not limited to):

• Identifying important words and topics from documents.
• Classifying text into categories like spam or non spam.
• Extracting opinions and emotions from social media posts.
• Discovering relationships / patterns hidden in text corpora.
• Summarizing long documents into a few sentences.

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Example

“The battery life of this phone is amazing.”

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Example – (How Text Mining Predicts
Sentiment)

“The battery life of this

phone is amazing.”

Text mining transforms raw text into numerical features, uses

learned patterns from past data, and automatically identifies that
words like “amazing” express a positive sentiment.

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Introduction - What is Information Extraction (IE)?

• IE is the process of automatically identifying key

facts from raw text.
• Converts text into entities, relations, and events.

Example:
“Dr. Meera visited Apollo Hospital on Monday.”
Entity: Dr. Meera ; Entity: Apollo Hospital
Relation: visited. ; Event: medical consultation

•Entity: The main actors or objects in the text.

•Relation: The connection between those entities.
•Event: The action or occurrence that involves the entities. 8

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Introduction - What is Information Extraction (IE)?

Example :
“Google acquired DeepMind in 2014 to advance
artificial intelligence research.”

Entities: Google, DeepMind, 2014, artificial intelligence research

Relations: acquired, aimed at
Event: Acquisition for AI research expansion

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Introduction - Why we need Information Extraction?

Why ?

v The web generates billions of text documents every

day; humans cannot manually read or organize them.
v IE turns unstructured text into structured,
analyzable knowledge.
v IE powers real-world systems: chatbots, search
engines, fraud detectors, medical decision systems.

10

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Introduction - Why Information Extraction Important?

Why IE Matters ?

v Helps machines “understand” language.

v Enables deeper text analytics, knowledge graphs,
intelligent search, and automation.

11

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Introduction - Difference Between IR, NLP and IE

Task Purpose Example

Google search for "COVID
IR (Information Retrieval) Finding relevant documents
symptoms"
NLP (Natural Language Understanding language POS tagging, parsing,
Processing) structure and meaning sentiment
IE (Information Extraction)
Pulling out facts and Extracting disease names
relationships from articles

12

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Introduction - Relationship Between IE, IR, and NLP

How They Work Together ?

v NLP provides linguistic tools (tokenization, POS tags).

v IR retrieves documents from massive corpora.
v IE extracts meaningful knowledge from those
documents.
v Together they form the backbone of modern
intelligent systems.

13

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Introduction - IE Pipeline Overview

Preprocessing Annotation Extraction Evaluation

Clean the text Label words with Identify entities, Measure accuracy
(tokenize, normalize, linguistic information relations, events, (precision, recall, F1
remove noise). (POS, Named Entity facts. score).
Recognition (NER
tags)).

14

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Introduction - Visual Pipeline of IE

Split Labelled
Identify Extract Stored in
into with
Named Knowledge
Text Tokens POS tags Relations
Entities graph

A sentence is gradually converted into structured knowledge

by identifying words, tagging them, finding entities, discovering
relations, and building a knowledge graph.

15

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Introduction - Visual Pipeline of IE (Example)
Split Labelled
Identify Extract Stored in
into with
Named Knowledge
Text Tokens POS tags Relations
Entities graph

“Virat Kohli scored 82 runs at the MCG during the 2022 T20 World Cup.”

Text
“Virat Kohli scored 82 runs at the MCG during the 2022 T20 World Cup.”

Tokens
["Virat", "Kohli", "scored", "82", "runs", "at", "the", "MCG",
"during", "the", "2022", "T20", "World", "Cup"]
16

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Introduction - Visual Pipeline of IE (Example)
Split Labelled
Identify Extract Stored in
into with
Named Knowledge
Text Tokens POS tags Relations
Entities graph

“Virat Kohli scored 82 runs at the MCG during the 2022 T20 World Cup.”
POS Tags
•Virat → Noun
•Kohli → Noun Named Entities
•scored → Verb •Virat Kohli → Person
•82 → Number •MCG → Location
•MCG → Proper noun •2022 T20 World Cup → Event
•2022 → Number •82 runs → Performance metric
•World Cup → Proper noun
phrase 17

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Introduction - Visual Pipeline of IE (Example)
Split Labelled
Identify Extract Stored in
into with
Named Knowledge
Text Tokens POS tags Relations
Entities graph

“Virat Kohli scored 82 runs at the MCG during the 2022 T20 World Cup.”

Relations
scored → relation between Virat Kohli and 82 runs Knowledge Graph Output
(Virat Kohli → scored → 82 runs) A structured graph would contain edges
at → relation between action and location such as:
(scored → at → MCG) Virat Kohli -scored→ 82 runs
during → relation between action and event Virat Kohli -played at→ MCG
(scored → during → 2022 T20 World Cup) Performance -during→ 2022 T20 World Cup

18

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Introduction - Visual Pipeline of IE (Example)
Split Labelled
Identify Extract Stored in
into with
Named Knowledge
Text Tokens POS tags Relations
Entities graph

“Dr. Arjun prescribed Metformin to a diabetic patient at Apollo

Clinic last Friday.”

19

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Introduction - Visual Pipeline of IE (Example)
Split Labelled
Identify Extract Stored in
into with
Named Knowledge
Text Tokens POS tags Relations
Entities graph

“Sara tweeted that the new iPhone 16 camera is amazing and

tagged Apple in her post.

20

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Introduction - Applications of Information Extraction

Business
Healthcare Social Media Cyber
Intelligence in
Analytics Security
Finance

21

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Introduction : Future Scope and Career Relevance

v IE is at the heart of search engines, voice assistants, recommender

systems, and generative AI.
v Every industry needs systems that convert text into actionable insights.

Skills in IE open pathways to careers in:

v Data science
v AI engineering
v Information security
v Health informatics
v Knowledge graph engineering

22

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Named Entity Recognition (NER)

23

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Named Entity Recognition (NER)

NER is the task of automatically identifying important

real-world entities in text.

• Extracts “who, where, what, when” from unstructured text.

• Key step in transforming text into structured knowledge
used by search engines, chatbots, RS, etc...

Example:
Text: “Elon Musk visited Bengaluru in January.”
NER Output: Elon Musk (Person), Bengaluru (Location), January (Date)

24

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 NER Process Steps

25

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Why NER is Important ?

• Reduces manual annotation effort by automatically

identifying key elements.
• Helps build knowledge graphs, question
answering systems, and summarization engines.
• Enables intelligent applications in healthcare,
finance, customer support, and cybersecurity.

26

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 NER - Common Entity Types

Person – “Virat Kohli”, “Dr. Meera”

Organization – “Google”, “WHO”, “Apollo Hospital”
Location – “Delhi”, “Japan”, “MCG Stadium”
Date / Time – “2022”, “last Friday”
Product – “iPhone 16”
Event – “Olympics”, “T20 World Cup”
Medical Entities – diseases, drugs, symptoms
(important in healthcare IE)

27

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 NER - Common Entity Types

“Tesla’s new autonomous vehicle just completed a

1000-mile journey across the US with zero driver
intervention. #AutonomousVehicles #Tesla”

•Tesla → Organization
•autonomous vehicle → Product
•1000-mile journey → Event
•US → Location

28

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 How NER Works – Approaches ?

A. Rule Based NER

• Uses handwritten patterns, dictionaries, regular expressions.
• Good for well structured text (ex: medical prescriptions).
• Limitation: hard to scale and maintain.
B. Statistical NER
• Uses probabilistic models trained on annotated datasets.
• Popular early method: Conditional Random Fields (CRF).
• Learns patterns - capital letters, context words, part of speech.
C. Deep Learning NER
• Current state-of-the-art.
• Learns contextual meaning from millions of sentences.
• Handles slang, typos, abbreviations better.

29

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 How CRF-Based NER Works ?

CRF-Based NER
• CRF models predict entity labels by considering
relationships between neighboring words.
• Effective for sequence labeling tasks.
• Still widely used for domain-specific NER (legal,
biomedical).

Example:
Tokens: [Dr., Meera, visited, Apollo, Hospital]
Labels: [Title, Person, O, Organization, Organization]

“O”. - represents outside (Not an entity) 30

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 How Neural NER Works ?

Neural NER – BiLSTM-CRF

• Combines:
• BiLSTM – learns long-term context from both left
and right directions.
• CRF – ensures coherent label sequences.

• Significantly improves accuracy in noisy text and social

media content.

BiLSTM reads sentences like humans: from left to right

and right to left to understand meaning.

31

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 How Transformer-Based NER Works?

Transformer-Based NER (Modern Approach)

Models like BERT, DistilBERT revolutionized NER.
Key strengths:
• Understand context deeply (ex: “Apple” as fruit vs company).
• Handle domain adaptation better.
• Work extremely well for large scale IE and knowledge
graph construction.

Sentence: “Apple announced new products in California.”

BERT knows “Apple” = Company, not fruit.

32

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 NER Challenges

1. Ambiguity. (Lexical / Syntactic)

2. Domain Adaptation
3. Multilingual NER
4. Noisy Text

33

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Relation Extraction (RE)

34

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Relation Extraction (RE)

• Relation Extraction (RE) is the process of identifying and classifying

relationships between entities in text.
• For example, recognizing that "Virat Kohli" plays for "Royal
Challengers Bangalore".

RE allows systems to construct structured knowledge from unstructured

text, which is crucial knowledge graph creation & information retrieval.

“Elon Musk founded SpaceX in 2002.”

Entities: “Elon Musk” (Person), “SpaceX” (Organization), “2002” (Date)

35

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Relation Extraction (RE) - Types of Relations

Types of Relations
Binary Relations: The simplest form of relations, involving two
entities.
Example: "Albert Einstein" was born in "Ulm.”

N-ary Relations: Relations involving more than two entities.

Example: "Apple Inc." released "iPhone 12" on "October 13, 2020."
Binary relations are more commonly extracted than n-ary, though n-
ary relations are crucial for capturing complex interactions.

36

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Relation Extraction (RE) - Types of Relations

v Classify a sentence: “Bill Gates co-founded

Microsoft with Paul Allen in 1975”
Entities: Bill Gates (Person), Microsoft
(Organization), Paul Allen (Person), 1975 (Date)
Relation: Co-founded

v Whether this is binary or n-ary ?

37

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Relation Extraction (RE) – Supervised ML

• Supervised Learning: Involves training

models on labeled data to classify relations
• Common algorithms include Decision Trees,
SVM, and Neural Networks.
• Training Process: Requires annotated
corpora where relationships are explicitly
labeled.

• Supervised models can be highly accurate but require large amounts of labeled data, which can be a
limitation. Example - Try to build a simple classifier.
38

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Neural Approaches for Relation Extraction (RE)

v Convolutional Neural Networks (CNN): Used for extracting local

patterns in the text to identify relations - used in sentence classification tasks.
v Recurrent Neural Networks (RNN): Effective for modeling sequences of
words & capturing contextual dependencies between words for relation
extraction.
v Transformers: Modern architectures like BERT, which are state-of-the-art for
RE tasks - Handle long-range dependencies and context better than RNNs
and CNNs.
Transformers has improved performance RE, with BERT-based models achieving
superior performance over traditional methods.

39

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Evaluation Metrics - Relation Extraction (RE)

40

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Evaluation Metrics - Relation Extraction (RE)

Question: Consider a dataset with 10 sentences. After performing

Relation Extraction (RE), the system identifies 6 relations (out of which 4
are correct, and 2 are incorrect). There are also 3 relations that the system
missed.

True Positives (TP): The correct relations identified by the system (4).
False Positives (FP): Incorrect relations identified by the system (2).
False Negatives (FN): Correct relations that were not identified by the
system (3).
True Negatives (TN): The number of cases where the system correctly
identified no relation (0 relations, as all sentences had relations).

41

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Evaluation Metrics - Relation Extraction (RE)

Using the provided information, calculate the following

metrics for the Relation Extraction (RE) task:
• Precision
• Recall
• F1 Score
• Accuracy

Summary of the Results:

•True Positives (TP) = 4
•Precision = 0.67
•False Positives (FP) = 2
•Recall = 0.57
•False Negatives (FN) = 3 •F1 Score = 0.61
•True Negatives (TN) = 0
•Accuracy = 0.44
42

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Relation Extraction (RE) – Use-cases

Knowledge Graphs:
RE helps build knowledge graphs by linking entities and their relations.
These graphs are essential for tasks like search engines and recommendation
systems.

Question Answering:
By extracting relations, RE helps answer complex questions by identifying the
key entities and their relations in the context.

Explore how RE is foundational in advanced AI systems like virtual

assistants (e.g., Siri, Alexa) and search engines (e.g., Google).

43

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Unsupervised Information Extraction

44

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Unsupervised Information Extraction - Introduction

What is Unsupervised Information Extraction

(IE)?
• Unsupervised IE refers to extracting useful
information (like entities, relations, or events) from
text without labeled training data.

• It involves techniques where the system learns

patterns and structures from raw data, without
predefined labels, making it particularly useful when
labeled data is scarce or expensive to acquire.

45

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Unsupervised Information Extraction - Advantages
• Limited Labeled Data: In many cases, annotating
large corpora is time-consuming and expensive.
• Scalability: Can scale better, especially in dynamic
environments where new information is
continuously generated.
• Flexibility: Can adapt to various domains without
needing domain-specific labeled data.

Example:
Extracting relations like "works_for" between entities
like "Steve Jobs" and "Apple" from raw news
articles, without having labeled datasets for training.
46

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Unsupervised Information Extraction - Introduction

Clustering-Based Information Extraction

v These methods group similar instances together based
on patterns or features.
v Once the data is clustered, relationships within the
same group can be inferred.
v Example: Cluster sentences containing similar
named entities and extract relations based on
co-occurrence patterns.

47

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Unsupervised Information Extraction - Introduction

Types of Clustering for IE:

• Entity Clustering: Grouping similar entities together, such as
companies, people, or locations.
• Relation Clustering: Identifying clusters of similar relations
from text.

Bag of Words Representation: Text is represented as a set of

words without considering grammar or word order, simplifying
the clustering process.

48

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Unsupervised Information Extraction - Steps
Step-by-Step Workflow for Unsupervised IE
Step 1: Data Collection
Goal: Collect unstructured data (e.g., product reviews).
Tool like Scrapy / BeautifulSoup for web scraping.

Step 2:Text Preprocessing

Goal: Clean and prepare data for extraction.
Steps:
• Tokenization: Break sentences into words (NLTK).
• Stopword Removal: Remove common words.
• Stemming/Lemmatization: Reduce words to their
base form.
49

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Unsupervised Information Extraction - steps

Step 3: Entity Recognition (Clustering) - Identify and group

similar words (e.g., products, features).
Technique: Any clustering algo for grouping entities.
Example: Cluster “iPhone”, “Samsung” as Products and
“camera”, “battery” as Features.

Step 4: Relation Extraction - Identify relationships between

entities (e.g., “iPhone 13 - has_feature - camera”).
Techniques:
Pattern-based Extraction: Identify patterns like "X such as Y"
OpenIE: Use frameworks OpenIE to extract triplets.
Example: Extract relation "Apple" has_feature "camera".
50

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Unsupervised Information Extraction - steps

Step 5: Post-Processing & Structuring Data -

Organize extracted relations
Example: Store relations in a Knowledge Graph or DB.

Step 6: Evaluation - Assess performance using metrics

like Precision, Recall, and F1 Score.
Example:
Precision = TP / (TP + FP)
Recall = TP / (TP + FN).

51

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Unsupervised Information Extraction - (OpenIE) Frameworks

Open Information Extraction (OpenIE) Frameworks

• Unlike traditional IE systems, OpenIE does not assume a
predefined set of relations.
• It extracts a wide variety of relations from text, making it
flexible and suitable for many domains.

Features:
• Extracts triplets: Subject - Relation - Object (e.g.,
"Steve Jobs - founded - Apple").
• Does not require predefined relations or labeled data.
• OpenIE frameworks can operate on raw text and extract
knowledge without domain-specific training.
52

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Unsupervised Information Extraction - Challenges

• Ambiguity: Words or phrases may have multiple

meanings, making it difficult to extract accurate relations.
• Context: The same relation might be expressed in many
ways across different contexts.
• Noise: Unsupervised methods are often prone to errors
due to noisy text or incorrect entity alignment.

53

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Unsupervised Information Extraction - Use Case

Extracting Product-Related Information from E-commerce Reviews

Scenario:
• You work at an e-commerce company and want to extract useful information from
customer reviews to understand the relations between products and features mentioned by
customers.
• For example, you want to identify relations like "product_name - has_feature - feature"
(e.g., "iPhone 13 - has_feature - camera") from customer reviews.
• This is an unsupervised information extraction (IE) task, where you don’t have labeled data
to train the model.
• You want to use unsupervised algorithms to extract key information (like product names
and their features) from thousands of customer reviews.

54

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Text Summarization

55

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Text Summarization - Introduction

v Text Summarization is the process of reducing a large

body of text into a shorter version, maintaining key
information and the essence of the original content.
v The goal is to create a concise summary while
preserving critical facts and meaning.
v Example: This is especially useful in applications like
news aggregation, research paper summarization, and
customer review analysis

56

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Types of Text Summarization

1. Extractive Summarization - Extracts key sentences,

phrases, or passages directly from the source text to create a
summary. No modification of the original sentences.
• Example: Selecting a sentence like "The iPhone 13's
camera is the best ever" directly from the article as part of
the summary.
2. Abstractive Summarization - Generates new sentences
by understanding the meaning of the text and rephrasing it in
a shorter form.
• Example: "The iPhone 13's camera has been improved
significantly" (paraphrased).

57

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Indicative vs Informative Summarization
• Indicative Summarization: Summarizes the
theme of the document. It indicates what the text
is about but doesn’t provide detailed information.
• Example: "This article is about the advancements in
smartphone camera technology.”

• Informative Summarization: Provides detailed

information about the key points in the text. It’s
more comprehensive than indicative summaries.
• Example: "This article discusses the new features of
the iPhone 13's camera, including improved low-light
performance and the new night mode."
58

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Generic vs Query-Focused Summarization

• Generic Summarization: Summarizes the entire

document without a specific focus. It cover all
major points but is not tailored for a specific question.
• Example: A summary of an entire news article or
research paper.
• Query-Focused Summarization: Summarizes
the text with a focus on answering a specific
query
• Example: "What new features does the iPhone 13
camera offer?" – The summary focuses only on
answering this question.

59

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Applications of Text Summarization

• News: Summarizing long news articles for quicker

consumption (e.g., Google News).
• Legal Domain: Summarizing long legal documents
to highlight key points (e.g., Case law summaries).
• Research Papers: Automatically summarizing
academic papers to assist researchers in quickly
understanding the content.
• Customer Reviews: Summarizing customer
feedback for products or services (e.g., Amazon
Reviews).

60

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Applications of Text Summarization

Which type of summarization (Extractive / Abstractive)

would be most effective for each of the following?

v News Articles:
v Legal Documents:
v Research Papers:
v Customer Reviews:

Answer:
News Articles: Extractive Summarization
Legal Documents: Extractive Summarization
Research Papers: Abstractive Summarization
Customer Reviews: Abstractive Summarization.
61

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Future Directions

Future Directions:
v Moving towards more sophisticated abstractive models
like GPT-3 and BERT for better quality summaries.
v Applications will continue to grow in domains like business
intelligence, education, and customer service.

62

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Topic Representation Approaches

63

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Topic Representation Approaches Topic Representation Approaches (TRA)

vTopic Representation refers to the techniques used to

represent and identify the topics within a collection of documents.
vThese approaches are fundamental in information retrieval, text
summarization, and natural language processing tasks.

Why is it important?
vHelps categorize large text datasets.
vExtracts the main topics from documents to aid in clustering,
summarization, and classification tasks.

64

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Term Frequency (TF) and Term Frequency-Inverse Document
Frequency (TF-IDF)
Topic Representation Approaches

Term Frequency (TF): Measures how frequently a term appears in a

document. It is computed as:
Number of times term t appears in document d
TF 𝑡 𝑑 =
Total number of terms in document d
Term Frequency-Inverse Document Frequency (TF-IDF): Weighs
terms by their frequency in a document and how rare the term is across the
entire corpus. It reduces the weight of common words (like "is", "the") and
increases the weight of rare, significant words.
Total number of documents
TF−IDF 𝑡 𝑑 = TF 𝑡 𝑑 × log
Number of documents containing t
65

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Example Problem for TF and TF-IDF

Consider the following 3 documents: Doc1: "Machine learning is a field of AI."

Doc2: "Deep learning is a subfield of machine learning."
Doc3: "AI is transforming industries."
1. Calculate the TF for the term "machine" in Doc1.
2. Calculate the TF-IDF for the term "learning" in Doc2 using a corpus of 3
documents.
TF Calculation:
TF for "machine" in Doc1 = 1/6 (since "machine" appears once out of 6 words in Doc1).
TF-IDF Calculation:
TF for "learning" in Doc2 = 1/6 (appears once in Doc2, total 6 words).
!
IDF for "learning" = log " , since "learning" appears in 2 out of 3 documents.
TF-IDF = TF * IDF = 0.068
66

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Bag-of-Words (BoW)

vBag-of-Words (BoW) is a text representation technique

where each document is represented as a collection of the
words it contains, disregarding grammar and word order, but
keeping track of word frequencies.
vBoW Model: Represents a document as a vector where
each dimension corresponds to a word in the vocabulary,
and the value in that dimension is the frequency (or
presence/absence) of the word in the document.

67

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Bag-of-Words (BoW)

Steps in BoW:
1. Tokenization: Split the text into individual words (tokens).
2. Vocabulary Creation: Create a list of unique words
(vocabulary) from the entire corpus (all documents).
3. Vector Representation: For each document, represent it as a
vector, where each entry in the vector corresponds to the
frequency of a word in the document.

68

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Bag-of-Words (BoW)

Example: Given these two sentences:

Doc1: "I love programming."
Doc2: "Programming is fun."
The vocabulary would be: ["I", "love", "programming", "is", "fun"]
The BoW vectors:
Doc1: [1, 1, 1, 0, 0] (I=1, love=1, programming=1, is=0, fun=0)
Doc2: [0, 0, 1, 1, 1] (I=0, love=0, programming=1, is=1, fun=1)
BoW does not account for word order or context, only the
presence/absence and frequency of words.

69

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Cosine Similarity

• Cosine Similarity is a measure of similarity between two non-zero

vectors of an inner product space.
• It is calculated by measuring the cosine of the angle between two
vectors. The smaller the angle, the more similar the vectors are.
Cosine Similarity Formula:
𝐴⋅𝐵
Cosine Similarity =
∥ 𝐴 ∥∥ 𝐵 ∥
Where:
A and B are vectors representing two documents,
𝐴 ⋅ 𝐵is the dot product of the vectors,
∥ 𝐴 ∥∥ 𝐵 ∥is the product of the magnitudes (norms) of the vectors.

70

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Cosine Similarity

• Range: Cosine similarity values range from 0 to 1, where:

• 1 means the vectors are identical (i.e., the documents are
identical),
• 0 means the vectors are orthogonal (i.e., no similarity).
• Usage: It’s widely used to measure the similarity between two text
documents by comparing the cosine of the angle between their
vector representations.

71

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Cosine Similarity

Example: Let’s calculate the cosine similarity between the following two
document vectors (BoW vectors):

v Doc1: [1, 2, 0, 0] (word1=1, word2=2, word3=0, word4=0)

v Doc2: [1, 0, 1, 1] (word1=1, word2=0, word3=1, word4=1)

Step 1: Define the Document Vectors

•Doc1 = [1, 2, 0, 0]
•Doc2 = [1, 0, 1, 1]

Step 2: Calculate the Dot Product

Dot Product = 1×1 + 2×0 + 0×1 + 0×1 = 1
72

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Cosine Similarity

Step 3: Calculate the Magnitudes of the Vectors

∥ Doc1 ∥= 1" + 2" + 0" + 0" = 5 ≈ 2.236
∥ Doc2 ∥= 1" + 0" + 1" + 1" = 3 ≈ 1.732

Step 4: Apply the Cosine Similarity Formula

1 1
Cosine Similarity = = ≈ 0.258
2.236×1.732 3.873

Final Answer:
•Cosine Similarity = 0.258.
73

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Latent Semantic Analysis (LSA)
LSA is a technique for extracting the underlying meaning or topics in a
set of documents by analyzing patterns of word co-occurrence.
LSA working:
1. Create a term-document matrix.
2. Apply Singular Value Decomposition (SVD) to reduce the
dimensions and extract latent topics.
3. The reduced matrix reveals patterns of associations between terms,
thus identifying topics in the text.
SVD breaks down the term-document matrix into singular values,
enabling dimensionality reduction & capturing hidden semantic structures.
Example: Given a set of news articles, LSA can identify topics like "Politics",
"Technology", and "Economy” based on word co-occurrences.
74

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Latent Dirichlet Allocation (LDA)

LDA is a generative probabilistic model that assumes each document

is a mixture of topics and each topic is a mixture of words.
It is widely used for topic modeling.
LDA working:
1. Each document has a mix of topics.
2. Each topic has a probability distribution over words.
3. LDA assigns each word in a document to a topic based on its likelihood.
LDA is typically trained using - Markov Chain Monte Carlo method
to infer the topic distributions.
Example: Use LDA on a collection of blog posts to discover latent
topics like "AI", "Healthcare", "Finance" from the text.
75

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Topic Modelling in Summarization

v Topic modeling techniques like LDA and LSA can be used to identify the
main themes in a document.
v These topics can be used as the basis for generating summaries, ensuring
that the most important content is included.

Summarization systems can use topic modelling to extract key sentences or

paragraphs that are most representative of the main topics.

Example: Given a set of research papers, LDA can identify the main topics
like "Deep Learning" or "Computer Vision", and the summary would
include sentences that describe these topics.

76

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Semantic Embeddings (Word2Vec, GloVe, BERT)

Semantic Embeddings (Word2Vec, GloVe, BERT)

v Word2Vec: A model that represents words in a continuous vector space,
capturing semantic relationships between words. Words with similar
meanings have similar vector representations.
v GloVe: Similar to Word2Vec, but it uses matrix factorization techniques to
learn word vectors by aggregating global word-word co-occurrence
statistics.
v BERT: A transformer-based model that generates contextualized
embeddings. Unlike Word2Vec and GloVe, BERT understands the meaning
of a word based on the context of the sentence.

77

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Semantic Embeddings (Word2Vec, GloVe, BERT)

Semantic Embeddings (Word2Vec, GloVe, BERT)

Technical Inference:
Word2Vec and GloVe are pre-trained on large corpora like Wikipedia,
while BERT is fine-tuned for specific tasks and generates dynamic word
embeddings.

Example: BERT embeddings can be used to improve topic modeling, as it

understands the context in which a word appears (e.g., the word "bank" in
finance vs river contexts).

78

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Example Problem for TF and TF-IDF

Identifying Key Terms in a Large Corpus Using TF-IDF

Problem:You have a large corpus of product reviews for a new smartphone.

After processing the corpus, you identify the following terms that appear
frequently:
Doc1: "This smartphone has an amazing camera and battery life."
Doc2: "The camera is great but the battery drains fast."
Doc3: "Camera quality is top-notch, but the phone heats up quickly."

1. Calculate the TF and TF-IDF for the term "camera" in Doc1, Doc2, and Doc3.
2. Analyze which terms have the highest TF-IDF values across the entire corpus and what
this indicates about their significance.

79

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Example Problem for TF and TF-IDF
Solution to Identifying Key Terms in a Large Corpus Using TF-IDF
Step 1: Calculate the TF for the Term "camera" in Each Document
TF (Term Frequency) measures how often a term appears in a document relative to the
total number of terms in that document. The formula for TF is:
Number of times term t appears in document d
TF =
Total number of terms in document d
TF Calculation for "camera"
Doc1: "This smartphone has an amazing camera and battery life."
Total number of terms in Doc1 = 7 ; "camera" appears 1 time.
!
TF(camera, Doc1) = = 0.1429
"
Doc2: "The camera is great but the battery drains fast."
Total number of terms in Doc2 = 8 ; "camera" appears 1 time.
!
TF(camera, Doc2) = # = 0.125
Doc3: "Camera quality is top-notch, but the phone heats up quickly."
Total number of terms in Doc3 = 9 ; "camera" appears 1 time.
!
TF(camera, Doc3) = = 0.1111 80

$
PRESENTATION BY: YUVARAJ , SCOPE, VIT
Presentation by:Yuvaraj , SCOPE,VIT
 Example Problem for TF and TF-IDF
Solution to Identifying Key Terms in a Large Corpus Using TF-IDF
Step 2: Calculate the IDF for the Term "camera” ; IDF measures how important a term is
within the entire corpus.
If a term appears in many documents, it has a lower IDF. The formula for IDF is:
Total number of documents
IDF 𝑡 = log
Number of documents containing term t
Total number of documents = 3 (Doc1, Doc2, Doc3)
"camera" appears in all three documents, so the number of documents containing "camera" = 3.
Since "camera" appears in all the documents, the IDF value is 0, which means it's not a
discriminating term in the corpus.

Step 3: Calculate the TF-IDF for the Term "camera"

The formula for TF-IDF is: TF−IDF = TF×IDF
Since the IDF for "camera" is 0, the TF-IDF for "camera" in all documents will be 0:
TF-IDF(camera, Doc1) = 0.1429 × 0 = 0 ; TF-IDF(camera, Doc2) = 0.125 × 0 = 0
TF-IDF(camera, Doc3) = 0.1111 × 0 = 0 81

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Example Problem for TF and TF-IDF

Step 4: Analyze the Significance of Terms Based on TF-IDF :

v Since "camera" appears in all documents and has an IDF of 0, it is a common term
across the corpus and doesn't help in distinguishing documents from each other.

v This is reflected in the TF-IDF value of 0.

v Let's consider another term, such as "battery", which might be more specific in
identifying the features mentioned in the reviews.

82

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Example Problem for TF and TF-IDF
Step 5: Calculate TF and TF-IDF for "battery"
TF Calculation for "battery"
Doc1: "This smartphone has an amazing camera and battery life."
!
Total terms = 7 ; "battery" appears 1 time. ; TF(battery, Doc1) = = 0.1429
"
Doc2: "The camera is great but the battery drains fast."
!
Total terms = 8. ; "battery" appears 1 time. ; TF(battery, Doc2) = = 0.125
#
Doc3: "Camera quality is top-notch, but the phone heats up quickly."
Total terms = 9. ; "battery" does not appear. ; TF(battery, Doc3) = 0
IDF Calculation for "battery” - "battery" appears in Doc1 and Doc2, so it is present
%
in 2 out of 3 documents. IDF(battery) = log ≈ 0.1761
&

TF-IDF Calculation for "battery"

TF-IDF(battery, Doc1) = 0.1429 × 0.1761 = 0.0251.
TF-IDF(battery, Doc2) = 0.125 × 0.1761 = 0.0220
TF-IDF(battery, Doc3) = 0 × 0.1761 = 0
83

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Example Problem for TF and TF-IDF
Solution to Identifying Key Terms in a Large Corpus Using TF-IDF
Step 6: Summary of Results
TF-IDF for "camera": Since "camera" is common in all documents, its TF-IDF is 0, indicating
it is not significant for distinguishing documents.
TF-IDF for "battery": "battery" has higher TF-IDF values in Doc1 and Doc2 compared
to Doc3. This indicates that "battery" is more significant in these two documents.

• TF-IDF helps identify key terms in a document by considering both their

frequency and their rarity across the entire corpus.
• A term that appears in many documents (like "camera") has a low TF-IDF, while
a term that appears in fewer documents (like "battery") can have a higher TF-
IDF, making it more important for differentiating documents.

84

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Example Problem for Bag of words and cosine similarity

Sample Question for Bag-of-Words (BoW) and Cosine

Similarity
Problem: Consider the following two documents:
Doc1: "I love programming with Python."
Doc2: "Python programming is fun and love."
Task:
Create the Bag-of-Words (BoW) representation for both documents.
Calculate the Cosine Similarity between Doc1 and Doc2 using their
BoW vectors.

85

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Example Problem for Bag of words and cosine similarity

Solution Steps:
Step 1: Create the Bag-of-Words (BoW) Representation
Vocabulary: ["I", "love", "programming", "with", "Python", "is", "fun", "and"]
BoW for Doc1:
Doc1 = "I love programming with Python."
BoW representation: [1, 1, 1, 1, 1, 0, 0, 0]
(I=1, love=1, programming=1, with=1, Python=1, is=0, fun=0, and=0)
BoW for Doc2:
Doc2 = "Python programming is fun and love."
BoW representation: [0, 1, 1, 0, 1, 1, 1, 1]
(I=0, love=1, programming=1, with=0, Python=1, is=1, fun=1, and=1)
86

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Example Problem for Bag of words and cosine similarity
Step 2: Calculate Cosine Similarity
!⋅#
Cosine Similarity Formula: Cosine Similarity = ∥!∥∥#∥
Where: A is the BoW vector for Doc1: [1, 1, 1, 1, 1, 0, 0, 0]. ;
B is the BoW vector for Doc2: [0, 1, 1, 0, 1, 1, 1, 1]
Dot Product: 𝐴 ⋅ 𝐵 = 1×0 + 1×1 + 1×1 + 1×0 + 1×1 + 0×1 + 0×1 + 0×1 = 3

Magnitude of A: ∥ 𝐴 ∥= 1% + 1% + 1% + 1% + 1% = 5 ≈ 2.236
Magnitude of B:∥ 𝐵 ∥= 0% + 1% + 1% + 0% + 1% + 1% + 1% + 1% = 6 ≈ 2.449
& &
Cosine Similarity: Cosine Similarity = = ≈ 0.547
%.%&(×%.**+ ,.*-*
v The Cosine Similarity between Doc1 and Doc2 is approximately 0.547.
v This indicates a moderate similarity between the two documents based on the Bag-of-
Words representation.
87

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Indicator Representations and Machine Learning for Summarization

88

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Indicator Representations and Machine Learning for Summarization

v Extractive Summarization: This technique involves selecting sentences

directly from the original document to form a summary.
v It is based on identifying the most important sentences that represent the
essence of the text.
v The goal is to choose the most relevant sentences based on specific
features to create a concise summary.

89

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Indicator Representations and Machine Learning for Summarization

Features Used for Extractive Summarization

• Sentence Position: Sentences appearing at the beginning or the end of a
document are often more important.
• Cue Phrases: Certain words or phrases (e.g., "in conclusion", "thus",
"therefore") often indicate important sentences that summarize key ideas.
• Length: Longer sentences may contain more information, but overly long
sentences can also be noise, so balance is needed.
• Similarity: Sentences that are more similar to the overall theme of the
document are likely to be important.
• TF-IDF: Measures how important a term is in the document relative to its
occurrence in the entire corpus.
• Topic Relevance: Sentences relevant to the main topics or themes in the
document are prioritized. 90

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Machine Learning-Based Summarization Approaches

Classification Approach:
• How it works: Each sentence is treated as a "class", and
the system classifies whether it should be included in the
summary.
• Feature Engineering: Features like sentence position,
length & similarity are extracted and fed into a classifier
(e.g., SVM or Logistic Regression) to predict important
sentences.

91

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Machine Learning-Based Summarization Approaches

Graph-Based Methods:
• TextRank: A graph-based ranking algorithm that assigns
a score to each sentence based on its importance, using
sentence similarity as the edge weights.
• Algorithm: Sentences are nodes, and edges represent
similarity between sentences. PageRank algorithm is
applied to determine sentence importance.

92

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Neural Methods for Summarization
vSeq2Seq: DL method where an encoder-decoder architecture is used
to generate summaries. Model is trained to map the input text to
summary.
vAttention Mechanism: Helps the model focus on different parts of
the input sequence, improving the generation of summaries by
weighting parts of the text more heavily.
vTransformers (T5, BART):
vT5: text-to-text transformer model that can be fine-tuned for
summarization tasks.
vBERT: A model that combines the best of both BERT
(bidirectional encoding) and GPT (autoregressive decoding) to
improve the quality of generated summaries.
93

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Summary of Methods for Summarization

Summary of Methods:
v Extractive Summarization focuses on selecting important sentences
using features like position, cue phrases, and similarity.
v Machine Learning Methods (e.g., classification, graph-based methods
like TextRank) help automate the extraction of key sentences.
v Neural Methods (e.g., Seq2Seq, Attention, T5, and BART) offer advanced
techniques for generating high-quality abstractive summaries.
Evaluation: Use ROUGE and BLEU scores to evaluate the quality of the
summaries generated by different models.
Future of Summarization: With advancements in Transformers,
abstractive summarization is becoming more accurate and context-aware.

95

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT
 Thank you

96

PRESENTATION BY: YUVARAJ , SCOPE, VIT

Presentation by:Yuvaraj , SCOPE,VIT