Module 7

TEXT SUMMARIZATION
Overview
principal approaches in summarization,
describes the design, implementation,
summary generation and
reviews methods of evaluating summaries.
Lexrank and Textrank
Conclusion
Summary??
A summary is a text that is produced from one
or more texts, that contains a significant
portion of the information in the original
text(s), and that is no longer than half of the
original text(s).
Types
indicative summaries (that provide an idea of
what the text is about without giving any
content) and
informative ones (that do provide some
shortened version of the content) are often
referenced.
Extracts are summaries created by reusing
portions (words, sentences, etc.) of the input
text verbatim, while
abstracts are created by re-generating the
extracted content.
Summarization Types - Examples
Original Text:
Amazon rainforest is the world’s largest tropical rainforest, famed for
its biodiversity. It produces about 20% of the world's oxygen and is
home to thousands of plant and animal species. Deforestation and
climate change pose serious threats to this vital ecosystem.
 Indicative summary:
A report discussing the Amazon rainforest's ecological importance
and threats from environmental issues
 Informative summary:
Amazon rainforest, the largest tropical forest, contributes 20% of
global oxygen and supports vast biodiversity. It faces severe threats
from deforestation and climate change.
 Extractive summary:
Amazon rainforest is the world’s largest tropical rainforest, famed
for its biodiversity. Deforestation and climate change pose serious
threats to this vital ecosystem.
 Abstractive summary:
Amazon, a major source of oxygen and biodiversity, is under threat
Stages of Automated Summarization
 topic identification, identifies the most
important unit(s) (words, sentences,
paragraphs, etc.).
 interpretation: fusion of concepts,
evaluation, and other processing.
 Summary generation, The results of
interpretation are usually unreadable
abstract representations. Systems therefore
include a stage of summary generation to
produce human-readable text.
Stage 1: Topic identification
To perform this stage, almost all systems employ
several independent modules.

Each module assigns a score to each unit of input

(word, sentence, or longer passage); then a
combination module combines the scores for each unit
to assign a single integrated score to it;

finally, the system returns the n highest-scoring units,

according to the summary length requested by the
user.
Recall and Precision
Given an input text, a human’s extract, and a system’s
extract, these scores quantify how closely the
system’s extract corresponds to the human’s.
correct = the number of sentences extracted by
the system and the human;
wrong = the number of sentences extracted by
the system but not by the human; and
missed = the number of sentences extracted by
the human but not by the system.
Precision = correct / (correct + wrong)
Recall = correct / (correct + missed)
Criteria to assign scores
Positional criteria. certain locations of the
text (headings, titles, first paragraphs, etc.)
tend to contain important information.
Cue phrase indicator criteria. Since in
some genres certain words and phrases
(‘significant’, ‘in this paper we show’)
explicitly signal importance, sentences
containing them should be extracted.
Criteria to assign scores
Word and phrase frequency criteria. if a
text contains some words unusually
frequently, then sentences containing these
words are probably important.
Query and title overlap criteria.
Combination of various module scores.
Stage 2 - Interpretation or topic fusion
• During interpretation, the topic identified as
important are fused, represented in new terms,
and expressed using a new formulation, using
concept or words not found in original text.
• Interpretation is what distinguishes extract type
summarization from abstract type systems.
• No system can perform interpretation without
prior knowledge about the domain.
• But acquiring enough prior domain knowledge
is so difficult that summarizer to date have only
attempted it in a small way.
Stage 3 - Generation
Challenges:
Result of abstraction are unreadable
Extracts are seldom coherent
Repetition of material
Omitted discourse linkages.

So there is a need of summary generation to

produce a human readable text
Text Planning
The facts must be organized so as to signal
the causal, logical and intentional
relationships between them.

The facts must be organized so as to signal

the causal, logical and intentional
relationships between them.
Example:
The system performs the enhancement. Before that, the
system resolves conflicts. First, the system asks the user to
tell it the characteristic of the program to be enhanced. The
system applies transformations to the program. It confirms
the enhancement with the user. It scans the program in
order to find opportunities to apply transformations to the
program.
The system asks the user to tell it the characteristic of the
program to be enhanced. Then the system applies
transformations to the program. In particular, the system
scans the program in order to find opportunities to apply
transformations to the program. Then the system resolves
conflicts. It confirms the enhancement with the user.
Finally, it performs the enhancement.
Sentence Planning
Generate semantic and syntactic
specifications that can actually be realized in
natural language.
a. This sentence contains two noun phrases.
b. It contains them.

Sentence Realization
 This is a purely linguistic level, which takes
choices about words and syntactic structures
made during sentence planning, and
constructs a sentence using them.
Dysfluencies:
Repetition of clauses
Repetition of named entities
Inclusion of less important material
Multi Document Summarization
Three Major Problems
1) Recognizing and coping with redundancy.
2) Identifying important differences among
documents.
3) Ensuring summary coherence
SUMMONS:
 Takes an information extraction approach
All input documents are parsed into templates
clusters the templates according to their
contents, and then applies rules to extract
items of major import.
Uses predefined rules.
Systems measure the similarity of a candidate
passage to that of already-selected passages
and retain it only if it contains enough new
(dissimilar) information.
Determination of additional material
Identify the units most relevant to the user’s query
MMR Technique – Maximum Marginal Relevance
Score
It selects the most relevant sentences at the same
time avoiding redundancy.
In extractive summarization, the final score of a
given sentence Si in MMR is calculated as follows:
MMR(Si) = λ×Sim1(Si, D) − (1−λ)×Sim2(Si,
Summ)
where: D  entire document
Summ  Existing Summary
Evaluating Summaries
Previous Evaluation Studies
Two approaches:
intrinsic evaluations as measuring output
quality (only) and
extrinsic as measuring user assistance in
task performance.
Intrinsic Evaluations
Create a set of ideal summaries, one for each
test text, and then
Compare the summarizer’s output to it,
measuring content overlap
often by sentence or phrase recall and
precision, but sometimes by simple word
overlap.
Intrinsic (cont..)
OR rate systems’ summaries according to some
scale
Readability
Informativeness
Fluency
coverage
Extrinsic Evaluation
easy to motivate
but have to ensure that the metric applied
correlates well with task performance
efficiency.
Examples of extrinsic evaluation can be found
in
Morris, Kasper, and Adams (1992) for GMAT
testing,
Miike et al. (1994) for news analysis, and
Mani and Bloedorn (1997) for information
retrieval.
Two basic measures
In general, to be a summary, the summary must
obey two requirements:
it must be shorter than the original input text;
it must contain the important information of
text T.
Two measures to capture the extent to which a
summary S conforms to these requirements with
regard to a text T:
Compression Ratio: CR = (length S) / (length
T)
Retention Ratio: RR = (info in S) / (info in T)
Fig (a) : as the summary gets longer , it includes more
information, until it equals the original.
Fig. (b): shows a more desirable situation: at some
special point, the addition of just a little more text to the
summary adds a disproportionately large amount of
information.
Fig. (c) shows another case: Most of the important
material is included in the summary quite early; as the
length grows, the added material is less interesting.
Approaches: Lexrank and Textrank
In both LexRank and TextRank, a graph is
constructed with
vertex as each sentence in the document.
The edges between sentences are based on
some form of semantic similarity or content
overlap.
How are edges formed??
LexRank uses cosine similarity of TF-
IDF vectors.
TextRank uses a very similar measure based
on the number of words two sentences have in
common.

where tfw,s is the number of occurrences of the word w in the sentence

s.
In the unweighted edges version, edges were
formed between the sentences having
similarity greater than threshold.
While in the weighted edges graph, similarity
scores were used as weights.
How are summaries formed?
In both algorithms, the sentences are ranked
by applying PageRank to the resulting graph. A
summary is formed by combining the top
ranking sentences, using a threshold or length
cutoff to limit the size of the summary.

Pagerank eqn:

d is damping factor, typically chosen from

[0.8,0.85]
Differences between lexrank and textrank
1. TextRank was applied to summarization
exactly as described here, while LexRank
combines the LexRank score with other
features like sentence position and length.
2. TextRank was used for single document
summarization, while LexRank has been
applied to multi-document summarization.
When summarizing multiple documents,
there is a greater risk of selecting duplicate
or highly redundant sentences to be placed in
the same summary.
 To address this issue, LexRank builds up a
summary by adding sentences in rank order,
but discards any sentences that are too
similar to ones already placed in the
summary.
The method used is called Cross-Sentence
Information Subsumption (CSIS).
Evaluation Metric
ROUGE stands for Recall-Oriented Understudy
for Gisting Evaluation.
Recall based score to compare system
generated summary with one or more human
generated summaries.
Can be with respect to n-gram matching.
Generally, it is evaluated on all 3 metrics: P, R
and F1.
i.e., ROUGE-1 Precision, ROUGE-1 Recall &
ROUGE-1 F1.
ROUGE-1: Examples
ROUGE-1 is computed as division of count of unigrams in
reference that appear in system and count of unigrams in
reference summary.

Example 1:
Reference-summary: Beijing hosted the summer
Olympics.
System-summary: The summer Olympics were held in
Beijing.

ROUGE-1 score: 0.75

Example 2:
Reference-summary: The policemen killed the gunman
System-summary: The gunman killed the policemen
ROUGE-1 score: 1
ROUGE-1
ROUGE score is generally averaged for
multiple references.

ROUGE-1 does not determine if the result is

coherent or if the sentences flow together in a
sensible manner.

A higher order n-gram ROUGE score can

measure fluency to some degree.
ROUGE-2 & ROUGE-L metrics
ROUGE-2 =
It focusses more on fluency and phrase-level
accuracy than ROGUE-1 metric.
ROUGE-L =
It focusses more on in-sequence matching for
sequential paraphrasing than other two ROGUE
metrics.
LCS – Longest Common Subsequence  It
considers word order, but allows skipping words.
ROUGE-1,2,L metrics: Example
Reference: Lalu is very particular about his
handwriting style.
Candidate: Lalu is very keen about his
handwriting technique.

ROGUE-L recall = 6/8 = 75%

ROGUE-1 recall = 6/8 = 75%
ROGUE-2 recall = 4/7 = 57%
ROUGE metrics
If ROUGE-1 is high, the model likely covers the
main vocabulary and concepts from the reference.
If ROUGE-2 is high, the model maintains phrase-
level structures, and not just words from the
reference  captures the local fluency.
If ROUGE-L is high, the model shows potential
well-structured sentences, and not just phrases
from the reference  evaluates sentence-level
fluency and coherence. It checks for same order
of common sequence of words, but not necessarily
consecutively.
ROUGE metrics - Range
Approximate range for ROGUE metrics for a good
model based on BERT / BART etc.,:
Metric Good Practical Range (minimum)
ROUGE-2 15 – 30
ROUGE-L 25 – 40
ROUGE-1 35 – 50
Note: ROUGE-L can be higher than ROUGE-2, since
ROUGE-2 looks for exact bigram matches, whereas
ROUGE-L looks for longest common subsequence
(which can be unigram match as well) but in same
order.
ROUGE metrics - Example
Reference: This is the very last week of current
winter semester. (Token total: 10)
Candidate: We are in the final week of this semester.
(Token total: 9)
ROUGE-2 recall = 1/9 (week of)
ROUGE-L recall = 4/10 (the, week of, semester)
ROUGE-1 recall = 5/10 (this, the, week, of, semester)
Similarly, ROUGE-n Precision and ROUGE-n F1 are
evaluated.
Note: ROUGE-L will not consider 'this‘ token, since
the sequence order is not maintained in the
candidate.
 Module 7
QUESTION ANSWERING SYSTEM
(QA)
QA System
A Question-Answering (QA) System automatically
answers questions posed in natural language. It
enables machines to understand and respond to
queries by retrieving or generating relevant
information from structured or unstructured data
sources.

It supports conversational AI by enabling intelligent

dialogues.

Provides automated customer service using chat-

bots.
QA System - Scope
QA systems play a crucial role across below
range of applications:
 Search engines (e.g., Google’s direct answers)
 Virtual assistants (e.g., Siri, Alexa, Google
Assistant)
 Customer support chat-bots
 Medical and legal document retrieval
 Academic research support
 Knowledge base navigation (e.g., Wikipedia-
based QA)
Common Datasets used for QA models
SQuAD (Stanford Question Answering
Dataset)
NQ (Natural Questions)
TriviaQA
HotpotQA – for multi-hop QA
BioASQ – for biomedical QA
QuAC and CoQA – for conversational QA
NLP models for QA chat-bots
BERT
- Pre-trained model fine-tuned for SQuAD
dataset.
RoBERTa
- An optimized variant of BERT, used for
reading comprehension tasks.
T5
- Converts all NLP tasks into Text-to-Text
format for generative QA.
GPT-3 / GPT-4
- used for conversational and generative QA
- used in chat-bots like ChatGPT
NLP models for QA chat-bots
XLNet –
- Permutation-based transformer model for
extractive QA.
ALBERT –
- used for large scale QA tasks with a lighter
version of BERT.
RAG –
- Combines document retrieval and text generation.
DrQA –
- Open-domain QA system that leverages Wikipedia
data source (Developed by Facebook AI)
GPT models for QA Systems
Generative Pre-trained Transformer (GPT) models
have below modules:
 Question Pre-processor
 Transformer Decoder Layers (GPT Core):
 Masked Multi-Head Self-Attention

 Feed Forward Neural Network

 Layer Norm & Residual Connections

 Positional Encoder Layer

 Output Decoder Layer
 Prompt Engineering & Few-Shot Learning
 Fine-tuning & Alignment modules:
 Supervised Fine-Tuning (SFT) using SQuAD & NQ datasets
 Reinforcement Learning from Human Feedback (RLHF) for
safe responses.
QA system models – Evaluation Metrics
Based on task type – text generation,
retrieval, answer span prediction or dialogue
quality, one or more of these metrics are
used:
 BLEU-n score

 METEOR score

 ROUGE-n score

 BERTScore

 Perplexity Score