Text summarization

process of generating short, fluent, and most importantly accurate & essential
summary of a longer text document.
How to do text summarization: Text cleaning, Sentence tokenization, Word
tokenization, Word-frequency table, Summarization

Why automatic text summarization?

• Summaries reduce reading time.
• When researching documents, summaries make the selection process easier.
• Automatic summarization improves the effectiveness of indexing.
• Automatic summarization algorithms are less biased than human
summarization.
• Personalized summaries are useful in question-answering systems as they
provide personalized information.
Type of summarization
Based on input type:
Single Document, where the input length is short.
Multi-Document, where the input length is long.
Based on the purpose:
Generic, where the model makes no assumptions about the domain or content of the text to be summarized and
treats all inputs as homogeneous.
Domain-specific, where the model uses domain-specific knowledge to form a more accurate summary. For
example, summarizing research papers of a specific domain, biomedical documents, etc.
Query-based, where the summary only contains information that answers natural language questions about the
input text.
Based on output type:
Extractive, where important sentences are identified & selected from the input text to form a summary. Most
summarization approaches today are extractive in nature. Eg; frequency-based methods will tend to rank the
sentences in a text in order of importance by how frequently different words are used.
Abstractive, where the model forms its own phrases and sentences like what a human would generate. This
approach is definitely more appealing, but much more difficult than extractive summarization Eg; Deep Learning
(casting summarization as a seq2seq problem) & Transformer
Bilingual Evaluation Understudy-(BLEU)
• Translating text from source to target language
• Calculated by comparing the n=grams of machine translated sentences to n-grams
of human translated sentences.
• Consider the two reference translations R1 and R2 produced by human experts,
and the candidate translation C1 produced by our translation system.
• R1: The cat is on the mat, R2: There is a cat on the mat, C1: The cat and the dog.
Precision=No. of correctly predicted words/total no. of predicted words
Perfect score is 1.0, whereas the worst score is 0.0. Let’s call this metric BLEU*.
In C1 there are three words (“the”, “cat”, “the”) that appear on the reference
translations, thus BLEU*(C1) = 3/5 = 0.6
The problem with repeating unigrams
• Let’s compute the BLEU* score of the new candidate translation C2:
• C2: The The The The The.
Recall-Oriented Understudy for Gisting Evaluation- ROUGE score, is a metric used
to evaluate text summarization and translation models.
ROUGE-N: N indicates the number of N grams which can be 1 and 2. For ROUGE-1 it is the number of words. In ROUGE-2 it is the number of bigrams
• for unigrams it is (I),(love),(Machine),(Learning)
• for bigrams it is (I love),(love Machine),(Machine Learning)

ROUGE-1
• Candidate 1 : Summarization is cool
Reference 1 : Summarization is beneficial and cool
Reference 2 : Summarization saves time.
Overlapping words(unigrams) for reference 1 is more than reference 2. ROUGE-2
Recall = 3/5 = 0.6 It’s the same process as above but right now bigrams are determined.
Precision = 3/3 = 1 candidate 1 : (Summarization is),(is cool)
Rouge_1= 2*(0.6)*(1)/((0.6)+1) = 0.75 reference 1: (Summarization is),(is beneficial),(beneficial and),(and cool)
reference 2: (Summarization saves),(saves time)
candidate 2 : I love Machine Learning
best reference : I think i love Machine Learning In terms of bigrams there is only one match in reference 1 and there is none at referenc
Recall = 1/4 = 0.25
Precision = 1/2 = 0.5
Recall = 4/6 = 0.66 Rouge_2 = (2*0.5*0.25)/(0.5+0.25) = 0.33
Precision = 4/4 = 1 candidate 2
Rouge_1 = 2*0.66*1/(1+0.66) = 0.795 (approximately) candidate 2 : (I love),(love Machine),(Machine Learning)
best reference = (I think),(think i),(i love),(love Machine),(Machine Learning)
candidate 3 : Good night Recall = 3/5 = 0.6
best reference : Good night everyone!
Precision = 3/3 = 1
Rouge_2 = (2*1*0.6)/(1+0.6) = 0.75
Recall = 2/3 = 0.66
Precision = 2/2 = 1 candidate 3
candidate 3 : (Good night)
Rouge_1 = 2*0.66*1/(1+0.66) = 0.795 (approximately) best reference : (Good night),(night everyone!)
 ROUGE-L Longest Common Subsequence (LCS)
LCS is the longest sequence of words that appear in both the candidates and
reference summaries. It is important to note that LCSes are not necessarily
consecutive but still in order. Example:
Model output: “A fast brown fox leaps over a sleeping dog.”
Reference summary: “The quick brown fox jumps over the lazy dog.”

LCS here is “brown fox over dog.”

Precision: Number of words in LCS /
Number of words in the model output =
4 / 9 ≈ 0.444
Recall: Number of words in LCS /
Number of words in the reference X is a reference summary sentence and Y is a
candidate summary sentence with sizes m,n
summary = 4 / 9 ≈ 0.444 respectively. Beta is a parameter in the ROUGE-L
score that controls the relative importance of
F1-score = (2 * 0.444 * 0.444) / (0.444 + precision and recall. When beta is 1 the ROUGE-L
score is simply the harmonic mean of precision and
0.444) ≈ 0.444 recall.
Advantages of ROUGE:
• correlate well with human evaluation.
• easy to calculate and understand.
• language independent. So it can be used to evaluate
summaries in any language.
• Disadvantages of ROUGE:
• ROUGE only measures n-gram overlap. So it does not take into
account the semantic meaning of the summary.
• It is sensitive to the choice of reference summaries.
• It can be biased towards summaries that are shorter or longer
than the reference summaries
 ROUGE-S
ROUGE-S stands for skip-grams. The order of the words in each sequence is
preserved, but arbitrary gaps are allowed between words. For example -
Sentence: police killed the gunman
(“police killed”, “police the”,
“police gunman”, “killed the”,
“killed gunman”, “the gunman”)

ROGUE vs BLEU:
BLEU focuses on precision
ROGUE focuses on recall
 Two kind of conversational agents
1. Chatbots: mimic informal human chatting; for fun or therapy
2. Task-based Dialogue Agents: interfaces to personal assistants; cars,
robots, appliances, booking flights or restaurants

Speech Acts (aka Dialogue

Acts)
Constatives: committing the speaker to something’s being the case (answering, claiming,
confirming, denying, disagreeing, stating)

Directives: attempts by the speaker to get the addressee to do something (advising, asking,
forbidding, inviting, ordering, requesting)

Commissives: committing the speaker to some future course of action (promising, planning,
vowing, betting, opposing)
Acknowledgments: express the speaker’s attitude regarding the hearer with respect to some
social action (apologizing, greeting, thanking, accepting an acknowledgment)
 Grounding
• Principle of closure. Agents performing an action require
evidence, sufficient for current purposes, that they have succeeded
in performing it
• Grounding: acknowledging that the hearer has understood. Eg ; elevator
buttons light up when you press them

System: Did you want to review some more of your profile?

User: No.
System: What’s next?
System: Did you want to review some more of your profile?
User: No.
System: Okay, what’s next?
• A reporter interviewing a chef asks questions, and the chef responds. This reporter has the
conversational initiative (Some conversations are controlled by one person)
• User initiative (user asks or commands, system responds)
• System initiative (system asks user questions to fill out a form, user can't change the
direction)
• Most human conversations have mixed initiative: I lead, then you lead, then I lead. Mixed
Inititative is difficult for NLP.

Chatbot Architectures
Rule-based - Pattern-action rules (ELIZA)
Corpus-based -
• Response by retrieval: Use information retrieval to grab a response (that is appropriate
to the context) from some corpus often using TF-IDF frequency count approach.(XiaoIce)
• Response by generation: Use a language model or encoder-decoder to generate the
response given the dialogue context. (BlenderBot)
Eliza Rules - It mimics human-like conversation using simple pattern-matching and substitution rules, without
actually understanding the content of the conversation. Rules are organized by keywords. Each keyword has a
pattern and a list of possible transform.
Pattern: "I feel *"
Response Template: "Why do you feel *?"
Pattern: "You * me"
Response Template: "What makes you think I * you?"
Pattern: "Can you *"
Response Template: "What makes you think I can't *?"
Pattern: (No matching keywords)
Response Template: "Tell me more about that."
 Frame-based / task-based dialogue agents
• Systems that have the goal of helping a user solve a task like
making a travel reservation or buying a product
• One or more frames (each consisting of slots with values)
• Eg: GUS architecture - System asks questions of user, filling any
slots that user specifies
 Dialogue-State or Belief-State Architecture
A more sophisticated version of the frame-based architecture
NLU: extracts slot fillers from the user’s utterance using ML
Dialogue state tracker: maintains the current state of the dialogue
(user’s most recent dialogue act, set of slot-filler constraints from
user
Dialogue policy: decides what the system should do or say next
• GUS policy: ask questions until the frame was full then report
back
• More sophisticated: know when to answer questions, when to
ask a clarification question, etc.
NLG: produce more natural, less templated utterances
Dialogue Acts
Combine the ideas of speech acts and grounding into a
representation
single Young et al., 2010:
Tag Sys User Description
HELLO (a = x, b = y, ...) X X Open a dialogue and give info a = x, b = y, ...
INFORM (a = x, b = X X Give info a = x, b = y, ...
y, ...)
REQUEST (a, b = x, ...) X X Request value for a given b = x, ...
REQALTS(a = x, ...) c X Request alternative with a = x, ...
CONFIRM (a = x, b = X X Explicitly confirm a = x, b = y, ...
y, ...)
CONFREQ (a = x,..., d) X c Implicitly confirm a = x, ... and request value of
d
SELECT (a = x, a = y) X c Implicitly confirm a = x, ... and request value of
d
AFFIRM(a = x, b = y, ...) X X Affirm and give further info a = x, b = y, ...
 Dialogue Young et al., 2010:

Acts
Utterance Dialogue act
hello(task = find,type=restaurant)
U: Hi, I am looking for somewhere to eat.
confreq(type = restaurant, food)
S: You are looking for a restaurant. What
type of food do you like?
U: I’d like an Italian somewhere near the inform(food = Italian, near=museum)
museum.
S: Roma is a nice Italian restaurant near inform(name = "Roma", type =
restaurant,
the museum. food = Italian, near = museum)
U: Is it reasonably priced? confirm(pricerange =
moderate)
S: Yes, Roma is in the moderate price affirm(name = "Roma",
pricerange =
range. moderate)
U: What is the phone number? request(phone
 Slot filling as sequence labeling: BIO tagging
Idea: Train a classifier to label each input word with a tag that tells us what
slot (if any) it fills and convert the training data to this format
OO O O O B- I-DES O B- I-DEPTIME O
DES DEPTIME
I wan to fly to San Francisco on Monday afternoon pleas
t e
 Features for detecting corrections in spoken
dialogueexamples
features
lexical words like “no”, “correction”, “I don’t”, swear words, utterance length
semantic similarity (word overlap or embedding dot product) between the
candidate
correction act and the user’s prior utterance
phonetic phonetic overlap between the candidate correction act and the user’s prior ut-
terance (i.e. “WhatsApp” may be incorrectly recognized as “What’s up”)

Two important mechanisms:

• confirming understandings
with the user
• rejecting utterances that the
system is likely to have
misunderstood.
 Using confidence to decide whether to
confirm:
confidence value indicates how likely system understood the user.
Systems could use set confidence thresholds:

<a low confidence reject

≥a above the threshold confirm explicitly
≥b high confidence confirm implictly
≥g very high confidence don’t confirm at all
Sentence Realization
• Delexicalization: replacing words in the training set that
represent slot values with a generic placeholder token:

recommend(restaurant name= Au Midi, neighborhood =

midtown, cuisine = french
1 Au Midi is in Midtown and serves French food.
2 There is a French restaurant in Midtown called Au Midi.

recommend(restaurant name= Au Midi, neighborhood =

midtown,
1 cuisine
restaurant= french
name is in neighborhood and serves cuisine food.
2 There is a cuisine restaurant in neighborhood called restaurant
name.
Sentence Realization: mapping from frames
to delexicalized sentences
[name] has decent service
Encoder-decoder models:
DECODER
ENCODER

RECOMMEND service: decent cuisine: null

Output:
restaurant_name has decent
service
Relexicalize to:
Au Midi has decent service