Natural Language Processing

By
Dr. L. Lakshmi
Course Code: DS311
Module-5
 Module-5 Contents

 Discourse -Reference Resolution

 Text Coherence - Discourse Structure – Coherence,
 Dialog and Conversational Agents - Dialog Acts
 Interpretation -Conversational Agents
 Language Generation – Architecture
 Discourse
• Discourse refers to a structured and coherent sequence of sentences or utterances that form a
meaningful whole, going beyond individual words or sentences.
• Discourse can be written or spoken and is commonly organized into genres like conversations,
narratives, instructions, or expositions.
• It includes monologues, where communication flows one-way (e.g., lectures or articles), and
dialogues, where participants take turns speaking (e.g., conversations).
• Coherence is the quality that makes a discourse logical, meaningful, and easy to follow.
• It connects sentences and ideas in a way that flows naturally, allowing readers or listeners to
understand how each part relates to others.
• Coherence is achieved through relationships like causation, contrast, or sequencing, which
link ideas smoothly.
• For example, in a story, coherence helps readers understand why certain actions happen, which
builds continuity and clarity within the text.
• Coherence Models in NLP: Techniques like segmentation and interpreting coherence relations
enable computational models to handle text more naturally, improving functions such as
summarization, pronoun resolution, and overall comprehension
 Discourse
• Discourse Types:
• Monologue: A one-way communication from speaker to listener (or writer to reader).
• Dialogue: An interactive, two-way conversation where speakers alternate turns.
• Human-Computer Interaction (HCI): Dialogue between humans and computers, limited by the
computer’s conversational capabilities.
• Coreference Resolution: This is the task of identifying which pronouns or expressions refer to
the same entities in a discourse.
• For example, in “The Tin Woodman went to the Emerald City to see the Wizard of Oz and ask
• for a heart. After he asked for it, the Woodman waited for the Wizard’s response.”
• pronouns like "he" and "it" refer to specific entities (the Tin Woodman and a heart,
respectively).
• But doing this disambiguation automatically is a difficult task.
• This goal of deciding what pronouns and other noun phrases refer to is called coreference
resolution.
• Coreference resolution is important for information extraction, summarization, and for
conversational agents.
 Discourse
• Coreference Resolution and Discourse Structures:
• In language processing, systems must accurately identify references like pronouns and their
corresponding nouns (e.g., identifying “their” as referring to “First Union Corp” rather than
“Paine Webber”).
• Such identification aids in creating coherent summaries and understanding text relationships.
• Example:
• Summarizing a news passage:
• Original: "First Union Corp is continuing to wrestle with severe problems. According to
industry insiders at Paine Webber, their president, John R. Georgius, is planning to announce
his retirement tomorrow.“
• Summary: "First Union President John R. Georgius is planning to announce his retirement
tomorrow."
 Discourse
• Coherence Relations:
• These are the logical connections that link sentences or passages within a text, such as cause-
effect or explanation.
• They are essential for tasks like summarization, automatic essay grading, and evaluating the
quality of generated texts.
• For instance:
• Example:
• "John hid Bill’s car keys. He was drunk." (Coherent, since "being drunk" explains "hiding the
keys.")
• "John hid Bill’s car keys. He likes spinach." (Incoherent without context, as no clear
relationship connects the sentences.)
 Discourse
• Entity-Based Coherence:
• This is a type of coherence that ensures focus on a main entity across sentences, establishing
"aboutness" by following one primary character or object smoothly.
• Texts with consistent entity-based coherence are easier to understand.
• Example:
• a. John went to his favorite music store to buy a piano.
• b. He had frequented the store for many years.
• c. He was excited that he could finally buy a piano.
• d. He arrived just as the store was closing for the day.
• Passage focuses on John's experience in a music store in a cohesive sequence.
• a. John went to his favorite music store to buy a piano.
• b. It was a store John had frequented for many years.
• c. He was excited that he could finally buy a piano.
• d. It was closing just as John arrived.
• Passage alternates between John and the store, reducing coherence and clarity.
 Discourse Segmentation
• Discourse segmentation, an approach for dividing texts into coherent units based on their
content.
• Different types of texts often follow conventional structures that guide readers through a
logical flow:
• Academic Papers:
• Typically divided into sections such as Abstract, Introduction, Methodology, Results, and
Conclusion. Each section serves a distinct purpose, helping readers quickly locate specific
information.
• Newspaper Stories:
• Often structured as an inverted pyramid, where the most critical information (the "lede")
appears in the opening paragraphs. This format ensures readers get the essential details early.
• Medical Reports (SOAP Format): SOAP stands for Subjective, Objective, Assessment, and
Plan, a format doctors follow to structure patient reports. This standardization aids in medical
review and continuity of care.
• Challenge: Automatically recognizing these structural boundaries across large texts is complex
and remains a largely unsolved problem in natural language processing (NLP).
• However, algorithms that detect simpler discourse structures, like topic-based segmentation,
provide a starting point.
 Discourse Segmentation
• Discourse segmentation involves splitting a document into linear segments that represent
subtopics or logically cohesive passages.
• Rather than complex hierarchical structures, it captures the broad flow, e.g., in a news
broadcast, distinguishing individual news stories.
• Applications: This segmentation supports tasks like:
• Information Retrieval: Helps separate segments to locate relevant content.
• Summarization: Ensures that summarized parts reflect diverse sections of the text.
• Information Extraction: Confines extracted data to a single cohesive segment for accuracy.
 Discourse Segmentation
• Unsupervised Discourse Segmentation
• The process of dividing text into cohesive subtopics based on lexical cohesion (word
relationships), without needing pre-labeled data.
• Cohesion:
• Cohesion is the glue connecting text segments, achieved by repeating words, synonyms, or
related terms (e.g., hypernyms, like "fruit" relating to "apples" and "pears").
• Types of Cohesion:
• Lexical Cohesion: Word relations that link sentences, e.g., synonyms, repetitions, hypernyms.
• Non-Lexical Cohesion: Relations like anaphora (referring back to previous entities with words
like "they").
• Example: Cohesion in a simple two-sentence text:
• Before winter, I built a chimney, and shingled the sides of my house. I have thus a tight
shingled and plastered house.
• Here, “house” and “shingled” tie the sentences together, showing cohesion..
 Discourse Segmentation
• Cohesion Chains:
• Instead of single word matches, related words form a "chain" that connects sentences or passages within the
same subtopic.
• Example: Peel, core, and slice the pears and apples. Add the fruit to the skillet. When they are soft...
• In this example, the chain includes "pears," "apples," and "fruit," showing lexical cohesion across sentences.
• TextTiling Algorithm:
• An unsupervised segmentation algorithm that uses cohesion dips to locate subtopic boundaries.
• Steps:
• Tokenization: The document is broken into "pseudo-sentences" (usually 20 words each), and common words
(stop words) are removed.
• Lexical Cohesion Scoring: A similarity score is calculated for gaps between pseudo-sentences, using cosine
similarity. The score measures how similar words are across neighboring pseudo-sentence blocks.
• Boundary Detection: Scores reveal "valleys" (low cohesion) between blocks. A boundary is placed at any
significant valley, dividing the text into segments.
• Example Calculation:
• If two blocks of words contain similar vocabulary, they have high cohesion, and the cosine similarity score
between their word vectors will be high.
• A "depth score" determines the valley depth in the cohesion graph. If it’s deeper than a threshold, the algorithm
assigns a boundary.
 Discourse Segmentation
• Supervised Discourse Segmentation
• When labeled data with discourse boundaries is available, supervised learning methods can
train models to identify segment boundaries more precisely.
• Applications:
• Broadcast News Segmentation: Separates news stories in TV/radio for tasks like
summarization.
• Monologue or Lecture Transcription: Creates paragraph breaks for readability and coherence
in speech transcriptions.
• Methods:
• Common classifiers like Support Vector Machines (SVM) and Decision Trees make yes-no
boundary decisions between sentences.
• Sequence-based models, like Hidden Markov Models (HMMs) and Conditional Random
Fields (CRFs), capture context by incorporating sentence sequences, improving segmentation
accuracy.
 Discourse Segmentation
• Features Used:
• Cohesion Features: These include word overlap, cosine similarity, and lexical chains to track
word connections across segments.
• Discourse Markers: Words or phrases that indicate boundaries, such as "good evening" or
"joining us now" in news broadcasts.
• Example:
• In the Wall Street Journal, the word “incorporated” often signals the beginning of a business
article where a company is introduced, providing a natural boundary feature for the algorithm.
• Discourse Markers:
• Domain-specific words or phrases signal transitions. Identifying them can rely on rules or
automatic methods.
• Example: For real estate ads, markers like neighborhood names or phone numbers denote
sections.
• Some segmentation models use named entity recognition to identify people or locations,
setting boundaries based on common structures (e.g., names in broadcast news segments).
 Text Coherence
• Coherence in a discourse relies on coherence relations, which are logical and semantic links between sentences
or passages that enable a reader to interpret the text as a unified whole.
• 1. Coherence Requirement:
• Coherence ensures that parts of a discourse logically or causally relate to each other.
• It goes beyond mere cohesion (e.g., repetition) and connects ideas in meaningful ways.
• 2. Coherence Relations:
• Coherence relations help define how two or more sentences are linked.
• Some primary types of coherence relations, based on Hobbs (1979), include:
• Result: When an event or state in one sentence (S0) causes an event or state in another sentence (S1).
• Example: "The Tin Woodman was caught in the rain. His joints rusted.“
• Explanation: When an event or state in S1 explains the event or state in S0.
• Example: "John hid Bill’s car keys. He was drunk.“
• Parallel: Similar elements or actions occur in both sentences.
• Example: "The Scarecrow wanted some brains. The Tin Woodman wanted a heart.“
• Elaboration: When both sentences describe the same proposition in different ways.
• Example: "Dorothy was from Kansas. She lived in the midst of the great Kansas prairies.“
• Occasion: Indicates a sequence of states or events.
• Example: "Dorothy picked up the oil-can. She oiled the Tin Woodman’s joints."
 Text Coherence
• 3. Hierarchical Structure in Discourse:
• Coherence relations create a hierarchical structure in discourse, where some sentences relate
more directly to others.
• John went to the bank to deposit his paycheck. (S1)
• He then took a train to Bill’s car dealership. (S2)
• He needed to buy a car. (S3)
• The company he works for now isn’t near any public transportation. (S4)
• He also wanted to talk to Bill about their softball league. (S5)
• In passage sentences form a tree-like structure, with main events and secondary details linked
through coherence relations like Explanation and Parallel.
 Text Coherence
• 4. Discourse Segments:
• Coherent discourse is organized into segments, similar to sentence constituents, with each
segment representing a coherent group of sentences or clauses.
• These segments form a tree structure that shows the primary and secondary relationships
between parts of a discourse.
• 5. Applications in Summarization and Information Extraction:
• Coherence relations are useful for summarizing or extracting information.
• For example, an Elaboration relation suggests that details expand on a central idea, allowing a
summarization system to merge details into a single event or concept.
• Identifying coherence relations like Elaboration can help a system condense or merge
redundant information for more effective summaries.
 Reference Resolution
• The concept of reference resolution in computational discourse, focusing on how natural language
expressions are linked to specific entities in a discourse.
• The goal of reference resolution is to identify the entities referred to by expressions like pronouns
or names, and to determine relationships between these expressions to ensure coherent discourse
interpretation.
• 1. Referring Expressions and Referents:
• Referring Expression: A linguistic expression used to denote an entity (e.g., "Victoria Chen",
"she").Referent: The actual entity being referred to by the expression.
• Victoria Chen, Chief Financial Officer of Megabucks Banking Corp since 2004, saw her pay jump
20%, to $1.3 million, as the 37-year-old also became the Denver-based financial-services company’s
president. It has been ten years since she came to Megabucks from rival Lotsabucks.
• In passage, "Victoria Chen" and "she" refer to the same person, making "she" an anaphoric
expression, as it references an entity introduced earlier.
• 2. Coreference and Anaphora:
• Coreference: When multiple expressions refer to the same entity, they are said to corefer, creating a
coreference chain (e.g., "Victoria Chen" and "she").
• Antecedent: An initial mention that allows subsequent anaphoric reference (e.g., "John" as the
antecedent for "he").
• Anaphora: The use of an expression to refer back to a previously mentioned entity (e.g., pronouns
like "she" and "her" in passage are anaphoric).
 Reference Resolution
• 3. Discourse Model:
• A mental model containing representations of entities and their relationships as they appear in
discourse.
• Two primary operations:
• Evoke: Introducing a referent into the model when it is first mentioned.
• Access: Accessing a referent already present in the model during subsequent mentions.
• The discourse model dynamically updates to track entities and references as the discourse
progresses.
• 4. Reference Resolution Tasks:
• Coreference Resolution: Identifying expressions in a text that refer to the same entity, creating
coreference chains (e.g., {Victoria Chen, Chief Financial Officer, her, the 37-year-old}).
• Pronominal Anaphora Resolution: Finding the antecedent of a specific pronoun, which is a
subset of coreference resolution (e.g., determining that "her" refers to "Victoria Chen").
 Reference Resolution
• 5. Algorithms for Reference Resolution:
• The text mentions multiple algorithms used for reference resolution:
• Hobbs Algorithm: A syntactic approach for finding antecedents.
• Centering Algorithm: Uses coherence to track referential focus in discourse.
• Log-Linear (MaxEnt) Algorithm: A statistical model for pronoun resolution.
• More advanced methods handle complex references beyond simple entities, including
references to events, propositions, or speech acts
• According to Doug, Sue just bought a 1961 Ford Falcon.
• a. But that turned out to be a lie.
• b. But that was false.
• c. That struck me as a funny way to describe the situation.
• d. That caused a financial problem for Sue.
• (e.g., in passage, "that" can refer to an event, a description, or a speech act depending on
context).
 Reference Resolution
• Example and Applications:
• In a passage about "Victoria Chen", reference resolution algorithms would identify all
expressions coreferring to her (e.g., "Chief Financial Officer", "she", "the 37-year-old") and link
them in a coreference chain.
• Accurate reference resolution is essential for tasks in NLP like summarization, information
extraction, and dialogue systems, as it enables systems to understand context and maintain
coherence across discourse.
 Dialog and Conversational Agents
• Dialog and conversational agents are systems designed to interact with humans using natural
language.
• Facilitate human-like conversations to achieve tasks or provide information.
• Types of Agents:
• Chatbots: Rule-based systems for simple tasks (e.g., FAQs).
• Virtual Assistants: AI-based, performs complex tasks (e.g., Alexa, Siri).
• Enhance user interaction, provide assistance, automate tasks.
• History and Evolution of Conversational Agents
• Early Days: ELIZA (1966), one of the first chatbots.
• Rule-Based Systems: Predefined responses, limited understanding.
• Advances in AI and ML: Machine learning improved language understanding.
• Modern Day Applications: NLP and deep learning models (e.g., GPT, BERT).
 Dialog Acts
• Dialog acts represent the function of an utterance in conversation, such as requesting, informing,
or confirming.
• Helps conversational agents understand user intent and manage interactions smoothly.
• Enhance the agent’s ability to respond appropriately based on the user’s intended meaning.

• Common Types of Dialog Acts

• Informative Acts: Provide information or clarification.
• Request Acts: Ask for information or action.
• Confirming Acts: Verify previously exchanged information.
• Acknowledge Acts: Indicate understanding or agreement.
• Questioning Acts: Inquire to elicit information from the user
• .
 Dialog Acts
• Importance of Dialog Act Interpretation
• Enhances Intent Recognition: Enables better understanding of user needs.
• Context Management: Helps agents manage multi-turn interactions.
• Personalized Responses: Leads to more nuanced and adaptive responses.
• Examples in Action:
• Distinguishing between a request for information (“Where can I find…?”) versus a confirmation
(“So this is where I find…?”).
• Techniques for Dialog Act Interpretation
• Rule-Based Approaches: Handcrafted rules classify dialog acts based on keywords or phrases.
• Machine Learning Approaches:
• Supervised Classification: Train on labeled datasets with dialog act annotations
• Common Algorithms: SVMs, Naive Bayes, neural networks.
• Deep Learning Approaches:
• RNNs, LSTMs, Transformers: Better at understanding sequential data and context.
 Conversational Agents - Dialog Acts – Interpretation
• Human conversation is complex, involving nuanced joint activity, unlike simpler and more
constrained human-machine interactions.
• Understanding human conversation is essential for designing effective conversational agents.
• Turns and Turn-Taking:
• Turn-Taking:
• Conversations are characterized by turn-taking, where speakers alternate turns.
• Efficient turn allocation relies on structured language that minimizes overlap, with typical
transition times between turns being short.
• Turn-Taking Rules:
• The foundational rules include:
• The current speaker selecting the next speaker.
• Any other speaker taking a turn if the current speaker doesn't specify.
• The current speaker continuing if no one else takes the turn.
• Conversation Analysis (CA): Studies turn-taking behaviors and how speakers coordinate to
identify who speaks next and when.
 Conversational Agents - Dialog Acts – Interpretation
• Transition-Relevance Place (TRP):
• Indicates points where speaker shifts are likely to occur, often at the end of utterances.
• Adjacency Pairs:
• Conversational structures where an initial utterance (e.g., question) is followed by an expected
response (e.g., answer). These pairs guide dialogue modeling and user expectations.
• Significant Silence:
• Silence can be meaningful, particularly after a question, where it can imply refusal or
hesitation. By contrast, silence at other times may not carry the same significance.
• Implications for UI Design:
• Recognizing the importance of pauses and transitions helps inform user-friendly designs for
spoken dialogue systems, addressing user discomfort with delays.
• Utterance Differences:
• Spoken utterances differ from written sentences, being shorter, often single clauses, and
including pronouns, pauses, and repairs, all of which a listener must consider to engage
effectively.
 Conversational Agents - Dialog Acts – Interpretation
• Language as Action: Speech Acts:
• Conversation is not just about exchanging words; each utterance can be viewed as an action
performed by the speaker. This concept is rooted in the idea of speech acts, introduced by
Wittgenstein and expanded by Austin.
• Performative Sentences: Certain sentences directly perform an action when spoken. Examples
include:
• "I name this ship the Titanic." (changes the ship's state by naming it)
• "I second that motion." (affirms support in a formal setting)
• "I bet you five dollars it will snow tomorrow." (initiates a bet)
• Types of Acts in Utterances:
• Locutionary Act: The act of saying something with a particular meaning.
• Illocutionary Act: The intended action performed through the utterance (e.g., promising,
questioning, commanding). Example: "You can’t do that" could be a protest.
• Perlocutionary Act: The effect on the listener's thoughts or actions (e.g., stopping or annoying
the addressee).Speech Act Focus: The term speech act generally refers to illocutionary acts,
which encompass the primary intention of the speaker.
 Conversational Agents - Dialog Acts – Interpretation
• Language as Action: Speech Acts:
• Classification of Speech Acts (based on Searle's taxonomy):
• Assertives: Commit the speaker to the truth of a statement (e.g., "I suggest that the meeting be
at noon.").Directives: Attempt to make the listener do something (e.g., "Please pass the salt.").
• Commissives: Commit the speaker to a future action (e.g., "I promise to call you.").
• Expressives: Express the speaker’s feelings or emotional state (e.g., "I apologize for being
late.").
• Declarations: Change the state of the world with the utterance (e.g., "I resign," or "You're
fired.").
 Conversational Agents - Dialog Acts – Interpretation
• Language as Joint Action: Grounding:
• Conversations are not independent acts; they are joint activities between speaker and hearer,
requiring mutual understanding or common ground (Stalnaker, 1978).
• Grounding: The hearer must confirm they have understood the speaker’s intent, ensuring effective
communication.
• Principle of Closure:
• People need evidence that their actions are successful. For example, elevator buttons light up when
pressed to confirm a successful action.
• Phases of Contribution:
• Presentation Phase: The speaker presents an utterance.
• Acceptance Phase: The hearer grounds the utterance, signaling comprehension.
• Methods of Grounding (from weakest to strongest):
• Continued Attention: The hearer indicates engagement (e.g., eye contact).
• Next Contribution: The hearer responds with the next relevant dialogue move.
• Acknowledgement: The hearer provides verbal or non-verbal feedback (e.g., "uh-huh").
• Demonstration: The hearer paraphrases or completes the speaker's statement.
• Display: The hearer repeats the speaker's words verbatim.
 Conversational Agents - Dialog Acts – Interpretation
• Language as Joint Action: Grounding:
• Examples of Grounding in Dialogue:
• Display: C1: "I need to travel in May.“
• A1: "And, what day in May did you want to travel?“
• Acknowledgement:
• C: "He wants to fly from Boston.“
• A: "Mm-hmm."Backchannels:Short utterances like "uh-huh" or "mm-hmm" signal the hearer’s
attention without requiring further response.
• Adjacency Pairs:
• Natural dialogues often include paired exchanges like question-answer or proposal-
acceptance/rejection.
• Significance in Human-Machine Interaction:
• Proper grounding is essential for making interactions sound natural and clear.
• Example: The response “Okay, what’s next?” (grounding) sounds more natural than jumping to
“What’s next?” without an acknowledgment.
• Errors Without Grounding: Systems that fail to provide clear acknowledgment can lead to
confusion.
 Conversational Agents - Dialog Acts – Interpretation
• Conversational Structure:
• Conversations are organized through elements like adjacency pairs and structured
contributions.
• Conversational Openings: Openings, especially in phone conversations, often follow a 4-stage
structure:
• Summons-Response Pair: Initiation by ringing and response by answering.
• Identification: Participants introduce themselves.
• Establish Joint Willingness: Indicating readiness to converse.
• First Topic: The caller typically introduces the main topic.
• Example of Task-Oriented Opening (from Clark, 1994):
• Stage 1: A1 (rings phone)Stage 1 & 2: B1: "Benjamin Holloway“
• Stage 2: A1: "This is Professor Dwight’s secretary, from Polymania College.“
• Stage 2 & 3: B1: "Ooh yes—“
• Stage 4: A1: "Um, about the lexicology seminar.“
• Stage 4: B1: "Yes."
 Conversational Agents - Dialog Acts – Interpretation
• Conversational Structure:
• Caller Expectations: Callers generally expect to introduce the first topic.
• If the answerer starts with the topic, it can lead to confusion.
• Example of Caller Confusion:
• Operator: "Directory Enquiries, for which town please?
• "Customer: "Could you give me the phone number of um, Mrs. Smithson?
• "The operator’s initial question ("for which town please?") was ignored by the customer, who
was expecting to introduce her own topic.
• Designing Conversational Prompts:
• Effective conversational agents use open-ended prompts like "How may I help you?" instead of
directive prompts (e.g., "For which town please?").
• This allows callers to express their topics and minimizes confusion.
• Other Structures:
• Conversations have more complex structures, including conversational closings and pre
sequences.
 Conversational Agents - Dialog Acts – Interpretation
• Conversational Implicature:
• In conversation, the literal meaning of an utterance often extends beyond what is explicitly
said due to inference.
• This means that speakers rely on the hearer's ability to infer intended meanings beyond the
words spoken.
• Examples from Dialogue:
• In a travel conversation, a client states, "I need to be there for a meeting that’s from the 12th to
the 15th" instead of directly stating a travel date.
• The agent infers that the client intends to travel before the 12th, relying on the context and the
Maxim of Relevance.
• An agent saying, "There’s three non-stops today" implies there are only three non-stops, even
though technically more could exist.
• The Maxim of Quantity helps the hearer infer the intended meaning that three is the total
number of non-stops.
• Grice's Theory of Conversational Implicature:
• Grice (1975, 1978) introduced the concept that conversations are governed by implicatures—
inferences that go beyond literal meaning, facilitated by the assumption that participants
adhere to conversational maxims.
 Conversational Agents - Dialog Acts – Interpretation
• Conversational Implicature:
• Grice’s Four Maxims:
• Maxim of Quantity:
• Be as informative as required for the purpose of the conversation.
• Avoid giving more information than necessary.
• Maxim of Quality:
• Contribute truthfully.
• Do not say anything believed to be false or for which there is inadequate evidence.
• Maxim of Relevance:
• Stay relevant in your contributions.
• Maxim of Manner:
• Be clear and orderly.
• Avoid obscurity, ambiguity, and unnecessary verbosity.
• Application of Maxims:
• Maxim of Quantity: The agent implies "only three" non-stops by stating "three," not "seven" (informative but not
overly so).
• Maxim of Relevance: The client’s mention of the meeting implies it is relevant to the travel discussion, helping the
agent infer intended travel dates.
• Grice’s maxims facilitate understanding and guide both speakers and listeners in inferring meaning and
maintaining effective communication.
 Language Generation – Architecture
• Commercial spoken dialogue systems typically consist of six main components:
• Speech Recognition and Natural Language Understanding (NLU), which process and interpret
user input.
• Dialogue Manager, which coordinates the system’s interactions and flow.
• Task Manager, which handles specific domain knowledge and tasks (e.g., air travel).
• Natural Language Generation (NLG) and Text-to-Speech (TTS), which produce and deliver spoken
responses.
 Language Generation – Architecture
• ASR component
• The ASR (automatic speech recognition) component in dialogue systems processes audio input
(e.g., from a phone or microphone) and outputs a transcribed string of words.
• While general ASR systems are designed to handle a wide range of vocabulary for various topics,
dialogue systems typically focus on specific domains and thus use specialized, restrictive language
models.
• Domain-Specific Optimization: ASR in dialogue systems doesn't need to transcribe every possible
English sentence, but only those relevant to the task. This is often achieved with hand-written
finite-state grammars that outline the responses the system can understand.
• Dialogue-State Dependent Models: ASR systems adjust language models based on the current
dialogue context. For instance, if the system asks, "What city are you departing from?", the ASR
can be set to recognize only city names or phrases like "I want to leave from [CITYNAME]."Types
of Language Models:
• Finite-state grammars: Hand-written and often used for restricted responses.
• N-gram models: Used when the system needs to understand a broader range of sentences,
conditioned by dialogue context.
 Language Generation – Architecture
• ASR component
• Restrictive Grammar: The system uses restrictive grammar to narrow down user responses based
on its last utterance.
• For more flexibility, it can mix restrictive models with more general ones, depending on how much
user initiative is allowed.
• Speaker Adaptation: ASR systems benefit from speaker adaptation techniques like MLLR
(Maximum Likelihood Linear Regression) and VTLN (Vocal Tract Length Normalization), which
improve recognition over time as the system receives more user speech.
• Real-Time Response and Confidence Values: ASR engines in dialogue systems need to process
input quickly to prevent user frustration.
• They also return confidence scores to help the system decide if user input should be confirmed,
ensuring better interaction quality.
• Example: If a user interacts with a travel booking system, the ASR might switch from general
recognition to a city-specific language model after asking, "Which city are you departing from?"
This helps improve accuracy and relevance in responses.
 Language Generation – Architecture
• NLP component
• The NLU (natural language understanding) component in dialogue systems is crucial for
transforming user input into semantic representations that align with the system’s task.
• 1. Semantic Representation with Frames and Slots: The NLU component often uses a frame-and-
slot approach, where user utterances fill specific slots in a frame relevant to the task (e.g., booking
a flight).
• For example, "Show me morning flights from Boston to San Francisco on Tuesday" would
populate a frame with ORIGIN: Boston, DEST: San Francisco, DATE: Tuesday, and TIME:
morning.
• 2. Methods of Semantic Analysis:
• Unification Grammars: Systems may use general-purpose grammars with semantic attachments,
parsing sentences to extract slot-fillers.
• Semantic Grammars: Domain-specific grammars where grammar rules correspond to semantic
entities. For instance, ORIGIN ! from CITY helps label input directly. They are parsed using
standard algorithms like CKY or Earley.
 Language Generation – Architecture
• NLP component
• 3. Ambiguity Handling:
• Probabilistic Extensions: To handle ambiguity, grammars can include probabilities. For example, the
TINA system mixes syntactic and semantic rules and trains probabilities using statistical CFGs.
• HMM Models: These probabilistic models, like the one by Pieraccini et al. (1991), use hidden states as
semantic slot labels and observed words as fillers. They decode sequences of hidden states from words
using the Viterbi algorithm.
• 4. HMM and Semi-HMM Models:
• Generative Model Components:
• P(C): A prior over semantic slots, representing the sequence of meanings.
• P(W|C): The likelihood of words expressing a given meaning, modeled by word N-grams conditioned
on slots.
• Semi-HMMs: Variants where hidden states generate multiple outputs (i.e., words filling a slot),
different from classic HMMs where one state produces one output.
• 5. Advanced Models:
• Hidden Understanding Model (HUM): Introduces hierarchical structures to integrate semantic
grammars and HMMs for more comprehensive understanding.
• Decision-List Methods: An alternative approach using decision lists for semantic interpretation.
 Language Generation – Architecture
• NLP component
• 3. Ambiguity Handling:
• Probabilistic Extensions: To handle ambiguity, grammars can include probabilities. For example, the
TINA system mixes syntactic and semantic rules and trains probabilities using statistical CFGs.
• HMM Models: These probabilistic models, like the one by Pieraccini et al. (1991), use hidden states as
semantic slot labels and observed words as fillers. They decode sequences of hidden states from words
using the Viterbi algorithm.
• 4. HMM and Semi-HMM Models:
• Generative Model Components:
• P(C): A prior over semantic slots, representing the sequence of meanings.
• P(W|C): The likelihood of words expressing a given meaning, modeled by word N-grams conditioned
on slots.
• Semi-HMMs: Variants where hidden states generate multiple outputs (i.e., words filling a slot),
different from classic HMMs where one state produces one output.
• 5. Advanced Models:
• Hidden Understanding Model (HUM): Introduces hierarchical structures to integrate semantic
grammars and HMMs for more comprehensive understanding.
• Decision-List Methods: An alternative approach using decision lists for semantic interpretation.
 Language Generation – Architecture
• Generation and TTS components
• The Figure shows components involved in a conversational agent, focusing on the generation and
text-to-speech (TTS) aspects

• 1. Generation Component
• Role: Chooses concepts to express, plans how to articulate them, and assigns prosody (intonation
and rhythm) to the words.
• Separation of Tasks:
• What to Say: Handled by the content planner, which decides the content (e.g., asking a question,
presenting an answer).
• How to Say It: Managed by the language generation module, which selects the syntax and words
to convey the meaning.
 Language Generation – Architecture
• Generation and TTS components
• 2. Language Generation Methods
• Template-Based Generation:
• Most words are predefined by the dialogue designer; some variables are filled in dynamically.
• Example Prompts:
• "What time do you want to leave CITY-ORIG?“
• "Will you return to CITY-ORIG from CITY-DEST?“
• Natural Language Generation:
• Involves building a representation of the intended meaning and using a generator with
components like a sentence planner, surface realizer, and prosody assigner.
• 3. Discourse and Conversational Constraints
• Coherence in Conversation: Dialogue should flow naturally, using discourse markers and
pronouns for better coherence.
• Example of Natural vs. Mechanical Dialogue:
• Less natural: “Please say the data... Please say the start time...”
• More natural: “First, tell me the date... Next, I’ll need the time it starts...”
 Language Generation – Architecture
• Generation and TTS components
• Tapered Prompts: Used when a system needs to prompt the user repeatedly, getting shorter with
each iteration.
• Example of Tapered Prompts:
• "Now, what’s the first company to add to your watch list?" followed by progressively shorter
prompts.
• Memory and Attention Considerations:
• Prompts should be structured to minimize cognitive load, such as placing the desired response at
the end of the prompt.
• Example: "To hear the list again, say ‘Repeat list’" is more effective than "Say ‘Repeat list’ to hear
the list again.“
• 4. Content Planning Rules
• Systems have rules for managing user responses, especially for long lists. For instance, if there are
more than three options, only a summary or the earliest item is detailed.
• The generation and TTS components of conversational agents work together to create coherent
and user-friendly dialogues.
 Language Generation – Architecture
• Dialogue Manager
• The dialogue manager oversees the structure and flow of the conversation, managing input from
Automatic Speech Recognition (ASR) and Natural Language Understanding (NLU) components,
maintaining the dialogue state, and interfacing with the task manager.
• It outputs information to the Natural Language Generation (NLG) and Text-to-Speech (TTS)
modules.
• Historical Context
• Early dialogue managers, like ELIZA, operated on a simple read-substitute-print loop without
maintaining state.
• Modern dialogue managers retain more complex state information and can model dialogue
structures beyond single responses.
 Language Generation – Architecture
• Dialogue Manager
• Types of Dialogue Management Architectures
• Finite-State Dialogue Manager:
• Uses a finite-state automaton (FSA) where states represent questions and arcs represent user
responses.
• Example: In a travel booking system, states correspond to questions about departure cities,
destinations, and times.
• This architecture is system-initiative, meaning the system controls the conversation flow and
expects direct responses to its questions.
• Frame-Based Dialogue Manager:
• Users fill in slots in a predefined frame, allowing for more flexible information entry.
• This architecture accommodates mixed initiative, where users can respond to multiple questions at
once.
• Slot Example:
• For travel planning:
• ORIGIN CITY: "From what city are you leaving?“
• DESTINATION CITY: "Where are you going?"
 Language Generation – Architecture
• Dialogue Manager
• Initiative in Dialogue Systems
• System Initiative: The system controls the conversation, asking questions that the user must
answer specifically.
• User Initiative: The user has control over the dialogue, asking questions without being restricted
to a predefined structure.
• Mixed Initiative: Both the system and user can take turns controlling the dialogue. This is
operationalized based on the type of prompts (open vs. directive) and the grammar used by the
ASR.
• Open Prompt: Allows for flexible user responses (e.g., "How may I help you?").
• Directive Prompt: Specifies how the user should respond (e.g., "Say yes if you accept the call;
otherwise, say no.").
• Dynamic Initiative Adjustment
• Systems can adjust their initiative based on user experience and speech recognition accuracy.
• For example, novice users may benefit from more structured interactions, while expert users may
prefer flexible dialogue options.
• The dialogue manager is crucial for effective conversational agents, providing the structure
necessary for managing user interactions.
 Dialog and Conversational Agents
• Conversational Agent Examples
• Task-Oriented Agents:
• Alexa, Google Assistant for smart home controls.
• Social Bots:
• Replika for companionship and social interaction.
• Customer Service Bots:
• Zendesk, Intercom for automated customer support.

• Applications of Conversational Agents

• E-commerce and Customer Support: Automated FAQs, personalized recommendations.
• Healthcare: Virtual health assistants, appointment scheduling.
• Education: Language learning apps, tutoring agents.
• Enterprise Use: Productivity tools, employee self-service.
 Dialog and Conversational Agents
• Conversational Agent Examples
• Task-Oriented Agents:
• Alexa, Google Assistant for smart home controls.
• Social Bots:
• Replika for companionship and social interaction.
• Customer Service Bots:
• Zendesk, Intercom for automated customer support.

• Applications of Conversational Agents

• E-commerce and Customer Support: Automated FAQs, personalized recommendations.
• Healthcare: Virtual health assistants, appointment scheduling.
• Education: Language learning apps, tutoring agents.
• Enterprise Use: Productivity tools, employee self-service.
 Discourse Planning
• Discourse planning is the process in natural language generation (NLG) that structures content
at a higher level to create coherent, cohesive, and logically sequenced communication.
• It involves organizing ideas, facts, or responses before language formulation.
• Ensures that the generated text is logical, contextually relevant, and flows in a way that makes
sense to the reader or listener.
• Key Objectives of Discourse Planning
• Coherence: Maintaining logical flow and thematic unity throughout the discourse.
• Cohesion: Using linguistic elements such as pronouns, conjunctions, and transition words to
connect ideas smoothly.
• Relevance: Prioritizing information that is most pertinent to the communication goals or user’s
query.
• Applications of Discourse Planning
• Dialogue Systems: Enables chatbots to generate multi-turn interactions that maintain context
and coherence.
• Text Summarization: Helps in structuring content for concise summaries that preserve essential
information.
• Report and Article Generation: Organizes content to produce structured outputs such as news
articles, scientific papers, or business reports.
 Discourse Planning
• Components of Discourse Planning
• Content Selection: Choosing what information or ideas to include in the output.
• Factors to Consider: User intent, task-specific needs, data sources, and audience.
• Content Structuring: Arranging selected content in a logical and effective sequence.
• Techniques:
• Topical Structuring: Grouping related information together under specific themes.
• Chronological Structuring: Presenting events or facts in time order.
• Problem-Solution Structuring: Stating a problem followed by proposed solutions.
• Rhetorical Strategies:
• Employing various strategies to strengthen communication and engagement.
• Examples:
• Exemplification: Using examples to clarify points.
• Contrast: Highlighting differences to emphasize certain aspects.
• Summarization: Providing concise summaries for clarity and reinforcement.
 Discourse Planning
• Stages in Discourse Planning
• Macro Planning (High-Level Planning):Focuses on the overall structure and main ideas.
• Determines the sequence of topics or main points.
• Micro Planning (Detailed Planning):Breaks down macro-level content into sub-components and
fine details.
• Decides on sentence structure, transitions, and specific phrasing.
• Frameworks for Discourse Planning
• Rhetorical Structure Theory (RST):
• Describes how text elements relate to each other in a hierarchical structure.
• Emphasizes nucleus-satellite relationships (e.g., main content vs. supporting details).Plan-Based
Models:
• Utilize plans or schemas to guide the discourse generation process.
• Often used in rule-based NLG systems where predefined templates and rules are applied.
• Task-Oriented Approaches:
• Plan discourse based on the goal of communication (e.g., persuasive, informative, instructional).
 Surface Realization
• Stages in Discourse Planning
• Surface realization is the process in NLP where structured content is transformed into natural
language sentences.
• It takes the syntactic structures or logical forms produced during the earlier stages of natural
language generation (NLG) and converts them into grammatically correct and contextually
appropriate sentences.
• Role in NLG Pipeline: The step following discourse planning, surface realization ensures that
abstract representations are rendered into text or speech that users can understand.
• Key Concepts in Surface Realization
• Grammatical Structure: Ensuring that the generated sentences follow the rules of grammar for
the chosen language.
• Lexicalization: Selecting appropriate words or phrases to express the content while maintaining
context and coherence.
• Morphological Processing: Adjusting word forms to fit the sentence structure, including tense,
agreement, and inflection.
• Fluency and Naturalness: Ensuring the generated text reads smoothly and is not awkward or
robotic.
 Surface Realization
• Approaches to Surface Realization
• Template-Based Methods:
• Simple and effective for controlled language generation.
• Limited in flexibility, best suited for well-defined, repetitive tasks.
• Rule-Based Systems:
• Use hand-crafted linguistic rules to convert structures into natural language.
• Offers more control but requires extensive domain knowledge and is not easily scalable.
• Statistical and Machine Learning Models:
• Utilize large datasets to learn patterns for sentence generation.
• Examples include sequence-to-sequence models that can handle variability in output.
• Neural Network-Based Approaches:
• Employ deep learning, including models such as RNNs, LSTMs, or Transformers, to learn
complex patterns for generating natural text.
• Advantages: Capable of handling nuanced linguistic structures.
• Challenges: High dependency on quality and size of training data
 Surface Realization
• Evaluation Metrics for Surface Realization
• BLEU (Bilingual Evaluation Understudy): Commonly used metric that compares the generated
text to human references based on n-gram matching.
• ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Measures overlap between
generated and reference texts, focusing on recall.
• Human Evaluation: Involves subjective assessment of fluency, coherence, and relevance by
human judges for qualitative feedback.
• Applications of Surface Realization
• Chatbots and Conversational Agents: Used for generating responses that are coherent and
contextually appropriate in conversations.
• Automatic Report Generation: Creating structured reports from data inputs (e.g., weather
reports, financial summaries).
• Assistive Writing Tools: Helping users draft texts by converting structured inputs or
suggestions into full sentences.