Module-IV DISCOURSE ANALYSIS AND DIALOGUE SYSTEMS

Discourse Coherence and Structure, Centering Theory and Coherence mechanisms,

introduction to question Answering system, classical QA models-conceptual overview –
dialogue system-types of architecture overview (frame based, state driven)

Definition of Discourse: Discourse is the coherent structure of language above the level of
sentences or clauses. A discourse is a coherent structured group of sentences.
Cohesion
Relations between words in two units (sentences, paragraphs) ―glue‖ them together.
Example: Before winter I built a chimney, and shingled the sides of my house… I have thus a
tight shingled and plastered house.
There are Three Main Classes of Features for Discourse Cohesion
• Lexical overlap/lexical chains
• Coreference chains
• Cue words/discourse markers
Discourse Processing
One of the major problems in NLP is discourse processing − building theories and
models of how utterances stick together to form coherent discourse. Actually, the language
always consists of collocated, structured and coherent groups of sentences rather than isolated
and unrelated sentences like movies. These coherent groups of sentences are referred to as
discourse.
Concept of Coherence
Coherence and discourse structure are interconnected in many ways. Coherence, along
with property of good text, is used to evaluate the output quality of natural language
generation system. The question that arises here is what does it mean for a text to be coherent?
Suppose we collected one sentence from every page of the newspaper, then will it be a
discourse? Of- course, not. It is because these sentences do not exhibit coherence. The coherent
discourse must possess the following properties −
Discourse structure
An important question regarding discourse is what kind of structure the discourse must
have. The answer to this question depends upon the segmentation we applied on discourse.
Discourse segmentations may be defined as determining the types of structures for large
discourse. It is quite difficult to implement discourse segmentation, but it is very important for
information retrieval, text summarization and information extraction kind of applications.
Algorithms for Discourse Segmentation
In this section, we will learn about the algorithms for discourse segmentation. The
algorithms are described below −
Unsupervised Discourse Segmentation
The class of unsupervised discourse segmentation is often represented as linear
segmentation. We can understand the task of linear segmentation with the help of an example.
In the example, there is a task of segmenting the text into multi-paragraph units; the units
represent the passage of the original text. These algorithms are dependent on cohesion that may
be defined as the use of certain linguistic devices to tie the textual units together. On the other
hand, lexicon cohesion is the cohesion that is indicated by the relationship between two or
more words in two units like the use of synonyms.

Supervised Discourse Segmentation

The earlier method does not have any hand-labeled segment boundaries. On the other
hand, supervised discourse segmentation needs to have boundary-labeled training data. It is very
easy to acquire the same. In supervised discourse segmentation, discourse marker or cue words
play an important role. Discourse marker or cue word is a word or phrase that functions to
signal discourse structure. These discourse markers are domain-specific.

Text Coherence
Lexical repetition is a way to find the structure in a discourse, but it does not satisfy the
requirement of being coherent discourse. To achieve the coherent discourse, we must focus on
coherence relations in specific. As we know that coherence relation defines the possible
connection between utterances in a discourse. Hebb has proposed such kind of relations as
follows
We are taking two terms S0 and S1 to represent the meaning of the two related sentences
Parallel
It infers p(a1,a2,…) from assertion of S0 and p(b1,b2,…) from assertion S1. Here ai
and bi are similar for all i. For example, two statements are parallel − Ram wanted car. Shyam
wanted money.
Elaboration
It infers the same proposition P from both the assertions − S0 and S1 For
example, two statements show the relation elaboration: Ram was from Chandigarh. Shyam
was from Kerala.
 Occasion
It happens when a change of state can be inferred from the assertion of S0, final state of
which can be inferred from S1 and vice-versa. For example, the two statements show the
relation occasion: Ram picked up the book. He gave it to Shyam.
Building Hierarchical Discourse Structure
The coherence of entire discourse can also be considered by hierarchical structure
between coherence relations. For example, the following passage can be represented as
hierarchical structure
 S1 − Ram went to the bank to deposit money.
 S2 − He then took a train to Shyam’s cloth shop.
 S3 − He wanted to buy some clothes.
 S4 − He do not have new clothes for party.
 S5 − He also wanted to talk to Shyam regarding his health

Introduction to question answering system

Natural Language Processing is one of the important and exciting fields in AI and Data
science. NLP applications are already used in many places — chatbots, sentiment analyzers,
recommender systems, translators, search engines, etc.
What is Question Answering? — The task of searching through a large collection of
documents for a piece of text that answers a question. Simply, answering the questions using a
set of documents as reference. QA systems are used for information retrieval, document search,
Real time FAQ etc..
Haystack is an open-source framework for building search systems that work intelligently over
large document collections.
Streamlit is an open-source framework for building Machine Learning and Data Science web
apps.
QA system — Overview

 Documents: Source of information. Word documents, Plain text documents, PDFs, etc..
 File Converter: Converts files on your computer into the documents that can be
processed by the Haystack pipeline.
 Preprocessor: Cleans and splits the text into sensible units.
 Document Store: The component in Haystack that stores the text documents and their
metadata in a way that optimizes retrieval time.
 Retriever: A lightweight filter that selects only the most relevant documents for the
Reader to further process.
 Reader: A trained Question Answering model that does the closest reading of a
document to extract the exact text which answers a question.
Introduction
Question Answering is a computer science discipline within the fields of information
retrieval and natural language processing, which focuses on building systems that automatically
answer questions posed by humans in a natural language. A computer understanding of natural
language consists of the capability of a program system to translate sentences into an internal
representation so that this system generates valid answers to questions asked by a user. Valid
answers mean answers relevant to the questions posed by the user. As the internal
representation of natural language, sentences must adequately map semantics of this statement,
the most natural approach is in the simulation of facts contained in the sentences using a
description of real objects as well as actions and events connected with these objects. To form
an answer it is necessary, in the first place, to execute the syntax and semantic analysis of a
question. This article covers the introduction to Question Answering, types and challenges
posed by the systems in real world.
Open Datasets available for Question Answering
1. Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset,
consisting of questions posed by crowd workers on a set of Wikipedia articles, where
the answer to every question is a segment of text, or span, from the corresponding
reading passage, or the question might be unanswerable.
2. WikiQA dataset is a publicly available set of question and answer pairs, collected and
annotated for research on open-domain question answering. It is constructed using a
more natural process and is more than an order of magnitude larger than the previous
dataset. In addition, the WikiQA dataset also includes questions for which there are no
correct sentences, enabling researchers to work on answer triggering, a critical
component in any QA system.
3. The TREC-QA dataset contains questions and answer patterns, as well as a pool of
documents returned by participating teams.
4. NewsQA dataset is to help the research community build algorithms that are capable of
answering questions requiring human-level comprehension and reasoning skills.
Leveraging CNN articles from the Deep Mind Q&A Dataset, authors prepared a crowd-
sourced machine reading comprehension dataset of 120K..
Types of Question Answering
There are three major modern paradigms of question answering:
a). IR-based Factoid Question Answering goal is to answer a user’s question by finding short
text segments on the Web or some other collection of documents. In the question-processing
phase a number of pieces of information from the question are extracted. The answer type
specifies the kind of entity the answer consists of (person, location, time, etc.). The query
specifies the keywords that should be used for the IR system to use in searching for documents.

b) Knowledge-based question answering is the idea of answering a natural language question

by mapping it to a query over a structured database. The logical form of the question is thus
either in the form of a query or can easily be converted into one. The database can be a full
relational database, or simpler structured databases like sets of RDF triples. Systems for
mapping from a text string to any logical form are called semantic parsers. Semantic parsers for
question answering usually map either to some version of predicate calculus or a query
language like SQL or SPARQL.

c) Using multiple information sources: IBM’s Watson system from IBM that won the
Jeopardy! Challenge in 2011 is an example of a system that relies on a wide variety of
resources to answer questions. The first stage is question processing. The DeepQA system runs
parsing, named entity tagging, and relation extraction on the question. Then, like the text-based
systems, the DeepQA system extracts the focus, the answer type (also called the lexical answer
type or LAT), and performs question classification and question sectioning.
Next DeepQA extracts the question focus. Finally the question is classified by type as
definition question, multiple-choice, puzzle or fill-in-the-blank. Next is the candidate answer
generation stage according to the question type, where the processed question is combined with
external documents and other knowledge sources to suggest many candidate answers. These
candidate answers can either be extracted from text documents or from structured knowledge
bases.
Then it is passed through the candidate answer scoring stage, which uses many sources
of evidence to score the candidates. One of the most important is the lexical answer type. In the
final answer merging and scoring step, it first merges the candidate answers that are equivalent.
The merging and ranking is actually run iteratively; first the candidates are ranked by the
classifier, giving a rough first value for each candidate answer, then that value is used to decide
which of the variants of a name to select as the merged answer, then the merged answers are re-
ranked.
Dialogue System
Introduction
A Dialogue is a conversation between two or more agents, be they human or machine.
Research on dialogue is on two topics human-human dialogue and human-computer dialogue.
The later is involved in a Dialogue System, a computerized system whose aim is to interact
with humans in a natural language. Today dialogue system is developing in text, graphical,
spoken and multimodal systems.

Dialogue System
A dialogue system is a computer program that communicates with a human user in a
natural way. The dialogue System provides an interface between the user and a computer-
based application that permits interaction with the application in a relatively natural manner.
The System can be CUI, GUI, VUI and multi model etc. it can be used in telephones, PDA
systems, cars, robot systems and web browsers. A text based dialogue system is in which we
chat with the system. A spoken dialogue systems is defined as a computer systems that human
interact on a turn-by-turn basic and in which spoken natural language interface plays an
important part in the c o m m u n i c a t i o n A m u l t i m o d a l d i a l o g u e systems
are those which are dialogue systems that process two or more combined user input modes -
such as speech, pen, touch, manual gestures, gaze, and head and body movements - in a
coordinated manner with multimedia system output.[3] Different Dialogue Systems have
different architectures but they have same set of phases which are Input Recognition, Natural
Language Understanding, Dialogue Management, Response Generation and Output .
Components of Dialogue System
A Dialogue system has mainly seven components these
components are following:
 Input Decoder
 Natural Language Understanding
 Dialogue Manager
 Domain Specific Component
 Response Generator
 Output Renderer

Fig.1 - Components of dialogue system

Input Decoder
Input Decoder component is the one which recognizes the input. It converts the input to
the simple text. This component is present only in which are not text base dialogue systems. This
component involves conversion of spoken sound (user utterances) to text (a string of words). This
requires the knowledge of phonetics and phonology. Phonetics is branch of linguistic which deals
with the sound of speech and their production, combination, description and representation by
written symbols. Phonology is study of speech sound in language or a language with reference to
their distribution and patterning and to tacit rules governing pronunciation. For this purpose
speech Recognition is needed. There are many systems available for this purpose. These are
called Automatic Speech Recognition (ASR), Computer Speech Recognition or simply Speech to
Text (STT). Besides speech the dialogue system can have other inputs like gesture, handwriting
etc.
Natural Language Understanding
As the name suggest this unit try to understand what user want to tell. It converts the
sequence of words into a semantic representation that can be used by the dialogue manager. This
component involves use of morphology, syntax and semantics. Morphology is the study of the
structure and content of word forms. After identifying the keywords and forming a meaning it
provide it to dialogue manager.
Dialogue Manager
The Dialogue Manager manages all aspects of the dialogue. It takes a semantic
representation of the user’s text, figures out how text fits in the overall context and creates a
semantic representation of the system response. It performs many tasks these are:
 Maintains the history of dialogue
 Adopts certain dialogue strategies
 Deal with malformed and unrecognized text
 Retrieve the contents stored in files or database
 Decides the best response for user
 Manage initiative and system response
 Handle issue of pragmatics
 Discourse analysis
 It also performs grounding
For these tasks dialogue manager has many components these components are:
 Dialogue Model
 User Model
 Knowledge Base
 Discourse Manager
 Reference Resolver
 Grounding Module
Domain Specific Component
The Dialogue Manager usually needs to interface with some external software such as a
database or an expert system. The query or plans thus have to be converted from the internal
representation used by the dialogue manager to the format used by the external domain specific
system (e.g. SQL). This interfacing is handled by the domain specific components. This can be
handled by Natural Language Query Processing system. This system generate SQL query from
natural language.
Response Generator
This component involves constructing the message that is to be given by the user. It takes
decision regarding what information should be included, how information should be structured,
choice of words and syntactic structure for message. Current systems use simple methods such as
insertion of retrieved data into predefined slots in a template.
Speech Generation
It translates the message constructed by the response generation component into spoken
form. For speech generation two approaches may be used. The first approach is to use
prerecorded canned speech may be used with spaces to be filled by retrieved or previously
recorded samples e.g. ―Welcome, h o w I c a n h e l p yo u .‖ The s e c o n d approach is use text
to speech synthesis. In this speech is generated of text. It is called Contaminative Speech
Synthesis, Text to Phoneme conversion and Phoneme to speech conversion or Text to Speech
(TTS).
Classification of Dialogue System
On the basis of method used to control dialogue a dialogue system can be classified in
three categories
 Finite State (or graph) based systems
  Frame based systems
 Agent based systems
Finite State based Systems
In these types of systems the user is taken through a dialogue consisting of a sequence of
predetermined steps or stages. The flow of dialogue is specified as a set of dialogue states.
Following is the example:

Fig. 2 – Example of Finite State based System

Advantages
 Simple to construct
 The required vocabulary and grammar for each state can be determined in advance.

Disadvantages
 Dialogues are not natural
 Do not allow over-informative answers
 Inhabits the user ability to ask questions and take initiative.
Frame Based Systems
Frame Based systems uses template filling from user response. In this system user is asked
questions that enable the system to fill slots in a template in order to perform tasks. The flow
of dialogue is not predetermined but depends upon the content of user input and the information
the user has to elicit. Example is shown in Fig.3, in which we can see there are two different
dialogues in first the dialogue goes like finite state system. In second dialogue user provide over
information in respond to a question but system fills its slots of from the user’s input and asks for
the remaining information. This is how frame based systems works.
 Fig. 3 – Example of Frame Based System

Advantages
 Allow more natural Dialogues
 User can provide over informative answers

Disadvantages
 These systems can’t handle complex dialogues
 Range of application is limited to the systems that elicit information from users and act on
the basis on the same
Agent Based Systems
These systems allow complex communication between the system, the user and the
application in order to solve some problem or task. The interaction is viewed as interaction
between two agents, each of which is capable of reasoning about its own actions and beliefs. The
dialogue model takes the preceding context into account. The dialogue evolves dynamically as a
sequence of related steps that build on top of each other.

Advantages
 Allow natural language in complex domain
 User friendly, like talking to human
Disadvantages
 These systems are hard to build
 The agent itself are usually very complex