Application of NLP
Question Answering System (QAS)
Definition:
A Question Answering System (QAS) is an advanced application of Natural Language Processing
(NLP) that automatically answers questions posed by humans in natural language. It extracts,
processes, and presents relevant answers from a large collection of documents or databases.
As shown in Fig. 6.5.1, a QAS works in three main phases:
1. Question Processing
2. Information Retrieval
3. Answer Processing
1. Question Processing
This is the first and most important phase of a QAS.
Its main goal is to understand the user’s query correctly and prepare it for the retrieval phase.
Steps in Question Processing:
(i) Question Formulation:
• The system converts the user’s natural language query into a structured form that the
information retrieval module can understand.
• It identifies key terms and constructs a formal query using those terms.
• Different retrieval systems may require different query formats.
• The formulated query is then sent to the Information Retrieval phase.
(ii) Question Type Identification:
• The system determines what kind of question has been asked (e.g., Who, What, When, Where,
Why, etc.).
• This helps the system decide which type of answer is most relevant.
• Machine learning and deep learning classification techniques (e.g., K-Nearest Neighbour,
SVM, Naïve Bayes, LSTM) are used to identify the question type.
• The identified type is then sent to the Answer Processing phase.
2. Information Retrieval
This phase acts as the brain of the QAS.
Its main goal is to locate and extract the most relevant information or documents related to the user’s
question.
Functions:
• Takes input from the Question Formulation stage.
• Searches a knowledge source (can be closed or open-ended):
o Closed-domain systems use pre-supplied documents or databases.
o Open-domain systems search the web or other large sources.
• Extracts pertinent information from the knowledge source.
• The retrieved information is then passed to the Answer Processing stage.
Knowledge Sources:
There are three ways to provide knowledge to the system:
1. Textual data (documents)
2. Knowledge base
3. Structured database
Retrieving information from a structured database or knowledge base is generally easier and
faster than from unstructured text.
3. Answer Processing
This is the final stage of the Question Answering System.
It refines the results obtained from the Information Retrieval phase to generate the most accurate
answer.
Functions:
• Takes the question type (from Question Processing) and retrieved data (from Information
Retrieval) as input.
• Filters multiple possible answers to select the most relevant and precise one.
• Uses linguistic and contextual analysis to extract the correct answer.
• Finally, presents the result to the user in a clear and concise form.
Flow of QAS:
1. User asks a question.
2. Question Processing analyzes and reformulates it.
3. Information Retrieval fetches relevant documents or passages.
4. Answer Processing extracts the final answer.
5. System outputs the answer to the user.
Challenges / Issues in QAS
1. Lexical Gap:
Difference between words used in the question and those in the documents (e.g., synonyms).
2. Ambiguity:
A word or phrase can have multiple meanings, making interpretation difficult.
3. Complex Data:
Large, unstructured, and diverse data sources complicate retrieval and processing.
4. Complex Queries:
Multi-part or context-based questions are hard to analyze accurately.
5. Partial Knowledge:
The system may not have complete information to answer every query.
6. Incorrect Knowledge:
Errors or outdated data in the knowledge base lead to wrong answers.
Write note on : Machine Translation approach in NLP
Definition: Machine Translation (MT) is the process of automatically translating text or speech
from one natural language (called the source language) to another (called the target language) using
computational methods.
The main goal of MT is to preserve the original meaning of the sentence while converting it into
another language.
Example:
Translating “How are you?” (English) → “तुम कैसे हो?” (Hindi)
Challenges in Machine Translation:
1. Large variety of languages, grammar, and alphabets.
2. Word meaning ambiguity — one word can have many meanings.
3. Context understanding — difficult for machines to preserve sense and tone.
4. Idioms and cultural expressions — not directly translatable.
5. Sentence structure differences between languages.
Approaches to Machine Translation
Machine Translation has evolved over time through three main approaches:
1. Rule-Based Machine Translation (RBMT):
• Uses linguistic rules, dictionaries, and grammatical structures of both source and target
languages.
• Translation is done using morphological, syntactic, and semantic analysis.
Example:
English → Hindi translation using predefined grammar rules.
Advantages:
• Works well for specific language pairs.
• Provides grammatically correct translations.
Disadvantages:
• Requires heavy manual effort to build rules.
• Not flexible for all languages.
2. Statistical Machine Translation (SMT):
• Based on probability and statistics rather than linguistic rules.
• Uses large amounts of bilingual text data (parallel corpora) to learn translation patterns.
• Translates by finding the most probable word or phrase in the target language.
Example:
Google Translate (earlier versions) used SMT.
Advantages:
• Learns automatically from data.
• Adapts better to multiple languages.
Disadvantages:
• Needs large parallel datasets.
• May produce grammatically incorrect sentences.
3. Neural Machine Translation (NMT):
• Uses deep learning and artificial neural networks.
• Translates sentences end-to-end, learning the relationship between source and target
sequences.
• Considers context of the whole sentence, not just individual words.
Example:
Modern Google Translate, Microsoft Translator use NMT.
Advantages:
• Produces fluent and natural translations.
• Learns context and meaning better.
Disadvantages:
• Needs powerful computing resources and large data.
• Sometimes fails with rare words or idioms.
4. Hybrid Machine Translation:
• Combines Rule-Based and Statistical/Neural approaches.
• Aims to get the accuracy of RBMT and fluency of NMT/SMT.
Example:
Using grammar rules to correct neural translations.
Explain Text Summerization in detail.
Definition:
Text Summarization is the process of reducing a large body of text into a shorter version while
preserving its key information and main meaning.
It aims to generate a concise and coherent summary that conveys the essential points of the original
content.
Need for Text Summarization:
In today’s Big Data era, a massive amount of textual data is produced every second — from news
articles, research papers, social media posts, and legal documents.
Reading and analyzing all of it manually is time-consuming and difficult, hence automatic text
summarization helps by providing the main idea quickly and efficiently.
Applications of Text Summarization:
• News summarization – condensing long news into short versions.
• Legal and medical document summarization.
• Search engines – showing summarized snippets of results.
• Question answering systems – summarizing relevant answers.
• Headline generation and report generation.
Types of Text Summarization
Text summarization can be classified into two main types:
1. Extractive Summarization
Definition:
Extractive summarization works by selecting and combining key sentences, phrases, or words
directly from the original text.
It uses statistical and linguistic features (like sentence importance, frequency of keywords, or
position in paragraph) to identify the most relevant parts.
How it Works:
• Assign a score to each sentence using algorithms (like TF-IDF, TextRank, etc.).
• Select top-ranked sentences.
• Combine them to form a summary.
Characteristics:
• Summary is made up of existing phrases from the source text.
• Simpler and more grammatical (since words come from original text).
• May lack smooth flow or coherence.
Example:
Source Text:
Joseph and Mary rode on a donkey to attend the annual event in Jerusalem. In the city, Mary gave
birth to a child named Jesus.
Extractive Summary:
Joseph and Mary attend event Jerusalem. Mary birth Jesus.
The sentences are picked and shortened from the original text, not re-written.
2. Abstractive Summarization
Definition:
Abstractive summarization aims to understand the meaning of the original text and then generate
new sentences that convey the same information in a shorter form — similar to how humans
summarize.
How it Works:
• Uses Natural Language Understanding (NLU) and Natural Language Generation
(NLG).
• Requires deep learning, neural networks, or sequence-to-sequence models (like
transformers, BERT, GPT, etc.).
• Rephrases and paraphrases content to produce a fluent, natural summary.
Example:
Source Text:
Joseph and Mary rode on a donkey to attend the annual event in Jerusalem. In the city, Mary gave
birth to a child named Jesus.
Abstractive Summary:
Joseph and Mary traveled to Jerusalem, where Jesus was born.
Demonstrate the working of machine translation system.
Definition:
Machine Translation is the process of automatically converting text from a source language to a
target language, preserving its meaning as accurately as possible.
Working of a Machine Translation System
1. Input (Source Language)
• The user provides the text in the source language.
• Example: “I am going to school.”
2. Preprocessing
Before translation, the text is cleaned and analyzed:
• Tokenization: Split text into words or sentences.
• Part-of-Speech (POS) Tagging: Identify nouns, verbs, etc.
• Morphological Analysis: Identify word roots, prefixes, and suffixes.
3. Analysis / Understanding
• The system analyzes the grammatical and semantic structure of the source sentence.
• Syntactic parsing: Builds a sentence structure tree.
• Semantic analysis: Understands word meanings and relationships.
4. Transfer / Mapping
• Maps elements of the source language to the target language:
o Words
o Phrases
o Grammar structures
Depending on the MT approach:
1. Rule-Based MT: Uses dictionaries and grammar rules.
2. Statistical MT: Uses probability models from parallel corpora.
3. Neural MT: Uses deep learning to generate translations based on context.
5. Generation (Target Language)
• Generates the final translated text in the target language.
• Adjusts word order, agreement, and idiomatic expressions.
Example:
• Source: “I am going to school.”
• Target (Hindi): “मैं स्कूल जा रहा हूँ ।”
6. Post-processing
• Corrects minor grammatical errors.
• Handles punctuation, capitalization, and formatting.
Diagram: Working of MT System
Source Text → Preprocessing → Analysis → Transfer/Mapping → Generation → Target Text
Flow Example:
1. Input: “She is reading a book.”
2. Tokenization: [She, is, reading, a, book]
3. POS Tagging: [(She, PRP), (is, VBZ), (reading, VBG), (a, DT), (book, NN)]
4. Semantic Analysis: Subject=She, Verb=reading, Object=book
5. Mapping: Apply target language grammar & word mapping
6. Output: “वह एक ककताब पढ़ रही है ।”
Information Retrieval (IR)
Definition:
Information Retrieval (IR) is a process of organizing, storing, and retrieving relevant information
from large document repositories, particularly textual information. It helps users find documents or
data related to their queries.
An IR system does not directly answer questions but identifies documents or data sources that may
contain the required information. The goal is to retrieve only the relevant documents that meet the
user’s information needs.
Working of a Typical Information Retrieval System
The working of an IR system involves several steps as shown in the figure (Fig. 6.4.1):
1. Acquisition
• In this step, the system collects documents and text-based content from various web
resources.
• Web crawlers or agents gather and store this data in a database.
• Example: Collecting webpages, articles, and reports for indexing.
2. Representation
• The collected documents are represented and indexed using human and automated
techniques.
• It includes controlled vocabulary, free-text phrases, summarizing, and bibliographic
descriptions such as author, title, and metadata.
• The purpose is to convert raw documents into a structured form suitable for retrieval.
3. File Organization
• Documents are organized in files to enable efficient searching.
• Two main types:
o Sequential: Organized by document data (document-wise).
o Inverted: Organized by terms with a list of documents in which they occur.
• Sometimes a hybrid of both is used.
4. Query
• When a user submits a query, the system starts searching for matching documents.
• A query expresses the user’s information need, often in keywords.
• The IR system finds items that partially or fully match the query terms.
5. Matching and Retrieval
• The system matches the user’s query with stored document representations.
• Relevant documents are retrieved and displayed to the user.
6. Feedback
• The user gives feedback on retrieved documents (relevant or not).
• The system uses this feedback to improve future searches and refine results.