U20CSST01 - NATURAL LANGUAGE PROCESSING

MODULE I: INTRODUCTION TO NLP

Overview of Natural Language Processing, Stages of NLP, NLP Pipeline. Morphology and Word Structure,
Challenges in NLP, Document Structure – Methods and Complexity, Performance and Feature Considerations
OVERVIEW OF NATURAL LANGUAGE PROCESSING
Natural Language Processing (NLP) refers to AI method of communicating with an intelligent systems using a
natural language such as English. Processing of Natural Language is required when you want an intelligent system
like robot to perform as per your instructions, when you want to hear decision from a dialogue based etc.

The field of NLP involves making computers to perform useful tasks with the natural languages humans use. The
input and output of an NLP system can be

• Speech
• Written Text
NLP trinity:

Applications of NLP or Use cases of NLP :

1. Sentiment analysis:
• Sentiment analysis, also referred to as opinion mining - identifies the emotional tone behind a body of
text.
• Helps organizations gather insights into real-time customer sentiment, customer experience and brand
reputation.
• Use text analytics to analyze online sources such as emails, blog posts, online reviews, news articles,
survey responses, case studies, web chats, tweets, forums and comments.
U20CSST01 - NATURAL LANGUAGE PROCESSING

• The metrics used are designed to detect whether the overall sentiment of a piece of text is positive,
negative or neutral.

2. Machine Translation :
• Sub-field of computational linguistics that investigates the use of software to translate text or speech
from one language to another.
• Corpus: A collection of written texts, especially the entire works of a particular author.

3. Text Extraction :
• Extract information from text or unstructured data.
• Extract information such as entity names, locations, quantities, and more.

4. Text Classification :
• Unstructured text - hard to extract value from this data unless it’s organized.
• Text classification also known as text tagging or text categorization is the process of categorizing text
into organized groups.
• Text classifiers can automatically analyze text and then assign a set of pre-defined tags or categories
based on its content.

5. Speech Recognition :
• An interdisciplinary subfield of computer science and computational linguistics that develops
methodologies and technologies that enable the recognition and translation of spoken language into text
by computers.
• Automatic speech recognition (ASR), computer speech recognition or speech to text (STT).

6. Chatbot :
• Computer programs that conduct automatic conversations with people. They are mainly used in
customer service for information acquisition.

7. Email Filter :
• Most fundamental and essential applications of NLP.
• It began with spam filters, which identified specific words or phrases that indicate a spam message.
But, like early NLP adaptations, filtering has been improved.

8. Search Autocorrect and Autocomplete :

• Search for anything, it displays a list of probable search keywords. Alternatively, if you search for
anything with mistakes, it corrects them for you while still returning relevant results

Components of NLP :
• There are two components of NLP as given –
➢ Natural Language Understanding (NLU)
➢ Natural Language Generation (NLG)
• Natural Language Understanding (NLU) :
Understanding involves the following tasks –
✓ Mapping the given input in natural language into useful representations.
✓ Analyzing different aspects of the language.
• Natural Language Generation (NLG):
It is the process of producing meaningful phrases and sentences in the form of natural language from
some internal representation.
It involves –
✓ Text planning − It includes retrieving the relevant content from knowledge base.
✓ Sentence planning − It includes choosing required words, forming meaningful
phrases, setting tone of the sentence.
✓ Text Realization − It is mapping sentence plan into sentence structure.
U20CSST01 - NATURAL LANGUAGE PROCESSING

NLP Terminology:

• Phonology : It is study of organizing sound systematically.

• Morphology: The study of the formation and internal structure of words.

• Morpheme : It is primitive unit of meaning in a language.

• Syntax: The study of the formation and internal structure of sentences.

• Semantics: The study of the meaning of sentences.

• Pragmatics : It deals with using and understanding sentences in different situations and how the
interpretation of the sentence is affected.

• Discourse : It deals with how the immediately preceding sentence can affect the interpretation of the
next sentence.

• World Knowledge : It includes the general knowledge about the world.

Steps/Stages in NLP :

There are general five steps :

1. Lexical Analysis
2. Syntactic Analysis (Parsing)
3. Semantic Analysis
4. Discourse Integration
5. Pragmatic Analysis

Lexical Analysis :
• Lexical analysis is the process of converting a sequence of characters into a sequence of tokens.
A lexer is generally combined with a parser, which together analyzes the syntax of
programming languages, web pages, and so forth.
• Lexers and parsers are most often used for compilers but can be used for other computer
languages tools, such as pretty printers or liters.
• Lexical analysis is also an important analysis during the early stage of natural language
processing, where text or sound waves are segmented into words and other units
• Lexeme: A lexeme is a basic unit of meaning. In linguistics, the abstract unit of morphological
analysis that corresponds to a set of forms taken by a single word is called lexeme.
U20CSST01 - NATURAL LANGUAGE PROCESSING

Syntactic Analysis :
• Parsing, syntax analysis, or syntactic analysis is the process of analyzing a string of symbols,
either in natural language, computer languages, or data structures, conforming to the rules of
formal grammar.
• It is used in the analysis of computer languages, referring to the syntactic analysis of the input
code into its component parts to facilitate the writing of compilers and interpreters.
• Grammatical rules are applied to categories and groups of words, not individual words.
Syntactic analysis is a very important part of NLP that helps in understanding the grammatical
meaning of any sentence.
Semantic Analysis :
• Semantic Analysis attempts to understand the meaning of Natural Language.
• Semantic Analysis of Natural Language captures the meaning of the given text while
considering context, logical structuring of sentences, and grammar roles.
• 2 parts of Semantic Analysis are (a) Lexical Semantic Analysis and (b) Compositional
Semantics Analysis. Semantic analysis can begin with the relationship between individual
words.
Discourse Integration:
• Researchers use Discourse analysis to uncover the motivation behind a text.
• It is useful for studying the underlying meaning of a spoken or written text as it considers the
social and historical contexts.
• Discourse analysis is a process of performing text or language analysis, involving text
interpretation, and understanding social interactions.
Pragmatic Analysis
• Pragmatic Analysis is part of the process of extracting information from text. It focuses on
taking a structured set of text and figuring out the actual meaning of the text.
• It also focuses on the meaning of the words of the time and context. Effects on interpretation
can be measured using PA by understanding the communicative and social content

NLP PIPELINE:
NLP is a set of steps followed to build an end-to-end NLP software. In simple terms, To create a complete NLP
software, we follow a series of steps and that is called the NLP pipeline.

NLP software consists of the following steps :

1. Data Acquisition
2. Text Preprocessing
3. Feature Engineering
4. Modelling
5. Evaluation
6. Deployment
U20CSST01 - NATURAL LANGUAGE PROCESSING

1. Data Acquisition :
• Data acquisition involves obtaining raw textual data from various sources to create a robust dataset for
NLP tasks.
• Data acquisition- face one of three situations:

(i) Data Available Scenarios

1. Data on Your Desk: The data needed for the NLP task is already in possession. Initiate the text
preprocessing stage immediately.
2. Data in Databases: The required data resides within company databases or repositories.
Collaborate with data engineers to retrieve the data.
3. Less Data: Insufficient data volume for robust model training or analysis. Employ data
augmentation techniques to enhance the dataset.
(ii) Data from Other Resources
• Public Datasets: Utilize publicly available datasets from repositories like Kaggle, UCI
Machine Learning Repository, or government databases, aligning them with the project’s
requirements.
• Web Scraping: Extract data from websites or forums by scraping relevant information. Tools
like BeautifulSoup or Scrapy assist in collecting data from various websites.
• APIs: Access data through Application Programming Interfaces (APIs) offered by various
platforms such as social media APIs (Twitter/X, Reddit), news aggregators, or linguistic
databases.
• PDFs: Extract text from PDF documents relevant to the project using libraries
like PyPDF2 or PDFMiner.
(iii) Nobody has the Data
• Engaging Trusted Clients: Collaborate with trustworthy clients or users willing to share
anonymized data relevant to the project’s goals.
• Data Generation: If viable, companies can generate synthetic data or collect information
through surveys, interviews, or user-generated content to build a dataset from scratch.

2. Text Preprocessing:
Text preprocessing is a critical phase in NLP, encompassing various steps to refine raw text data for meaningful
analysis and model training.
(i) Basic Cleaning:
Eliminating irrelevant or disruptive elements from the text:
• HTML Tag Removal: Stripping out HTML tags is crucial when working with web-based text
sources. These tags contain formatting information and are unnecessary for linguistic analysis.
• Handling Emojis: Managing emojis involves converting them to textual representations
using emoji library or removing them entirely, depending on their relevance to the analysis.
• Basic Spell Checks: Performing rudimentary spell checks to rectify common typographical
errors and ensure consistency in the text.
(ii) Basic Preprocessing
Prepare the text for further analysis by applying fundamental transformations:
• Tokenization: Segmenting text into smaller units such as words or sentences (word
tokenization and sentence tokenization). This step breaks down the text into manageable
chunks.
• Stop Word Removal: Eliminating common and less meaningful words (stop words) like “the,”
“is,” etc., which don’t contribute significantly to the meaning of the text.
• Stemming/Lemmatization: Reducing words to their root forms — stemming removes
prefixes/suffixes, while lemmatization maps words to their base or dictionary form, aiding in
standardization.
• Lowercasing: Converting all text to lowercase to ensure uniformity in text analysis, as case
sensitivity can affect certain NLP tasks.
• Language Detection: Identifying the language of the text, is especPally useful when dealing
with multilingual content.
U20CSST01 - NATURAL LANGUAGE PROCESSING

(iii) Advanced Preprocessing:

Intricate linguistic analysis to delve deeper into the structural and semantic aspects of the text:
• Part-of-Speech (POS) Tagging: Assigning grammatical categories (like nouns, verbs,
adjectives) to words in the text, providing insights into the syntactic structure.
• Parsing: Analyzing the grammatical structure of sentences to identify relationships between
words, and determining the syntactic roles and dependencies.
• Coreference Resolution: Resolving references within the text, linking pronouns or noun phrases
to their respective entities for coherent understanding and analysis.

3. Feature Engineering :
• Transforming raw text data into numerical features that machine learning models can comprehend and
utilize effectively.
• The goal is to represent text in a format that captures semantic meaning, contextual information, and
relationships between words.
It can be done through various techniques:
(i) Bag of Words (BoW)
• Represents text as a collection of unique words disregarding grammar or word order.
• Creates a matrix where rows represent documents and columns represent unique words, with
values indicating word occurrence frequencies.
• Simple yet effective, but loses sequence information and context.
(ii) Term Frequency-Inverse Document Frequency (TF-IDF)
• Measures the importance of words in a document relative to a corpus.
• Considers both the frequency of a term in a document and its rarity across the corpus.
• Assigns higher weights to rare terms that are more discriminative.
(iii) Word Embeddings (Word2Vec, GloVe, FastText)
• Techniques that map words or phrases to dense vector representations in a continuous vector
space.
• Capture semantic relationships between words by placing similar words closer in the vector
space.
• Retain semantic meaning and context, useful for capturing word analogies and semantic
similarities.
(iv) N-Gram Models
• Captures sequences of adjacent words (bigrams, trigrams, etc.) as features.
• Preserves some sequence information, aiding in capturing context in language.
(v) Dependency Parsing
• Represents the grammatical structure of sentences as features.
• Captures relationships between words through syntactic dependencies.

4. Modelling :
The heart of the pipeline - models are applied and evaluated using different approaches:

(i) Heuristic Approaches

Heuristic models rely on predefined rules or strategies based on expert knowledge to make decisions.
Application: Commonly used in simple text-based tasks where rule-based systems can effectively handle
specific patterns or tasks, like keyword matching for sentiment analysis or rule-based chatbots.
(ii) Machine Learning (ML) Approaches
ML models learn patterns and relationships from data to make predictions or classifications.
Applications:
• Support Vector Machines (SVM): Effective for text classification tasks by finding the best
separation between classes in a high-dimensional space.
• Random Forests: Suitable for tasks like sentiment analysis or text categorization, leveraging
ensemble learning for improved accuracy.
(iii) Deep Learning (DL) Approaches
U20CSST01 - NATURAL LANGUAGE PROCESSING

DL models use neural networks with multiple layers to learn complex patterns and representations from raw
data.

Applications:
• Recurrent Neural Networks (RNNs): Effective for sequence-based tasks like language
modelling, sentiment analysis, or machine translation.
• Transformers: Particularly potent for attention-based mechanisms, excelling in tasks like
language translation, text generation, and summarization due to their ability to capture long-
range dependencies efficiently.
(iv) Cloud APIs
Cloud-based APIs offer pre-built, scalable models accessible via APIs, saving time and resources.
Applications:
• Google Cloud Natural Language API: Offers sentiment analysis, entity recognition, and
language detection.
• Microsoft Azure Text Analytics API: Provides sentiment analysis, key phrase extraction, and
named entity recognition.

5. Evaluation:
Encompassing intrinsic and extrinsic assessments to comprehensively gauge model performance from both
technical and practical standpoints.
(i) Intrinsic Evaluation
Intrinsic evaluation focuses on assessing the technical aspects and capabilities of the model in isolation, without
considering its real-world application.
Examples of Intrinsic Metrics:
• Accuracy: Measures the ratio of correctly predicted instances to the total instances in the
dataset.
• Precision, Recall, F1-score: Assess the model’s performance in binary or multi-class
classification tasks.
• Perplexity: Evaluates the language model’s predictive capability in language generation tasks.
• BLEU Score: Measures the quality of machine-translated text against a reference translation.
(ii) Extrinsic Evaluation
Extrinsic evaluation measures the model’s performance in real-world applications or business contexts,
considering its impact and utility in practical scenarios.
Examples of Extrinsic Evaluation Metrics:
• Business Metrics: Metrics aligned with specific business goals or outcomes, such as customer
satisfaction scores, revenue impact, or user engagement rates.
• Task-Specific Metrics: Metrics directly relevant to the NLP task at hand, like sentiment
analysis accuracy for customer feedback or document classification precision for information
retrieval systems.
• User-Centric Evaluation: Soliciting user feedback, surveys, or usability testing to assess user
satisfaction and experience with the NLP application.

6. Deployment :
NLP pipeline marks the transition of the developed model from the development environment to a production
environment, followed by continuous monitoring and adaptation to ensure sustained performance and relevance.
(i) Deployment
• Rolling out the Model: Moving the trained NLP model from the development environment to
a production environment where it can be utilized in real-world applications.
• Infrastructure Setup: Configuring the necessary infrastructure, integrating the model into the
existing systems, and ensuring scalability and reliability.
• Testing and Validation: Thoroughly testing the deployed model to ensure it functions as
expected and delivers accurate results in the production environment.
(ii) Monitoring
• Continuous Performance Oversight: Constantly monitoring the model’s performance,
including its accuracy, efficiency, and response time in real-time or at regular intervals.
U20CSST01 - NATURAL LANGUAGE PROCESSING

• Alert Systems: Implementing alert systems or triggers to notify about deviations or anomalies
in the model’s behaviour, ensuring timely interventions.

(iii) Update
• Adaptation to Dynamic Data: Adapting the model to changing data patterns or evolving
requirements by periodically updating and retraining the model.
• Improvement Iterations: Incorporating feedback, identifying areas for improvement, and
fine-tuning the model to enhance its performance or address changing user needs.
• Version Control: Maintaining version control to track model iterations and changes, ensuring
transparency and reproducibility.

MORPHOLOGY AND WORD STRUCTURE

Word Structure :

• Smallest linguistic units that can form a complete utterance by themselves.

• Minimal parts of words with meaning - morphemes.
• Internal organization of words, including their morphological, syntactic, and semantic properties.

Word structure Components:

i) Tokens :
• Words in English are delimited only by whitespace and punctuation (the marks, such as full stop,
comma, and brackets)
• Example: Will you read the newspaper? Will you read it? I won’t read it.
twowords here: words newspaper and won’t.
newspaper- derivational structure- distinguished from the isolated news and paper.
The structure of won’t could be parsed as – will not .

ii) Lexemes :
• One linguistic form- in the given context, but also the concept behind the form and the set of alternative
forms that can express it.
• Set of alternative forms- lexemes or lexical items.
• Divided by their behaviour into the lexical categories- verbs, nouns, adjectives, conjunctions or other
parts of speech.
• Citation form of a lexeme - lemma.
• Example: Did you see him? I didn’t see him. I didn’t see anyone.
tokenization of didn’t and investigation of the internal structure of anyone.

iii) Morphemes:
• Associate the properties of word forms with their structural components. These components are usually
called segments or morphs.
• The morphs that by themselves represent some aspect of the meaning of a word are called morphemes
of some function.

Two types of morphemes:

i Stems: play, cat, friend
ii Affixes: -ed, -s, un-, -ly

Two main types of affixes:

i Prefixes - precede the stem: un
ii Suffixes- follow the stem: -ed, -s, un-, -ly
U20CSST01 - NATURAL LANGUAGE PROCESSING

Stemming = find the stem by stripping off affixes

For example:
• The word “unhappiness” can be divided into three morphemes:
• “un-”: a prefix meaning “not”
• “happy”: the Stem
• “-ness”: a suffix that turns an adjective into a noun.

MORPHOLOGY :
• Morphology is the study of morphemes – the smallest units of meaning in a language.
• Morphemes can be either free (stand-alone words) or bound (prefixes, suffixes, and infixes).
• The combination of morphemes forms complex words, conveying nuances of tense, plurality, and other
grammatical properties.

Types of Morphology:

i) Inflectional Morphology :
• Inflected forms are constructed from base forms and inflectional affixes.
• Inflection relates different forms of the same word.

Lemma Singular Plural

Cat Cat Cats
Dog Dog Dogs
knife knife Knives

ii) Derivational Morphology:

• Words are constructed from roots (or stems) and derivational affixes:
inter+national = international
international+ize = internationalize
internationalize+ation = internationalization

Importance of Morphological Analysis

Morphological analysis is a critical step in NLP for several reasons:
1. Understanding Word Formation: It helps in identifying the basic building blocks of words,
which is crucial for language comprehension.
2. Improving Text Analysis: By breaking down words into their roots and affixes, it enhances the
accuracy of text analysis tasks like sentiment analysis and topic modeling.
3. Enhancing Language Models: Morphological analysis provides detailed insights into word
formation, improving the performance of language models used in tasks like speech recognition
and text generation.
4. Facilitating Multilingual Processing: It aids in handling the morphological diversity of
different languages, making NLP systems more robust and versatile.

Key Techniques used in Morphological Analysis :

• Morphological analysis involves breaking down words into their constituent morphemes (the smallest
units of meaning) and understanding their structure and formation.
• Various techniques can be employed to perform morphological analysis, each with its own strengths
and applications.

1. Stemming
Stemming reduces words to their base or root form, usually by removing suffixes. The resulting stems are
not necessarily valid words but are useful for text normalization.
Common ways to implement stemming in python:
U20CSST01 - NATURAL LANGUAGE PROCESSING

• Porter Stemmer: One of the most popular stemming algorithms, known for its simplicity and
efficiency.
• Snowball Stemmer: An improvement over the Porter Stemmer, supporting multiple languages.
• Lancaster Stemmer: A more aggressive stemming algorithm, often resulting in shorter stems.

2. Lemmatization
Lemmatization reduces words to their base or dictionary form (lemma). It considers the context and part of
speech, producing valid words. To implement lemmatization in python, WordNet Lemmatizer is used,
which leverages the WordNet lexical database to find the base form of words.

3. Morphological Parsing
Morphological parsing involves analyzing the structure of words to identify their morphemes (roots,
prefixes, suffixes). It requires knowledge of morphological rules and patterns. Finite-State Transducers
(FSTs) is uses as a tool for morphological parsing.
Finite-State Transducers (FSTs)
FSTs are computational models used to represent and analyze the morphological structure of words. They
consist of states and transitions, capturing the rules of word formation.
Applications:
• Morphological Analysis: Parsing words into their morphemes.
• Morphological Generation: Generating word forms from morphemes.

4. Neural Network Models

Neural network models, especially deep learning models, can be trained to perform morphological analysis
by learning patterns from large datasets.
Types of Neural Network
• Recurrent Neural Networks (RNNs): Useful for sequential data like text.
• Convolutional Neural Networks (CNNs): Can capture local patterns in the text.
• Transformers: Advanced models like BERT and GPT that understand context and semantics.

5. Rule-Based Methods
Rule-based methods rely on manually defined linguistic rules for morphological analysis. These rules can
handle specific language patterns and exceptions.
Applications:
• Affix Stripping: Removing known prefixes and suffixes to find the root form.
• Inflectional Analysis: Identifying grammatical variations like tense, number, and case.

6. Hidden Markov Models (HMMs)

Hidden Markov Models (HMMs) are probabilistic models that can be used to analyze sequences of data,
such as morphemes in words. HMMs consist of a set of hidden states, each representing a possible state of
the system, and observable outputs generated from these states. In the context of morphological analysis,
HMMs can be used to model the probabilistic relationships between sequences of morphemes, helping to
predict the most likely sequence of morphemes for a given word.
Components of Hidden Markov Models (HMMs):
• States: Represent different parts of words (e.g., prefixes, roots, suffixes).
• Observations: The actual characters or morphemes in the words.
• Transition Probabilities: Probabilities of moving from one state to another.
• Emission Probabilities: Probabilities of an observable output being generated from a state.
Applications:
• Morphological Segmentation: Breaking words into morphemes.
• Part-of-Speech Tagging: Assigning parts of speech to each word in a sentence.
• Sequence Prediction: Predicting the most likely sequence of morphemes for a given word.
U20CSST01 - NATURAL LANGUAGE PROCESSING

CHALLENGES IN NLP:

1. Ambiguity and Polysemy

One of the fundamental challenges in NLP is dealing with the ambiguity and polysemy inherent in
natural language. Words often have multiple meanings depending on context, making it challenging
for NLP systems to accurately interpret and understand text.
2. Data Sparsity and Quality
NLP models require large amounts of annotated data for training, but obtaining high-quality labeled
data can be challenging. Furthermore, data sparsity and inconsistency pose significant hurdles in
building robust NLP systems, leading to suboptimal performance in real-world applications.
[Link] and Understanding
Understanding context is crucial for NLP tasks such as sentiment analysis, summarization, and
language translation. However, capturing and representing context accurately remains a challenging
task, especially in complex linguistic environments.
[Link] and Language Variations
NLP systems must be able to handle multiple languages and dialects to cater to diverse user
populations. However, language variations, slang, and dialectical differences pose challenges in
developing universal NLP solutions that work effectively across different linguistic contexts.
5. Lack of Domain-Specific Data
Many Natural Language Processing applications require domain-specific knowledge and
terminology, but obtaining labeled data for specialized domains can be difficult. This lack of domain-
specific data limits the performance of NLP systems in specialized domains such as healthcare, legal,
and finance.
6. Semantic Understanding and Reasoning
NLP systems often struggle with semantic understanding and reasoning, especially in tasks that
require inferencing or commonsense reasoning. Capturing the subtle nuances of human language
and making accurate logical deductions remain significant challenges in NLP research.
7. Handling Noise and Uncertainty
Natural language data is often noisy and ambiguous, containing errors, misspellings, and
grammatical inconsistencies. NLP systems must be robust enough to handle such noise and
uncertainty while maintaining accuracy and reliability in their outputs.
8. Ethical and Bias Concerns
NLP models can inadvertently perpetuate biases present in the training data, leading to unfair or
discriminatory outcomes. Addressing ethical concerns and mitigating biases in NLP systems is
crucial to ensuring fairness and equity in their applications.
9. Scalability and Performance
Scalability is a critical challenge in NLP, particularly with the increasing complexity and size of
language models. Building scalable NLP solutions that can handle large datasets and complex
computations while maintaining high performance remains a daunting task.
10. Interdisciplinary Collaboration
NLP research requires collaboration across multiple disciplines, including linguistics, computer
science, cognitive psychology, and domain-specific expertise. Bridging the gap between these
disciplines and fostering interdisciplinary collaboration is essential for advancing the field of NLP
and addressing NLP challenges effectively.
U20CSST01 - NATURAL LANGUAGE PROCESSING

DOCUMENT STRUCTURE:

• Document Structuring is a subtask of Natural language generation, which involves deciding the order
and grouping (for example intoparagraphs) of sentences in a generated text .
• It is closely related to the Content determination NLG task.
1. Sentence Boundary Detection
2. Topic Boundary Detection

Sentence boundary detection:

• Sentence detection is an important task, which should be performed at the beginning of a text processin
pipeline.
• Sentence boundary detection (also called sentence segmentation) deals with automatically segmenting
sequence of word tokens into sentence units.
• Natural language processing tools often require their input to be divided into sentences; however, sentenc
boundary identification can be challenging due to the potential ambiguity of punctuation marks.
• In written text in English and some other languages, the beginning of a sentence is usually marked with
an uppercase letter, and the end of a sentence is explicitly marked with a period (.), a question mark (?),
an exclamation mark or another type of punctuation.
• Capitalized initial letters are used to distinguish proper nouns, periods are used in abbreviations and
numbers and other punctuation marks are usec inside proper names.
• A character-wise analysis of text allows for a distinction between period characters that are enclosed
betweer two alphanumeric characters, and period characters that are followed by at least one, non-
alphabetic character_ such as a further punctuation sign, a space, tab or new line.
• There are various challenges associated with SBD, for written as well as spoken text and code switching.

Topic Boundary Detection:

• Topic segmentation (sometimes called discourse or text segmentation) is the task of automatically
dividing a stream of text or speech into topically homogeneous blocks.
• That is, given a sequence of (written or spoken) words, the aim of topic segmentation is to find the
boundaries where topics.
• Topic segmentation is an important task for various language-understanding applications, such as
information extraction and retrieval and text summarization.
U20CSST01 - NATURAL LANGUAGE PROCESSING

• In information retrieval, if long documents can be segmented into shorter, topically coherent segments,
then only the segment that is about the user's query could be retrieved.
• For multiparty meetings, the task of topic segmentation is inspired by discourse analysis.
• For official and well-structured meetings, the topics are segmented according to the agenda items,
whereas for more casual conversational-style meetings, the boundaries are less clear.
• For conversational speech, the topic boundaries may not be absolute. Hence they are more complex.
• In text, topic boundaries are usually marked with distinct segmentation cues, such as headlines and
paragraph breaks. These cues are absent in speech. However, speech provides other cues, such as pause
duration and speaker changes.
• Topic segmentation is a nontrivial problem without a very high human agreement because of many
natural-language-related issues and hence requires a good definition of topic categories and their
granularities

Techniques for Sentence Boundary Detection

Several techniques and approaches are used for Sentence Boundary Detection:
1. Rule-Based Methods: These methods rely on a set of rules and heuristics to determine sentence
boundaries.
Rules may consider punctuation marks, abbreviations, and contextual information.
2. Machine Learning: Machine learning techniques, such as sequence labeling using Conditional Random
Fields (CRF) or Recurrent Neural Networks (RNNs), can be trained to predict sentence boundaries based on
labeled training data.
3. Language-Specific Models: Some languages may require language-specific models and resources (e.g.,
dictionaries, grammatical rules) to perform accurate Sentence Boundary Detection.
4. Pre-trained Models: Pre-trained language models, like BERT or GPT, can be fine-tuned for Sentence
Boundary Detection tasks, leveraging their contextual understanding of text.

Techniques for Topic Boundary Detection

Several techniques and approaches are used for Topic Boundary Detection.
1. Rule-Based Methods: Rule-based methods use predefined rules and heuristics to identify topic
boundaries.
2. Machine Learning: Machine learning techniques, such as supervised classification or sequence labeling,
can be trained to predict topic boundaries based on labeled training data.
3. Topic Models: Topic modelling techniques like Negative Matrix Factorization (NMF), can be used to
identify topic transitions based on changes in topic distributions within a document

Methods for Sentence segmentation and Topic segmentation

• Sentence segmentation and topic segmentation have been considered as a boundary classification
problems.
• Given a boundary candidate( between two word tokens for sentence segmentation and between two
sentences for topic segmentation), the goal is to predict whether or not the candidate is an actual
boundary (sentence or topic boundary).
• Formally, let x & X be the vector of features (the observation) associated with a candidate and y &
Y be the label predicted for that candidate.
• The label y can be b for boundary and b' for non-boundary.
• Classification problem: Given a set of training examples (x, y), find a function that will assign the
most accurate possible label y of unseen examples x which is unseen.
• Alternatively to the binary classification problem, it is possible to model boundary types using finer-
grained categories.
• For segmentation the text be framed as a three-class problem: sentence boundary ba, without an
abbreviation ba' and abbreviation not as a boundary ba
• Similarly for spoken language, a two way classification can be made between non-boundaries
statements b, and question boundaries ba
U20CSST01 - NATURAL LANGUAGE PROCESSING

• For sentence or topic segmentation, the problem is defined as finding the most probable sentence or
topic boundaries.
• The natural unit of sentence segmentation is words and of topic segmentation is sentence, as we can
assume that topics typically do not change in the middle of a sentences.
• The words or sentences are then grouped into categories belonging to one sentences or topic as
boundaries and non-boundaries.
• The classification can be done at each potential boundary i (local modelling); then, the aim is to
estimate the most probable boundary type ŷ¡ for each candidate X₁

Machine learning algorithm: Generative versus Discriminative.

• Generative sequence models estimate the joint distribution of the observations P(X,Y) (words,
punctuation) and the labels (sentence boundary, topic boundary).
• Discriminative sequence models, however, focus on features that categorize the differences between
the labelling of the examples
Generative and Discriminative Sequence Models
• NLP encompasses various tasks, such as text classification, sentiment analysis, machine translation,
and language generation.
• Several models have been developed to address these tasks, the two primary models in NLP being
generative and discriminative models.
• Generative and discriminative sequence models are two categories of statistical models used in natural
language processing (NLP) and in various other fields.
• They have different approaches to modeling and solving sequence-related tasks.
• Discriminative models map an input to output and are trained on labeled data following the supervised
learning paradigm.
• These models learn to identify patterns and correlations between the input and output, allowing them
to make highly accurate predictions.
• The discriminative model indirectly learns certain features of the dataset that makes the task easier.
• Discriminative models work by learning the decision boundary that separates the input data into
different classes.
• Discriminative modelling learns to model the conditional probability of class label y given set of
features x as P(Y|X).
• Generative models are probabilistic models that can generate new text based on the input given to
them.

• These models are trained on large amounts of unlabeled data and can be fine-tuned to perform various
NLP tasks.
• They work by learning the probability distribution of words in a language and use this knowledge to
generate new text that matches the input's context.
• Generative modelling defines how a dataset is generated.
U20CSST01 - NATURAL LANGUAGE PROCESSING

• It tries to understand the distribution of data points, providing a model of how the data is actually
generated in terms of a probabilistic model.

Complexity of Approaches
• Discriminative approach
In terms of complexity, training of discriminative approaches is more complex than training of
generative ones.
Since, they require multiple passes over the training data to adjust for their feature weights.
• Generative models
Generative models such as HELMs can handle multiple orders of magnitude larger training sets.
But, they do not cope well with unseen events.
• Discriminative classifiers
They allow for a wider variety of features and perform better on smaller training sets.
Predicting with discriminative classifiers is also slower, even though the models are relatively simple
(linear or log-linear).
• Sequence approaches
Compared to local approaches, sequence approaches bring the additional complexity of decoding:
To find the best sequence of decisions requires evaluating all possible sequences of decisions.

Performance of the Approaches

The performance of the sentence segmentation approaches can be analyzed in:
• Sentence segmentation in text
• Sentence segmentation in speech
A. Sentence segmentation in text
For sentence segmentation in text, researchers have reported error rate results on a subset of the Wall
Street Journal Corpus of about 27,000 sentences.
For instance, Mikheev reports that have rule-based system performs at an error rate of 1.41%.
Even though the error rates presented seem low, sentence segmentation is one of the first processing
steps for any NLP task, and each error impacts subsequent steps, especially if the resulting sentences
are presented to the user.
B. Sentence segmentation in speech
For sentence segmentation in speech, performance is usually evaluated using:
1. The error rate (ratio of number of errors to the numbers of examples).
2. F1-measure (the harmonic mean of recall and precision).
Recall is defined as the ratio of the number of correctly returned sentence boundaries to the number of
sentence boundaries in the reference annotations.