BCSE409L
Natural Language Processing
Dr. Biji C. L
Associate Professor
Department of Analytics, SCOPE
Text Processing
• Text processing is an essential step in NLP.
• The task of converting a raw text file into a well-defined
sequence of linguistically meaningful units.
Why is text preprocessing important?
• Data quality significantly influences the performance of a
machine-learning model. Inadequate or low-quality data can
lead to lower accuracy and effectiveness of the model.
• In general, text data derived from natural language is
unstructured and noisy.
• Text preprocessing is a critical step to transform messy,
unstructured text data into a form that can be effectively used
to train machine learning models, leading to better results and
insights.
Text Processing Types
• Document triage:
process of converting
a set of digital files
into well-defined text
documents.
• text segmentation:
process of converting
a well-defined text
corpus into its
component words,
and sentences
Text Processing Types
•Character Encoding Recognition: Ensures the text is correctly
interpreted (e.g., UTF-8 vs. ASCII).
It prevents issues like garbled characters from messing up
downstream tasks.
•Language Identification: Crucial for multilingual data;
it helps select language-specific tokenizers and models.
•Text Sectioning: Divides content into logical regions (title, metadata,
body) using layout or rule-based methods.
Text Segmentation
• Once the document’s routed and structured, segmentation gets into
the granular details.
Word Segmentation / Tokenization: Breaks down strings into words or
subwords—essential for modeling and feature extraction.
Text Normalization: Includes lowercasing, removing accents or Unicode
symbols, handling contractions—anything to standardize the input.
Sentence Segmentation: Splits the text into sentence units, helping
with parsing, entity recognition, and semantic tasks.
Text processing includes:
✓Tokenization
✓Stemming
✓Lemmatization
✓Stop-word removal
✓Part-of-speech tagging
Text Preprocessing Techniques
• Text preprocessing refers to a series of techniques used to clean, transform and
prepare raw textual data into a format that is suitable for NLP or ML tasks.
• The goal of text preprocessing is to enhance the quality and usability of the text
data for subsequent analysis or modeling.
1. Tokenization
• Tokenization is the process of breaking down text into smaller
units called tokens, which can be words, phrases, or even
characters.
• This step is essential for analyzing and processing text data.
Example:
•Input Sentence: ""I am waiting for the deployment of our chatbot soon."."
•Tokens: ["I", "am", "waiting", "for", "the", "deployment", "of", "our", "chatbot", "soon", "."]
2. Stemming
• Stemming reduces words to their base or root form by
removing suffixes.
• This helps in normalizing words to their core meaning, which is
useful for text analysis.
Example:
•Input Words: ["I", "am", "waiting", "for", "the", "deployment", "of", "our", "chatbot", "soon", "."]
•Stemmed Tokens: ["i", "am", "wait", "for", "the", "deploy", "of", "our", "chatbot", "soon", "."]
3. Lemmatization
• Lemmatization is similar to stemming but more accurate.
• It reduces words to their base or dictionary form, considering
the context and part of speech.
• This ensures that words are transformed into their meaningful
base form.
Lemmatized Tokens: ["I", "am", "waiting", "for", "the", "deployment", "of", "our", "chatbot", "soon", "."]
Note
[Link] word "waiting" is already in its base form as a present participle. The lemmatizer recognizes it as a
verb in its continuous form, so it doesn't change it to "wait".
2."deployment": The word "deployment" is a noun, and its base form is "deployment". The lemmatizer
doesn't change it to "deploy" because "deploy" is a verb, and lemmatization keeps the part of speech
intact.
4. Stop-word Removal
• Stop-word removal involves eliminating common words that
add little value to the analysis, such as "and", "the", "is". This
helps in focusing on the more meaningful words in the text.
Tokens after Stop-word Removal: ["waiting", "deployment", "chatbot", "soon", "."]
5. Part-of-Speech Tagging
• Part-of-speech (POS) tagging assigns grammatical tags to
each word in a sentence, such as nouns, verbs, adjectives, etc.
• This helps in understanding the syntactic structure and
meaning of the text.
POS Tags: [("I", "PRP"), ("am", "VBP"), ("waiting", "VBG"), ("for", "IN"), ("the", "DT"), ("deployment",
"NN"), ("of", "IN"), ("our", "PRP"), ("chatbot", "NN"), ("soon", "RB"), (".", ".")]
•Part-of-speech (POS) tagging assigns grammatical tags to each word in the sentence, such as
pronouns (PRP), verbs (VBP, VBG), nouns (NN), adjectives (JJ), etc. This helps in understanding the
syntactic structure and meaning of the text.
Corpus
• A corpus is a large collection of texts that are considered to be
representative of a language or language variety. The plural of corpus
is corpora.
• A corpus is a collection of naturally occurring language text, chosen to
characterize the state of variety language.
• In Natural Language Processing (NLP), a corpus is a collection of text
or audio data that can be used to train AI and machine learning
systems.
• Corpora can be made up of a wide range of documents, texts, or
voices in one or more specific languages. They can include
newspapers, novels, recipes, radio broadcasts, television shows,
movies, and tweets.
Structure in a Corpus (Brown Corpus Style)
Each sentence in the corpus is annotated with:
Example Meaning
Word POS Tag
In Context
Verb: to move
"run" VB swiftly ("He will
run tomorrow")
Noun: a streak or
"run" NN series ("A run of
good luck")
Adjective: not
"light" JJ heavy ("A light
meal")
Noun: illumination
"light" NN ("Turn on the
light")
Benefit of Corpus
What should be a corpus?
• A corpus should be machine readable text.
• It should be collected in a large amount.
• It should contain authentic text not an artificial one.
• Corpus should be the sample of the particular language.
• It should be purposefully collected.
• Corpus should have proper structure from the start to end.
What should not be a corpus?
Considerations in Corpus Creation
• Corpus Size
• Representativeness
• Balance & Sampling
• Data Capture & Copyright
• Markup and Annotation
• Multilingual and Multimodal Corpora
Text Processing Activity
• For the given corpus, perform text processing
Corpus
[Link] 1: We can use health chatbots for treating stress.
[Link] 2: We can use NLP to create chatbots and we will be making health chatbots now!
[Link] 3: Health Chatbots cannot replace human counsellors now.
Text Processing Activity