BCSE409L Natural
Language Processing
Dr. Biji C. L
Associate Professor
Department of Analytics, SCOPE
Text Segmentation
Process of Converting well defined text corpus into its
component words and sentence.
• Word Segmentation
• Tokens - The point when one word ends another word begins
• Normalization - Merging the different forms of a token into a
canonical form. Ex: Mr. , Mr, Mister, mister – Normalized to a single
form.
• Sentence Segmentation
• Finding the sentence boundary between the words in a different
sentences.
Sentence segmentation
• Sentence boundary detection
• Sentence boundary disambiguation
• Sentence boundary recognition
Challenges in Text pre-processing
Text pre-processing in NLP presents unique challenges
depending on the writing system.
1. Logographic Systems (e.g., Chinese, Japanese):
Challenge:
• Each character is a word or morpheme, making tokenization (splitting
text into meaningful units) complex. There's no direct one-to-one
mapping between characters and sounds.
Example:
• Chinese characters, which can represent words or parts of words,
require sophisticated techniques to identify word boundaries.
Logographic and Syllabic
2. Syllabic Systems (e.g., Japanese kana,
Cherokee):
• Challenge:
Symbols represent syllables, which are units of
pronunciation. Tokenization needs to consider these syllabic units,
which may not directly align with word boundaries.
Example:
• In Japanese, kana characters can be combined to form words, but the
individual kana characters are still syllabic units.
3. Alphabetic Systems (e.g., English,
Spanish):
• Challenge:
• While generally straightforward, alphabetic systems still present
challenges in handling contractions, hyphenated words, and non-
standard spellings.
• Example:
• "can't" needs to be processed as "cannot" or "can not", and words
like "multi-tasking" need to be handled appropriately.
Text preprocessing
1. Tokenization
-Tokenization is the process that splits an input sequence into so-called
tokens.
- Useful units for the semantic processing
- Can be a word, paragraph, sentence, etc..
-Tokenization also referred as the text segmentation or analysis
Breaking a stream of input text into words and sentences for
subsequent processing
2. Token Normalization
▪ Converting all text to the same case (upper or lower), removing
punctuation, converting numbers to their word equivalents, and so
on.
▪ Normalization puts all words on equal footing, and allows processing
to proceed uniformly.
1. Stemming
2. Lemmatization
2.1 Stemming
• Process of removing and replacing suffixes to get the root form of the
word. (Prefix, suffix, Infix, Circumfix)
Running → Run
Running ,Runs, Runned, Runly → Run
• if the word ends in 'ed', remove the 'ed'
• if the word ends in 'ing', remove the 'ing'
• if the word ends in 'ly', remove the 'ly'
2.2 Lemmatization
• Lemmatization related to stemming differing in that is able to capture
a canonical form based on a word lemma.
• A more complex approach to the problem of determining a stem of a
word is lemmatization
Better → Good
Returns to based or dictionary form of a word which is known as lemma.
Stemming and Lemmatization
The goal of both stemming and lemmatization is to reduce inflectional forms and
sometimes derivationally related forms of a word to a common base form. For
instance:
am, are, is → be
car, cars, car's, cars' → car
Stemming vs. Lemmatization
Feature Stemming Lemmatization
Reduce to dictionary root
Goal Remove suffixes/prefixes
(lemma)
Often crude (e.g., "running" → Clean and meaningful (e.g.,
Output
"run" or "runn") "running" → "run")
Algorithm Type Rule-based Dictionary + context-aware
Accuracy Lower (fast but imprecise) Higher (slower but smarter)
Common Tools Porter, Snowball WordNet Lemmatizer
Character Encoding
Character Encoding
• Encoding - Process of Putting a sequence of characters (Letters,
Numbers, Punctuation, and Symbols)
• Character - Character is a smallest unit of writing system which is
capable of conveying information.
• Character Encoding – It is the way that letter, digits and other binary
symbols are expressed as binary values that a computer can
understand.
ASCII and UNICODE
• Character Encoding - ASCII was the first character encoding standard
(also called character set). ASCII defined 128 different
alphanumeric characters that could be used on the internet: numbers
(0-9), English letters (A-Z), and some special characters like ! ...
• UTF-8 (Unicode) covers almost all of the characters and symbols in
the world.
ASCII -Character Set
List of characters recognised by the computer hardware and the software.
Each character is represented by a number.
ASCII Character set:
• American Standard Code for Information Interchange (English)
• Numbers in 0 to 127 is to represent all English characters as well as the
special character
• The ASCII table has 128 characters with values from 0 through 127.
• Thus, 7 bits are sufficient to represent a character in ASCII; however, most
computers typically reserve 1 byte, (8 bits), for an ASCII character.
ASCII
• It is an acronym for the American Standard Code for Information
Interchange.
• It is a standard seven-bit code that was first proposed by the
American National Standards Institute or ANSI in 1963, and finalized
in 1968 as ANSI Standard X3.4.
• The purpose of ASCII was to provide a standard to code various
symbols ( visible and invisible symbols)
Text Segmentation
• Text segmentation is the process of dividing written text into
meaningful units, such as words, sentences, or topics.
• The term applies both to mental processes used by humans when
reading text, and to artificial processes implemented in computers,
which are the subject of natural language processing.
• In Natural Language Processing (NLP), text segmentation is a
technique that divides a document into smaller, meaningful units.
These units are often called "segments".
Word segmentation
• Word segmentation, also known as decompounding, is the process of splitting
a word into its constituent parts. It can also be defined as the task of adding
spaces between words.
• Word segmentation:
• is the process of splitting a word into its constituent parts.
• is the problem of dividing a string of written language into its component words.
• Word Segmentation is an NLP task for languages that doesn’t have white
space between its words like, for example, Chinese, Japanese, and Thai. Take
this Chinese sentence as example:
• 今天我很高兴 - I'm very happy today.
• The words 今天 (today), 我 (I), 很 (very), and 高兴 (happy) don’t have any
separation between them, and to process this sentence in a programmatically
way can be hard. For these kinds of languages, to perform NLP tasks we need
to be able to tokenize the sentences into its parts.
Word Segmentation Challenges
Statistical Learning Approach
Statistical Learning Approach
Word Segmentation Example
Word Segmentation Example
Perform Word Segmentation “isit”
• Generate all possible compositions
List the composition
• Two of the compositions consist
of genuine English words and
would be valid segmentation
suggestions.
• probability(is it)> probability(i sit)
• Result – is it
Algorithm
Sentence Segmentation and Decision Trees
Sentence segmentation
• Sentence segmentation is the process of breaking down a sentence into its
individual words. It can also refer to the process of dividing a string of written
language into its component sentences.
• Sentence segmentation is a fundamental process in natural language processing
(NLP) that involves dividing a text into individual sentences. It's the process of
determining where sentences start and end, and involves identifying sentence
boundaries between words.
• The process of deciding from where the sentences actually start or end in NLP or
we can simply say that here we are dividing a paragraph based on sentences. This
process is known as Sentence Segmentation.
Example:
we have the following paragraph: "I Love Coding. Geeks for Geeks helped me in this regard very much. I
Love Geeks for Geeks."
here there are 3 sentences.
1. I Love Coding.
2. Geeks for Geeks helped me in this regard very much.
3. I Love Geeks for Geeks
Sentence segmentation
• Sentence segmentation is the analysis of texts based on sentences. In NLP
analysis, we either analyze the text data based on meaningful words which is
tokens or we analyze them based on sentences.
To split the data to its main components i.e tokens we can do that through spaCy library as follows: