NLP
Text Preprocessing
anumalhan1@[Link]
2ODC0LSA47
Part II
Week- 1: Natural Language Processing
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
● Accented and Special Characters
● Lowercasing
anumalhan1@[Link]
2ODC0LSA47 Agenda
● Removing stop words
● Tokenization
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Common
Text Preprocessing
anumalhan1@[Link]
2ODC0LSA47
Methods
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Common Text Preprocessing Methods
● Some of the common text preprocessing steps used in every NLP project include stemming,
lemmatization, spell-checking, NER (Named Entity Recognition), word embedding, and data cleansing
such as removal of HTML tags.
● In this section we will be covering the following preprocessing steps:
1 Treating accented and special characters
anumalhan1@[Link]
2ODC0LSA47
2 Lowercasing
3 Removing stop words
4 Tokenization
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Treating
accented and special
anumalhan1@[Link]
2ODC0LSA47
characters
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Treating accented and special characters
● Let’s consider the following example of text data with accented and special characters:
Accented
characters
Using Nātùrăl Làñgûägè Processing, we make use of the text data available
anumalhan1@[Link]
2ODC0LSA47
across the internet to generate insights for the business. To make this huge
amount of #$ ^_^ $# data usable for a Natural Language Processing task, we use
text preprocessing.
Special
characters
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Treating accented and special characters
● In English text preprocessing, we usually remove accented characters. If we didn’t do so, our model will
consider words with and without an accent symbol as separate words, even though they may be the
same word. This would of course be different for other languages that rely more heavily on accented
characters.
Word with accented
characters
UsingNātùrăl Làñgûägè Processing, we make use of the text data available
anumalhan1@[Link]
2ODC0LSA47
across the internet to generate insights for the business. To make this huge
amount of #$ ^_^ $# data usable for a Natural Language Processing task, we
use text-preprocessing.
The same word without accented
characters
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Treating accented and special characters
● Another important technique in text preprocessing is the removal of special characters.
● A special character is a character that is not an alphabetic or numeric character.
For example, ! , #, $, &, @, etc are considered special characters.
Using Natural Language Processing, we make use of the text data available
anumalhan1@[Link]
2ODC0LSA47
across the internet to generate insights for the business. To make this huge
amount of #$ ^_^ $# data usable for a Natural Language Processing task, we use
text-preprocessing.
Special
characters
● These special characters add to the noise in unstructured text, and add
no value to the meaning of the text. Removing them is usually preferable.
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
anumalhan1@[Link]
2ODC0LSA47
Lowercasing
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Lowercasing
● Lowercasing converts all the words into lowercase to ensure that repeated occurrences of the same
word in different cases are still treated as the same word.
● The word "text" is written in two different cases in the example below, giving the model redundant
information. This can be avoided by applying lowercasing.
First letter is
anumalhan1@[Link] capitalized
2ODC0LSA47
Using Natural Language Processing, we make use of the Text data available
across the internet to generate insights for the business. To make this huge
amount of data usable for a Natural Language Processing task,
we use text preprocessing.
No capitalization
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Removing stop words
anumalhan1@[Link]
2ODC0LSA47
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Removing stop words
● We have learned in the previous section that, stop words are a set of commonly used intervening
words in a language. Some examples of stop words in English are “a”, “the”, “is”, “are”, etc.
● Let’s identify the stop words in the following example:
Usingnatural language processing, we make use of the text data available
anumalhan1@[Link]
2ODC0LSA47
across the internet to generate insights for the business. to make this huge
amount of data usable for a natural language processing task,
we use text preprocessing.
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Removing stop words
● The idea behind removing stop words is that by eliminating low-information parts of the text, we can
concentrate on the key words.
● Note however that the incorrect removal of stop words can even alter the meaning of our text.
usingnatural language processing, we make use of the text data available
anumalhan1@[Link]
2ODC0LSA47
across the internet to generate insights for the business. to make this huge
amount of data usable for a natural language processing task,
we use text preprocessing.
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Removing stop words
● The idea behind removing stop words is that by eliminating low-information parts of the text, we can
concentrate on the key words.
● Note however that the incorrect removal of stop words can even alter the meaning of our text.
For example, consider the below text -
anumalhan1@[Link] ‘The movie was not bad’
2ODC0LSA47
● If we remove the stop word i.e ‘not’, then the sentence becomes ‘The movie was bad’, which
completely changes the meaning of our original sentence.
● One way to avoid this issue is to manually choose the list of stop words we require according to
the task at hand.
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
anumalhan1@[Link]
2ODC0LSA47
Tokenization
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Tokenization
● Tokenization is simply the process of breaking a stream of raw text into small chunks of words or
sentences known as tokens.
● We can specify our own conditions to divide the input text into the relevant tokens depending on the
task at hand.
Two of the main tokenization techniques are:
anumalhan1@[Link]
2ODC0LSA47
1 Word Tokenization
2 Sentence Tokenization
● The tokenization process is important because the meaning of text can
be interpreted through analysis of the words present in the text.
We will discuss both of the above types of tokenization with examples.
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Word Tokenization
● This is the more popular tokenization method in NLP. It divides a piece of text into individual words
based on a specific delimiter.
NLP is very
NLP is very popular.
anumalhan1@[Link] popular It’s a
Word Tokenization
It’s a subfield of AI.
2ODC0LSA47
subfield of AI
● One disadvantage of word tokenization is that it cannot handle Out-of-
Vocabulary words.
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Sentence Tokenization
● This is analogous to word tokenization, but where we break down a piece of text into separate
sentences based on different delimiters.
NLP is very popular
NLP is very popular.
anumalhan1@[Link] Sentence Tokenization
It’s a subfield of AI.
2ODC0LSA47
It’s a subfield of AI
● In the above example, ' . ' is used to indicate the termination of a sentence
and divide it into two distinct parts.
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
anumalhan1@[Link]
2ODC0LSA47
Summary
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Summary
● Text Preprocessing techniques, as we have seen, vary from task to task. Some of the most popular text
preprocessing techniques used in NLP are:
○ Treating accented and special characters
○ Lowercasing
○ Removing stop words
○ Tokenization
anumalhan1@[Link]
● These techniques
2ODC0LSA47 all have their own advantages and disadvantages, so the application of these
techniques can be case-specific and may depend on the task at hand.
● In the following section, we will learn about more significant text preprocessing strategies that deal with
word inflection in text data.
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action.
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.
Happy Learning !
anumalhan1@[Link]
2ODC0LSA47
This file is meant for personal use by anumalhan1@[Link] only.
Sharing or publishing the contents in part or full is liable for legal action. 21
Proprietary content. © Great Learning. All Rights Reserved. Unauthorized use or distribution prohibited.