Why Text Preprocessing is Needed?
Text preprocessing is an important and one the most essential step before building any model in
Natural Language Processing. A raw text corpus, collected from one or many sources, may be full
of inconsistencies and ambiguity that requires preprocessing for cleaning it up.
Even after cleaning, more text preprocessing is required to reshape the data in a manner that it can
be fed directly to the model. If text preprocessing is not done properly the data will be as good as
garbage and the NLP model produced will be as bad as garbage only.
Whenever we have textual data, we need to apply several pre-processing steps to the data to
transform words into numerical features that work with machine learning algorithms. The pre-
processing steps for a problem depend mainly on the domain and the problem itself, hence, we
don’t need to apply all steps to every problem.
In this article, we are going to see text preprocessing in Python. We will be using the NLTK
(Natural Language Toolkit) library here.
Common Text Preprocessing Steps
We have listed below some of the most commons text preprocessing steps. So these are the different
types of text preprocessing steps which we can do on text data. But we need not do all of these all
the times. We need to carefully choose the preprocessing steps based on our use case since that also
play an important role.
For example, in sentiment analysis use case, we need not remove the emojis or emoticons as it will
convey some important information about the sentiment. Similarly we need to decide based on our
use cases.
1. Lowercasing of Text
2. Uppercasing of Text
3. Remove Numbers
4. Convert the Numbers into Words
5. Removing Extra Whitespaces
6. Sentence Segmentation
7. Word Tokenization
8. Spelling Correction
9. Removal of Stopwords
[Link] of Punctuations
[Link] of Frequent Words
[Link] of Rare words
[Link]
[Link]
[Link] of Speech Tagging
[Link]
17. Named Entity Recognition
[Link] of URLs
[Link] of HTML Tags
[Link] of emojis
[Link] of emoticons
[Link] of emoticons to words
[Link] of emojis to words
[Link] words conversion
[Link] correction
Example of Text Preprocessing using NLTK Python
Let us now see an example of how to carry out text preprocessing using NLTK library.
1. Lowercasing of Text
The lowercasing is an important text preprocessing step in which we convert the text into the same
casing, preferably all in lowercase so that the words America, America, and AMERICA can be
treated in the same way as “america”.
It is helpful in text featurization techniques like term frequency, TFIDF since it prevents duplication
of same words having different casing.
Lower casing is a common text preprocessing technique. The idea is to convert the input text into
same casing format so that 'text', 'Text' and 'TEXT' are treated the same way.
This is more helpful for text featurization techniques like frequency, tfidf as it helps to combine the
same words together thereby reducing the duplication and get correct counts / tfidf values.
This may not be helpful when we do tasks like Part of Speech tagging (where proper casing gives
some information about Nouns and so on) and Sentiment Analysis (where upper casing refers to
anger and so on)
In our example, we have used the lower() function of python to convert all the text in the dataframe
to the lower case.