[Tips] List of preprocessing techniques in NLP

Preprocessing of strings is an important task in NLP and greatly reduces bias from the strings.
However all of them may or maynot be used dependeing on the use case. In this notebook I'll breif
your about some basic and but most commonly used techniques of all.

 Removing HTML Tags

 Removing emojis
 Changing you're -> you are
 Removing contractions
 Removing Punctuations
 Remove Abbreviation
 Change Plural forms
 Remove Patterns
 Parts of speech
In [1]:
!pip install bs4
!pip install contractions

import re
from bs4 import BeautifulSoup
import contractions
import nltk
from [Link] import stopwords
from [Link] import WordNetLemmatizer

In [2]:
a = "I luv my &lt ;3 iphone & you're awsm apple. Display Is Awesome, sooo happppppy
🙂 \n<a herf = [Link]

Remove html tags

While scraping data from web, it is common to encounter tags like < html > , < br > , < p > , < href >
etc.
These does not add any information to out data and it is advisable to remove them.

In [3]:
soup = BeautifulSoup(a)
a = soup.get_text()
a

Out[3]:
"I luv my &lt ;3 iphone & you're awsm apple. Display Is Awesome, sooo happppppy 🙂 \n"
Remove emojis and errors
In [4]:
a = [Link]('ascii','ignore')
a = [Link]()
a

Out[4]:
"I luv my &lt ;3 iphone & you're awsm apple. Display Is Awesome, sooo happppppy \n"
Changing you're -> you are
These are known as contactions and shall be removed.
This helps in making consistency in the vocabulary as depending on the NLP model, it might
assume you're as completely different from you are, whoever we want to retain meaning of words
to generate the output.

In [5]:
a = [Link](a)
a

Out[5]:
'I love my &lt ;3 iphone & you are awsm apple. Display Is Awesome, sooo happppppy \
n'
Remove stopwards
Stopwards are words like articles, verbs etc that are useful in building a sentence however do not
add much information to our data.

In [6]:
a = ' '.join([word for word in [Link]() if not word in set([Link]('english'))])
a

Out[6]:
'I love &lt ;3 iphone & awsm apple. Display Is Awesome, sooo happppppy'
Remove punctuations and numbers
In most cases punctuations are not required and act as an overhead in the string. Numbers may or
maynot be useful depending on the usecase.
To prevent numbers from removal use [Link]('[^a-zA-Z0-9]', ' ', a)

In [7]:
a = [Link]('[^a-zA-Z]', ' ', a)
a = ' '.join([Link]())
a

Out[7]:
'I love lt iphone awsm apple Display Is Awesome sooo happppppy'
Removing Abbreviation
Most data includes short forms of various words that are difficult to remove.
Some known abbreviations can be removed using this method, however, we may also use
autocorrect packages to correct the words.

In [8]:
lookup_dict = { 'dm':'direct message', "awsm" : "awesome", "luv" :"love" }
a = ' '.join( [lookup_dict[word] if word in lookup_dict.keys() else word for word in [Link]()])
a

Out[8]:
'I love lt iphone awesome apple Display Is Awesome sooo happppppy'
Words to singular forms
This step is again optional depending on the usecase. It helps in reducing vocabulary size as certain
words are just plural forms of others.
Most used methods are Porter Stemmer and Lemmatizer. More can be found here

In [9]:
lem = WordNetLemmatizer()
a = [Link](a)
a

Out[9]:
'I love lt iphone awesome apple Display Is Awesome sooo happppppy'
Removing patterns
Most patterns can easily be removed using Regex these may include words between brackets,
words starting with # etc. You may refer Docs to learn more.

In [10]:
# removing words staring with #
string = "Kaggle is an #awesome platform to learn"
pattern = "#[\w]*"
[Link](pattern,'',string)

Out[10]:
'Kaggle is an platform to learn'
Parts of speech tagging
Certain words within a string are of more importance than other. This words are using Nouns,
pronouns etc. Extracting specific words from the strings can help us reduce vocabulary size, helping
us train faster

In [11]:
from nltk import word_tokenize, pos_tag
text = "Kaggle is great platform to learn and explore."
tokens = word_tokenize(text)
pos_tag(tokens)

Out[11]:
[('Kaggle', 'NNP'),
('is', 'VBZ'),
('great', 'JJ'),
('platform', 'NN'),
('to', 'TO'),
('learn', 'VB'),
('and', 'CC'),
('explore', 'VB'),
('.', '.')]
In [12]:
[Link].upenn_tagset('NNP')
NNP: noun, proper, singular
Motown Venneboerger Czestochwa Ranzer Conchita Trumplane Christos
Oceanside Escobar Kreisler Sawyer Cougar Yvette Ervin ODI Darryl CTCA
Shannon A.K.C. Meltex Liverpool ...
In [13]:
import spacy
from spacy import displacy
from collections import Counter
!python3 -m spacy download en
import en_core_web_sm
nlp = en_core_web_sm.load()
You can now load the model via [Link]('en_core_web_sm')
✔ Linking successful
You can now load the model via [Link]('en')
In [14]:
doc = nlp("The World Health Organization declared the outbreak a Public Health Emergency
of international concern in January 2020 and a pandemic in March 2020. As of 28 January
2021, more than 100 million cases have been confirmed, with more than 2.17 million deaths
attributed to COVID-19.")
print([([Link], X.label_) for X in [Link]])
[('The World Health Organization', 'ORG'), ('January 2020', 'DATE'), ('March 2020',
'DATE'), ('28 January 2021', 'DATE'), ('more than 100 million', 'CARDINAL'), ('more than
2.17 million', 'CARDINAL')]
In [15]:
[Link](doc, jupyter=True, style='ent')
The World Health Organization ORG declared the outbreak a Public Health Emergency of

international concern in January 2020 DATE and a pandemic in March 2020 DATE . As of 28 January

2021 DATE , more than 100 million CARDINAL cases have been confirmed, with more than 2.17

million CARDINAL deaths attributed to COVID-19.