Content
• Data Preprocessing
• Text Data Preprocessing Techniques
• Uncapitalizing
• Removing Punctuation
• Removing Stopwords
• Standardizing text
• Correcting spelling
• Tokenizing
• Stemming and Lemmatization
• Text Data Exploring
• Building pre-processing model for text data
1. Data Preprocessing
Source: [Link]
Motivation
• Text data come from many
resources (and heterogeneous!)
• Web, social network, documents,
etc.
• Noise, redundancy
• ==> Need to be transformed into
understandable format
2. Text Data Preprocessing Techniques
2.1. Uncapitalizing
• Problem: “NLP” and “nlp” is different or not?
• All the text needs to be represented in the same format
• Solution: using lower() function in python
2.2. Removing Punctuation
• Punctuation has no
meaning!
• Reducing data dimension as
well as improving the
computational performance
• Solution: Regular Expression
+ replace() function in
python
2.3. Removing Stop words
• Stop words are commonly used in text
document but meaningless
• Removing stop words allows to reduce the
data dimension and to improve the model
performance
• Solution:
• Using the library NLTK
• Building a list of stop-words and removing them
from the input document
• Example: [Link]
2.4. Standardizing Text
• Transforming acronym and
abbreviation in the document text
• Solution:
• Building a dictionary for common
acronym and abbreviation in the text
2.5. Correcting Spelling
• Data comes from social network (e.g., user comments, blogs, tweets
etc.) may contain spelling errors
• Correcting spelling allows to remove redundant words
• Solution:
• Using library TextBlod
2.6. Tokenizing
• Motivation: split the input text into
“terms” ➔ numerical representation
• Input: document after some pre-
processing techniques
• Output: A list of terms
• Solution
• For English documents: NLTK, TextBlod,
Spacy
• For Vietnamese documents: VnCoreNLP,
underthesea, coccoc-tokenizer
VnCoreNLP: [Link]
Underthesea: [Link]
Coccoc-tokenizer: [Link]
2.7. Stemming
• Extract the base form of a
word by removing affixes
from them
• Solution:
• NLTK
• TextBlod
2.8. Lemmatization
• Identify the derived
forms of a word then
convert them to the base
form
• Input: a word
• Output: base form of this
word
• Solution
• NLTK
• TextBlod
2.9. Statistical Analysis of Text Document
• Input: a text document
• Output:
• Counting the number of words of this
document
• Counting the frequency of each word
• Counting words with specified
constraint on length
• Building word cloud
• Solution:
• NLTK
• TextBlod
3. Pre-processing Text Data Model
Solution: Build a step-by-step preprocessing model with all the steps