■ Natural Language Processing (NLP) – Preprocessing
Techniques
Natural Language Processing (NLP) is a subfield of Artificial Intelligence that helps
machines understand, interpret, and generate human language. Before analyzing or
building models, we must clean and preprocess text data so that the computer can handle
it efficiently.
1. Punctuation in NLP
Definition: Punctuation refers to characters like . , ! ? ; : ' " - ( ) etc., that help structure and
clarify sentences in natural language.
In NLP: Punctuation often does not carry semantic meaning, so they are usually removed
or handled carefully during preprocessing.
Example: text = 'Hello!!! How are you doing today?' → Cleaned Text: 'Hello How are you
doing today'
Why remove punctuation?
• To simplify text
• To reduce noise in data
• To ensure accurate tokenization and word frequency counting
2. Removing Punctuation
Definition: It is the process of deleting punctuation marks from the text using regular
expressions or Python string utilities.
Example (Using Python):
import string
text = 'Wow! This NLP course is amazing, isn't it?'
cleaned_text = [Link]([Link]('', '', [Link]))
print(cleaned_text)
Output: Wow This NLP course is amazing isnt it
Why it’s done:
• To reduce unnecessary symbols
• Helps in better tokenization
• Standardizes text input
3. Word Tokenization
Definition: Tokenization is the process of splitting a text into smaller units called tokens.
Word Tokenization specifically splits text into words.
Example:
from [Link] import word_tokenize
text = 'NLP is fun and exciting!'
tokens = word_tokenize(text)
print(tokens)
Output: ['NLP', 'is', 'fun', 'and', 'exciting', '!']
4. Sentence Tokenization
Definition: Sentence tokenization splits a paragraph into individual sentences.
Example:
from [Link] import sent_tokenize
text = 'NLP is fun. It helps machines understand language.'
sentences = sent_tokenize(text)
print(sentences)
Output: ['NLP is fun.', 'It helps machines understand language.']
5. Stop Words Removal
Definition: Stop words are common words (like 'the', 'is', 'in', 'and') that are removed to
focus on meaningful terms.
Example:
from [Link] import stopwords
from [Link] import word_tokenize
text = 'This is a simple example to show stopword removal.'
tokens = word_tokenize(text)
filtered_words = [w for w in tokens if [Link]() not in [Link]('english')]
print(filtered_words)
Output: ['This', 'simple', 'example', 'show', 'stopword', 'removal', '.']
6. Stemming
Definition: Stemming reduces a word to its base or root form by removing suffixes and
prefixes. It is rule-based and may not always result in valid English words.
Example:
from [Link] import PorterStemmer
ps = PorterStemmer()
words = ['running', 'runs', 'easily', 'fairly']
stemmed = [[Link](w) for w in words]
print(stemmed)
Output: ['run', 'run', 'easili', 'fairli']
Note: Stemming is fast but less accurate. Common stemmers include Porter, Snowball,
and Lancaster.
7. Lemmatization
Definition: Lemmatization reduces a word to its base or dictionary form (lemma) using
vocabulary and morphological analysis. Output is a real English word.
Example:
from [Link] import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
words = ['running', 'better', 'studies']
lemmas = [[Link](w, pos='v') for w in words]
print(lemmas)
Output: ['run', 'good', 'study']
Aspect Stemming Lemmatization
Approach Rule-based Dictionary + Grammar-based
Output May not be valid Always valid
Example studies → studi studies → study
Speed Fast Slower
Accuracy Less accurate More accurate
8. NLP Pipeline Operations
Definition: An NLP Pipeline is a sequence of text preprocessing steps applied to raw text
data before model training.
Steps in NLP Pipeline:
1. Text Cleaning
2. Tokenization
3. Stop Word Removal
4. Stemming/Lemmatization
5. Vectorization
6. Modeling
Example (Full Pipeline):
import string
from [Link] import word_tokenize
from [Link] import stopwords
from [Link] import WordNetLemmatizer
text = 'Cats, running faster than dogs, are amazing animals!'
text = [Link]().translate([Link]('', '', [Link]))
tokens = word_tokenize(text)
filtered = [w for w in tokens if w not in [Link]('english')]
lemmatizer = WordNetLemmatizer()
lemmas = [[Link](w) for w in filtered]
print(lemmas)
Output: ['cat', 'running', 'faster', 'dog', 'amazing', 'animal']
Operation Purpose Example Input Example Output
Remove punctuation Clean text Hello!!! Hello
Word Tokenization Split into words I love NLP ['I','love','NLP']
Sentence Tokenization Split sentences NLP is fun. Try it! ['NLP is fun.','Try it!']
Stop Words Removal Remove common words I am learning NLP ['learning','NLP']
Stemming Reduce to root running run
Lemmatization Reduce to base studies study
Conclusion
Text preprocessing is the foundation of NLP tasks. By performing operations like
punctuation removal, tokenization, stop word removal, stemming, and lemmatization, we
make text data clean and structured, allowing NLP models to understand language
efficiently.