Scribd
Upload
14 views1 page

NLP Text Normalization Techniques

The document outlines an NLP lab assignment focused on text normalization techniques including tokenization, stopword removal, stemming, and lemmatization. It provides a step-by-step guide for normalizing raw text and includes sample outputs for cleaned, stemmed, and lemmatized tokens. The objective is to prepare text for effective processing by a model.

Uploaded by

officialmoosamemon
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
14 views1 page

NLP Text Normalization Techniques

The document outlines an NLP lab assignment focused on text normalization techniques including tokenization, stopword removal, stemming, and lemmatization. It provides a step-by-step guide for normalizing raw text and includes sample outputs for cleaned, stemmed, and lemmatized tokens. The objective is to prepare text for effective processing by a model.

Uploaded by

officialmoosamemon
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

■ NLP Lab Assignment – Text Normalization

Objective: Normalize raw text using tokenization, stopword removal, stemming, and lemmatization.

■ Input Text:
Natural Language Processing (NLP) is a fascinating field of Artificial Intelligence! In this lecture, we
explored tokenization – splitting text into words or subwords. We also discussed removing
punctuation, converting to lowercase, and eliminating stop words like 'is', 'the', and 'an'. Let's
normalize text properly before passing it to a model.

■ Steps to Perform:
1. Convert all text to lowercase.
2. Remove all digits (if any).
3. Strip punctuation using Python string operations.
4. Trim extra whitespace.
5. Tokenize the text using NLTK.
6. Remove stop words using NLTK stopword list.
7. Apply stemming using PorterStemmer.
8. Apply lemmatization using WordNetLemmatizer.
9. Compare and interpret the difference between stemmed and lemmatized tokens.

■ Sample Output Snippets:


Cleaned & Tokenized:
['natural', 'language', 'processing', 'nlp', 'fascinating', 'field', 'artificial', 'intelligence', 'lecture',
'explored', 'tokenization', 'splitting', 'text', 'words', 'subwords', 'also', 'discussed', 'removing',
'punctuation', 'converting', 'lowercase', 'eliminating', 'stop', 'words', 'like', 'lets', 'normalize', 'text',
'properly', 'passing', 'model']
Stemmed Tokens:
['natur', 'languag', 'process', 'nlp', 'fascin', 'field', 'artifici', 'intellig', 'lectur', 'explor', 'token', 'split', 'text',
'word', 'subword', 'also', 'discuss', 'remov', 'punctuat', 'convert', 'lowercas', 'elimin', 'stop', 'word',
'like', 'let', 'normal', 'text', 'properli', 'pass', 'model']
Lemmatized Tokens:
['natural', 'language', 'processing', 'nlp', 'fascinating', 'field', 'artificial', 'intelligence', 'lecture',
'explored', 'tokenization', 'splitting', 'text', 'word', 'subwords', 'also', 'discussed', 'removing',
'punctuation', 'converting', 'lowercase', 'eliminating', 'stop', 'word', 'like', 'lets', 'normalize', 'text',
'properly', 'passing', 'model']

You might also like