Scribd
Upload
8 views2 pages

Python NLP Text Preprocessing Guide

The document provides a Python program for preprocessing text in Natural Language Processing (NLP) using the NLTK library. It includes steps such as tokenization, filtration of non-alphabetic words, script validation for English, stop word removal, and stemming. An example text is processed through these steps, demonstrating the transformation of the original text into a list of stemmed words.

Uploaded by

savinsreenu588
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
8 views2 pages

Python NLP Text Preprocessing Guide

The document provides a Python program for preprocessing text in Natural Language Processing (NLP) using the NLTK library. It includes steps such as tokenization, filtration of non-alphabetic words, script validation for English, stop word removal, and stemming. An example text is processed through these steps, demonstrating the transformation of the original text into a list of stemmed words.

Uploaded by

savinsreenu588
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Program 1.

Write a Python program for the following


preprocessing of text in NLP: ming
import nltk
from [Link] import word_tokenize
from [Link] import stopwords
from [Link] import PorterStemmer
import re
import unicodedata

# Download required resources


[Link]('punkt')
[Link]('stopwords')

def preprocess_text(text):
print("\nOriginal Text:\n", text)

# 1. Tokenization
tokens = word_tokenize(text)
print("\nTokenized Words:\n", tokens)

# 2. Filtration (Removing non-alphabetic words)


filtered_tokens = [word for word in tokens if [Link]()]
print("\nFiltered Tokens (Only Alphabetic Words):\n",
filtered_tokens)

# 3. Script Validation (Checking if text is English)


def is_english(text):
return all(ord(char) < 128 for char in text) # ASCII
validation

valid_tokens = [word for word in filtered_tokens if


is_english(word)]
print("\nScript Validated Tokens (Only English Words):\n",
valid_tokens)

# 4. Stop Word Removal


stop_words = set([Link]('english'))
tokens_without_stopwords = [word for word in valid_tokens if
[Link]() not in stop_words]
print("\nAfter Stop Word Removal:\n", tokens_without_stopwords)

# 5. Stemming
stemmer = PorterStemmer()
stemmed_words = [[Link](word) for word in
tokens_without_stopwords]
print("\nStemmed Words:\n", stemmed_words)

return stemmed_words
# Example Text
text = "The quick brown fox jumps over the lazy dog. NLP preprocessing
is important!"
processed_text = preprocess_text(text)

[nltk_data] Downloading package punkt to


[nltk_data] C:\Users\siraj\AppData\Roaming\nltk_data...
[nltk_data] Unzipping tokenizers\[Link].
[nltk_data] Downloading package stopwords to
[nltk_data] C:\Users\siraj\AppData\Roaming\nltk_data...
[nltk_data] Unzipping corpora\[Link].

Original Text:
The quick brown fox jumps over the lazy dog. NLP preprocessing is
important!

Tokenized Words:
['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy',
'dog', '.', 'NLP', 'preprocessing', 'is', 'important', '!']

Filtered Tokens (Only Alphabetic Words):


['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy',
'dog', 'NLP', 'preprocessing', 'is', 'important']

Script Validated Tokens (Only English Words):


['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy',
'dog', 'NLP', 'preprocessing', 'is', 'important']

After Stop Word Removal:


['quick', 'brown', 'fox', 'jumps', 'lazy', 'dog', 'NLP',
'preprocessing', 'important']

Stemmed Words:
['quick', 'brown', 'fox', 'jump', 'lazi', 'dog', 'nlp', 'preprocess',
'import']

You might also like