Scribd
Upload
14 views2 pages

Python Text Processing Techniques

The document provides Python programs for text processing tasks using the NLTK library, including tokenization, stop word removal, and stemming. It demonstrates how to tokenize text into sentences and words, remove stop words, and apply the Porter stemmer algorithm. Sample texts are used to illustrate each task, along with the necessary imports and NLTK data downloads.

Uploaded by

truparanicse
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
14 views2 pages

Python Text Processing Techniques

The document provides Python programs for text processing tasks using the NLTK library, including tokenization, stop word removal, and stemming. It demonstrates how to tokenize text into sentences and words, remove stop words, and apply the Porter stemmer algorithm. Sample texts are used to illustrate each task, along with the necessary imports and NLTK data downloads.

Uploaded by

truparanicse
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

1 (a) Write a Python Program to perform following tasks on text Tokenization

# Import necessary modules


import nltk
from [Link] import word_tokenize, sent_tokenize

# Download NLTK data files (only needed once)


[Link]('punkt')
[Link]('punkt_tab') # Added to resolve the LookupError for punkt_tab

# Sample text
text = """Python is a popular programming language. It is widely used for web
development, data analysis, artificial intelligence, and more. Learning Python can be
fun and rewarding."""

# 1. Sentence Tokenization
sentences = sent_tokenize(text)
print("Sentence Tokenization:")
for i, sentence in enumerate(sentences, 1):
print(f"{i}: {sentence}")

# 2. Word Tokenization
words = word_tokenize(text)
print("\nWord Tokenization:")
print(words)

# 3. Count number of sentences and words


print("\nNumber of sentences:", len(sentences))
print("Number of words:", len(words))

# 4. Find unique words


unique_words = set(words)
print("\nUnique words:", unique_words)

1 (b) Write a Python Program to perform following tasks on Stop word Removal
# Import necessary modules
import nltk
from [Link] import stopwords
from [Link] import word_tokenize

# Download NLTK data files (only needed once)


[Link]('punkt')
[Link]('stopwords')

# Sample text
text = """Python is a popular programming language. It is widely used for web
development, data analysis, artificial intelligence, and more."""

# 1. Tokenize the text into words


words = word_tokenize(text)
print("Original Words:")
print(words)

# 2. Get English stopwords from NLTK


stop_words = set([Link]('english'))
print("\nStop Words:")
print(stop_words)

# 3. Remove stop words from the tokenized words


filtered_words = [word for word in words if [Link]() not in stop_words]
print("\nFiltered Words (Stop words removed):")
print(filtered_words)

2. Write a Python program to implement Porter stemmer algorithm for stemming


# Import necessary modules
import nltk
from [Link] import PorterStemmer
from [Link] import word_tokenize

# Download NLTK data files (only needed once)


[Link]('punkt')

# Sample text
text = """Python programmers are writing programs. He enjoys coding and studies
various programming techniques."""

# 1. Tokenize the text into words


words = word_tokenize(text)
print("Original Words:")
print(words)

# 2. Create a PorterStemmer object


stemmer = PorterStemmer()

# 3. Apply Porter Stemmer to each word


stemmed_words = [[Link](word) for word in words]
print("\nStemmed Words:")
print(stemmed_words)

You might also like