Scribd
Upload
3 views2 pages

NLP Python Programs Explained

The document provides Python programs for various Natural Language Processing (NLP) tasks using the NLTK library, including tokenization, stopword removal, stemming, lemmatization, and part-of-speech (POS) tagging. Each section includes code snippets and explanations for processing text data effectively. The examples demonstrate how to manipulate and analyze text to enhance understanding of human language.

Uploaded by

Venkata Ranga Rao Kommineni
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
3 views2 pages

NLP Python Programs Explained

The document provides Python programs for various Natural Language Processing (NLP) tasks using the NLTK library, including tokenization, stopword removal, stemming, lemmatization, and part-of-speech (POS) tagging. Each section includes code snippets and explanations for processing text data effectively. The examples demonstrate how to manipulate and analyze text to enhance understanding of human language.

Uploaded by

Venkata Ranga Rao Kommineni
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

NLP Python Programs with Explanation

Tokenization
Tokenization is the process of breaking text into words or sentences.

import nltk
from [Link] import word_tokenize, sent_tokenize

[Link]('punkt')

text = "Natural Language Processing is interesting. It helps computers understand human language."

# Sentence Tokenization
sentences = sent_tokenize(text)
print("Sentence Tokenization:")
print(sentences)

# Word Tokenization
words = word_tokenize(text)
print("\nWord Tokenization:")
print(words)

Stopword Removal
Stopwords are common words that add little meaning to a sentence (e.g., 'is', 'the').
import nltk
from [Link] import stopwords
from [Link] import word_tokenize

[Link]('stopwords')
[Link]('punkt')

text = "This is an example sentence demonstrating stopwords removal."

words = word_tokenize(text)

filtered = [word for word in words if [Link]() not in [Link]('english')]

print("Original Words:", words)


print("After Stopword Removal:", filtered)

Stemming
Stemming reduces words to their base or root form.

from [Link] import PorterStemmer


from [Link] import word_tokenize
import nltk

[Link]('punkt')

stemmer = PorterStemmer()

words = ["playing", "played", "player", "plays"]

print("Stemming Results:")
for word in words:
print(word, "→", [Link](word))

Lemmatization
Lemmatization converts words to base form while keeping meaning intact.

import nltk
from [Link] import WordNetLemmatizer
from [Link] import word_tokenize

[Link]('wordnet')
[Link]('punkt')

text = "birds flying flies better than other flying animals"

lemmatizer = WordNetLemmatizer()
words = word_tokenize(text)

print("Lemmatization Result:")
for word in words:
print(word, "→", [Link](word))

POS Tagging
POS tagging assigns grammatical roles like noun, verb, adjective to words.

import nltk
from [Link] import word_tokenize

[Link]('averaged_perceptron_tagger')
[Link]('punkt')

sentence = "Artificial Intelligence will change the future."

words = word_tokenize(sentence)
pos_tags = nltk.pos_tag(words)

print("POS Tags:")
print(pos_tags)

You might also like