EXPERIMENT-1 Natural Language Processing
1. Write a Python Program to perform the following tasks on text
a) Tokenization b) Stop word Removal
Natural Language Processing (NLP) is a fascinating field that empowers machines to comprehend and interact
with human language. In this blog, we’ll unravel the concepts of Tokenization, Stop Words, Stemming, and
Lemmatization, essential pillars of NLP.
Difference between corpus, documents, vocabulary, words
Corpus — A paragraph as our data
Documents — Sentences as our data
Vocabulary — Unique words present in the corpus
Words — All words present in a corpus
a) Tokenization:
Definition: Tokenization is the process of breaking down a text into individual words or tokens. These tokens are
the building blocks for further analysis.
Tokenization are of 2 types:
Word tokenizer — Breaking the corpus into individual words
Sentence tokenizer — Breaking the corpus into individual sentences
Example:
Consider the sentence: “Natural Language Processing is amazing!”
Tokenization Result: [“Natural”, “Language”, “Processing”, “is”, “amazing”, “!”]
Prereqisites:
1)NLTK
NLTK (Natural Language Toolkit) is used in Python to implement programs under the domain of Natural
Language Processing. It contains a variety of libraries for various purposes like text classification, parsing,
stemming, tokenizing, etc.
pip install nltk
2) PUNKT
In NLTK, PUNKT is an unsupervised trainable model, which means it can be trained on unlabeled [Link]
generates a list of sentences from a text by developing a model for words that start sentences, prepositional
phrases, and abbreviations using an unsupervised technique. Without first being put to use, it has to be trained on
a sizable amount of plaintext in the intended language.
import nltk
[Link]('punkt')
Python Program to perform Tokenization on text
i) Word tokenizer
# Word tokenizer
import nltk [Link]('punkt')
# Download the necessary tokenization models
from [Link] import word_tokenize
def tokenize_words(text):
words = word_tokenize(text)
return words # Example text
text = "NLTK is a leading platform for building Python programs to work with human language data."
# Tokenize words
words = tokenize_words(text)
# Print tokenized words
print(words)
ii) Sentence tokenizer
# Sentence tokenizer
import nltk
[Link]('punkt') # Download the necessary tokenization models
from [Link] import sent_tokenize
def tokenize_sentences(text):
sentences = sent_tokenize(text)
return sentences
# Example text
text = "NLTK is a leading platform for building Python programs to work with human language data.
It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along
with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and
semantic reasoning, wrappers for industrial-strength NLP libraries, and an active discussion forum."
# Tokenize sentences
sentences = tokenize_sentences(text)
# Print tokenized sentences
for i, sentence in enumerate(sentences):
print(f"Sentence {i+1}: {sentence}")
b) Stop word Removal
Definition: Stop words are common words (e.g., “the,” “is,” “and”) that are often removed during
NLP tasks as they don’t carry significant meaning.
Here we will first tokenize the sentence or the data (consisting of multiple sentences), Then from these
tokenized sentences we will remove those tokens which are stopwords.
Example:
Original: “The quick brown fox jumps over the lazy dog.” After Removing Stop Words: [“quick”,
“brown”, “fox”, “jumps”, “lazy”, “dog.”]
Python Program to perform Stop word Removal on text
# Stopwords
import nltk from [Link]
import stopwords
from [Link] import word_tokenize
# Download NLTK stopwords and tokenizer models
[Link]('stopwords')
[Link]('punkt')
def remove_stopwords(text):
# Tokenize the text into words
words = word_tokenize(text)
# Get English stopwords
english_stopwords = set([Link]('english'))
# Remove stopwords from the tokenized words
filtered_words = [word for word in words if [Link]() not in english_stopwords]
# Join the filtered words back into a single string
filtered_text = ' '.join(filtered_words)
return filtered_text
# Example text
text = "NLTK is a leading platform for building Python programs to work with human language data."
# Remove stopwords
filtered_text = remove_stopwords(text)
# Print filtered text
print(filtered_text)