1.
Text segmentation: Segment a text into linguistically meaningful units, such as
paragraphs, sentences, or words. Write programs to segment text (in different formats) into
tokens (words and word-like units) using regular expressions. Compare an automatic
tokenization with a gold standard
PROGRAM:
import re
text = """NLP is easy.
It is useful for students.""" # Input Text
print("Original Text:")
print(text)
paragraphs = [Link]("\n") # Paragraph Segmentation
print("\nParagraphs:")
print(paragraphs)
sentences = [] # Sentence Segmentation
for s in [Link]('.'):
if [Link]() != "":
[Link]([Link]())
print("\nSentences:")
print(sentences)
auto_tokens = [Link](r'\W+', text) # Automatic Tokenization using Regex
auto_tokens = [w for w in auto_tokens if w != ""]
print("\nAutomatic Tokens:")
print(auto_tokens)
gold_tokens = ["NLP", "is", "easy", "It", "is", "useful", "for", "students"] # Gold Standard
Tokens (Manual)
print("\nGold Standard Tokens:")
print(gold_tokens)
if auto_tokens == gold_tokens: # Comparison
print("\nAutomatic tokenization matches Gold Standard")
else:
print("\nAutomatic tokenization does NOT match Gold Standard")
OUTPUT:
Original Text:
NLP is easy.
It is useful for students.
Paragraphs:
['NLP is easy.', 'It is useful for students.']
Sentences:
['NLP is easy', 'It is useful for students']
Automatic Tokens:
['NLP', 'is', 'easy', 'It', 'is', 'useful', 'for', 'students']
Gold Standard Tokens:
['NLP', 'is', 'easy', 'It', 'is', 'useful', 'for', 'students']
Automatic tokenization matches Gold Standard
2. Part-of-speech tagging: Label words (tokens) with parts of speech such as noun,
adjective, and verb using a variety of tagging methods, e.g., default tagger, regular
expression tagger, unigram tagger, and n-gram taggers.
PROGRAM:
import nltk
from [Link] import word_tokenize
from [Link] import DefaultTagger, RegexpTagger, UnigramTagger, BigramTagger,
TrigramTagger
from [Link] import treebank
sentence = input("Enter a sentence: ") # Input sentence
tokens = word_tokenize(sentence)
print("Tokens:")
print(tokens)
default_tagger = DefaultTagger("NN") # Default Tagger
print("\nDefault Tagger:")
print(default_tagger.tag(tokens))
patterns = [ # Regex Tagger
(r'^(am|are|is|was|were|be|been|being)$', 'VBP'),
(r'.*ing$', 'VBG'), # gerund
(r'.*ed$', 'VBD'), # past tense
(r'.*s$', 'NNS'), # plural noun
(r'.*', 'NN') # default noun
regex_tagger = RegexpTagger(patterns)
print("\nRegex Tagger:")
print(regex_tagger.tag(tokens))
train_data = treebank.tagged_sents()[:2000] # Training data (Penn Treebank)
# Unigram Tagger (with backoff)
unigram_tagger = UnigramTagger(train_data, backoff=regex_tagger)
print("\nUnigram Tagger:")
print(unigram_tagger.tag(tokens))
# Bigram Tagger (with backoff)
bigram_tagger = BigramTagger(train_data, backoff=unigram_tagger)
print("\nBigram Tagger:")
print(bigram_tagger.tag(tokens))
# Trigram Tagger (with backoff)
trigram_tagger = TrigramTagger(train_data, backoff=bigram_tagger)
print("\nTrigram Tagger:")
print(trigram_tagger.tag(tokens))
OUTPUT:
Enter a sentence: students are learning NLp
Tokens:
['students', 'are', 'learning', ' NLP']
Default Tagger:
[('students', 'NN'), ('are', 'NN'), ('learning', 'NN'), (' NLP', 'NN')]
Regex Tagger:
[('students', 'NNS'), ('are', 'VBP'), ('learning', 'VBG'), (' NLP', 'NN')]
Unigram Tagger:
[('students', 'NNS'), ('are', 'VBP'), ('learning', 'NN'), (' NLP', 'NN')]
Bigram Tagger:
[('students', 'NNS'), ('are', 'VBP'), ('learning', 'NN'), (' NLP', 'NN')]
Trigram Tagger:
[('students', 'NNS'), ('are', 'VBP'), ('learning', 'NN'), ('NLP', 'NN')]
3. Text classification: Categorize text documents into predefined classes using Naive Bayes
Classifier and the Perceptron model
PROGRAM:
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import Perceptron
texts = [ # Training data
"I love this movie", # positive
"This product is very good", # positive
"I hate this movie", # negative
"This product is bad", # negative
"The team won the match", # sports
"The player scored a goal", # sports
labels = [
"positive",
"positive",
"negative",
"negative",
"sports",
"sports",
vectorizer = CountVectorizer() # Convert text into numbers
X = vectorizer.fit_transform(texts)
nb = MultinomialNB() # Train classifiers
[Link](X, labels)
perc = Perceptron()
[Link](X, labels)
user_input = input("Enter a sentence: ") # User input
test_vector = [Link]([user_input])
print("\nUser sentence:", user_input) # Predictions
print("Naive Bayes Prediction:",
[Link](test_vector)[0])
print("Perceptron Prediction:",
[Link](test_vector)[0])
OUTPUT:
Enter a sentence: the team scored a goal
User sentence: the team scored a goal
Naive Bayes Prediction: sports
Perceptron Prediction: sports
4. Chunk extraction, or partial parsing: Extract short phrases from a part-of-speech tagged
sentence. This is different from full parsing in that we're interested in standalone chunks,
or phrases, instead of full parse trees
PROGRAM:
import nltk
from nltk import word_tokenize, pos_tag
from [Link] import RegexpParser
sentence = input( "Enter Sentence:") # Input sentence
tokens = word_tokenize(sentence) # Tokenization
pos_tags = pos_tag(tokens) # POS tagging (Penn Treebank)
print("POS Tagged Sentence:")
print(pos_tags)
chunk_grammar = r""" # Chunk grammar (Noun Phrase)
NP: {<DT>?<JJ>*<NN.*>+}
"""
chunk_parser = RegexpParser(chunk_grammar) # Create chunk parser
chunked_output = chunk_parser.parse(pos_tags) # Apply chunking
print("\nChunked Output:")
print(chunked_output)
OUTPUT:
Enter Sentence: The students are learning NLP concepts
POS Tagged Sentence:
[('The', 'DT'), ('students', 'NNS'), ('are', 'VBP'), ('learning', 'VBG'), ('NLP', 'NNP'), ('concepts',
'NNS')]
Chunked Output:
(S
(NP The/DT students/NNS)
are/VBP
learning/VBG
(NP NLP/NNP concepts/NNS))
5. Parsing: parsing specific kinds of data, focusing primarily on dates, times, and HTML..
Make use of the following preprocessing libraries:
date util which provides date time parsing and time zone conversion
Ixml and BeautifulSoup which can parse, clean, and convert HTML
charade and Unicode Dammit which can detect and convert text character encoding
PROGRAM:
from dateutil import parser
from dateutil import tz
date_string = "21st August 2025 10:30 PM IST" # Raw date string
parsed_date = [Link](date_string) # Parse date and time
print("Original Date String:")
print(date_string)
print("\nParsed Date-Time:")
print(parsed_date)
from_zone = [Link]("Asia/Kolkata") # Convert time zone (IST to UTC)
to_zone = [Link]("UTC")
parsed_date = parsed_date.replace(tzinfo=from_zone)
utc_time = parsed_date.astimezone(to_zone)
print("\nConverted to UTC:")
print(utc_time)
OUTPUT:
Original Date String:
21st August 2025 10:30 PM IST
Parsed Date-Time:
2025-08-21 22:30:00
Converted to UTC:
2025-08-21 17:00:00+00:00
6. Sentiment Analysis: Using Libraries TextBlob and nitk, give the sentiment of a document
PROGRAM:
from textblob import TextBlob # Sentiment Analysis using TextBlob and NLTK
from [Link] import SentimentIntensityAnalyzer # Import libraries
import nltk
text = input("Enter input: ")
print("\nInput:")
print(text)
blob = TextBlob(text) # Sentiment Analysis using TextBlob
polarity = [Link]
subjectivity = [Link]
print("\n--- TextBlob Sentiment ---")
print("Polarity:", polarity)
print("Subjectivity:", subjectivity)
if polarity > 0:
print("TextBlob Overall Sentiment: Positive")
elif polarity < 0:
print("TextBlob Overall Sentiment: Negative")
else:
print("TextBlob Overall Sentiment: Neutral")
sia = SentimentIntensityAnalyzer() # Sentiment Analysis using NLTK (VADER)
scores = sia.polarity_scores(text)
print("\n--- NLTK VADER Sentiment ---")
print("Scores:", scores)
if scores['compound'] > 0:
print("NLTK Overall Sentiment: Positive")
elif scores['compound'] < 0:
print("NLTK Overall Sentiment: Negative")
else:
print("NLTK Overall Sentiment: Neutral")
OUTPUT:
Enter input: I really love this product. It is amazing.
Input:
I really love this product. It is amazing.
--- TextBlob Sentiment ---
Polarity: 0.55
Subjectivity: 0.75
TextBlob Overall Sentiment: Positive
--- NLTK VADER Sentiment ---
Scores: {'neg': 0.0, 'neu': 0.376, 'pos': 0.624, 'compound': 0.8516}
NLTK Overall Sentiment: Positive