SIRMVIT Natural Language Processing (BAI601)

1. Write a Python program for the following preprocessing of text in NLP:

● Tokenization
● Filtration

● Script Validation
● Stop Word Removal
● Stemming

import re
import nltk
from [Link] import stopwords
from [Link] import PorterStemmer

# Download necessary NLTK data

[Link]('punkt')
[Link]('stopwords')

def preprocess_text(text):
# Step 1: Tokenization
tokens = nltk.word_tokenize(text)
print("Tokens:", tokens)

# Step 2: Filtration
filtered_tokens = [word for word in tokens if [Link](r'^[a-zA-Z]+$', word)]
print("Filtered Tokens:", filtered_tokens)

# Step 3: Script Validation (English in this case)

# Assuming we want to keep only English words
english_tokens = [word for word in filtered_tokens if [Link]()]
print("English Tokens:", english_tokens)

# Step 4: Stop Word Removal

stop_words = set([Link]('english'))
cleaned_tokens = [word for word in english_tokens if [Link]() not in stop_words]
print("Tokens after Stop Word Removal:", cleaned_tokens)

# Step 5: Stemming
stemmer = PorterStemmer()
stemmed_tokens = [[Link](word) for word in cleaned_tokens]
print("Stemmed Tokens:", stemmed_tokens)

return stemmed_tokens

Dept of AI & ML Page 1

SIRMVIT Natural Language Processing (BAI601)

# Example text
text = "The quick brown fox jumps over the lazy dog! 123"

# Preprocess the text

preprocessed_text = preprocess_text(text)

# Output the final preprocessed text

print("Final Preprocessed Text:", preprocessed_text)

OUTPUT:

[nltk_data] Downloading package punkt to C:\Users\VVIT-

[nltk_data] AIML\AppData\Roaming\nltk_data...

[nltk_data] Package punkt is already up-to-date!

[nltk_data] Downloading package stopwords to C:\Users\VVIT-

[nltk_data] AIML\AppData\Roaming\nltk_data...

[nltk_data] Package stopwords is already up-to-date!

Tokens: ['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog', '!', '123']

Filtered Tokens: ['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog']

English Tokens: ['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog']

Tokens after Stop Word Removal: ['quick', 'brown', 'fox', 'jumps', 'lazy',

'dog'] Stemmed Tokens: ['quick', 'brown', 'fox', 'jump', 'lazi', 'dog']

Final Preprocessed Text: ['quick', 'brown', 'fox', 'jump', 'lazi', 'dog']

Dept of AI & ML Page 2

SIRMVIT Natural Language Processing (BAI601)

2. Demonstrate the N-gram modeling to analyze and establish the probability distribution
across sentences and explore the utilization of unigrams, bigrams, and trigrams in diverse
English sentences to illustrate the impact of varying n-gram orders on the calculated
probabilities.

from collections import Counter

import random

# Sample sentences
sentences = [
"I love ice cream",
"I love chocolate",
"Ice cream is delicious"
]

# Tokenize sentences into words

words = []
for sentence in sentences:
[Link]([Link]().split())

# Total number of words in the corpus

total_words = len(words)

# Unigrams: word frequencies

unigrams = Counter(words)

# Function to calculate unigram probabilities

def calculate_unigram_probabilities(unigrams, total_words):
unigram_probabilities = {word: count / total_words for word, count in [Link]()}
return unigram_probabilities

# Bigrams: consecutive word pairs

bigrams = [(words[i], words[i + 1]) for i in range(len(words) - 1)]
bigram_counts = Counter(bigrams)

# Function to calculate bigram probabilities

def calculate_bigram_probabilities(bigrams, bigram_counts, unigrams):
bigram_probabilities = {}
for (word1, word2), count in bigram_counts.items():
bigram_probabilities[(word1, word2)] = count / unigrams[word1]
return bigram_probabilities

# Trigrams: consecutive word triplets

Dept of AI & ML Page 3

SIRMVIT Natural Language Processing (BAI601)
trigrams = [(words[i], words[i + 1], words[i + 2]) for i in range(len(words) - 2)]
trigram_counts = Counter(trigrams)

Dept of AI & ML Page 4

SIRMVIT Natural Language Processing (BAI601)

# Function to calculate trigram probabilities

def calculate_trigram_probabilities(trigrams, trigram_counts, bigrams):
trigram_probabilities = {}
for (word1, word2, word3), count in trigram_counts.items():
bigram_count = bigrams[(word1, word2)]
trigram_probabilities[(word1, word2, word3)] = count / bigram_count
return trigram_probabilities

# Calculate probabilities for unigrams, bigrams, and trigrams

unigram_probabilities = calculate_unigram_probabilities(unigrams, total_words)
bigram_probabilities = calculate_bigram_probabilities(bigrams, bigram_counts, unigrams)
trigram_probabilities = calculate_trigram_probabilities(trigrams, trigram_counts, bigram_counts)

# Print results
print("Unigram Probabilities:")
for word, prob in unigram_probabilities.items():
print(f"P({word}) = {prob:.4f}")

print("\nBigram Probabilities:")
for (word1, word2), prob in bigram_probabilities.items():
print(f"P({word2} | {word1}) = {prob:.4f}")

print("\nTrigram Probabilities:")
for (word1, word2, word3), prob in trigram_probabilities.items():
print(f"P({word3} | {word1}, {word2}) = {prob:.4f}")

OUTPUT:

Unigram Probabilities:

P(i) = 0.1818

P(love) = 0.1818

P(ice) = 0.1818

P(cream) = 0.1818

P(chocolate) = 0.0909

P(is) = 0.0909

P(delicious) = 0.0909

Dept of AI & ML Page 5

SIRMVIT Natural Language Processing (BAI601)

Bigram Probabilities:

P(love | i) = 1.0000

P(ice | love) = 0.5000

P(cream | ice) = 1.0000

P(i | cream) = 0.5000

P(chocolate | love) = 0.5000

P(ice | chocolate) = 1.0000

P(is | cream) = 0.5000

P(delicious | is) = 1.0000

Trigram Probabilities:

P(ice | i, love) = 0.5000

P(cream | love, ice) = 1.0000

P(i | ice, cream) = 0.5000

P(love | cream, i) = 1.0000

P(chocolate | i, love) = 0.5000

P(ice | love, chocolate) =

1.0000

P(cream | chocolate, ice) = 1.0000

P(is | ice, cream) = 0.5000

P(delicious | cream, is) = 1.0000

Dept of AI & ML Page 6

SIRMVIT Natural Language Processing (BAI601)

3. Investigate the Minimum Edit Distance (MED) algorithm and its application in string
comparison and the goal is to understand how the algorithm efficiently computes the minimum
number of edit operations required to transform one string into another. ● Test the algorithm on
strings with different type of variations (e.g., typos, substitutions, insertions, deletions) ●
Evaluate its adaptability to different types of input variations

def min_edit_distance(X, Y):

m, n = len(X), len(Y)
D = [[0] * (n + 1) for _ in range(m + 1)]

for i in range(m + 1):

D[i][0] = i
for j in range(n + 1):
D[0][j] = j

for i in range(1, m + 1):

for j in range(1, n + 1):
if X[i - 1] == Y[j - 1]: D[i]
[j] = D[i - 1][j - 1]
else:
D[i][j] = min(D[i][j - 1], D[i - 1][j], D[i - 1][j - 1]) + 1

return D[m][n]

# Test cases
print(min_edit_distance("kitten", "sitten")) # Output: 1 (substitution)
print(min_edit_distance("kitten", "kittens")) # Output: 1 (insertion)
print(min_edit_distance("kitten", "kiten")) # Output: 1 (deletion)
print(min_edit_distance("sitting", "kitten")) # Output: 3 (substitution + substitution + deletion)
print(min_edit_distance("kitten", "kitten")) # Output: 0 (no changes)
print(min_edit_distance("abc", "xyz")) # Output: 3 (substitution x 3)

Dept of AI & ML Page 7

SIRMVIT Natural Language Processing (BAI601)

OUTPUT:

PS C:\Users\VVIT-AIML\Desktop\shivleela\NLP> python -u "c:\Users\VVIT- AIML\Desktop\

shivleela\NLP\[Link]"

Dept of AI & ML Page 8

SIRMVIT Natural Language Processing (BAI601)

4. Write a program to implement top-down and bottom-up parser using appropriate context free
grammar.

Top-Down Parser (Recursive Descent):

class TopDownParser:
def init (self, grammar):
[Link] = grammar
self.input_string = ""
[Link] = 0

def parse(self, input_string):

self.input_string = input_string
[Link] = 0
return self.E()

def E(self):
result = self.T()
if result is None:
return None
while [Link] < len(self.input_string) and self.input_string[[Link]] == '+':
[Link] += 1 # consume '+'
result2 = self.T()
if result2 is None:
return None
return result

def T(self):
result = self.F()
if result is None:
return None
while [Link] < len(self.input_string) and self.input_string[[Link]] == '*':
[Link] += 1 # consume '*'

Dept of AI & ML Page 9

SIRMVIT Natural Language Processing (BAI601)

result2 = self.F()
if result2 is None:
return None
return result

def F(self):
if [Link] < len(self.input_string) and self.input_string[[Link]] == '(':
[Link] += 1 # consume '('
result = self.E()
if result is None or [Link] >= len(self.input_string) or self.input_string[[Link]] != ')':
return None
[Link] += 1 # consume ')'
return result
elif [Link] < len(self.input_string) and self.input_string[[Link]].isalpha():
[Link] += 1 # consume id
return True
return None

# Example usage:
topdown_parser = TopDownParser(grammar=None)
input_expr = "id+id*id"
print("Top-Down Parser Result:", topdown_parser.parse(input_expr))

OUTPUT:

PS C:\Users\VVIT-AIML\Desktop\shivleela\NLP> python -u "c:\Users\VVIT- AIML\Desktop\

shivleela\NLP\[Link]"

Top-Down Parser Result: True

Dept of AI & ML Page 10

SIRMVIT Natural Language Processing (BAI601)

Bottom-Up Parser (Shift-Reduce):

class BottomUpParser:
def init (self, grammar):
[Link] = grammar
[Link] = []
self.input_string = ""
[Link] = 0

def parse(self, input_string):

self.input_string = input_string
[Link] = 0
while [Link] < len(self.input_string):
[Link]()
while [Link]():
pass
return [Link] == ['S']

def shift(self):
if [Link] < len(self.input_string):
[Link](self.input_string[[Link]])
[Link] += 1

def reduce(self):
# Look for possible reductions based on the grammar rules
if len([Link]) >= 3 and [Link][-3:] == ['id', '+', 'id']:
[Link] = [Link][:-3] # reduce to E -> E + T
[Link]('E')
return True
elif len([Link]) >= 3 and [Link][-3:] == ['id', '*', 'id']:
[Link] = [Link][:-3] # reduce to T -> T * F
[Link]('T')

Dept of AI & ML Page 11

SIRMVIT Natural Language Processing (BAI601)

return True
elif len([Link]) >= 1 and [Link][-1] == 'id':
[Link] = [Link][:-1] # reduce to F -> id
[Link]('F')
return True
return False

# Example usage:
bottomup_parser = BottomUpParser(grammar=None)
input_expr = "id+id*id"
print("Bottom-Up Parser Result:", bottomup_parser.parse(input_expr))

OUTPUT:

PS C:\Users\VVIT-AIML\Desktop\shivleela\NLP> python -u "c:\Users\VVIT- AIML\Desktop\

shivleela\NLP\[Link]\[Link]"

Bottom-Up Parser Result: False

Dept of AI & ML Page 12

SIRMVIT Natural Language Processing (BAI601)

5. Given the following short movie reviews, each labeled with a genre, either comedy or action:
● fun, couple, love, love comedy ● fast, furious, shoot action ● couple, fly, fast, fun, fun comedy
● furious, shoot, shoot, fun action ● fly, fast, shoot, love action and A new document D: fast,
couple, shoot, fly Compute the most likely class for D. Assume a Naive Bayes classifier and
use add-1 smoothing for the likelihoods.

from collections import Counter

# Training data: reviews and their corresponding genres

reviews = [
("fun, couple, love, love", "comedy"),
("fast, furious, shoot", "action"),
("couple, fly, fast, fun, fun", "comedy"),
("furious, shoot, shoot, fun", "action"),
("fly, fast, shoot, love", "action")
]

# New document to classify

new_document = "fast, couple, shoot, fly"

# Preprocess the data (tokenize and prepare for counting)

def preprocess_text(text):
return [Link]().split(", ")

# Step 1: Count the word frequencies for each class (comedy and action)
class_word_counts = {"comedy": Counter(), "action": Counter()}
class_doc_counts = {"comedy": 0, "action": 0}

# Step 2: Build vocabulary and word counts for each class

for review, genre in reviews:
class_doc_counts[genre] += 1
words = preprocess_text(review)
class_word_counts[genre].update(words)

Dept of AI & ML Page 13

SIRMVIT Natural Language Processing (BAI601)

# Step 3: Calculate priors (probability of each class)

total_docs = len(reviews)
prior_comedy = class_doc_counts["comedy"] / total_docs
prior_action = class_doc_counts["action"] / total_docs

# Step 4: Build vocabulary

all_words = set()
for review, genre in reviews:
words = preprocess_text(review)
all_words.update(words)

vocab_size = len(all_words)

# Step 5: Add-1 Smoothing - Calculate likelihoods (P(w | class))

def calculate_likelihood(word, genre):
# P(w | genre) = (count(w, genre) + 1) / (total words in genre + vocab size)
word_count = class_word_counts[genre][word]
total_words_in_class = sum(class_word_counts[genre].values())
return (word_count + 1) / (total_words_in_class + vocab_size)

# Step 6: Classify the new document using Naive Bayes

def classify_document(new_document):
# Preprocess the new document
words_in_document = preprocess_text(new_document)

# Step 7: Calculate the likelihood for each class

log_prob_comedy = 0
log_prob_action = 0

# Add the log of the prior for each class

Dept of AI & ML Page 14

SIRMVIT Natural Language Processing (BAI601)

log_prob_comedy += prior_comedy
log_prob_action += prior_action

# Multiply the likelihoods for each word in the new document

for word in words_in_document:
# Calculate likelihood for both classes
likelihood_comedy = calculate_likelihood(word, "comedy")
likelihood_action = calculate_likelihood(word, "action")

# Multiply the log-likelihoods

log_prob_comedy += likelihood_comedy
log_prob_action += likelihood_action

# Step 8: Compare the probabilities and classify

if log_prob_comedy > log_prob_action:
return "comedy"
else:
return "action"

# Step 9: Classify the new document

result = classify_document(new_document)
print(f"The predicted class for the new document is: {result}")

OUTPUT:

PS C:\Users\VVIT-AIML\Desktop\shivleela\NLP> python -u "c:\Users\VVIT- AIML\Desktop\

shivleela\NLP\[Link]\[Link]"

The predicted class for the new document is: action

Dept of AI & ML Page 15

SIRMVIT Natural Language Processing (BAI601)

6. Demonstrate the following using appropriate programming tool which illustrates the
use of information retrieval in NLP:
● Study the various Corpus – Brown, Inaugural, Reuters, udhr with various methods like
filelds, raw, words, sents, categories 3
● Create and use your own corpora (plaintext, categorical)
● Study Conditional frequency distributions
● Study of tagged corpora with methods like tagged_sents, tagged_words
● Write a program to find the most frequent noun tags
● Map Words to Properties Using Python Dictionaries
● Study Rule based tagger, Unigram Tagger
Find different words from a given plain text without any space by comparing this text with a
given corpus of words. Also find the score of words.

import nltk
from [Link] import brown, inaugural, reuters, udhr
from [Link] import ConditionalFreqDist
from [Link] import pos_tag
#[Link]('all') # Ensure all required corpora are downloaded

# Display file IDs or categories for each corpus

print("Brown Corpus File IDs:", [Link]())
print("Inaugural Corpus File IDs:", [Link]())
print("Reuters Categories:", [Link]())
print("UDHR Raw (English) Snippet:", [Link]('English-Latin1')[:100])

# Display sample words and sentences

print("First 20 words of Brown Corpus:", [Link]()[:20])
print("First 5 sentences of Inaugural Corpus:", [Link]()[:5])

# Task 2: Custom Corpus

corpus_custom = {
"news": "Today the stock market soared as investors cheered positive earnings.",
"sports": "The local team won the championship in a thrilling final game."
}

Dept of AI & ML Page 16

SIRMVIT Natural Language Processing (BAI601)

print("Custom Corpus Categories:", corpus_custom.keys())

for category, text in corpus_custom.items():
print(f"Category: {category}, Sample Text: {text}")

# Task 3: Conditional Frequency Distribution (CFD)

cfd = ConditionalFreqDist()
for category in [Link]():
for word in [Link](categories=category):
cfd[category][[Link]()] += 1
print("Common words in 'news':", cfd["news"].most_common(5))
print("Common words in 'romance':", cfd["romance"].most_common(5))

# Task 4: Tagged Corpora

news_tagged_sents = brown.tagged_sents(categories="news")
news_tagged_words = brown.tagged_words(categories="news")
print("First 5 tagged sentences in 'news':", news_tagged_sents[:5])
print("First 10 tagged words in 'news':", news_tagged_words[:10])

# Task 5: Most Frequent Noun Tags

noun_tag_freq = {}
for word, tag in news_tagged_words:
if [Link]("NN"):
noun_tag_freq[tag] = noun_tag_freq.get(tag, 0) + 1
most_frequent_noun_tag = max(noun_tag_freq, key=noun_tag_freq.get)
print("Most frequent noun tag:", most_frequent_noun_tag, "with count:",
noun_tag_freq[most_frequent_noun_tag])

# Task 6: Mapping Words to Properties

sample_text = "This is a sample text with sample words and sample analysis"
words_list = sample_text.split()
word_properties = {}

Dept of AI & ML Page 17

SIRMVIT Natural Language Processing (BAI601)

for word in words_list:

word_lower = [Link]()
if word_lower not in word_properties:
word_properties[word_lower] = {"length": len(word_lower), "frequency": 1}
else:
word_properties[word_lower]["frequency"] += 1
print("Word Properties:", word_properties)

# Task 7: Text Segmentation

def segment_text(text, valid_words):
if not text:
return [[]]
segmentation_results = []
for i in range(1, len(text) + 1):
prefix = text[:i]
if prefix in valid_words:
suffix_results = segment_text(text[i:], valid_words)
for segmentation in suffix_results:
segmentation_results.append([prefix] + segmentation)
return segmentation_results

valid_words = ["hello", "this", "is", "a", "test", "his", "s", "isatest"]

input_text = "hellothisisatest"
segmentations = segment_text(input_text, valid_words)
print("Possible segmentations for", input_text)
for segmentation in segmentations:
score = 1 / len(segmentation)
print(segmentation, "Score:", score)

OUTPUT:

Dept of AI & ML Page 18

SIRMVIT Natural Language Processing (BAI601)

Brown Corpus File IDs: ['ca01', 'ca02', 'ca03', 'ca04', 'ca05', 'ca06', 'ca07', 'ca08', 'ca09', 'ca10',
'ca11', 'ca12', 'ca13', 'ca14', 'ca15', 'ca16', 'ca17', 'ca18', 'ca19', 'ca20', 'ca21', 'ca22', 'ca23', 'ca24',
'ca25', 'ca26', 'ca27', 'ca28', 'ca29', 'ca30', 'ca31', 'ca32', 'ca33', 'ca34', 'ca35', 'ca36', 'ca37', 'ca38',
'ca39', 'ca40', 'ca41', 'cg20', 'cg21', 'cg22', 'cg23', 'cg24', 'cg25', 'cg26', 'cg27', 'cg28', 'cg29', 'cg30',
'cg31', 'cg32', 'cg33', 'cg34', 'cg35', 'cg36', 'cg37', 'cg38', 'cg39', 'cg40', 'cg41', 'cg42', 'cg43',
'cg44', 'cg45', 'cg46', 'cg47', 'cg48', 'cg49', 'cp22', 'cp23', 'cp24', 'cp25', 'cp26', 'cp27', 'cp28',
'cp29', 'cr01', 'cr02', 'cr03', 'cr04', 'cr05', 'cr06', 'cr07', 'cr08', 'cr09']

Inaugural Corpus File IDs: ['[Link]', '[Link]', '[Link]',

'[Link]', '[Link]', '[Link]', '[Link]', '1817-
[Link]', '[Link]', '[Link]', '[Link]', '[Link]', '1837-
[Link]', '[Link]', '[Link]', '[Link]', '[Link]', '1857-
[Link]', '[Link]', '[Link]', '[Link]', '[Link]', '1877-
[Link]', '[Link]', '[Link]', '[Link]', '[Link]',
'[Link]', '[Link]', '[Link]', '[Link]', '[Link]',
'[Link]', '[Link]', '[Link]', '[Link]', '1933-
[Link]', '[Link]', '[Link]', '[Link]', '[Link]',
'[Link]', '[Link]', '[Link]', '[Link]', '1969-
[Link]', '[Link]', '[Link]', '[Link]', '[Link]', '1989-
[Link]', '[Link]', '[Link]', '[Link]', '[Link]', '2009-
[Link]', '[Link]', '[Link]', '[Link]', '[Link]']

Reuters Categories: ['acq', 'alum', 'barley', 'bop', 'carcass', 'castor-oil', 'cocoa', 'coconut',
'coconut-oil', 'coffee', 'copper', 'copra-cake', 'corn', 'cotton', 'cotton-oil', 'cpi', 'cpu', 'crude', 'dfl',
'dlr', 'dmk', 'earn', 'fuel', 'gas', 'gnp', 'gold', 'grain', 'groundnut', 'groundnut-oil', 'heat', 'hog',
'housing', 'income', 'instal-debt', 'interest', 'ipi', 'iron-steel', 'jet', 'jobs', 'l-cattle', 'lead', 'lei', 'lin-oil',
'livestock', 'lumber', 'meal-feed', 'money-fx', 'money-supply', 'naphtha', 'nat-gas', 'nickel', 'nkr',
'nzdlr', 'oat', 'oilseed', 'orange', 'palladium', 'palm-oil', 'palmkernel', 'pet-chem', 'platinum', 'potato',
'propane', 'rand', 'rape-oil', 'rapeseed', 'reserves', 'retail', 'rice', 'rubber', 'rye', 'ship', 'silver',
'sorghum', 'soy-meal', 'soy-oil', 'soybean', 'strategic-metal', 'sugar', 'sun-meal', 'sun-oil', 'sunseed',
'tea', 'tin', 'trade', 'veg-oil', 'wheat', 'wpi', 'yen', 'zinc']

UDHR Raw (English) Snippet: Universal Declaration of Human Rights

Preamble

Whereas recognition of the inherent dignity and of th

First 20 words of Brown Corpus: ['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', 'Friday',
'an', 'investigation', 'of', "Atlanta's", 'recent', 'primary', 'election', 'produced', '``', 'no', 'evidence',
"''", 'that']

Dept of AI & ML Page 19

SIRMVIT Natural Language Processing (BAI601)

First 5 sentences of Inaugural Corpus: [['Fellow', '-', 'Citizens', 'of', 'the', 'Senate', 'and', 'of',
'the', 'House', 'of', 'Representatives', ':'], ['Among', 'the', 'vicissitudes', 'incident', 'to', 'life', 'no',
'event', 'could', 'have', 'filled', 'me', 'with', 'greater', 'anxieties', 'than', 'that', 'of', 'which', 'the',
'notification', 'was', 'transmitted', 'by', 'your', 'order', ',', 'and', 'received', 'on', 'the', '14th', 'day', 'of',
'the', 'present', 'month', '.'], ['On', 'the', 'one', 'hand', ',', 'I', 'was', 'summoned', 'by', 'my', 'Country',
',', 'whose', 'voice', 'I', 'can', 'never', 'hear', 'but', 'with', 'veneration', 'and', 'love', ',', 'from', 'a',
'retreat', 'which', 'I', 'had', 'chosen', 'with', 'the', 'fondest', 'predilection', ',', 'and', ',', 'in', 'my',
'flattering', 'hopes', ',', 'with', 'an', 'immutable', 'decision', ',', 'as', 'the', 'asylum', 'of', 'my',
'declining', 'years', '--', 'a', 'retreat', 'which', 'was', 'rendered', 'every', 'day', 'more', 'necessary', 'as',
'well', 'as', 'more', 'dear', 'to', 'me', 'by', 'the', 'addition', 'of', 'habit', 'to', 'inclination', ',', 'and', 'of',
'frequent', 'interruptions', 'in', 'my', 'health', 'to', 'the', 'gradual', 'waste', 'committed', 'on', 'it', 'by',
'time', '.'], ['On', 'the', 'other', 'hand', ',', 'the', 'magnitude', 'and', 'difficulty', 'of', 'the', 'trust', 'to',
'which', 'the', 'voice', 'of', 'my', 'country', 'called', 'me', ',', 'being', 'sufficient', 'to', 'awaken', 'in',
'the', 'wisest', 'and', 'most', 'experienced', 'of', 'her', 'citizens', 'a', 'distrustful', 'scrutiny', 'into', 'his',
'qualifications', ',', 'could', 'not', 'but', 'overwhelm', 'with', 'despondence', 'one', 'who', '(',
'inheriting', 'inferior', 'endowments', 'from', 'nature', 'and', 'unpracticed', 'in', 'the', 'duties', 'of',
'civil', 'administration', ')', 'ought', 'to', 'be', 'peculiarly', 'conscious', 'of', 'his', 'own', 'deficiencies',
'.'], ['In', 'this', 'conflict', 'of', 'emotions', 'all', 'I', 'dare', 'aver', 'is', 'that', 'it', 'has', 'been', 'my',
'faithful', 'study', 'to', 'collect', 'my', 'duty', 'from', 'a', 'just', 'appreciation', 'of', 'every',
'circumstance', 'by', 'which', 'it', 'might', 'be', 'affected', '.']]

Custom Corpus Categories: dict_keys(['news', 'sports'])

Category: news, Sample Text: Today the stock market soared as investors cheered positive
earnings.

Category: sports, Sample Text: The local team won the championship in a thrilling final game.

Common words in 'news': [('the', 6386), (',', 5188), ('.', 4030), ('of', 2861), ('and', 2186)]

Common words in 'romance': [(',', 3899), ('.', 3736), ('the', 2988), ('and', 1905), ('to', 1517)]

First 5 tagged sentences in 'news': [[('The', 'AT'), ('Fulton', 'NP-TL'), ('County', 'NN-TL'),
('Grand', 'JJ-TL'), ('Jury', 'NN-TL'), ('said', 'VBD'), ('Friday', 'NR'), ('an', 'AT'), ('investigation',
'NN'), ('of', 'IN'), ("Atlanta's", 'NP$'), ('recent', 'JJ'), ('primary', 'NN'), ('election', 'NN'),
('produced', 'VBD'), ('``', '``'), ('no', 'AT'), ('evidence', 'NN'), ("''", "''"), ('that', 'CS'), ('any', 'DTI'),
('irregularities', 'NNS'), ('took', 'VBD'), ('place', 'NN'), ('.', '.')], [('The', 'AT'), ('jury', 'NN'),
('further', 'RBR'), ('said', 'VBD'), ('in', 'IN'), ('term-end', 'NN'), ('presentments', 'NNS'), ('that',
'CS'), ('the', 'AT'), ('City', 'NN-TL'), ('Executive', 'JJ-TL'), ('Committee', 'NN-TL'), (','Common
words in 'news': [('the', 6386), (',', 5188), ('.', 4030), ('of', 2861), ('and', 2186)]

Common words in 'romance': [(',', 3899), ('.', 3736), ('the', 2988), ('and', 1905), ('to', 1517)]

Dept of AI & ML Page 20

SIRMVIT Natural Language Processing (BAI601)

First 5 tagged sentences in 'news': [[('The', 'AT'), ('Fulton', 'NP-TL'), ('County', 'NN-TL'),
('Grand', 'JJ-TL'), ('Jury', 'NN-TL'), ('said', 'VBD'), ('Friday', 'NR'), ('an', 'AT'), ('investigation',
'NN'), ('of', 'IN'), ("Atlanta's", 'NP$'), ('recent', 'JJ'), ('primary', 'NN'), ('election', 'NN'),
('produced', 'VBD'), ('``', '``'), ('no', 'AT'), ('evidenceCommon words in 'news': [('the', 6386), (',',
5188), ('.', 4030), ('of', 2861), ('and', 2186)]

Common words in 'romance': [(',', 3899), ('.', 3736), ('the', 2988), ('and', 1905), ('to', 1517)]

First 5 tagged sentences in 'news': [[('The', 'AT'), ('Fulton', 'NP-TL'), ('County', 'NN-TL'),
('Grand', 'JJ-TL'), ('Jury', 'NN-TL'), ('said', 'VBD'), ('Friday', 'NR'), ('an', 'AT'Common words in
'news': [('the', 6386), (',', 5188), ('.', 4030), ('of', 2861), ('and', 2186)]

Common words in 'romance': [(',', 3899), ('.', 3736), ('the', 2988), ('and', 1905), ('to', 1517)]

First 5 tagged sentences in 'news': [[('The', 'AT'), ('Fulton', 'NP-TL'), ('County', 'NN-TL'),
('Grand', 'JJ-TL'), ('Jury', 'NN-TL'), ('said', 'VBD'), ('Friday', 'NR'), ('an', 'AT'Common words in
'news': [('the', 6386), (',', 5188), ('.', 4030), ('of', 2861), ('and', 2186)]

Common words in 'news': [('the', 6386), (',', 5188), ('.', 4030), ('of', 2861), ('and', 2186)]

Common words in 'romance': [(',', 3899), ('.', 3736), ('the', 2988), ('and', 1905), ('to', 1517)]

First 5 tagged sentences in 'news': [[('The', 'AT'), ('Fulton', 'NP-TL'), ('County', 'NN-TL'),
('Grand', 'JJ-TL'), ('Jury', 'NN-TL'), ('said', 'VBD'), ('Friday', 'NR'), ('an', 'AT'), ('investigation',
'NN'), ('of', 'IN'), ("Atlanta's", 'NP$'), ('recent', 'JJ'), ('primary', 'NN'), ('election', 'NN'),
('produced', 'VBD'), ('``', '``'), ('no', 'AT'), ('evidence', 'NN'), ("''", "''"), ('that', 'CS'), ('any', 'DTI'),
('irregularities', 'NNS'), ('took', 'VBD'), ('place', 'NN'), ('.', '.')], [('The', 'AT'), ('jury', 'NN'),
('further', 'RBR'), ('said', 'VBD'), ('in', 'IN'), ('term-end', 'NN'), ('presentments', 'NNS'), ('that',
'CS'), ('the', 'AT'), ('City', 'NN-TL'), ('Executive', 'JJ-TL'), ('Committee', 'NN-TL'), (','),
('investigation', 'NN'), ('of', 'IN'), ("Atlanta's", 'NP$'), ('recent', 'JJ'), ('primary', 'NN'), ('election',
'NN'), ('produced', 'VBD'), ('``', '``'), ('no', 'AT'), ('evidence', 'NN'), ("''", "''"), ('that', 'CS'), ('any',
'DTI'), ('irregularities', 'NNS'), ('took', 'VBD'), ('place', 'NN'), ('.', '.')], [('The', 'AT'), ('jury', 'NN'),
('further', 'RBR'), ('said', 'VBD'), ('in', 'IN'), ('term-end', 'NN'), ('presentments', 'NNS'), ('that',
'CS'), ('the', 'AT'), ('City', 'NN-TL'), ('Executive', 'JJ-TL'), ('Committee', 'NN-TL'), (','', 'NN'),
("''", "''"), ('that', 'CS'), ('any', 'DTI'), ('irregularities', 'NNS'), ('took', 'VBD'), ('place', 'NN'), ('.',
'.')], [('The', 'AT'), ('jury', 'NN'), ('further', 'RBR'), ('said', 'VBD'), ('in', 'IN'), ('term-end', 'NN'),
('presentments', 'NNS'), ('that', 'CS'), ('the', 'AT'), ('City', 'NN-TL'), ('Executive', 'JJ-TL'),
('Committee', 'NN-TL'), (',', ','), ('which', 'WDT'), ('had', 'HVD'), ('over-all', 'JJ'), ('charge', 'NN'),
('of', 'IN'), ('the', 'AT'), ('election', 'NN'), (',', ','), ('``', '``'), ('deserves', 'VBZ'), ('the', 'AT'), ('praise',
'NN'), ('and', 'CC'), ('thanks', 'NNS'), ('of', 'IN'), ('the', 'AT'), ('City', 'NN-TL'), ('of', 'IN-TL'),
('Atlanta', 'NP-TL'), ("''", "''"), ('for', 'IN'), ('the', 'AT'), ('manner', 'NN'), ('in', 'IN'), ('which',
'WDT'), ('the', 'AT'), ('election', 'NN'), ('was', 'BEDZ'), ('conducted', 'VBN'), ('.', '.')], [('The',
'AT'), ('September-October', 'NP'), ('term', 'NN'), ('jury', 'NN'), ('had', 'HVD'), ('been', 'BEN'),

Dept of AI & ML Page 21

SIRMVIT Natural Language Processing (BAI601)

('charged', 'VBN'), ('by', 'IN'), ('Fulton', 'NP-TL'), ('Superior', 'JJ-TL'), ('Court', 'NN-TL'),
('Judge', 'NN-TL'), ('Durwood', 'NP'), ('Pye', 'NP'), ('to', 'TO'), ('investigate', 'VB'), ('reports',
'NNS'), ('of', 'IN'), ('possible', 'JJ'), ('``', '``'), ('irregularities', 'NNS'), ("''", "''"), ('in', 'IN'), ('the',
'AT'), ('hard-fought', 'JJ'), ('primary', 'NN'), ('which', 'WDT'), ('was', 'BEDZ'), ('won', 'VBN'),
('by', 'IN'), ('Mayor-nominate', 'NN-TL'), ('Ivan', 'NP'), ('Allen', 'NP'), ('Jr.', 'NP'), ('.', '.')], [('``',
'``'), ('Only', 'RB'), ('a', 'AT'), ('relative', 'JJ'), ('handful', 'NN'), ('of', 'IN'), ('such', 'JJ'), (''), ('Ivan',
'NP'), ('Allen', 'NP'), ('Jr.', 'NP'), ('.', '.')], [('``', '``'), ('Only', 'RB'), ('a', 'AT'), ('relative', 'JJ'),
('handful', 'NN'), ('of', 'IN'), ('such', 'JJ'), ('reports', 'NNS'), ('was', 'BEDZ'), ('received', 'VBN'),
("''", "''"), (',', ','), ('the', 'AT'), ('jury', 'NN'), ('said', 'VBD'), (',', ','), ('``', '``'), ('considering', 'IN'),
(reports', 'NNS'), ('was', 'BEDZ'), ('received', 'VBN'), ("''", "''"), (',', ','), ('the', 'AT'), ('jury', 'NN'),
('said', 'VBD'), (',', ','), ('``', '``'), ('considering', 'IN'), ('the', 'AT'), ('widespread', 'JJ'), ('interest',
'NN'), ('in', 'IN'), ('the', 'AT'), ('election', 'NN'), (',', ','), ('the', 'AT'), ('number', 'NN'), ('of', 'IN'),
('voters', 'NNS'), ('and', 'CC'), ('the', 'AT'), ('size', 'NN'), ('of', 'IN'), ('this', 'DT'), ('city', 'NN'), ("''",
"''"), ('.', '.')], [('The', 'AT'), ('jury', 'NN'), ('said', 'VBD'), ('it', 'PPS'), ('did', 'DOD'), ('find', 'VB'),
('that', 'CS'), ('many', 'AP'), ('of', 'IN'), ("Georgia's", 'NP$'), ('registration', 'NN'), ('and', 'CC'),
('election', 'NN'), ('laws', 'NNS'), ('``', '``'), ('are', 'BER'), ('outmoded', 'JJ'), ('or', 'CC'),
('inadequate', 'JJ'), ('and', 'CC'), ('often', 'RB'), ('ambiguous', 'JJ'), ("''", "''"), ('.', '.')]]

First 10 tagged words in 'news': [('The', 'AT'), ('Fulton', 'NP-TL'), ('County', 'NN-TL'),
('Grand', 'JJ-TL'), ('Jury', 'NN-TL'), ('said', 'VBD'), ('Friday', 'NR'), ('an', 'AT'), ('investigation',
'NN'), ('of', 'IN')]

Most frequent noun tag: NN with count: 13162

Word Properties: {'this': {'length': 4, 'frequency': 1}, 'is': {'length': 2, 'frequency': 1}, 'a':
{'length': 1, 'frequency': 1}, 'sample': {'length': 6, 'frequency': 3}, 'text': {'length': 4, 'frequency':
1}, 'with': {'length': 4, 'frequency': 1}, 'words': {'length': 5, 'frequency': 1}, 'and': {'length': 3,
'frequency': 1}, 'analysis': {'length': 8, 'frequency': 1}}

Possible segmentations for hellothisisatest

['hello', 'this', 'is', 'a', 'test'] Score: 0.2

['hello', 'this', 'isatest'] Score: 0.3333333333333333.

Dept of AI & ML Page 22

SIRMVIT Natural Language Processing (BAI601)

7. Write a Python program to find synonyms and antonyms of the word "active" using WordNet.

import nltk
import nltk
#[Link]('wordnet')
#[Link]('omw-1.4')
from [Link] import wordnet

# Define the target word

word = "active"

# Initialize empty sets for synonyms and antonyms

synonyms = set()
antonyms = set()

# Iterate through synsets of the word

for synset in [Link](word):
for lemma in [Link]():
[Link]([Link]())
if [Link]():
for ant in [Link]():
[Link]([Link]())

# Print synonyms and antonyms

print("Synonyms of 'active':", synonyms)
print("Antonyms of 'active':", antonyms)

Dept of AI & ML Page 23

SIRMVIT Natural Language Processing (BAI601)

OUTPUT:

Synonyms of 'active': {'participating', 'alive', 'combat-ready', 'active', 'fighting', 'dynamic',

'active_agent', 'active_voice'}

Antonyms of 'active': {'extinct', 'stative', 'dormant', 'passive', 'quiet', 'passive_voice', 'inactive'}

Dept of AI & ML Page 24

SIRMVIT Natural Language Processing (BAI601)

8. Implement the machine translation application of NLP where it needs to train a

machine translation model for a language with limited parallel corpora. Investigate and
incorporate techniques to improve performance in low-resource scenarios.

import nltk
import tensorflow as tf
import numpy as np
from [Link] import Tokenizer
from [Link] import pad_sequences
from [Link] import Embedding, LSTM, Dense, Input
from [Link] import Model

# Download necessary NLTK data

[Link]('punkt')

def load_parallel_corpus(dataset_name):
# Placeholder: Load real parallel dataset
source_sentences = ["hello world", "how are you", "good morning"]
target_sentences = ["hola mundo", "como estas", "buenos dias"]
return source_sentences, target_sentences

def preprocess_sentences(sentences):
# Tokenization and lowercasing
return [nltk.word_tokenize([Link]()) for sent in sentences]

def train_translation_model(src, tgt):

Dept of AI & ML Page 25

SIRMVIT Natural Language Processing (BAI601)

# Placeholder: Simulating back-translation (dummy function)

return lambda x: x # Identity function for now

# Load data
source_sentences, target_sentences = load_parallel_corpus("low_resource_dataset")
preprocessed_source = preprocess_sentences(source_sentences)
preprocessed_target = preprocess_sentences(target_sentences)

# Data Augmentation: Back-Translation

reverse_model = train_translation_model(preprocessed_target, preprocessed_source)
synthetic_source_sentences = [reverse_model(tgt) for tgt in preprocessed_target]
augmented_source = preprocessed_source + synthetic_source_sentences
augmented_target = preprocessed_target + preprocessed_target

# Tokenization
source_tokenizer = Tokenizer()
source_tokenizer.fit_on_texts(augmented_source)
source_vocab_size = len(source_tokenizer.word_index) + 1

target_tokenizer = Tokenizer()
target_tokenizer.fit_on_texts(augmented_target)
target_vocab_size = len(target_tokenizer.word_index) + 1

# Convert texts to sequences

source_sequences = source_tokenizer.texts_to_sequences(augmented_source)
target_sequences = target_tokenizer.texts_to_sequences(augmented_target)

Dept of AI & ML Page 26

SIRMVIT Natural Language Processing (BAI601)

# Padding
max_length = max(max(len(seq) for seq in source_sequences), max(len(seq) for seq in
target_sequences))
source_sequences = pad_sequences(source_sequences, maxlen=max_length, padding='post')
target_sequences = pad_sequences(target_sequences, maxlen=max_length, padding='post')

# Define Seq2Seq Model

embedding_dim = 128
hidden_size = 256

# Encoder
encoder_inputs = Input(shape=(max_length,))
enc_embedding = Embedding(source_vocab_size, embedding_dim, mask_zero=True)
(encoder_inputs)
encoder_lstm = LSTM(hidden_size, return_state=True)
encoder_outputs, state_h, state_c = encoder_lstm(enc_embedding)
encoder_states = [state_h, state_c]

# Decoder
decoder_inputs = Input(shape=(max_length,))
dec_embedding = Embedding(target_vocab_size, embedding_dim,
mask_zero=True)(decoder_inputs)
decoder_lstm = LSTM(hidden_size, return_sequences=True, return_state=True)
decoder_outputs, _, _ = decoder_lstm(dec_embedding, initial_state=encoder_states)
decoder_dense = Dense(target_vocab_size, activation='softmax')
decoder_outputs = decoder_dense(decoder_outputs)

Dept of AI & ML Page 27

SIRMVIT Natural Language Processing (BAI601)

# Define Model
model = Model([encoder_inputs, decoder_inputs], decoder_outputs)
[Link](optimizer='adam', loss='sparse_categorical_crossentropy')

# Train Model
epochs = 10
y_train = np.expand_dims(target_sequences, -1)
[Link]([source_sequences, target_sequences], y_train, epochs=epochs)

# Inference Function
def translate_sentence(sentence):
seq = source_tokenizer.texts_to_sequences([nltk.word_tokenize([Link]())])
seq = pad_sequences(seq, maxlen=max_length, padding='post')
prediction = [Link]([seq, seq])
predicted_ids = [Link](prediction[0], axis=-1)
return ' '.join([target_tokenizer.index_word[idx] for idx in predicted_ids if idx > 0])

# Example Translation
test_sentence = "hello world"
translation = translate_sentence(test_sentence)
print("Source:", test_sentence)
print("Translation:", translation)

OUTPUT:

To enable the following instructions: SSE SSE2 SSE3 SSE4.1 SSE4.2 AVX AVX2 AVX_VNNI
FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.

Dept of AI & ML Page 28

SIRMVIT Natural Language Processing (BAI601)

Epoch 1/10

1/1 [==============================] - 6s 6s/step - loss: 1.9479

Epoch 2/10

1/1 [==============================] - 0s 14ms/step - loss: 1.9305

Epoch 3/10

1/1 [==============================] - 0s 19ms/step - loss: 1.9130

Epoch 4/10

1/1 [==============================] - 0s 17ms/step - loss: 1.8948

Epoch 5/10

1/1 [==============================] - 0s 15ms/step - loss: 1.8756

Epoch 6/10

1/1 [==============================] - 0s 16ms/step - loss: 1.8548

Epoch 7/10

1/1 [==============================] - 0s 16ms/step - loss: 1.8319

Epoch 8/10

1/1 [==============================] - 0s 16ms/step - loss: 1.8065

Epoch 9/10

1/1 [==============================] - 0s 13ms/step - loss: 1.7780

Epoch 10/10

1/1 [==============================] - 0s 14ms/step - loss: 1.7459

1/1 [==============================] - 2s 2s/step

Source: hello world

Translation: hola mundo

mundo

Dept of AI & ML Page 29