JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
ASSIGNMENT 1:
Write a Python program to perform tokenization by word and sentence using NLTK.
Soln- import nltk
from [Link] import word_tokenize, sent_tokenize
[Link]('punkt')
text = """
Natural Language Processing (NLP) is a field of Artificial Intelligence.
It helps computers understand human language.
Tokenization is the first step in NLP.
"""
sentences = sent_tokenize(text)
print("Sentence Tokenization:")
for i, sentence in enumerate(sentences, 1):
print(f"{i}. {sentence}")
words = word_tokenize(text)
print("\nWord Tokenization:")
print(words)
OUTPUT-
Name: University Roll No.: 123231211 Page No.:
JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
ASSIGNMENT 2:
Write a Python program to eliminate stopwords using NLTK.
Soln- import nltk
from [Link] import stopwords
from [Link] import word_tokenize
[Link]('punkt')
[Link]('stopwords')
[Link]('punkt_tab')
text = "Natural Language Processing is a fascinating field of Artificial Intelligence."
words = word_tokenize(text)
stop_words = set([Link]('english'))
filtered_words = [word for word in words if [Link]() not in stop_words]
print("Original Words:")
print(words)
print("\nWords after removing stopwords:")
print(filtered_words)
OUTPUT-
Name: University Roll No.: 123231211 Page No.:
JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
ASSIGNMENT 3:
Write a Python program to perform stemming using NLTK.
Soln- import nltk
from [Link] import stopwords
from [Link] import word_tokenize
[Link]('punkt')
[Link]('stopwords')
[Link]('punkt_tab')
text = "Natural Language Processing is a fascinating field of Artificial Intelligence."
words = word_tokenize(text)
stop_words = set([Link]('english'))
filtered_words = [word for word in words if [Link]() not in stop_words]
print("Original Words:")
print(words)
print("\nWords after removing stopwords:")
print(filtered_words)
OUTPUT-
Name: University Roll No.: 123231211 Page No.:
JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
ASSIGNMENT 4:
Write a Python program to perform Parts of Speech (POS) tagging using NLTK.
Soln- import nltk
from [Link] import word_tokenize
[Link]('punkt')
[Link]('punkt_tab')
[Link]('averaged_perceptron_tagger')
[Link]('averaged_perceptron_tagger_eng')
text = "The quick brown fox jumps over the lazy dog."
words = word_tokenize(text)
pos_tags = nltk.pos_tag(words)
print("Words with POS Tags:\n")
for word, tag in pos_tags:
print(f"{word} --> {tag}")
OUTPUT-
Name: University Roll No.: 123231211 Page No.:
JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
ASSIGNMENT 5:
Write a Python program to perform lemmatization using NLTK.
Soln- import nltk
from [Link] import WordNetLemmatizer
from [Link] import word_tokenize
[Link]('punkt')
[Link]('punkt_tab')
[Link]('wordnet')
[Link]('omw-1.4')
text = "running studies leaves better"
words = word_tokenize(text)
lemmatizer = WordNetLemmatizer()
lemmatized_words = [[Link](word) for word in words]
print("Original Words:")
print(words)
print("\nLemmatized Words:")
print(lemmatized_words)
OUTPUT-
Name: University Roll No.: 123231211 Page No.:
JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
ASSIGNMENT 6:
Write a Python program for chunking using NLTK.
Soln- import nltk
from [Link] import word_tokenize
[Link]('punkt')
[Link]('punkt_tab')
[Link]('averaged_perceptron_tagger')
[Link]('averaged_perceptron_tagger_eng')
[Link]('maxent_ne_chunker')
[Link]('maxent_ne_chunker_tab')
[Link]('words')
text = "The quick brown fox jumps over the lazy dog."
words = word_tokenize(text)
pos_tags = nltk.pos_tag(words)
grammar = "NP: {<DT>?<JJ>*<NN>}"
chunk_parser = [Link](grammar)
chunk_result = chunk_parser.parse(pos_tags)
print("Chunked Tree:\n")
print(chunk_result)
chunk_result.draw()
OUTPUT-
Name: University Roll No.: 123231211 Page No.:
JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
ASSIGNMENT 7:
Write a Python program to perform Named Entity Recognition (NER) using NLTK.
Soln- import nltk
from [Link] import word_tokenize
[Link]('punkt')
[Link]('punkt_tab')
[Link]('averaged_perceptron_tagger')
[Link]('averaged_perceptron_tagger_eng')
[Link]('maxent_ne_chunker')
[Link]('maxent_ne_chunker_tab')
[Link]('words')
text = "Barack Obama was born in Hawaii and worked at Microsoft."
words = word_tokenize(text)
pos_tags = nltk.pos_tag(words)
ner_result = nltk.ne_chunk(pos_tags)
print("Named Entity Recognition Output:\n")
print(ner_result)
ner_result.draw()
OUTPUT-
Name: University Roll No.: 123231211 Page No.:
JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
ASSIGNMENT 8:
Write a Python program to find Term Frequency and Inverse Document Frequency (TF-IDF).
Soln- from sklearn.feature_extraction.text import TfidfVectorizer
documents = [
"Natural language processing is interesting",
"Machine learning is a part of artificial intelligence",
"Natural language processing and machine learning are related"
]
vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(documents)
feature_names = vectorizer.get_feature_names_out()
tfidf_array = tfidf_matrix.toarray()
print("TF-IDF Matrix:\n")
for i, doc in enumerate(tfidf_array):
print(f"Document {i+1}:")
for word, score in zip(feature_names, doc):
print(f"{word} : {score:.3f}")
print()
OUTPUT-
Name: University Roll No.: 123231211 Page No.:
JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
ASSIGNMENT 9:
Write a Python program for CYK parsing (Cocke-Younger-Kasami Parsing) or Chart Parsing.
Soln- from collections import defaultdict
grammar = {
('NP', 'VP'): ['S'],
('Det', 'N'): ['NP'],
('V', 'NP'): ['VP'],
('she',): ['NP'],
('eats',): ['V'],
('a',): ['Det'],
('fish',): ['N']
}
sentence = "she eats a fish"
words = [Link]()
n = len(words)
table = [[set() for j in range(n)] for i in range(n)]
for i, word in enumerate(words):
for rhs, lhs in [Link]():
if len(rhs) == 1 and rhs[0] == word:
table[i][i].update(lhs)
for length in range(2, n + 1):
for i in range(n - length + 1):
j = i + length - 1
for k in range(i, j):
left = table[i][k]
right = table[k + 1][j]
for B in left:
for C in right:
for rhs, lhs in [Link]():
if rhs == (B, C):
table[i][j].update(lhs)
print("CYK Parsing Table:\n")
for row in table:
print(row)
if 'S' in table[0][n - 1]:
print("\nThe sentence is grammatically correct.")
else:
print("\nThe sentence is NOT grammatically correct.")
Name: University Roll No.: 123231211 Page No.:
JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
OUTPUT-
Name: University Roll No.: 123231211 Page No.:
JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
ASSIGNMENT 10:
Write a Python program to find all unigrams, bigrams, and trigrams present in the given corpus.
Soln- import nltk
from [Link] import ngrams
from [Link] import word_tokenize
[Link]('punkt')
[Link]('punkt_tab')
text = "Natural Language Processing is very interesting"
tokens = word_tokenize(text)
unigrams = list(ngrams(tokens, 1))
bigrams = list(ngrams(tokens, 2))
trigrams = list(ngrams(tokens, 3))
print("Unigrams:")
print(unigrams)
print("\nBigrams:")
print(bigrams)
print("\nTrigrams:")
print(trigrams)
OUTPUT-
Name: University Roll No.: 123231211 Page No.:
JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
ASSIGNMENT 11:
Write a Python program to find the probability of the statement “This is my cat” using an example
corpus.
Soln- from collections import Counter
corpus = """
this is my dog
this is my cat
my cat is cute
this dog is friendly
"""
words = [Link]().split()
word_counts = Counter(words)
total_words = len(words)
sentence = "this is my cat"
sentence_words = [Link]().split()
probability = 1
for word in sentence_words:
word_prob = word_counts[word] / total_words
probability *= word_prob
print(f"P({word}) = {word_counts[word]}/{total_words} = {word_prob:.4f}")
print("\nProbability of the sentence:")
print(probability)
OUTPUT-
Name: University Roll No.: 123231211 Page No.:
JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
ASSIGNMENT 12:
Use the Stanford Named Entity Recognizer to extract entities from documents and identify their types.
Soln- import nltk
from [Link] import StanfordNERTagger
from [Link] import word_tokenize
classifier = r"C:\Users\kunda\Desktop\NLP\stanford-ner-2020-11-
17\classifiers\[Link]"
jar = r"C:\Users\kunda\Desktop\NLP\stanford-ner-2020-11-17\[Link]"
st = StanfordNERTagger(classifier, jar, encoding='utf-8')
text = "Barack Obama was born in Hawaii and worked at Microsoft."
tokens = word_tokenize(text)
classified_text = [Link](tokens)
print("Named Entities and their Types:\n")
for word, entity in classified_text:
print(f"{word} --> {entity}")
OUTPUT-
Name: University Roll No.: 123231211 Page No.:
JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
ASSIGNMENT 13:
Design and implement a Transformer-based NLP system (e.g., using BERT or GPT) for multi-class text
classification (such as fake news, sentiment, and spam combined). Evaluate performance using
accuracy, F1-score, and confusion matrix, and optimize for inference latency.
Soln- import pandas as pd
import torch
from transformers import (
BertTokenizer,
BertForSequenceClassification,
Trainer,
TrainingArguments
)
from datasets import Dataset
from [Link] import accuracy_score, f1_score, confusion_matrix
import numpy as np
import [Link] as plt
data = {
"text": [
"Win money now",
"I love this product",
"Breaking fake political news",
"This movie is amazing",
"Claim your free reward",
"The news is completely false",
"Excellent customer support",
"Spam offers available"
],
"label": [0,1,2,1,0,2,1,0]
}
df = [Link](data)
dataset = Dataset.from_pandas(df)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
def tokenize_function(example):
return tokenizer(
example["text"],
padding="max_length",
truncation=True,
max_length=32
)
tokenized_dataset = [Link](tokenize_function)
split_dataset = tokenized_dataset.train_test_split(test_size=0.25)
Name: University Roll No.: 123231211 Page No.:
JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
train_dataset = split_dataset['train']
test_dataset = split_dataset['test']
model = BertForSequenceClassification.from_pretrained(
'bert-base-uncased',
num_labels=3
)
def compute_metrics(eval_pred):
logits, labels = eval_pred
predictions = [Link](logits, axis=-1)
acc = accuracy_score(labels, predictions)
f1 = f1_score(labels, predictions, average='weighted')
return {
'accuracy': acc,
'f1_score': f1
}
training_args = TrainingArguments(
output_dir="./results",
eval_strategy="epoch",
save_strategy="epoch",
per_device_train_batch_size=2,
per_device_eval_batch_size=2,
num_train_epochs=3,
logging_dir="./logs",
logging_steps=10,
load_best_model_at_end=True
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=train_dataset,
eval_dataset=test_dataset,
compute_metrics=compute_metrics
)
[Link]()
predictions = [Link](test_dataset)
pred_labels = [Link]([Link], axis=-1)
true_labels = predictions.label_ids
accuracy = accuracy_score(true_labels, pred_labels)
Name: University Roll No.: 123231211 Page No.:
JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
f1 = f1_score(true_labels, pred_labels, average='weighted')
print("\nModel Evaluation")
print("-----------------------")
print("Accuracy :", accuracy)
print("F1 Score :", f1)
cm = confusion_matrix(true_labels, pred_labels)
print("\nConfusion Matrix")
print(cm)
[Link](cm)
[Link]("Confusion Matrix")
[Link]("Predicted Labels")
[Link]("True Labels")
[Link]()
[Link]()
print("\nInference Latency Optimization Techniques:")
print("1. Use DistilBERT instead of BERT")
print("2. Apply Dynamic Quantization")
print("3. Reduce input token length")
print("4. Use GPU/ONNX Runtime")
OUTPUT-
Name: University Roll No.: 123231211 Page No.:
JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
ASSIGNMENT 14:
Develop an end-to-end Explainable NLP pipeline that integrates deep learning (LSTM/Transformer)
with XAI techniques (such as SHAP or LIME) to interpret model predictions on real-world datasets
(e.g., healthcare or social media text). Include deployment using a web interface (Streamlit/Gradio).
Soln- import pandas as pd
import torch
import shap
from transformers import (
pipeline,
AutoTokenizer,
AutoModelForSequenceClassification
)
model_name = "distilbert-base-uncased-finetuned-sst-2-english"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(model_name)
classifier = pipeline(
"text-classification",
model=model,
tokenizer=tokenizer
)
texts = [
"I absolutely love this product!",
"This is the worst service ever.",
"The hospital staff were very supportive.",
"Social media is spreading fake information."
]
print("\nPredictions:\n")
for text in texts:
result = classifier(text)
print("Text :", text)
print("Prediction :", result)
print()
print("\nGenerating SHAP explanations...\n")
explainer = [Link](classifier)
shap_values = explainer(texts)
[Link](shap_values)
Name: University Roll No.: 123231211 Page No.:
JIS COLLEGE OF ENGINEERING
DEPARTMENT OF B Tech CSE (AIML)
Experiment Name: ____________________________________________________________
df = [Link]({
"Text": texts,
"Prediction": [classifier(t)[0]['label'] for t in texts]
})
df.to_csv("[Link]", index=False)
print("\nPredictions saved to [Link]")
print("\nOptimization Techniques:")
print("1. Use DistilBERT instead of BERT")
print("2. Dynamic Quantization")
print("3. ONNX Runtime")
print("4. Reduce sequence length")
print("5. GPU inference")
OUTPUT-
Name: University Roll No.: 123231211 Page No.: