Scribd
Upload
5 views13 pages

NLP

The document contains multiple Python experiments demonstrating Natural Language Processing (NLP) techniques, including word-level analysis, word generation using Markov chains and bigrams, morphological analysis using NLTK and Hugging Face, and N-Gram models for text analysis. Each experiment includes code snippets, outputs, and explanations of the methods used. The experiments cover various NLP tasks such as tokenization, filtering stopwords, and generating text based on learned patterns.

Uploaded by

modiyampurushothamreddy
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
5 views13 pages

NLP

The document contains multiple Python experiments demonstrating Natural Language Processing (NLP) techniques, including word-level analysis, word generation using Markov chains and bigrams, morphological analysis using NLTK and Hugging Face, and N-Gram models for text analysis. Each experiment includes code snippets, outputs, and explanations of the methods used. The experiments cover various NLP tasks such as tokenization, filtering stopwords, and generating text based on learned patterns.

Uploaded by

modiyampurushothamreddy
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

REG NO:24095A3213

EXPERIMNET 1:Write a Python program to perform word-level analysis on a sample


paragraph using Natural Language Processing (NLP) techniques.
import string
from collections import Counter
text = """Natural Language processing is a branch of Artificial Intelligence.
It helps computers interpret and generate human understanding.
Widely used in sentiment analysis, chatbots, and search engines."""
print("Original Text:\n", text)
text_lower = [Link]()
text_no_punct = text_lower.translate([Link]("", "", [Link]))
print("\nText without punctuation:\n", text_no_punct)
words = text_no_punct.split()
print("\nTokenized Words:\n", words)
stopwords = {"and", "it", "to", "is", "a", "of", "in", "the"}
filtered_words = [word for word in words if word not in stopwords]
print("\nFiltered Words (no stopwords):\n", filtered_words)
word_freq = Counter(filtered_words)
print("\nWord Frequency:\n", word_freq)

OUTPUT:
Original Text:
Natural Language processing is a branch of Artificial Intelligence.
It helps computers interpret and generate human understanding.
Widely used in sentiment analysis, chatbots, and search engines.
Text without punctuation:
natural language processing is a branch of artificial intelligence
it helps computers interpret and generate human understanding
widely used in sentiment analysis chatbots and search engines

Tokenized Words:
['natural', 'language', 'processing', 'is', 'a', 'branch', 'of', 'artificial', 'intelligence', 'it', 'helps',
'computers', 'interpret', 'and', 'generate', 'human', 'understanding', 'widely', 'used', 'in',
'sentiment', 'analysis', 'chatbots', 'and', 'search', 'engines']

Filtered Words (no stopwords):


['natural', 'language', 'processing', 'branch', 'artificial', 'intelligence', 'helps', 'computers',
'interpret', 'generate', 'human', 'understanding', 'widely', 'used', 'sentiment', 'analysis',
'chatbots', 'search', 'engines']

Word Frequency:
Counter({'natural': 1, 'language': 1, 'processing': 1, 'branch': 1, 'artificial': 1, 'intelligence': 1,
'helps': 1, 'computers': 1, 'interpret': 1, 'generate': 1, 'human': 1, 'understanding': 1, 'widely': 1,
'used': 1, 'sentiment': 1, 'analysis': 1, 'chatbots': 1, 'search': 1, 'engines': 1})
EXPERIMNET 1:Write a Python program to perform word-level analysis on a sample
paragraph using Natural Language Processing Tool Kit (NLTK)
import nltk
import string
from [Link] import stopwords
from [Link] import word_tokenize
# Download required resources
[Link]('punkt')
[Link]('punkt_tab') # <-- add this line
[Link]('stopwords')
text = "Hello there! This is an example sentence, showing how to remove stop words and
punctuation."
tokens = word_tokenize(text1)
print(tokens)
stop_words = set([Link]('english'))
clean_tokens = [
word for word in tokens
if [Link]() not in stop_words and word not in [Link]
]
print(clean_tokens)
clean_text = " ".join(clean_tokens)
print("Cleaned Tokens:", clean_tokens)
print("Cleaned Text:", clean_text)
OUTPUT:
['Hello', 'there!', 'This', 'an', 'example', 'sentence,', 'showing', 'how', 'remove', 'stop', 'words',
'punctuation.']
['Hello', 'there!', 'This', 'an', 'example', 'sentence,', 'showing', 'how', 'remove', 'stop', 'words',
'punctuation.']
Tokens: ['Hello', 'there', '!', 'This', 'is', 'an', 'example', 'sentence', ',', 'showing', 'how', 'to',
'remove', 'stop', 'words', 'and', 'punctuation', '.']
Cleaned Tokens: ['Hello', 'example', 'sentence', 'showing', 'remove', 'stop', 'words',
'punctuation']
Cleaned Text: Hello example sentence showing remove stop words punctuation
EXPERIMNET 1:Write a Python program to perform word-level analysis on a sample
paragraph using Natural Language Processing USING SPACY
import spacy
# Load the small English model
nlp = [Link]("en_core_web_sm")
# Sample text
text = "Natural Language Processing (NLP) is a field that combines computer science,
artificial intelligence and language studies."
# Process text with spaCy
doc = nlp(text)
print(doc)
# --- Tokenization ---
tokens = [[Link] for token in doc]
print("All Tokens:", tokens)
# --- Remove punctuation tokens ---
tokens_no_punct = [[Link] for token in doc if not token.is_punct]
print("Without Punctuation:", tokens_no_punct)
# --- Remove stop words + punctuation ---
tokens_clean = [[Link] for token in doc if not token.is_stop and not token.is_punct]
print("Clean Tokens:", tokens_clean)
# --- Lemmatized clean tokens (optional) ---
lemmas_clean = [token.lemma_ for token in doc if not token.is_stop and not token.is_punct]
print("Clean Lemmas:", lemmas_clean)
lemma_freq = Counter(lemmas_clean)
print(lemma_freq)

OUTPUT:

Natural Language Processing (NLP) is a field that combines computer science, artificial
intelligence and language [Link] Tokens: ['Natural', 'Language', 'Processing', '(', 'NLP', ')',
'is', 'a', 'field', 'that', 'combines', 'computer', 'science', ',', 'artificial', 'intelligence', 'and',
'language', 'studies', '.']

Without Punctuation: ['Natural', 'Language', 'Processing', 'NLP', 'is', 'a', 'field', 'that',
'combines', 'computer', 'science', 'artificial', 'intelligence', 'and', 'language', 'studies']

Clean Tokens: ['Natural', 'Language', 'Processing', 'NLP', 'field', 'combines', 'computer',


'science', 'artificial', 'intelligence', 'language', 'studies']

Clean Lemmas: ['Natural', 'Language', 'Processing', 'NLP', 'field', 'combine', 'computer',


'science', 'artificial', 'intelligence', 'language', 'study']
Counter({'Natural': 1, 'Language': 1, 'Processing': 1, 'NLP': 1, 'field': 1, 'combine': 1,
'computer': 1, 'science': 1, 'artificial': 1, 'intelligence': 1, 'language': 1, 'study': 1})
REG NO:24095A3213

EXPERIMNET 2:Implement a Python program to perform word generation using Natural


Language Processing techniques USING MARKOV CHAIN
Import random
# Sample text
text = """I love NLP. NLP is fun. I love learning new things related to NLP and machine
learning."""
# Tokenize into words
tokens = [Link]()
# Build Markov chain dictionary
markov_chain = {}
for i in range(len(tokens) - 1):
current_word = tokens[i]
next_word = tokens[i + 1]
if current_word not in markov_chain:
markov_chain[current_word] = []
markov_chain[current_word].append(next_word)

# Function to generate text


def generate_text(chain, start_word, length=10):
word = start_word
output = [word]
for _ in range(length - 1):
if word in chain:
word = [Link](chain[word])
[Link](word)
else:
break
return " ".join(output)

# Example usage
print(generate_text(markov_chain, "I", 12))

OUTPUT:
I love NLP. NLP and machine learning.
EXPERIMNET 2:Implement a Python program to perform word generation using Natural
Language Processing techniques USING BIGRAM
import nltk
from [Link] import word_tokenize
from collections import Counter
[Link]('punkt')
text = "I love machine learning NLP and I love NLP"
# Tokenize into words
words = word_tokenize(text)
print("Words:", words)
# Build bigrams
bigrams = [(words[i], words[i+1]) for i in range(len(words)-1)]
print("Bigrams:", bigrams)
# Count bigram frequencies
bigram_freq = Counter(bigrams)
print("\nBigram Frequency:")
for bg, count in bigram_freq.items():
print(bg, ":", count)

OUTPUT:
Words: ['I', 'love', 'machine', 'learning', 'NLP', 'and', 'I', 'love', 'NLP']
Bigrams: [('I', 'love'), ('love', 'machine'), ('machine', 'learning'), ('learning', 'NLP'), ('NLP',
'and'), ('and', 'I'), ('I', 'love'), ('love', 'NLP')]

Bigram Frequency:
('I', 'love') : 2
('love', 'machine') : 1
('machine', 'learning') : 1
('learning', 'NLP') : 1
('NLP', 'and') : 1
('and', 'I') : 1
('love', 'NLP') : 1
REG NO:24095A3213

EXPERIMENT 3:Write a Python program to perform morphological analysis on a given set


of words using NLP techniques. Your task is to analyze the internal structure of words by
identifying and extracting the root words (lemmas), prefixes, and suffixes where applicable
implement USING NLTK
import nltk
from [Link] import WordNetLemmatizer
# Download required resources
[Link]('wordnet')
[Link]('omw-1.4')
# Initialize lemmatizer
lemmatizer = WordNetLemmatizer()
# Sample set of words
words = ["running", "unhappy", "happiness", "replaying", "cats", "better"]
def morphological_analysis(word):
# Lemma (root word)
lemma = [Link](word)
# Simple prefix detection (common English prefixes)
prefixes = ["un", "re", "in", "dis", "pre", "mis", "non"]
prefix = None
for p in prefixes:
if [Link](p):
prefix = p
break
# Simple suffix detection (common English suffixes)
suffixes = ["ing", "ed", "ly", "s", "es", "ness", "ment", "er"]
suffix = None
for s in suffixes:
if [Link](s):
suffix = s
break
return {
"Word": word,
"Lemma": lemma,
"Prefix": prefix if prefix else "None",
"Suffix": suffix if suffix else "None"
}
# Perform analysis
for w in words:
result = morphological_analysis(w)
print(result)
OUTPUT:
{'Word': 'running', 'Lemma': 'running', 'Prefix': 'None', 'Suffix': 'ing'}
{'Word': 'unhappy', 'Lemma': 'unhappy', 'Prefix': 'un', 'Suffix': 'None'}
{'Word': 'happiness', 'Lemma': 'happiness', 'Prefix': 'None', 'Suffix': 's'}
{'Word': 'replaying', 'Lemma': 'replaying', 'Prefix': 're', 'Suffix': 'ing'}
{'Word': 'cats', 'Lemma': 'cat', 'Prefix': 'None', 'Suffix': 's'}
{'Word': 'better', 'Lemma': 'better', 'Prefix': 'None', 'Suffix': 'er'}
EXPERIMENT 3:Write a Python program to perform morphological analysis on a given set
of words using NLP techniques. Your task is to analyze the internal structure of words by
identifying and extracting the root words (lemmas), prefixes, and suffixes where applicable
implement using HUGGING FACE
!pip install torch transformers
from transformers import AutoTokenizer, AutoModelForTokenClassification, pipeline
# Load a pretrained model for token classification (POS tagging / morphological analysis)
# Here we use a BERT model fine-tuned for POS tagging
model_name = "vblagoje/bert-english-uncased-finetuned-pos"
# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name)
# Create pipeline
nlp = pipeline("token-classification", model=model, tokenizer=tokenizer,
aggregation_strategy="simple")
# Sample text
text = "The childrens are playing"
# Run analysis
output = nlp(text)
# Print results
for token in output:
print(token)
OUTPUT:
{'entity_group': 'DET', 'score': np.float32(0.9994991), 'word': 'the', 'start': 0, 'end': 3}
{'entity_group': 'NOUN', 'score': np.float32(0.99824), 'word': 'children', 'start': 4, 'end': 12}
{'entity_group': 'PRON', 'score': np.float32(0.48870438), 'word': '##s', 'start': 12, 'end': 13}
{'entity_group': 'AUX', 'score': np.float32(0.9975871), 'word': 'are', 'start': 14, 'end': 17}
{'entity_group': 'VERB', 'score': np.float32(0.99942446), 'word': 'playing', 'start': 18, 'end':
25}
REG NO:24095A3213

EXPERIMENT 4:Write a Python program to implement N-Gram models (Unigram, Bigram,


and Trigram) for analyzing and generating sequences from a given text corpus USING NTLK
tool
from collections import Counter
rom [Link] import word_tokenize
from [Link] import ngrams
import nltk
[Link]('punkt')
[Link]('punkt_tab') # <-- required for word_tokenize
text = "I love NLP. I love AI. I love NLP."
tokens = word_tokenize([Link]())
print("Tokens:", tokens)
def generate_ngrams(tokens, n):
return list(ngrams(tokens, n))
unigrams = generate_ngrams(tokens, 1)
bigrams = generate_ngrams(tokens, 2)
trigrams = generate_ngrams(tokens, 3)
unigram_counts = Counter(unigrams)
bigram_counts = Counter(bigrams)
trigram_counts = Counter(trigrams)
print("\nUnigrams:", unigrams)
print("\nBigrams:", bigrams)
print("\nTrigrams:", trigrams)
# Vocabulary size (unique words)
V = len(set(tokens))
def unigram_prob(word, k=1.0):
return (unigram_counts[(word,)] + k) / (sum(unigram_counts.values()) + k * V)
def bigram_prob(prev_word, word, k=1.0):
return (bigram_counts[(prev_word, word)] + k) / (unigram_counts[(prev_word,)] + k * V)
def trigram_prob(word1, word2, word3, k=1.0):
return (trigram_counts[(word1, word2, word3)] + k) / (bigram_counts[(word1, word2)] + k
* V)
print("\n--- UNIGRAM PROBABILITIES ---")
print("P('nlp') =", unigram_prob("nlp"))
print("P('python') =", unigram_prob("python")) # unseen word
print("\n--- BIGRAM PROBABILITIES ---")
print("P('nlp' | 'love') =", bigram_prob("love", "nlp"))
print("P('python' | 'love') =", bigram_prob("love", "python")) # unseen
print("\n--- TRIGRAM PROBABILITIES ---")
print("P('nlp' | 'i love') =", trigram_prob("i", "love", "nlp"))
print("P('python' | 'i love') =", trigram_prob("i", "love", "python")) # unseen

OUTPUT:
Tokens: ['i', 'love', 'nlp', '.', 'i', 'love', 'ai', '.', 'i', 'love', 'nlp', '.']
Unigrams: [('i',), ('love',), ('nlp',), ('.',), ('i',), ('love',), ('ai',), ('.',), ('i',), ('love',), ('nlp',), ('.',)]
Bigrams: [('i', 'love'), ('love', 'nlp'), ('nlp', '.'), ('.', 'i'), ('i', 'love'), ('love', 'ai'), ('ai', '.'), ('.', 'i'), ('i',
'love'), ('love', 'nlp'), ('nlp', '.')]
Trigrams: [('i', 'love', 'nlp'), ('love', 'nlp', '.'), ('nlp', '.', 'i'), ('.', 'i', 'love'), ('i', 'love', 'ai'), ('love',
'ai', '.'), ('ai', '.', 'i'), ('.', 'i', 'love'), ('i', 'love', 'nlp'), ('love', 'nlp', '.')]

--- UNIGRAM PROBABILITIES ---


P('nlp') = 0.17647058823529413
P('python') = 0.058823529411764705
--- BIGRAM PROBABILITIES ---
P('nlp' | 'love') = 0.375
P('python' | 'love') = 0.125
EXPERIMENT 4:Write a Python program to implement N-Gram models (Unigram, Bigram,
and Trigram) for analyzing and generating sequences from a given text corpus USING
SPACY
import spacy

from collections import Counter


from itertools import islice
# Load spaCy
nlp = [Link]("en_core_web_sm")
# Training text
text = "I love NLP. I love AI. I love NLP."
doc = nlp([Link]())
tokens = [[Link] for t in doc if t.is_alpha]
V = len(set(tokens))
# Build n-grams
unigrams = list(zip(tokens))
bigrams = list(zip(tokens, islice(tokens, 1, None)))
trigrams = list(zip(tokens, islice(tokens, 1, None), islice(tokens, 2, None)))
# Count frequencies
uni_counts = Counter(unigrams)
bi_counts = Counter(bigrams)
tri_counts = Counter(trigrams)
# Generic add-k smoothing function
def prob(counts, context_counts, ngram, context, k=1.0):
return (counts[ngram] + k) / (context_counts[context] + k * V)
print("Unigrams:", unigrams)
print("P('nlp') =", prob(uni_counts, Counter(), ("nlp",), None, k=1))
print("P('python') =", prob(uni_counts, Counter(), ("python",), None, k=1))
print("\nBigrams:", bigrams)
print("P('nlp' | 'love') =", prob(bi_counts, uni_counts, ("love","nlp"), ("love",), k=1))
print("P('python' | 'love') =", prob(bi_counts, uni_counts, ("love","python"), ("love",), k=1))
print("\nTrigrams:", trigrams)
print("P('nlp' | 'i love') =", prob(tri_counts, bi_counts, ("i","love","nlp"), ("i","love"), k=1))
print("P('python' | 'i love') =", prob(tri_counts, bi_counts, ("i","love","python"), ("i","love"), k=1))
OUTPUT:
Unigrams: [('i',), ('love',), ('nlp',), ('i',), ('love',), ('ai',), ('i',), ('love',), ('nlp',)]
P('nlp') = 0.75
P('python') = 0.25

Bigrams: [('i', 'love'), ('love', 'nlp'), ('nlp', 'i'), ('i', 'love'), ('love', 'ai'), ('ai', 'i'), ('i', 'love'), ('love', 'nlp')]
P('nlp' | 'love') = 0.42857142857142855
P('python' | 'love') = 0.14285714285714285

Trigrams: [('i', 'love', 'nlp'), ('love', 'nlp', 'i'), ('nlp', 'i', 'love'), ('i', 'love', 'ai'), ('love', 'ai', 'i'), ('ai', 'i', 'love'),
('i', 'love', 'nlp')]
P('nlp' | 'i love') = 0.42857142857142855
P('python' | 'i love') = 0.14285714285714285
EXPERIMENT 4:Write a Python program to implement N-Gram models (Unigram, Bigram,
and Trigram) for analyzing and generating sequences from a given text corpus USING
HUGGING FACE
using hugging face:
from transformers import AutoTokenizer
from [Link] import ngrams
# Load a pretrained Hugging Face tokenizer (BERT base uncased)
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
# Sample text
text = "Natural language processing is important"
# Tokenize text
tokens = [Link](text)
print("Tokens:", tokens)
# Generate unigrams
unigrams = list(ngrams(tokens, 1))
print("\nUnigrams:", unigrams)
# Generate bigrams
bigrams = list(ngrams(tokens, 2))
print("\nBigrams:", bigrams)
# Generate trigrams
trigrams = list(ngrams(tokens, 3))
print("\nTrigrams:", trigrams)
# General function for n-grams
def generate_ngrams(tokens, n):
return list(ngrams(tokens, n))
# Example: 4-grams
fourgrams = generate_ngrams(tokens, 4)
print("\n4-grams:", fourgrams)
output:
Tokens: ['natural', 'language', 'processing', 'is', 'important']
Unigrams: [('natural',), ('language',), ('processing',), ('is',), ('important',)]
Bigrams: [('natural', 'language'), ('language', 'processing'),
('processing', 'is'), ('is', 'important')]
Trigrams: [('natural', 'language', 'processing'),

You might also like