NLP – Task 2:
import nltk
from nltk import word_tokenize, pos_tag
from [Link] import ne_chunk
from [Link] import FreqDist
from [Link] import bigrams, trigrams
from [Link] import NaiveBayesClassifier
from [Link] import accuracy
[Link]('punkt_tab')
[Link]('averaged_perceptron_tagger')
[Link]('averaged_perceptron_tagger_eng')
[Link]('maxent_ne_chunker')
[Link]('maxent_ne_chunker_tab')
[Link]('words')
[Link]('stopwords')
[Link]('treebank')
[Link]('conll2000')
[Link]('wordnet')
[Link]('omw-1.4')
# Sample Feedback
feedback = """
Dr. Ravi at Apollo Hospital in Chennai provided excellent treatment.
The staff were friendly and the service was outstanding.
"""
# --------------------------------------------------
# 1. Parsing (POS Tagging + Simple Parsing)
# --------------------------------------------------
tokens = word_tokenize(feedback)
tagged = pos_tag(tokens)
print("POS Tags:")
print(tagged)
# --------------------------------------------------
# 2. Named Entity Recognition (Chunking)
# --------------------------------------------------
print("\nNamed Entity Recognition:")
ner_tree = ne_chunk(tagged)
print(ner_tree)
# --------------------------------------------------
# 3. Classification
# --------------------------------------------------
# Training Data
train_data = [
({"excellent": True, "friendly": True}, "Positive"),
({"outstanding": True, "good": True}, "Positive"),
({"poor": True, "bad": True}, "Negative"),
({"terrible": True, "rude": True}, "Negative")
classifier = [Link](train_data)
# Feature Extraction
features = {
"excellent": "excellent" in [Link](),
"friendly": "friendly" in [Link](),
"outstanding": "outstanding" in [Link]()
}
category = [Link](features)
print("\nFeedback Category:")
print(category)
# --------------------------------------------------
# 4. Evaluation Metrics
# --------------------------------------------------
test_data = [
({"excellent": True, "friendly": True}, "Positive"),
({"poor": True, "bad": True}, "Negative")
acc = accuracy(classifier, test_data)
print("\nModel Accuracy:")
print(round(acc * 100, 2), "%")
# --------------------------------------------------
# 5. Frequency Distribution
# --------------------------------------------------
words = [[Link]() for word in tokens if [Link]()]
fdist = FreqDist(words)
print("\nWord Frequency Distribution:")
for word, freq in fdist.most_common(10):
print(word, ":", freq)
# --------------------------------------------------
# 6. Probability Distribution
# --------------------------------------------------
total_words = len(words)
print("\nProbability Distribution:")
for word, freq in fdist.most_common(5):
probability = freq / total_words
print(f"P({word}) = {freq}/{total_words} = {probability:.2f}")
# --------------------------------------------------
# 7. Semantic Analysis (Simple Meaning Mapping)
# --------------------------------------------------
semantic_dict = {
"excellent": "Positive Quality",
"friendly": "Positive Behaviour",
"treatment": "Medical Service",
"service": "Hospital Facility"
print("\nSemantic Analysis:")
for word in words:
if word in semantic_dict:
print(word, "->", semantic_dict[word])
# --------------------------------------------------
# 8. Bigrams
# --------------------------------------------------
print("\nBigrams:")
for bg in bigrams(words):
print(bg)
# --------------------------------------------------
# 9. Trigrams
# --------------------------------------------------
print("\nTrigrams:")
for tg in trigrams(words):
print(tg)
Explanation:
word_tokenize → splits sentence into words
pos_tag → assigns grammar tags (noun, verb, etc.)
ne_chunk → finds names like Person, Location, Organization
FreqDist → counts word frequency
bigrams/trigrams → creates word pairs/groups
NaiveBayesClassifier → machine learning classifier
accuracy → evaluates model performance
tokens = word_tokenize(feedback)
tagged = pos_tag(tokens)
Splits text into words:
Example:
["Dr.", "Ravi", "at", "Apollo", "Hospital", ...]
POS Tagging
Assigns grammar role:
Dr. → NNP (Proper noun)
Ravi → NNP
provided → VBD (Verb)
excellent → JJ (Adjective)
ner_tree = ne_chunk(tagged)
What it does:
Finds:
Person → Dr. Ravi
Organization → Apollo Hospital
Location → Chennai
Classification
Words present Sentiment
excellent + friendly Positive
poor + bad Negative
1. [Link] (Parsing)
Meaning:
Parsing is the process of analyzing a sentence to understand its grammatical structure (syntax).
Important Terms:
Syntax: Sentence structure
Parse Tree: Tree representation of grammar
NP: Noun Phrase
VP: Verb Phrase
S: Sentence
Types:
1. Shift-Reduce Parsing
2. Recursive Descent Parsing
3. Chart Parsing
Example:
Sentence: “Ravi reads a book”
Parse Output:
S → NP VP
NP → Ravi
VP → reads a book
Explanation:
“Ravi” is NP (subject)
“reads a book” is VP (action + object)
Applications:
Grammar checking
Chatbots
Machine translation
2. [Link] (Chunking / NER)
Meaning:
Chunking groups words into meaningful chunks (phrases or entities).
Important Terms:
Chunk: Group of words
NER: Named Entity Recognition
POS Tags: Part-of-Speech tags
NNP: Proper noun
NN: Noun
Types:
1. NP Chunking (Noun Phrase)
2. NER (Named Entity Recognition)
3. Regex Chunking
Example:
Sentence: “Dr. Ravi works at Apollo Hospital in Chennai”
Output:
Person → Dr. Ravi
Organization → Apollo Hospital
Location → Chennai
Explanation:
NNP tags help identify proper names
Groups words into entities
Applications:
Resume extraction
News analysis
Information retrieval
3. [Link] (Classification)
Meaning:
Classification assigns a label/category to text using machine learning.
Important Terms:
Feature: Input word information
Class/Label: Output category
Training Data: Data used to teach model
Test Data: Data used for checking
Types:
1. Naive Bayes Classifier
2. Decision Tree Classifier
3. Maximum Entropy (MaxEnt)
Example:
Sentence: “The service is excellent”
Output:
Category → Positive
Formula (Naive Bayes):
P(C|X) = (P(X|C) × P(C)) / P(X)
Where:
C = class (Positive/Negative)
X = features (words)
Applications:
Sentiment analysis
Spam detection
Topic classification
4. [Link] (Evaluation)
Meaning:
Used to measure performance of NLP models.
Important Terms:
TP: True Positive
FP: False Positive
FN: False Negative
TN: True Negative
Types & Formulas:
Accuracy:
Accuracy = Correct Predictions / Total Predictions
Precision:
Precision = TP / (TP + FP)
Recall:
Recall = TP / (TP + FN)
F1 Score:
F1 = 2 × (Precision × Recall) / (Precision + Recall)
Example:
If 90 correct out of 100:
Accuracy = 90%
Applications:
Model evaluation
Performance comparison
5. [Link]
Meaning:
Analyzes frequency and probability of words.
Important Terms:
Frequency: Number of occurrences
Distribution: Word count pattern
Corpus: Collection of text
Types:
1. FreqDist (Frequency Distribution)
2. Conditional Frequency Distribution
3. Probability Distribution
Formula:
P(word) = Frequency of word / Total words
Example:
Text: “service service good”
service = 2
good = 1
P(service) = 2/3 = 0.67
Applications:
Keyword extraction
Language modeling
Text analysis
6. [Link] (Semantics)
Meaning:
Deals with meaning of words and sentences.
Important Terms:
Semantics: Meaning of language
Lexical: Word-level meaning
Compositional: Sentence-level meaning
Logical Form: Structured meaning
Types:
1. Lexical Semantics
2. Compositional Semantics
3. Logical Semantics
Example:
excellent → positive quality
friendly → positive behavior
Formula:
Meaning(Sentence) = f(words)
Applications:
Chatbots
Question answering
Sentiment interpretation
7. [Link]
Meaning:
Provides tools for text processing and n-gram generation.
Important Terms:
n-gram: Sequence of n words
Bigram: 2-word sequence
Trigram: 3-word sequence
Types:
1. Bigrams
2. Trigrams
3. N-grams
Formulas:
Bigram: (wi, wi+1)
Trigram: (wi, wi+1, wi+2)
n-gram: (wi ... wi+n-1)
Example:
Sentence: “I love NLP”
Bigrams:
(I, love), (love, NLP)
Trigrams:
(I, love, NLP)
Applications:
Next word prediction
Speech recognition
Language modeling
Module Meaning Key Term Example
parse sentence structure NP, VP Ravi → reads book
chunk entity grouping NER Dr. Ravi → Person
classify labeling text feature, class Positive
metrics evaluation TP, FP accuracy 90%
probability word stats freq, P(word) P=2/3
sem meaning lexical, logical excellent → positive
util word patterns bigram, trigram (I, love)
Code example
import nltk
from nltk import word_tokenize, pos_tag, ne_chunk
[Link]('maxent_ne_chunker')
[Link]('maxent_ne_chunker_tab')
[Link]('words')
text = "Dr. Ravi works at Apollo Hospital in Chennai"
tokens = word_tokenize(text)
tagged = pos_tag(tokens)
print("Named Entities:")
print(ne_chunk(tagged))
Outpupt:
Named Entities:
(S
Dr./NNP
Ravi/NNP
works/VBZ
at/IN
(ORGANIZATION Apollo/NNP Hospital/NNP)
in/IN
(GPE Chennai/NNP))
Tag Meaning Example
NNP Proper noun Ravi, Apollo
VBZ Verb (3rd person singular) works
IN Preposition at, in
Classification
from [Link] import NaiveBayesClassifier
from [Link] import accuracy
train_data = [
({"good": True, "excellent": True}, "Positive"),
({"bad": True, "poor": True}, "Negative")
classifier = [Link](train_data)
test_data = [
({"good": True}, "Positive"),
({"bad": True}, "Negative")
print("Accuracy:", accuracy(classifier, test_data))
sample = {"good": True, "excellent": False, "bad": False}
print("Prediction:", [Link](sample))
Output:
Accuracy: 1.0
Prediction: Positive
Probability / Frequency Distribution
import nltk
from nltk import word_tokenize
from [Link] import FreqDist
[Link]('punkt')
[Link]('punkt_tab')
text = "service service good excellent service"
tokens = word_tokenize(text)
fd = FreqDist(tokens)
print("Frequency Distribution:")
print(fd)
total = len(tokens)
print("\nProbability:")
for word, freq in [Link]():
print(word, "=", freq/total)
Utilities
import nltk
from nltk import word_tokenize
from [Link] import bigrams, trigrams
[Link]('punkt')
[Link]('punkt_tab')
text = "I love natural language processing"
tokens = word_tokenize(text)
print("Bigrams:")
print(list(bigrams(tokens)))
print("\nTrigrams:")
print(list(trigrams(tokens)))
Output:
Bigrams:
[('I', 'love'), ('love', 'natural'), ('natural', 'language'), ('language', 'processing')]
Trigrams:
[('I', 'love', 'natural'), ('love', 'natural', 'language'), ('natural', 'language', 'processing')]