Experiment - 1
Aim:- Study of Text Mining Tools.
a) IBM SPSS Tool
b) IBM Watson Tool
Solution
Study of Text Mining Tools
Text mining tools help extract meaningful information from unstructured text. They support tasks such as
classification, clustering, sentiment analysis, keyword extraction, topic modelling, and pattern discovery.
Among the widely used industry tools, IBM SPSS Text Analytics and IBM Watson are prominent because
they combine linguistic rules, machine learning, and advanced NLP capabilities. Below is a detailed, human-
like explanation of both tools.
a) IBM SPSS Text Analytics for Surveys (SPSS Text Mining Tool)
IBM SPSS Text Analytics is designed mainly for converting unstructured text—customer feedback,
survey responses, reviews, interview transcripts—into structured categories that can be analysed
statistically.
Key Features
1. Linguistic-Based Text Analysis
The tool uses a rich linguistic library. It does not just search for keywords; it understands word
forms, synonyms, compound nouns, abbreviations, and context.
Example: “delayed”, “delay”, “got late” may be grouped under one concept.
2. Automatic Category Creation
SPSS can automatically create categories from text and place responses into these categories. Users
can manually refine them for higher accuracy.
3. Integration With SPSS Statistics
After processing text, the structured data can be exported directly to SPSS Statistics for deeper
statistical analysis like cross-tabulation, frequency distribution, and predictive modeling.
4. Text Preprocessing Capabilities
Includes stemming, lemmatization, stop-word removal, phrase detection, and entity extraction
(names, locations, dates).
5. Domain Dictionaries
SPSS provides pre-built dictionaries for industries such as healthcare, education, retail, and customer
service. These help in quickly categorizing domain-specific terms.
How SPSS Supports Text Mining
Converts raw textual responses into measurable variables.
Helps discover hidden patterns in complaints or survey data.
Supports sentiment analysis through recognition of positive and negative expressions.
Helps organizations identify common themes in customer feedback to improve products or services.
Typical Use Cases
Market Research: Understanding customer satisfaction themes.
Academic Research: Analyzing open-ended responses in surveys.
Customer Service: Identifying complaint categories to improve service quality.
Human Resources: Analysing employee feedback in performance reviews.
b) IBM Watson Tool (Watson Natural Language Understanding & Watson Discovery)
IBM Watson is a suite of AI-powered services that offer advanced text mining through machine learning and
deep learning. It is more powerful and modern compared to SPSS because it can analyze real-time large-
scale data, understand natural language deeply, and learn from patterns automatically.
Key Components Used for Text Mining
1. Watson Natural Language Understanding (NLU)
A cloud-based NLP service that extracts meaning from text. It can analyze articles, reviews,
documents, or social media posts.
2. Watson Discovery
A search-and-analysis engine that is used when organizations want to mine large document
collections, PDFs, webpages, legal files, research papers, etc.
3. Watson Assistant
Though mainly for chatbots, it uses text mining to understand intent, extract entities, and manage
conversational data.
Advanced Capabilities
1. Entity Extraction
Automatically identifies names, locations, organizations, dates, products, keywords, and custom
domain entities.
2. Sentiment and Emotion Analysis
Watson can analyze overall sentiment and emotion categories such as joy, anger, sadness, fear, and
disgust.
Example: “The service was slow and frustrating” → negative sentiment + frustration emotion.
3. Concept Detection
Even if a specific keyword is not mentioned, Watson can detect related concepts using deep semantic
understanding.
4. Relationship Extraction
Identifies relationships between entities.
Example: “IBM acquired Red Hat” → Detects an acquisition event.
5. Topic Modeling and Classification
Helps group text into logical themes using machine learning and user-defined training models.
6. Search + AI Combination (Watson Discovery)
Enables intelligent search on huge document sets. Instead of showing just matching keywords, it
shows the most relevant answers derived from context.
How Watson Supports Text Mining
Provides high-accuracy insights on mixed and large-scale text datasets.
Works well for industries needing intelligent automation, such as healthcare, finance, law, and
customer support.
Supports integration with applications through APIs.
Learns continuously as more data is added.
Typical Use Cases
1. Legal Document Analysis - Extracting case details, laws refer, entities, and summarizing long docs.
2. Healthcare - Analysing patient records, medical literature, symptoms, and treatments.
3. Customer Support Automation - Mining chat logs and emails to detect common issues and
improve support quality.
4. Business Intelligence - Extract insights from social media, review, or news articles to see trends.
5. Enterprise Search - Helping employees find relevant information from large document repositories.
Experiment - 2
Aim:- Perform Text Preprocessing using Python Programming with one real time dataset.
Solution
Code
import pandas as pd
import re
import string
import nltk
from [Link] import stopwords
from [Link] import WordNetLemmatizer
[Link]('stopwords', quiet=True)
[Link]('wordnet', quiet=True)
df = pd.read_csv("[Link]")
def to_lower(text):
return [Link]()
def remove_urls(text):
return [Link](r"http\S+|www\S+", "", text)
def remove_mentions_hashtags(text):
text = [Link](r"@\w+", "", text)
text = [Link](r"#\w+", "", text)
return text
def remove_punctuation(text):
return [Link]([Link]("", "", [Link]))
def remove_numbers(text):
return [Link](r"\d+", "", text)
def remove_extra_spaces(text):
return " ".join([Link]())
stop_words = set([Link]("english"))
def remove_stopwords(text):
words = [Link]()
filtered = [w for w in words if w not in stop_words]
return " ".join(filtered)
lemmatizer = WordNetLemmatizer()
def apply_lemmatization(text):
words = [Link]()
lemma_words = [[Link](w) for w in words]
return " ".join(lemma_words)
def preprocess(text):
if [Link](text):
return ""
text = str(text)
text = to_lower(text)
text = remove_urls(text)
text = remove_mentions_hashtags(text)
text = remove_punctuation(text)
text = remove_numbers(text)
text = remove_extra_spaces(text)
text = remove_stopwords(text)
text = apply_lemmatization(text)
return text
df['clean_text'] = df['text'].apply(preprocess)
pd.set_option('display.max_columns', None)
pd.set_option('display.max_colwidth', 60)
pd.set_option('[Link]', None)
print(df[['text', 'clean_text', 'airline_sentiment']])
Dataset
Output
Experiment - 3
Aim:- Demonstrate Text Mining Steps using Python programming.
Solution
Code
import pandas as pd
import re
import nltk
from [Link] import stopwords
from [Link] import WordNetLemmatizer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from [Link] import classification_report, accuracy_score
[Link]("stopwords", quiet=True)
[Link]("wordnet", quiet=True)
df = pd.read_csv('[Link]')
[Link] = [Link]()
def clean_text(text):
text = [Link]()
text = [Link](r"<.*?>", "", text)
text = [Link](r"http\S+|www\S+", "", text)
text = [Link](r"[^a-zA-Z\s]", "", text)
text = [Link]()
return text
df["cleaned"] = df["review"].apply(clean_text)
df["tokens"] = df["cleaned"].apply(lambda x: [Link]())
stop_words = set([Link]("english"))
df["no_stop"] = df["tokens"].apply(lambda x: [w for w in x if w not in stop_words])
lemmatizer = WordNetLemmatizer()
df["lemma"] = df["no_stop"].apply(lambda x: [[Link](w) for w in x])
df["final_text"] = df["lemma"].apply(lambda x: " ".join(x))
tfidf = TfidfVectorizer()
X = tfidf.fit_transform(df["final_text"])
y = df["sentiment"]
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.3, random_state=42, stratify=y
)
model = LogisticRegression(max_iter=1000)
[Link](X_train, y_train)
y_pred = [Link](X_test)
print("\n" + "="*60)
print("SENTIMENT ANALYSIS MODEL EVALUATION")
print("="*60)
print(f"\nDataset Size: {len(df)} reviews")
print(f"Training Set: {X_train.shape[0]} reviews")
print(f"Test Set: {X_test.shape[0]} reviews")
print(f"\nAccuracy: {accuracy_score(y_test, y_pred):.2%}")
print("\nClassification Report:")
print(classification_report(y_test, y_pred, zero_division=0))
print("\n" + "="*60)
print("TEST SET PREDICTIONS")
print("="*60)
test_indices = y_test.[Link]()
for idx in test_indices:
actual = [Link][idx, 'sentiment']
review = [Link][idx, 'review']
predicted = [Link]([Link]([[Link][idx, 'final_text']]))[0]
status = "✓" if actual == predicted else "✗"
print(f"\n{status} Review: {review[:60]}...")
print(f" Actual: {actual} | Predicted: {predicted}")
print("="*60)
Output Dataset
Experiment - 4
Aim:- Demonstrate Text Mining Steps using Python programming.
Solution
Code
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
from [Link] import make_pipeline
texts = [
"I love programming.",
"Python is awesome!",
"I hate bugs.",
"Sometimes coding is hard.",
"I enjoy coding.",
"This is great!",
"I like solving problems.",
"Programming is fun.",
"This is terrible.",
"I dislike errors.",
"This is frustrating.",
"Bugs are annoying."
]
labels = ["positive", "positive", "negative", "negative", "positive", "positive", "positive", "positive", "negative",
"negative", "negative", "negative"]
model = make_pipeline(CountVectorizer(), MultinomialNB())
[Link](texts, labels)
new_texts = [
"I enjoy solving coding problems.",
"Debugging is so frustrating."
]
predictions = [Link](new_texts)
print("Predicted Classes:", predictions)
Output
Experiment - 5
Aim:- Demonstrate Text Mining Steps using Python programming.
Solution
Code
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
from [Link] import make_pipeline
categories = ['Technology', 'Sports', 'Politics', 'Health']
texts = [
"Apple released a new iPhone.",
"Samsung launches new smartphone.",
"Microsoft updates Windows software.",
"Google announces AI technology.",
"The local team won the football match.",
"Basketball game ended in overtime.",
"Cricket tournament starts next week.",
"Tennis player wins championship.",
"Government passed a new law.",
"Elections results were announced.",
"Parliament debates new policy.",
"President signs executive order.",
"Doctors recommend regular exercise.",
"Yoga benefits for mental health.",
"New vaccine shows promising results.",
"Healthy diet reduces disease risk.",
]
labels = [0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3]
model = make_pipeline(TfidfVectorizer(), MultinomialNB())
[Link](texts, labels)
new_texts = [
"Samsung launches advanced smartphone.",
"Elections results were announced.",
"Yoga benefits for mental health.",
]
predicted_labels = [Link](new_texts)
for text, label in zip(new_texts, predicted_labels):
print(f"Text: {text}\nPredicted Category: {categories[label]}\n")
Output
Experiment - 6
Aim:- Demonstrate the working of Text Mining Mini Model (Project).
Solution
Code
import nltk
from [Link] import word_tokenize
from [Link] import stopwords
from [Link] import PorterStemmer
from [Link] import FreqDist
[Link]('punkt', quiet=True)
[Link]('punkt_tab', quiet=True)
[Link]('stopwords', quiet=True)
text = """
Yuvraj is a talented cricket player. Yuvraj's performance in the last match was outstanding.
He hit multiple sixes and was applauded by the [Link] is Class Cr and Developer of this code. Yuvraj trains
hard every day to improve his skills.
"""
tokens = word_tokenize([Link]())
stop_words = set([Link]('english'))
filtered_tokens = [word for word in tokens if [Link]() and word not in stop_words]
stemmer = PorterStemmer()
stemmed_tokens = [[Link](word) for word in filtered_tokens]
freq_dist = FreqDist(stemmed_tokens)
print("Original Tokens:\n", tokens, "\n")
print("Filtered Tokens (Stopwords Removed):\n", filtered_tokens, "\n")
print("Stemmed Tokens:\n", stemmed_tokens, "\n")
print("Frequency Distribution:\n")
for word, frequency in freq_dist.items():
print(f"{word}: {frequency}")
Output