Topic 4 :-Text Analytics

Text Mining/Text Analytics

Text mining, text data mining (TDM) or text analytics is the process of deriving high-quality
information from text. It involves "the discovery by computer of new, previously unknown
information, by automatically extracting information from different written resources.

History and Evolution of Text Mining :-

Early Roots: Counting Words to Understand Meaning (1800s - Early 20th Century)
Before computers, scholars had only one tool for analyzing text—manual counting. In the early
1800s, researchers studied religious and literary texts by tracking word frequencies to identify
recurring themes and sentiments.
The Rise of Content Analysis: Measuring Meaning in Text (Mid-20th Century)
By the mid-20th century, text analysis had moved beyond manual word counting into a more
systematic approach—content analysis. One of the pioneers, Harold Lasswell, developed
methods to study political propaganda by categorizing words, phrases, and themes in speeches
and media. His work helped governments and researchers understand how language shaped
public opinion, particularly during World War II.
Computational Text Analysis & Machine Translation (1940s-1960s)
One of the key figures in this movement, Warren Weaver, envisioned a future where machine
translation could automatically convert one language to another. His ideas sparked some of the
earliest natural language processing (NLP) efforts, relying on syntax rules and statistical models
to analyze text. However, early machine translation systems struggled with accuracy,
highlighting the complexity of human [Link] the 1960s, text analytics methods evolved to
include rule-based parsing, part-of-speech tagging, and statistical modeling.
Statistical and Symbolic Approaches (1970s-1980s): Teaching Machines to Read
As computers became more powerful, text analytics moved from simple rules to more structured
computational methods. Researchers developed part-of-speech tagging, stemming, and parsing
techniques, allowing machines to break text into structured components. These methods helped
with categorizing and coding qualitative data, making text more machine-readable.
TF-IDF and latent semantic analysis (LSA), introduced a statistical way to quantify the meaning
of [Link] Mining & the Rise of Big Data (1990s-2000s): Extracting Insights at Scale
By the 2000s, machine learning-powered sentiment analysis and entity recognition became
mainstream, allowing businesses to track public opinion and detect important entities in text
automatically. These advancements blurred the line between quantitative and qualitative text
analysis, making it easier to process large-scale unstructured data. AI, Deep Learning, and
Modern Text Analytics (2010s-Present): The AI Revolution
The rise of deep learning and large language models (LLMs), like GPT and BERT, has redefined
text analytics. Unlike earlier rule-based or statistical approaches, these AI-driven models
understand context, detect emotions, and generate human-like responses with unprecedented
accuracy.

Roots of Text Mining

Text mining originates from multiple disciplines, including computational linguistics,
information retrieval, machine learning, and statistics. It evolved alongside natural language
processing (NLP), artificial intelligence (AI), and big data analytics, enabling businesses and
researchers to extract valuable insights from large volumes of unstructured text.
1.​ Information extraction (IE): Identification and extraction of relevant facts and
relationships from unstructured text; the process of making structured data from unstructured and
semistructured text. Identifying specific pieces of information, such as names, dates, locations,
and relationships (e.g., Named Entity Recognition, or NER).
2.​ Information Retrieval (IR)
IR is the process of extracting out pertinent information and connected patterns from the given
set of words or phrases. In information retrieval, different algorithms are deployed for tracking
the user’s behaviour and discover relevant data and information accordingly.
Extracting relevant documents from large text collections using search engines and ranking
algorithms (e.g., Google Search).
3.​ Natural language processing (NLP): Low-level language processing and understanding
tasks (e.g., tagging part of speech); often used synonymously with computational linguistics.
Concept extraction: Grouping of words and phrases into semantically similar groups.
NLP deals with the automatic processing and analysis of unstructured textual information and
allows computers to read via analyzing sentence structure and grammar. It performs various
types of analysis such as NER, summarization, sentiment analysis.
Summarization: To give synopsis of huge textual data for making a concise, and intelligible
summary of substantial points of a document.
Part-of-Speech (PoS) tagging: To allocate a tag for each word/token in a document on the basis
of its part of speech as specifying nouns, verbs, adjectives, etc. PoS tagging permits semantic
analysis over unstructured text.
Text categorization: To analyze text documents and classify them on the basis of predefined
topics or categories and benefits when categorizing synonyms and [Link] is also known
as text classification.
Sentiment analysis: To determine positive or negative sentiment from inside/outside data
sources, and allow users to trace changes in customer behaviour over a specific time period.
4.​ Text Classification & Categorization-
Under the categorization method, one or more categories of independent (free format) text
documents are assigned. Depending on the input-output examples to discriminate new
documents , categorization is considered as a supervised learning method. Based on the texts
content, predefined classes are assigned to each text documents,
The process of text categorization involves methods such as pre-processing, indexing,
dimensionality reduction, and classification with the objective to train classifiers on the basis of
recognized examples and then unrecognized examples would be categorized automatically. Also,
text categorization faces the difficulty of high dimensionality of feature space.
Some useful analytical classification models, used to categorize text, are naive bayesian
classifier, nearest neighbor classifier, decision trees, and support vector machines.
Applications included in categorization are document organization, spam filtering, SMS
categorization, and hierarchical categorization of web pages.

5.​ Clustering & Topic Modeling

As a result, distinct partitions, called clusters, are generated and each cluster has a number of
documents. The content of each document in a single cluster is very similar and content in
different clusters are dissimilar such that the quality of clustering is accounted for better.
A fundamental clustering algorithm keeps track of topics for each document and measures the
weightage of how better the documents fit into each cluster.
The quality of a clustering result relies on similarity measures of text content used by the
clustering method and its implementation such that a good clustering method generates a great
quality of clusters with high intra-cluster similarity and low inter-cluster similarity.
It is different from categorization as in clustering, text contents are clustered without previous
knowledge of classes. The main advantage of clustering is that text content can be relevant to
multiple classes.
Different clustering techniques are hierarchical, distribution, density centroid, and k-means
clustering, used for analyzing unstructured text documents.
Grouping similar documents or identifying hidden topics within text using algorithms like
k-means and Latent Dirichlet Allocation (LDA).
6.​ Text Summarization
With the fundamental aim to decrease the length, details and complexity of a document while
keeping significant points and actual meaning, text summarization helps in dealing whether a
lengthy document accomplishes the user’s requirements or not and also in resolving whether it is
worthwhile reading for further information or not, and hence text summary could be replaced by
groups of documents.
Text summarization is three steps process;
Pre-processing: This step makes a structured representation of actual text. Tokenization, stop
word removal, and stemming are some methods, applied for pre-processing.
Processing: Algorithms are applied in order to translate and interpret summary structure out of
text structure.
Development state: This step includes retrieving the final summary from summary structure.
7.​ Sentiment Analysis
 It is the process of identifying and extracting emotional tone, opinions, or subjective information
from text. It aims to determine the attitude or feeling expressed by a writer or speaker — whether
it's positive, negative, or neutral.

Stemmer:-
A stemmer is a tool or algorithm in Natural Language Processing (NLP) that reduces a word to
its base or root form, called the stem.
Example:
Words like:
"running", "runs", "runner"are all reduced to: "run"
A.​ Porter Stemmer
Most widely used.
Rule-based and fairly aggressive.
Often used in search engines.
Can produce non-dictionary words.
B.​ Lancaster Stemmer
More aggressive than [Link] but less accurate.
Can lead to over-stemming (removing too much).
Good for compact indexing, not ideal for [Link]-Based Stemmers
Custom rules for domain-specific text (e.g., removing "-ing" in biomedical texts).
Custom stemmer using regular expressions. You define the pattern and replacement.
All Types of Stemming in NLP
Lemmitizer:-
A lemmatizer reduces a word to its lemma — its base or dictionary form — with proper
meaning. Unlike stemming, lemmatization considers the context, including part of speech
(POS), to return valid words.
Examples of lemmatization:
-> rocks : rock
-> corpora : corpus
-> better : good
1.​ Rule Based Lemmatization
2.​ Dictionary-Based Lemmatization
3.​ Machine Learning-Based Lemmatization
Types of Lemmitizers
 Bag of Words
The Bag of Words is a fundamental technique in Natural Language Processing (NLP) for
converting text into a numerical representation suitable for machine learning algorithms.
It treats a document as a “bag” of words, where the order or structure of the words doesn’t
matter. Instead, the focus is on the presence or frequency of individual words.
for example, imagine you have 2 documents and you want to vectorize those documents.
Document 1: “The cat is sleeping on the mat.”
Document 2: “The dog is running in the garden.”

Step 1: Tokenization
Tokenization involves splitting each sentence into individual words. After tokenization, we have
the following list of words for each sentence
Document 1: [“The”, “cat”, “is”, “sleeping”, “on”, “the”, “mat”]
Document 2: [“The”, “dog”, “is”, “running”, “in”, “the”, “garden”]

Step 2: Create Vocabulary

we create a vocabulary, which is a list of unique words present in the sentences:
Vocabulary: [“The”, “cat”, “is”, “sleeping”, “on”, “the”, “mat”, “dog”, “running”, “in”, “garden”]

Step 3: Remove stop words from Vocabulary

Vocabulary after removing stop words :

[‘cat’ ‘sleeping’ ‘mat’ ‘dog’ ‘running’ ‘garden’ ]

Step 4: Count Word Occurrences

Now, for each document, we count how many times each word appears in the vocabulary.
TFIDF
TF-IDF is a statistical measure used to evaluate the importance of a word in a document
relative to a collection of documents (corpus). TF-IDF combines two components: Term
Frequency (TF) and Inverse Document Frequency (IDF).
Terminology:
Corpus: The entire text data given to us. A Corpus can have many documents
Document: Single sentence inside a Text Corpus
Term: Single word inside a Document / Sentence
TF: Term Frequency
IDF: Inverse Document Frequency

Term Frequency (TF): How often a word appears in a document.

Term Frequency (TF): Measures how often a word appears in a document. A higher frequency
suggests greater importance. If a term appears frequently in a document, it is likely relevant to
the document’s content. Formula:

Inverse Document Frequency (IDF): How rare the word is across all [Link] the
weight of common words across multiple documents while increasing the weight of rare words.
If a term appears in fewer documents, it is more likely to be meaningful and specific. Formula:

Document Frequency(DF): It is the number of documents in a corpus that contain a given word
(term).

TF-IDF Score

Combines TF and IDF:

TF-IDF(t,d,D)=TF(t,d)×IDF(t,D)
Example of TFIDF:-

Let us take the same Example and calculate Term Frequency and IDF:

Document A: I do not like Vanilla Cake

Document B: I do not like Vanilla Icecream

No. of words in Document A: 6

No. of words in Document B: 6

Advantages of TFIDF:-
[Link] Important Terms
2 Filtering Common Words
3 Highlighting Unique Terms

Disadvantages of TFIDF:-
[Link] Context & Semantics
[Link] Handling of Word Order
[Link] Vectors
 Steps involved in Sentiment Analysis:-
1.​ Data Collection
2.​ Text Preprocessing
3.​ Feature Extraction
4.​ Model Training
5.​ Evaluation Metrics
1. Data Collection
Gather text data from sources like emails, social media, customer reviews, surveys, documents,
etc.
Sources: Databases, APIs, web scraping, PDFs, etc.
2. Text Preprocessing
Prepare raw text for analysis by cleaning and normalizing it:
Tokenization: Split text into words, phrases, or sentences (tokens).
Lowercasing: Convert all text to lowercase for consistency.
Stopword Removal: Remove common words (e.g., "the," "is") that add little meaning.
Stemming/Lemmatization: Reduce words to their base form (e.g., "running" → "run").
Removing Punctuation/Special Characters: Clean irrelevant symbols.
text = "This Is A SAMPLE Text"
text = [Link]()

import string
text = "Hello, world! Let's clean this text."
text = [Link]([Link]('', '', [Link]))

import nltk
[Link]('stopwords')
stop_words = set([Link]('english'))
text="Hello I am in ratnagiri?"
tokens=[Link]().split()
for token in tokens:
if(token not in stop_words):
print(token)

import nltk
from [Link] import PorterStemmer
from [Link] import word_tokenize
[Link]('punkt')
stemmer = PorterStemmer()
text="running playing effectively"
text=[Link]()
words = [Link]()
for word in words:
stemmed_words =[Link](word)
print(stemmed_words)

import spacy
nlp = [Link]("en_core_web_sm")
text = "The cats are running better than the dogs."
doc = nlp(text)
lemmatized = [token.lemma_ for token in doc]
print(lemmatized)

3.​ Text Representation

Transform text into a structured format:
Bag of Words (BoW) – Frequency of words in documents.
TF-IDF (Term Frequency–Inverse Document Frequency) – Weighs words based on importance.

from sklearn.feature_extraction.text import CountVectorizer

# Example Corpus
corpus = [
"Hello world! This is a sample text.",
"Bag of words model is simple.",
"This is another example of text processing."]
# Convert text to BoW matrix
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(corpus)
# Get Vocabulary
print("Vocabulary:", vectorizer.get_feature_names_out())
# Print BoW matrix
print("\nBoW Matrix:\n", [Link]())​

from sklearn.feature_extraction.text import TfidfVectorizer

# Example Corpus
corpus = [
"This is a sample document.",
 "This document is another example.",
"We are learning TF-IDF in NLP."]
# Initialize TF-IDF Vectorizer
vectorizer = TfidfVectorizer()
# Transform the corpus into TF-IDF vectors
X = vectorizer.fit_transform(corpus)
# Get the vocabulary (unique words)
print("Vocabulary:", vectorizer.get_feature_names_out())
# Convert the TF-IDF matrix to an array and print
print("\nTF-IDF Matrix:\n", [Link]())​ ​ ​

4.​ Exploratory Data Analysis (EDA)

Analyze word frequencies, n-grams, and key terms.
Visualize trends using word clouds, bar charts, or network graphs.
Bar charts of top terms,Co-occurrence and correlation of terms
5.​ Text Modeling
Apply machine learning or statistical techniques:
Classification (e.g., spam vs. not spam)
Clustering (e.g., group similar documents)​
Topic Modeling (e.g., LDA – Latent Dirichlet Allocation)
Sentiment Analysis (e.g., positive/negative/neutral text)​
Named Entity Recognition (NER) (e.g., extract people, places, organizations)

6.​ Model Evaluation

Use metrics like:
Accuracy, Precision, Recall, F1-score (for classification)
Silhouette Score (for clustering)
Coherence Score (for topic modeling)​
 Categorizing Documents by Topic​ ➖
Topic modeling is a way of abstract modeling to discover the abstract ‘topics’ that occur in the
collections of documents. The idea is that we will perform unsupervised classification on
different documents.

Latent Dirichlet Allocation(LDA)

●​ The aim behind the LDA is to find topics that the document belongs to, on the basis of
words contained in it. It assumes that documents with similar topics will use a similar
group of words. This enables the documents to map the probability distribution over
latent topics and topics are probability distribution
●​ The term latent conveys something that exists but is not yet developed. In other words,
latent means hidden or [Link], the topics that we want to extract from the data
are also “hidden topics”. It is yet to be discovered. Hence, the term “latent” in LDA. The
Dirichlet allocation is after the Dirichlet distribution and process.
●​ LDA generates probabilities for words, which are then used to form topics, and these
topics are subsequently classified into documents.
●​ The LDA makes two key assumptions:
Documents are a mixture of topics, and
Topics are a mixture of tokens (or words)

LDA applies the above two important assumptions to the given corpus. Let’s say we have the
corpus with the following five documents:

●​ Document 1: I want to watch a movie this weekend.

 ●​ Document 2: I went shopping yesterday. New Zealand won the World Test Championship
by beating India by eight wickets at Southampton.
●​ Document 3: I don’t watch cricket. Netflix and Amazon Prime have very good movies to
watch.
●​ Document 4: Movies are a nice way to chill however, this time I would like to paint and
read some good books. It’s been so long!
●​ Document 5: This blueberry milkshake is so good! Try reading Dr. Joe Dispenza’s books.
His work is such a game-changer! His books helped to learn so much about how our
thoughts impact our biology and how we can all rewire our brains.
●​ D1, D2, D3, D4, and D5 are the five documents, and
●​ the words are represented by the Ws, say there are 8 unique words from W1, to W8.

Hence, the shape of the matrix is 5 * 8 (five rows and eight columns):
LDA converts this document-word matrix into two other matrices: Document Term
matrix and Topic Word matrix as shown below:

The Document-Topic matrix already contains the possible topics (represented by K

above) that the documents can contain. Here, suppose we have 5 topics and had 5
documents so the matrix is of dimension 5*6
The Topic-Word matrix has the words (or terms) that those topics can contain. We have 5
topics and 8 unique tokens in the vocabulary hence the matrix had a shape of 6*8.
 Aspect Bag-of-Words (BoW) TF-IDF

Treats all words equally, counts Adjusts word importance by considering

Word Frequency
occurrences without considering both term frequency and rarity across the
vs. Importance
their rarity. corpus.

Common words (stop words) can

Handling Reduces the weight of common words,
dilute the model’s focus on
Common Words focusing on distinctive terms.
meaningful terms.

Document length affects word

Sensitivity to Normalizes the effect of document length,
frequency, potentially skewing
Document Length less sensitive to total word count.
results.

Simple to implement,
More complex and computationally
computationally inexpensive, but
Complexity expensive, but provides a more informative
may result in high-dimensional,
representation.
sparse vectors.
 Stemming
Aspect Lemmatization

Converts words to their base Reduces words to their root form (stem),
Definition
or dictionary form (lemma). which may not be a valid word.

Higher complexity,
Complexity Lower complexity, context-agnostic.
context-aware.

Uses dictionaries and Uses rule-based algorithms like Porter,

Algorithms
morphological analysis. Snowball, and Lancaster Stemmers.

Produces more accurate and Less accurate, may produce

Accuracy
meaningful words. non-meaningful stems.

"Running" → "run", "Better" "Running" → "run" or "runn", "Better" →

Output
→ "good". "bett".
Example
 Slower due to more complex
Speed Faster due to simpler rules.
processing.

Better search results

Use in Search
through understanding Useful for quick search indexing
Engines
context.

Essential for tasks needing

accurate word forms (e.g., Used for initial stages of preprocessing
Text Analysis
sentiment analysis, topic to reduce word variability
modeling)

Helps in producing
Machine Less common due to potential
grammatically correct
Translation inaccuracy
translations