MODULE 4: TEXT ANALYTICS

Explain Term Frequency and Document Frequency with a suitable

example and how they are used in information retrieval.
1. Term Frequency (TF)
Term Frequency refers to the number of mes a specific word or phrase appears within a
single document. It is used to determine the importance or weight of a term in that specific
context.
In Extrac ve Summariza on, so ware iden fies major text segments by performing a
sta s cal analysis of "words/phrases frequency" to detect which sentences are the most
significant to include in a summary.
This is the core of the Term-Based Method (TBM), which iden fies the distribu on and
rela onships of terms to classify documents or model topics.
Imagine a collec on of medical research papers, in one specific paper about a new flu
treatment, the word "vaccine" appears 20 mes. Because of this high term frequency, a text
summarizer might extract sentences containing "vaccine" as key points of that document.
2. Document Frequency (DF)
Document Frequency (DF) refers to the number of documents in a collec on that contain a
par cular term. It helps in understanding how common or rare a word is across the en re
dataset.
In a Document Term Matrix (DTM), there may be too many variables (terms). Words that
appear in very few documents (low DF) are o en removed, as they add li le value to
analysis. This reduces dimensionality and improves efficiency.
Example:
If we analyze 1000 medical documents and the word “pa ent” appears in 950 of them, it has
high document frequency and is considered common. On the other hand, a word like
“Zycov-D” appearing in only 2 documents has low document frequency and may be removed
during preprocessing.
Usage in Informa on Retrieval (IR)
Informa on Retrieval is the process of extrac ng relevant pa erns and documents based on
a specific set of words or phrases. Term and document frequencies are used in IR in the
following ways:
 Relevance Ranking: IR systems like Google Search rank documents based on how
o en query terms appear in a document (TF) and how unique those terms are across
all documents (DF). This helps in iden fying the most relevant results.
 Pa ern Recogni on: IR tries to process available documents into a structured form
(like a Document Term Matrix) so that analy cal processes can be applied to find
"per nent informa on and connected pa erns".
  User Behavior Tracking: IR algorithms use these frequencies to track user behavior
and discover data that matches their specific interests.
 Search Op miza on: By categorizing and indexing documents based on term
pa erns, IR systems improve performance and provide faster user experiences.

Explain TF-IDF with a suitable example.

TF-IDF (Term Frequency–Inverse Document Frequency) is a sta s cal measure used in
Informa on Retrieval and text mining to determine the importance of a word in a document
rela ve to a collec on of documents (corpus). It converts unstructured text into structured
data and is widely used in tasks like document classifica on, topic modeling, and
summariza on.
1. Term Frequency (TF)
Term Frequency refers to the number of mes a specific word or phrase appears within a
single document. It is used to determine the importance or weight of a term in that specific
context.
In Extrac ve Summariza on, so ware iden fies major text segments by performing a
sta s cal analysis of "words/phrases frequency" to detect which sentences are the most
significant to include in a summary.
This is the core of the Term-Based Method (TBM), which iden fies the distribu on and
rela onships of terms to classify documents or model topics.
Imagine a collec on of medical research papers, in one specific paper about a new flu
treatment, the word "vaccine" appears 20 mes. Because of this high term frequency, a text
summarizer might extract sentences containing "vaccine" as key points of that document.
2. Inverse Document Frequency (IDF)
Inverse Document Frequency measures how unique or rare a term is across all documents.
Common words (like “the” or “pa ent”) appear in many documents, so they have low
importance. Rare words (like specific technical terms) are more useful for dis nguishing
documents, so they get higher importance.
Thus, IDF reduces the weight of common words and increases the weight of rare words.
Working of TF-IDF
TF-IDF combines both concepts:
 High TF → word is important in that document
 Low DF (high IDF) → word is rare across documents
So, words that are frequent in a document but rare in the corpus get the highest weight.
Example: Medical Research Library
Consider a digital library with many research papers.
 The word “Biomarker” appears 50 mes in one document → high TF
  It appears in only 15 documents overall → low DF → high IDF
On the other hand, the word “Pa ent” appears in almost all documents → low IDF
Result:
“Biomarker” gets a high TF-IDF score and helps retrieve the most relevant documents when
searched, while “Pa ent” is given less importance.

What is text mining? Enlist and explain seven practice areas of

text analytics.
Text mining is a part of data mining that works with unstructured text like documents,
emails, and reviews. It uses Natural Language Processing and machine learning to extract
useful informa on and convert text into a structured form for analysis.
The following are the seven prac ce areas of text analy cs iden fied in the sources:
1. Informa on Extrac on (IE)
It is the first step where important informa on like keywords, en es (names, places), and
rela onships are iden fied from text.
This processed data is then stored for further analysis.
2. Informa on Retrieval (IR)
Informa on Retrieval focuses on finding relevant documents based on a user’s query. It uses
algorithms to search through large collec ons of data and return the most useful results. For
example, search engines like Google Search use IR techniques to display the most relevant
web pages.
3. Natural Language Processing (NLP)
NLP enables computers to "read" by automa cally analyzing the structure and grammar of
unstructured text. It encompasses several sub-tasks:
 Part-of-Speech (PoS) Tagging: Assigning tags (noun, verb, etc.) to words to permit
seman c analysis.
 Sen ment Analysis: Determining if the tone of the text is posi ve, nega ve, or
neutral.
 Named En ty Recogni on (NER): Iden fying and classifying en es like loca ons or
organiza ons.
4. Clustering
Clustering is an unsupervised learning method that groups similar documents together
based on their content. It does not require predefined labels. Documents within the same
cluster are highly similar, which helps in iden fying hidden pa erns.
5. Categoriza on
Categoriza on is a supervised learning method where documents are assigned to predefined
categories. A model is trained using labeled data and then used to classify new documents.
It is commonly used in spam filtering and document organiza on.
6. Visualiza on
Visualiza on presents text data in graphical form using charts, graphs, and colors. It helps
users understand large amounts of text data quickly and easily by highligh ng pa erns and
trends.
7. Text Summariza on
Text summariza on reduces the size of a document while keeping its main meaning.
 Extrac ve summariza on: selects important sentences from the text
 Abstrac ve summariza on: rewrites the content in a shorter and meaningful way

List and explain methods that can be used for sentiment analysis.
Sen ment analysis, also called opinion mining, is a technique in Natural Language
Processing used to understand the feeling or tone of text. It tells whether the text is
posi ve, nega ve, or neutral. It helps in understanding people’s opinions from data like
reviews, comments, or social media posts.
1. Core Text Mining Methods
The main methods used in sen ment analysis are:
 Phrase-Based Method (PBM):
This method looks at groups of words (phrases) instead of single words to
understand the exact meaning.
 Concept-Based Method (CBM):
This method is used for deeper analysis. It connects words and phrases to
understand the overall idea and meaning of the text.
 Term-Based Method (TBM):
This method focuses on how o en words appear and how they are related to each
other to help classify the text.
2. Classifica on Approaches
Sen ment analysis is treated as a classifica on problem and uses these approaches:
 Rule-based Method:
It uses predefined rules (like grammar or keywords) made by humans to decide
whether the text is posi ve, nega ve, or neutral.
 Machine-based Method:
It learns from past data. For example, in movie reviews, it checks words like “funny,”
“exci ng,” or “boring” and their frequency to decide the sen ment.
 Hybrid Approach:
This combines both rule-based and machine learning methods. It uses rules for ini al
tagging and then improves results using learning and feedback.
3. Specialized Types of Sen ment Analysis
Different types are used based on how detailed the analysis needs to be:
 Emo on Detec on:
It iden fies specific emo ons like happiness, anger, sadness, or frustra on instead of
just posi ve or nega ve.
 Graded Analysis:
It gives more detailed results using levels like very posi ve, posi ve, neutral,
nega ve, and very nega ve.
 Mul lingual Analysis:
It first iden fies the language of the text and then performs sen ment analysis
accordingly.
4. Suppor ng Algorithms
Different algorithms are used to perform sen ment analysis:
 Naive Bayes:
A simple and commonly used method for classifying sen ment based on probability.
 Support Vector Machines (SVM):
A powerful algorithm that separates data into different sen ment categories.
 Decision Trees:
Used to classify data by making decisions step-by-step (like a flowchart).
 Neural Networks (NN):
Advanced methods (like CNN and RNN) used for deep learning-based sen ment
analysis.
5. Essen al Pre-processing for Accuracy
Before analysis, some important steps are needed:
 Part-of-Speech (PoS) Tagging:
It iden fies whether a word is a noun, verb, or adjec ve, which helps in
understanding meaning.
 Syntax Parsing:
It checks the structure of a sentence. This is very important because changing word
order can completely change the sen ment.

What is a text summarizer? How does it work?

A text summarizer is a tool that shortens a long text into a smaller version by keeping only
the main points. It reduces the length and complexity of the content so that users can
quickly understand it and decide whether they need to read the full document.
How a Text Summarizer Works
A text summarizer generally works in three steps:
1. Pre-processing:
The tool first converts the text into a structured form. It breaks the text into words
 (tokeniza on), removes common words like “the” and “is” (stop word removal), and
finds the root form of words (stemming).
2. Processing:
Then, algorithms analyze the text to understand important informa on and create a
structure for the summary.
3. Development State:
Finally, the tool generates the actual summary from the processed informa on.
Main Approaches to Summariza on
 Extrac ve Summariza on:
This method selects important words, phrases, or sentences directly from the
original text and combines them to form a summary. It is based on things like word
frequency and sentence importance.
 Abstrac ve Summariza on:
This method uses Natural Language Processing to understand the meaning of the
text and then writes a new summary in its own words. It gives a more natural and
human-like result.
Key Features and Func ons
Modern text summarizers (o en AI-based) have useful features like:
 Control Summariza on:
Users can choose how short or detailed they want the summary.
 Bullet Point Forma on:
The tool can convert the summary into bullet points, which is useful for notes or
presenta ons.
 Text Ranking:
It can highlight the most important sentence or “best line” from the text.
 Mul lingual Support:
It can work with different languages by first iden fying the language and then
summarizing the text.
Typical User Workflow
To use a text summarizer, a user usually follows these steps:
1. Insert or Upload the text (like an ar cle or research paper).
2. Select Op ons such as summary length or features.
3. Generate Summary by clicking a bu on to get the final short version.

Difference between extractive and abstractive summarization.

Feature Extrac ve Summariza on Abstrac ve Summariza on

It creates a summary by directly It creates a summary by

Core Defini on picking important parts from the understanding the meaning and
original text. wri ng it in a new way.

It uses linguis c techniques and

It uses sta s cal methods like word
Methodology Natural Language Processing to
frequency and posi on of sentences.
understand text.

It selects important words or It understands the text and generates

Mechanism sentences and joins them to form a new sentences with the same
summary. meaning.

Sentence Keeps the same sentence structure Changes the sentence structure to
Structure as the original text. make it shorter and clearer.

Produces a summary using original Produces new sentences that may

Output Type
text parts. not be in the original text.

More complex and advanced but

Complexity Simple and easy to implement.
gives be er quality summaries.

What is text analysis? Enlist and explain the steps of text analysis
Text analysis (also called text analy cs) is the process of understanding and extrac ng useful
informa on from text data. It converts unstructured text (like ar cles, reviews, or social
media posts) into structured data so it can be analyzed easily.
While text mining mainly focuses on finding hidden pa erns, text analysis focuses more on
understanding the meaning of the text and using it to make decisions. It works by breaking
sentences into smaller parts and studying their meaning using algorithms.
Steps of Text Analysis
Text analysis follows several important steps to convert raw text into meaningful insights:
1. Language Iden fica on
The first step is to iden fy the language of the text. This is important because each language
has its own grammar rules, vocabulary, and sentence structure.
If the language is iden fied correctly, the system can apply the right processing techniques
and tools for accurate analysis.
2. Tokenisa on
In this step, the text is divided into smaller units called tokens, which include words,
numbers, or punctua on marks.
This makes the text easier to process and analyze, as the system can now work with
individual elements instead of long sentences.
3. Sentence Breaking
Here, the system iden fies where each sentence starts and ends.
Although punctua on marks like full stops usually help, there are excep ons (like “Dr.” or
“Mr.”), so the system uses addi onal rules and analysis to correctly separate sentences.
4. Part of Speech (PoS) Tagging
Each word in the sentence is labeled based on its gramma cal role, such as noun, verb,
adjec ve, or adverb.
This helps the system understand how words func on in a sentence, which is important for
interpre ng meaning and performing tasks like sen ment analysis.
5. Chunking
In chunking, words are grouped together into meaningful phrases, such as noun phrases or
verb phrases.
For example, instead of trea ng “the black cat” as separate words, it is grouped as one unit,
making it easier to understand the structure of the sentence.
6. Syntax Parsing
This step analyzes the overall structure of the sentence and how different words are related
to each other.
It is very important because even small changes in word order can change the meaning or
sen ment of the sentence completely.
7. Sentence Chaining
In the final step, sentences that are related to the same topic are connected together.
This helps in understanding the overall meaning of the text, even if related informa on is
spread across different parts of the document.

Application and use case of text mining.

Major Applica on Fields
 Digital Libraries and Academic Research:
Used to find pa erns in research papers and analyze student performance and
research trends.
 Life Science and Healthcare:
Analyzes pa ent records and reports to improve treatment and discover medical
insights.
 Social Media Monitoring:
Helps understand public opinion by analyzing social media, blogs, and news.
 Business Intelligence:
Tools like IBM Text Analy cs are used to analyze customers and compe tors to
improve business performance.
Use Cases
 Marke ng:
Helps in market segmenta on and personalized campaigns by predic ng customer
behavior.
 Banking and Insurance:
Used for fraud detec on, risk analysis, and be er pricing strategies.
 Retail and E-commerce:
Companies like Amazon use it for product recommenda ons and analyzing buying
pa erns.
 Manufacturing and Medicine:
Used to predict machine failures and iden fy health risks.
 Crime Inves ga on:
Helps predict crime areas and improve security planning.