Summary: Text Analytics and Text Mining Overview

In today’s information age, organizations collect massive amounts of data, and most of
it—about 85%—is unstructured text (emails, reports, social media, documents). These
unstructured text data are growing rapidly, doubling every 18 months. Since business
decisions depend on converting data into knowledge, companies that can analyze their text
data effectively gain a strategic advantage.
Text analytics and text mining help transform unstructured text into meaningful and
actionable information using techniques such as natural language processing (NLP).
●​ Text Analytics is a broad field. It includes:
o​ Searching and retrieving relevant documents
o​ Extracting key information from text
o​ Applying data mining and web mining techniques
●​ Text Mining is more specific. It focuses on:
o​ Discovering new patterns and knowledge hidden in text data
Thus, text mining is considered a part of the larger area of text analytics.
Difference Between Text Analytics and Text Mining
●​ Text Analytics = Information Retrieval + Information Extraction + Data Mining +
Web Mining​
(or simply Text Analytics = Information Retrieval + Text Mining)
●​ Usage Context:
o​ Text Analytics → term commonly used in business
o​ Text Mining → term commonly used in academic and research environments
Even though definitions differ slightly, both terms are often used interchangeably.

What is Text Mining?

Text mining (also called text data mining or knowledge discovery in text) is a
semi-automated process of finding useful patterns and knowledge from large collections of
unstructured text.
It follows the same goal as data mining — discover valid, new, useful, and understandable
patterns, but:

Data Mining Text Mining

Works on structured data (tables, Works on unstructured text (PDFs, Word docs, XML,
databases) emails)

Two major steps in text mining:

1.​ Convert unstructured text into structured data.
2.​ Apply data mining techniques to extract patterns and knowledge.
Where is Text Mining Applied?
Useful in fields that generate huge text data, such as:
●​ Law (court records)
●​ Finance (annual/quarterly reports)
●​ Research (papers/articles)
●​ Medicine (hospital discharge summaries)
●​ Marketing (customer feedback, social media)
●​ Technology (patent files)
Example:​
Using customer complaints/reviews to identify product defects or improve services.

Common Applications of Text Mining

Application What it does

Information Extraction Find key phrases or relationships in text.

Topic Tracking Recommends documents based on user interest.

Summarization Creates a shorter version of a long document.

Categorization Assigns documents to predefined categories.

Clustering Groups documents based on similarity (no predefined labels).

Concept Linking Connects documents with common ideas or themes.

Question Answering Automatically finds answers based on text content.

Natural Language Processing (NLP)

Earlier text mining systems used a simple method called the bag-of-words, where documents
are treated as unordered collections of words, ignoring grammar and meaning. This works for
simple tasks like spam detection but does not capture the real meaning of text. In many
cases (e.g., medical research papers), bag-of-words fails to give accurate results, showing the
need for more advanced techniques.
What is NLP?
Natural Language Processing (NLP) is a field of Artificial Intelligence that enables
computers to understand and process human language.​
NLP aims to go beyond word counting and consider:
●​ Grammar
●​ Meaning (semantics)
●​ Context
NLP converts text into structured, machine-understandable data.

Challenges in NLP
Understanding language is difficult because human language is complex, ambiguous, and
context-dependent. Key challenges include:
NLP Challenge Explanation

Part-of-speech tagging Same word can be noun/verb depending on context.

Some languages (like Chinese) don’t have spaces between

Text segmentation
words.

Word sense One word can have multiple meanings; context decides
disambiguation meaning.

Syntactic ambiguity A sentence can be interpreted in multiple ways.

Imperfect input Errors, accents, and informal text complicate understanding.

Words may imply actions (e.g., “Pass the salt” is not a yes/no
Speech acts
question).

NLP Benefits and Applications

NLP helps computers automatically extract knowledge from text.​
Stanford researchers enhanced WordNet, a large database of English words and semantic
relationships, using NLP. This helps build more intelligent language models.
One impactful business application is:
●​ Sentiment Analysis (measuring customer opinions from reviews, social media posts,
etc.)
Companies use NLP-based analytics to understand customer emotions and improve products
and services. NLP has successfully been applied to a variety of domains for a wide range of
tasks via computer programs to automatically process natural human language that previously
could only be done by humans.
Following are among the most popular of these tasks:
• Question answering. The task of automatically answering a question posed in
natural language; that is, producing a human language answer when given a human
language question. To find the answer to a question, the computer program may
use either a prestructured database or a collection of natural language documents (a
text corpus such as the World Wide Web).
• Automatic summarization. The creation of a shortened version of a textual
document by a computer program that contains the most important points of the
original document.
• Natural language generation. Systems convert information from computer databases
into readable human language.
• Natural language understanding. Systems convert samples of human language
into more formal representations that are easier for computer programs to
manipulate.
• Machine translation. The automatic translation of one human language to another.
• Foreign language reading. A computer program that assists a nonnative language
speaker to read a foreign language with correct pronunciation and accents
on different parts of the words.
• Foreign language writing. A computer program that assists a nonnative language
user in writing in a foreign language.
• Speech recognition. Converts spoken words to machine-readable input. Given a
sound clip of a person speaking, the system produces a text dictation.
• Text-to-speech. Also called speech synthesis, a computer program automatically
converts normal language text into human speech.
• Text proofing. A computer program reads a proof copy of a text to detect and
correct any errors.
• Optical character recognition. The automatic translation of images of handwritten,
typewritten, or printed text (usually captured by a scanner) into machine-editable
textual documents.

6.4 Text Mining Applications

Why Text Mining?
●​ Organizations collect huge amounts of unstructured data (emails, reports, customer
chats, reviews).
●​ Text mining helps convert this unstructured text into meaningful knowledge for
decision-making.
1. Marketing Applications
●​ Companies analyze customer conversations (call center notes, chat transcripts) using
text mining.
●​ Helps understand customer sentiments, complaints, satisfaction, etc.
●​ Text from blogs, reviews, discussion forums gives insights into public opinion about
products.
●​ Helps improve:
o​ Cross-selling & up-selling (suggesting related products to customers)
o​ Customer Relationship Management (CRM)
●​ Used to predict customer churn (customers likely to leave) so companies can take
action to retain them.
●​ Can automatically extract product attributes from product descriptions on websites;
useful for:
o​ Product recommendations
o​ Demand forecasting
o​ Comparing products across retailers
2. Security Applications
●​ Used for surveillance and intelligence gathering.
●​ ECHELON system (highly classified) is believed to intercept and analyze global
communications (emails, calls, fax, etc.).
●​ EUROPOL's OASIS system integrates data/text mining to track transnational
organized crime.
●​ The FBI and CIA, earlier working in separate databases, are now developing a
combined massive data warehouse with text mining modules to support law
enforcement across different levels.
3. Deception Detection (Truth vs. Lie Identification)
●​ Text mining can detect whether a written statement is truthful or deceptive.
●​ Researchers analyzed criminal statements and built a model that predicted deception
with 70% accuracy.
●​ Key advantages:
●​ Uses only text (no voice tone or body language needed).
●​ More practical and less intrusive compared to techniques like the polygraph.
Biomedical Applications of Text Mining
Why Text Mining is Powerful in Biomedicine
●​ Biomedical research produces huge volumes of literature (research papers, journals,
reports).
●​ Medical terminology is standardized and consistent, making it easier to extract
information.
●​ Text mining helps researchers find useful knowledge quickly from thousands of
documents.
How Text Mining Helps in Biomedical Research
1.​ Analyzing large experimental datasets
o​ Techniques like DNA microarrays, SAGE, and proteomics generate massive
gene/protein data.
o​ Text mining helps scientists compare this new experimental data with older
published research.
o​ Saves time during experiment validation and interpretation.
2.​ Predicting protein locations inside cells
o​ Protein location reveals its biological role and potential as a drug target.
o​ Shatkay et al. (2007) developed a system that uses:

▪​ Text-based features (from research papers)

▪​ Sequence-based features (from protein data)

o​ Their system outperformed earlier models in predicting protein location.

3.​ Finding disease–gene relationships
o​ Chun et al. (2006) created a system that scans MEDLINE to extract
relationships between diseases and genes.
o​ Uses:

▪​ A dictionary of disease/gene names from public databases

▪​ Machine learning (Named Entity Recognition – NER) to remove false

matches
o​ Result: Improved accuracy by 26.7% in identifying true relationships.
4.​ Extracting gene–protein / protein–protein interactions from literature
o​ Text mining can analyze biomedical sentences to identify interactions.
o​ Process:

▪​ The text is tokenized (split into words)

 ▪​ Part-of-speech tagging and shallow parsing are applied

▪​ Words are matched against a domain ontology (hierarchical biomedical

knowledge base)
o​ Helps derive relationships between genes and proteins.
5.​ Contribution to large scientific initiatives
o​ This text mining method (Nakov et al., 2005) helps decode biological
relationships.
o​ Offers great potential to understand complex data in projects like the Human
Genome Project.

Academic Applications of Text Mining

1.​ Publishers use text mining to improve information retrieval
o​ Scientific publishers store huge databases of research papers.
o​ Text mining helps index and organize this information so researchers can
find specific content quickly.
2.​ Initiatives to support text mining in academic publishing
o​ Nature proposed an Open Text Mining Interface so computers can extract
meaning from articles.
o​ The National Institutes of Health (NIH) created a standard structure called
Journal Publishing Document Type Definition.
o​ These initiatives give machines semantic cues (meaning-based hints) to
answer queries from texts, without violating publishing restrictions.
3.​ Universities and research centers are adopting text mining
o​ The National Centre for Text Mining (NaCTeM) in the UK (University of
Manchester + University of Liverpool):

▪​ Offers customized tools and research support.

▪​ Initially focused on biomedicine; now expanded to social sciences.

4.​ Academic projects supporting scientific research

o​ The University of California, Berkeley is developing a project called BioText.
o​ It helps bioscience researchers perform text mining and analyze large sets of
scientific literature.
6.5 Text Mining Process
1.​ Need for a Standard Process
 o​ Like data mining uses CRISP-DM, text mining also needs a structured
methodology.
o​ Text mining projects require more complex preprocessing because data is
unstructured (text, sentences, documents).
2.​ Context Diagram Explanation (High-Level View)
o​ Shows what is included and excluded in the text mining process.
3.​ Components of the Text Mining Process
o​ Input (left side):

▪​ Text documents (unstructured data)

▪​ Sometimes structured data (tables, databases)

o​ Output (right side):

▪​ Knowledge or insights that help in decision-making

o​ Controls / Constraints (top):

▪​ Software and hardware limitations

▪​ Privacy issues

▪​ Natural language complexity (difficulties in understanding human

language)
o​ Mechanisms / Resources (bottom):

▪​ Proper techniques and algorithms

▪​ Text mining software tools

▪​ Domain experts (people with knowledge of the field)

4.​ Goal of Text Mining

o​ Convert unstructured text into meaningful, actionable knowledge.
o​

5.​ Three Major Tasks (high-level process)

o​ Though detailed steps are not yet shown here, the process generally includes:
0.​ Preprocessing (prepare and clean the text)
1.​ Text Mining / Pattern Extraction
2.​ Evaluation & Interpretation (turn patterns into useful knowledge)
Task 1: Establish the Corpus (Collect Documents)
●​ Gather all documents related to the topic or domain.
●​ These documents can be in different formats:
o​ Text files, emails, web pages, XML files, notes
o​ Voice recordings can be converted into text using speech recognition
●​ After collecting, convert all documents into a common format (like plain text) so the
computer can process them.
●​ Documents can be stored:
o​ In folders as text files
o​ As links to web pages
●​ Text mining software can take these files and prepare them for further analysis.
Task 2: Create the Term–Document Matrix (TDM)
●​ A Term–Document Matrix (TDM) is created from the collected documents.
o​ Rows = documents
o​ Columns = terms (words)
o​ Cell values = how many times a term appears in a document
 ●​ Not all words are useful, so unnecessary terms are removed:
o​ Stop Words (e.g., “the”, “is”, “a”) — common words with no meaning for
analysis
o​ Synonyms can be grouped together (e.g., “car” and “automobile”)
o​ Phrases are treated as a single term (e.g., “Eiffel Tower”)
●​ Stemming is used:
o​ Converts words to their root form (e.g., modeling, modeled → model)
●​ After cleaning, the TDM becomes smaller and more meaningful.

Representing the Indices (Normalization)

●​ After creating the Term–Document Matrix (TDM), each cell shows how many times a
word appears in a document.
●​ Higher frequency ≠ always more important.​
(Example: Just because a word appears 3 times in one document and once in another
doesn't mean it's 3× more important.)
●​ Therefore, raw word counts need to be normalized to make comparisons more
meaningful.
●​ Common normalization methods:
o​ Log frequency (reduces the effect of very high word counts)
o​ Binary frequency (just marks whether word appears or not: 0 or 1)
o​ Inverse Document Frequency (IDF) (gives more weight to rare but
important words)
Reducing the Dimensionality of the Matrix
●​ TDMs are usually very large and sparse (most cells are zero).
●​ Goal: Reduce the size of the matrix to make analysis easier and faster.
●​ Methods to reduce size:
1.​ Manual filtering – domain expert removes irrelevant terms.
2.​ Remove rare terms – eliminate words that appear very few times.
 3.​ Apply SVD (Singular Value Decomposition) – mathematical method to
reduce dimensions.
●​ SVD:
o​ Similar to Principal Component Analysis (PCA)
o​ Reduces many terms into fewer "concept dimensions"
o​ Helps reveal hidden meanings (latent concepts/semantic relationships)

Task 3: Extract the Knowledge

Knowledge Extraction Methods
Main techniques used to find patterns from structured data (like TDM):​
1. Classification​
2. Clustering​
3. Association​
4. Trend Analysis
1. Classification
●​ Goal: Assign data (or text) to predefined categories or classes.
●​ Examples: Spam filtering, web page categorization, automatic tagging.
●​ Approaches:
o​ Knowledge Engineering: Uses expert-defined rules.
o​ Machine Learning: Learns automatically from labeled examples (now more
popular).
●​ In Text Mining: Called text categorization, used to find the right topic for each
document.
2. Clustering
●​ Goal: Group similar (unlabeled) items into natural clusters.
●​ Type: Unsupervised learning (no predefined labels).
●​ Use Cases: Document organization, web content grouping, search optimization.
●​ Benefits:
o​ Improved recall: Finds related documents beyond single-term matches.
o​ Improved precision: Groups related results for easier browsing.
●​ Popular Methods:
o​ Scatter/Gather Clustering: Dynamically groups documents to aid browsing.
o​ Query-Specific Clustering: Builds hierarchies of relevant documents for
better search relevance.
3. Association
 ●​ Meaning: Finds relationships or patterns between concepts/terms in large text
datasets.
●​ Goal: Discover which items or ideas often occur together.
●​ Key Measures:
o​ Support: % of documents containing both concept sets (A and C).
o​ Confidence: % of documents with A that also contain C.
●​ Example:
o​ “Software Implementation Failure” often appears with “ERP” and “CRM”
o​ Support = 4%, Confidence = 55%.
●​ Use Case:
o​ Used to detect links between ideas in literature, e.g., tracking the spread of
bird flu through related terms (regions, species, treatments).
4. Trend Analysis
●​ Meaning: Studies how concept frequencies change over time or across document
collections.
●​ Goal: Identify evolving topics, interests, or emerging trends.
●​ Example:
o​ Comparing research papers from different years to see how key ideas in
information systems have developed.
●​ Use Case:
o​ Helps in understanding topic evolution in academic or news datasets.
6.6 Sentiment Analysis & Topic Modeling
●​ Sentiment Analysis identifies opinions or emotions expressed in text (e.g., positive,
negative, neutral).
●​ Topic Modeling automatically discovers the main themes or topics present in a large
collection of documents.
●​ Both techniques are widely used because organizations generate large volumes of text
(emails, reviews, social media posts, etc.).
●​ These methods help businesses and researchers understand what people are talking
about and how they feel about those topics.
Sentiment Analysis
●​ People rely on others’ opinions (online reviews, social media posts, blogs) before
making decisions (e.g., buying a car, choosing a restaurant).
●​ Sentiment analysis (also called opinion mining) automatically identifies opinions and
emotions in text.
 ●​ It tries to answer the question: “What do people feel about a certain topic?”
●​ It classifies text usually into positive or negative, sometimes with levels (e.g., star
ratings).
●​ Sentiment can be:
o​ Explicit → clearly expressed opinion (e.g., “This product is amazing.”)
o​ Implicit → opinion implied indirectly (e.g., “The handle broke easily.”)
●​ Used in many fields → especially business, marketing, and customer relationship
management.
●​ Helps companies analyze customer feedback from websites, reviews, tweets, etc.
●​ Real-time sentiment analysis enables companies to understand customers quickly and
take action.
Sentiment Analysis Applications
Sentiment analysis uses text from online sources to understand people's opinions and
feelings. It is widely used in business, marketing, finance, politics, and more.
Key Application Areas:
1.​ Voice of the Customer (VOC)
o​ Understands customer feedback from reviews, complaints, and social media.
o​ Helps improve products/service and react quickly to negative comments.
2.​ Voice of the Market (VOM)
o​ Looks at overall market trends and competitor opinions.
o​ Useful for market research and product positioning.
3.​ Voice of the Employee (VOE)
o​ Analyzes employee opinions from surveys, emails, and internal messages.
o​ Helps improve work culture and employee satisfaction.
4.​ Brand Management
o​ Monitors social media for brand reputation.
o​ Identifies threats (negative buzz) and opportunities (positive feedback).
5.​ Financial Markets
o​ Tracks market sentiment (news, blogs, tweets) to predict stock movements.
o​ Helps traders make informed decisions.
6.​ Politics
o​ Analyzes public opinion before elections.
o​ Helps understand voter concerns and predict election results.
7.​ Government Intelligence
o​ Detects hostile or negative communication spikes.
o​ Helps in security and monitoring potential threats.
8.​ Other Uses
o​ Improves e-commerce product suggestions.
o​ Filters emails (e.g., urgent negative emails go to priority folder).
o​ Opinion-based search engines (summarizing reviews).