BUSINESS ANALYTICS Module 6

PREDICTIVE ANALYTICS II: TEXT, WEB, AND SOCIAL

MEDIA ANALYTICS

6. 1 TEXT ANALYTICS AND TEXT MINING OVERVIEW

The modern information age is marked by a rapid increase in digital data, most of which exists in
unstructured text form. Studies show that around 85% of corporate data is unstructured, and its
volume continues to grow rapidly. Since valuable business knowledge is derived from data,
organizations that can effectively analyze and extract insights from this large amount of text data
gain a competitive advantage. This growing challenge highlights the importance of text analytics
and text mining in helping businesses make better, data-driven decisions.

Text analytics and text mining both aim to transform unstructured text into useful information
using natural language processing. However, text analytics is a broader concept that includes
tasks such as information retrieval, information extraction, data mining, and web mining. In
contrast, text mining focuses more specifically on discovering new and valuable knowledge from
textual data.
The bottom of Figure 6.1 lists the main disciplines (the foundation of the house) That play a
critical role in the development of these increasingly more popular application areas. Based on
this definition of text analytics and text mining, one could simply formulate the difference
between the two as follows:

RLJIT CSE [DATA SCIENCE] 1

BUSINESS ANALYTICS Module 6

Fig1 : Text Analytics , Realted Application Areas, and Enabling Disciplnes

Text Analytics = Information Retrieval + Information Extraction + Data Mining + Web Mining,

Or simply

Text Analytics = Information Retrieval + Text Mining

Text analytics is a newer term that has gained popularity alongside the growing focus on
analytics in various fields. It is more commonly used in business contexts, while text mining is
often preferred in academic research. Despite minor differences in how they are sometimes
defined, both terms are generally used interchangeably, and the authors consider them
equivalent.

Text mining, also known as text data mining or knowledge discovery in text, is the process of
extracting useful and meaningful patterns from large volumes of unstructured text data. It is
similar to traditional data mining in purpose and methodology but differs in that it works with
unstructured sources such as documents, PDFs, and text files. The process involves first
converting unstructured text into a structured form and then applying data mining techniques to
discover valuable information and insights.

Text mining is highly valuable in fields that generate large volumes of unstructured text, such as
law, research, finance, healthcare, biology, technology, and marketing. It helps organizations
extract meaningful insights from documents like reports, customer feedback, emails, and
research papers. For example, customer complaints and feedback can reveal product weaknesses
and guide improvements, while marketing and focus group data provide insights into customer
opinions. Text mining is also widely used in email systems to filter spam, prioritize messages,
and generate automated responses, making it an important tool for managing and analyzing
large-scale textual information.
• Information extraction. Identification of key phrases and relationships within text by looking
for predefined objects and sequences in text by way of pattern matching.

• Topic tracking. Based on a user profile and documents that a user views, text mining can
predict other documents of interest to the user.

• Summarization. Summarizing a document to save time on the part of the reader.

• Categorization. Identifying the main themes of a document and then placing the document into
a predefined set of categories based on those themes.

• Clustering. Grouping similar documents without having a predefined set of categories.

RLJIT CSE [DATA SCIENCE] 2

BUSINESS ANALYTICS Module 6

• Concept linking. Connects related documents by identifying their shared concepts and, by
doing so, helps users find information that they perhaps would not have found using traditional
search methods.

• Question answering. Finding the best answer to a given question through knowledge-driven
pattern matching..

6.2 NATURAL LANGUAGE PROCESSING (NLP)

Early text mining systems often used the bag-of-words model, which represents text as an
unordered collection of words, ignoring grammar and word order. This approach was commonly
used for tasks like document classification and clustering. In applications such as spam filtering,
emails are compared against predefined word groups—one representing spam and the other
legitimate messages. The classification is based on how closely the words in an email match each
group, allowing the system to decide whether the message is spam or not.

Humans use language with clear structure and meaning, so modern text mining aims to move
beyond simple bag-of-words models. To better understand text, current approaches increasingly
incorporate semantic and syntactic information using advanced natural language processing
(NLP) techniques.

The bag-of-words approach often fails to capture enough meaningful information for complex
text mining tasks such as classification and clustering. This limitation is evident in fields like
evidence-based medicine, where studies using bag-of-words models on large medical research
datasets produced poor predictive results—no better than random guessing. These findings
suggest that bag-of-words representations are insufficient for understanding complex textual
content, highlighting the need for more advanced techniques such as natural language processing
(NLP).

Natural Language Processing (NLP) is a key part of text mining and artificial intelligence that
focuses on enabling computers to understand human language. It aims to convert natural
language into structured, machine-readable forms while moving beyond simple word counting to
capture grammar, meaning, and context for deeper and more accurate text understanding.

Understanding natural language remains a major challenge in NLP because human language is
complex, ambiguous, and deeply dependent on context and background knowledge. Although
NLP has advanced far beyond simple word-counting methods, achieving human-level
understanding is still unlikely. Significant challenges remain in fully capturing meaning, context,
and intent in natural language processing.

RLJIT CSE [DATA SCIENCE] 3

BUSINESS ANALYTICS Module 6

• Part-of-speech tagging: Determining whether a word is a noun, verb, adjective, etc., is

difficult because its role depends on context.
• Text segmentation: Some languages lack clear word boundaries, making it hard to
identify individual words; similar challenges occur in spoken language where sounds
blend together.
• Word sense disambiguation: Many words have multiple meanings, and selecting the
correct one requires understanding the surrounding context.
• Syntactic ambiguity: Sentences can have multiple grammatical structures, and identifying
the correct one depends on semantic and contextual clues.
• Imperfect or irregular input: Accents, speech impairments, and spelling or grammar
errors make language processing more complex.
• Speech acts: The intended meaning of a sentence may differ from its literal form,
requiring understanding of context and intent.

Artificial intelligence aims to enable computers to automatically read and extract knowledge
from text. Researchers have developed methods that use NLP to identify concepts and
relationships within large text collections, helping expand resources like WordNet—a
comprehensive lexical database of English. Automating this process reduces the high cost of
manual knowledge creation and enhances NLP capabilities. One major application of these
advances is in customer relationship management (CRM), particularly through sentiment
analysis, which uses textual data such as customer reviews to understand opinions and
improve products and services.
Text analytics and text mining help organizations understand customer sentiments by
analyzing user-generated content from the internet and social media. This insight allows
businesses to better capture the “voice of the customer,” enabling them to improve products,
services, and customer loyalty while increasing profitability.

Natural Language Processing (NLP) supports many applications that enable computers to
understand and generate human language, including:

• Question answering: Automatically providing answers to natural language questions.

• Automatic summarization: Creating shorter versions of documents while preserving
key information.
• Natural language generation: Converting structured data into readable text.
• Natural language understanding: Transforming human language into structured,
machine-processable representations.
• Machine translation: Translating text from one language to another.
• Foreign language assistance: Helping users read, write, and pronounce foreign
languages correctly.
RLJIT CSE [DATA SCIENCE] 4
BUSINESS ANALYTICS Module 6

• Speech recognition: Converting spoken language into text.

• Text-to-speech: Converting written text into spoken language.
• Text proofing: Detecting and correcting errors in written text.
• Optical character recognition (OCR): Converting scanned or handwritten text into
editable digital text.

Text mining relies heavily on advancements in NLP, which allows features to be

extracted from unstructured text. By combining NLP with data mining techniques, text
mining uncovers novel and useful patterns and knowledge from textual data.

6.3 TEXT MINING APPLICATIONS

As organizations collect more unstructured data, text mining tools are becoming
increasingly valuable. Businesses are recognizing the importance of extracting
knowledge from document-based data, with numerous practical application areas for text
mining.

Marketing Applications

Text mining of call center notes, voice transcriptions, blogs, reviews, and discussion
boards can reveal customer sentiments and perceptions. Analyzing this unstructured data
helps businesses identify opportunities for cross-selling and up-selling, improve customer
satisfaction, and enhance overall customer lifetime value.

Text mining is highly valuable for CM, enabling companies to analyze unstructured text
alongside structured data to predict customer behavior. It has been successfully used to
improve models that identify customers at risk of leaving, allowing targeted retention
strategies.
Security Applications
A major text mining application in the security domain is the classified EHELON
surveillance system, which is rumored to intercept and analyze the content of calls, faxes,
emails, and other data transmitted via satellites, telephone networks, and microwave
links.

In 2007, EUROPOL developed OASIS, an integrated system that accesses, stores, and
analyzes large amounts of structured and unstructured data. By leveraging dvanced data
and text mining technologies, OASIS helps EUROPOL track transnational organized
crime and supports international law enforcement efforts.

RLJIT CSE [DATA SCIENCE] 5

BUSINESS ANALYTICS Module 6

The FBI and CIA, under the Department of Homeland Security, are jointly developing a
supercomputer system for data and text mining. This system will create a massive data
warehouse and incorporate multiple data and text mining modules to support knowledge
discovery for federal, state, and local law enforcement agencies. Previously, the FBI and
CIA operated separate databases with little or no interconnection, limiting the ability to
share and analyze information collaboratively.

Text mining is also applied in security for deception detection. Researchers Fuller, Biros,
and Delen (2008) developed prediction models using text mining to distinguish deceptive
statements from truthful ones in criminal (person-of-interest) statements. By extracting a
rich set of textual cues, their model achieved 70% accuracy on holdout samples, which is
considered significant given that only textual data—without verbal or visual cues—was
used. Compared to traditional deception-detection methods like polygraphs, this approach
is nonintrusive and can be idely applied not only to text but also potentially to
transcriptions of voice recordings.

Biomedical Applications

Text mining has significant potential in medicine and biomedicine because the literature
is expanding rapidly, is more standardized than in many other fields, and uses relatively
consistent terminology with standardized ontologies. These characteristics make
biomedical literature highly suitable for extracting novel patterns using text mining
techniques.

Experimental techniques like DNA microarrays, SAGE, and mass spectrometry generate
large volumes of data on genes and proteins. To interpret and validate these experimental
results, researchers need to analyze them in the context of existing knowledge in the
literature. Developing automated text mining tools to assist with this interpretation is a
major challenge in bioinformatics research.

Determining a protein’s location within a cell is critical for understanding its biological
role and drug potential. Shatkay et al. (2007) developed a system combining sequence-
based and text-based features to predict protein locations. Their method innovatively
selects and integrates text sources with sequence data, consistently outperforming
previous systems when tested on both existing and newly devised datasets.

Chun et al. (2006) created a system to extract disease–gene relationships from MEDLINE
literature. They used dictionaries of disease and gene names from public databases and
applied dictionary matching to identify candidate relations. To reduce false positives, they
implemented machine-learning–based named entity recognition (NER), which improved

RLJIT CSE [DATA SCIENCE] 6

BUSINESS ANALYTICS Module 6

precision in relation extraction by 26.7% with only a slight decrease in recall, showing
that NER filtering is critical to the system’s success.

Figure 6.2 illustrates a multilevel text analysis process for identifying gene–protein or
protein–protein interactions in biomedical literature. The process involves tokenizing text
using part-of-speech tagging and shallow parsing, then mapping the terms to a
hierarchical domain ontology to extract relationships. This approach holds significant
potential for interpreting complex data from projects like the Human Genome Project.

Fig 6.2 Multilevel Analysis of Text for Gene/Protein Interaction Identification

Academic Applications

Text mining is crucial for publishers with large information databases that need indexing
for efficient retrieval, especially in scientific fields where detailed information is
embedded in text. Initiatives like Nature’s Open Text Mining Interface and the NIH’s
Journal Publishing Document Type Definition aim to provide semantic cues to machines
to answer queries from text without compromising public access barriers.

Academic institutions are also advancing text mining initiatives. The National Centre for
Text Mining, a collaboration between the Universities of Manchester and Liverpool,
offers tools, research support, and guidance to the academic community. Initially focused
on biomedical sciences, their research has expanded to social sciences. In the U.S., UC
Berkeley’s School of Information is developing BioText, a program to support bioscience
researchers in text mining and analysis.

RLJIT CSE [DATA SCIENCE] 7

BUSINESS ANALYTICS Module 6

6.4 TEXT MINING PROCESS

Successful text mining requires a sound methodology based on best practices. While the
CRISP-DM process for data mining is largely applicable, text mining projects require
more extensive data preprocessing. Figure 6.5 presents a high-level context diagram that
defines the scope of a typical text mining process, clarifying its boundaries and interfaces
with the larger environment.
The context diagram shows that the input to the text mining process includes both
unstructured and structured data. The output is context-specific knowledge that supports
decision-making. Controls (constraints) include hardware/software limitations, privacy
concerns, and the challenges of processing natural language text. Mechanisms involve
proper techniques, tools, and domain expertise. The main goal of text mining is to
process unstructured text (and relevant structured data) to extract meaningful patterns for
informed decision-making.
At a high level, the text mining process can be divided into three consecutive tasks, each
with specific inputs and outputs, as illustrated in Figure 6.3

Fig 6.3 Context Diagram for the Text Mining Process.

RLJIT CSE [DATA SCIENCE] 8

BUSINESS ANALYTICS Module 6

Task 1: Establish the Corpus

The first task in text mining is to collect all relevant documents for the domain of interest,
including textual documents, XML files, emails, web pages, short notes, and transcribed
voice recordings.

After collection, the documents are converted into a uniform format (e.g., ASCII text) to
enable computer processing.

The organized corpus can be a simple collection of digitized excerpts or a list of web
page links, which can be input directly into text mining software or preprocessed into a
flat file for analysis.

Task 2: Create the Term–Document Matrix

In Task 2, the organized corpus is converted into a term–document matrix (TDM), where
rows represent documents, columns represent terms, and the cells contain relational
measures, such as term frequency. The TDM captures the essence of documents through
the terms they contain.

Not all terms are useful for indexing; common words like articles or auxiliary verbs
(called stop terms/words) should be excluded. Domain experts identify these stop terms,
and predetermined important terms (the include terms or dictionary) as well as synonyms
and specific phrases can be used to improve indexing accuracy.

Stemming is applied to reduce words to their root forms so that different grammatical
variants are treated as the same term (e.g., “modeling” and “modeled” become “model”),
improving term consistency in the TDM.
The first-generation TDM includes all unique terms (excluding stop terms) as columns,
documents as rows, and term occurrences as cell values. Large corpora result in very
large TDMs, which can be time-consuming to process and may reduce pattern accuracy.
At this stage, decisions are needed about the best representation of indices and methods
to reduce matrix dimensionality.

REPRESENTING THE INDICES

After indexing documents and calculating raw term frequencies, the importance of terms
may not be accurately reflected by their counts alone. To create a more consistent term–
document matrix (TDM), these raw frequencies are normalized using methods such as
log frequencies, binary frequencies, or inverse document frequencies, ensuring that the

RLJIT CSE [DATA SCIENCE] 9

BUSINESS ANALYTICS Module 6

numerical representation better reflects each term’s significance in describing a

document.

REDUCING THE DIMENSIONALITY OF THE MATRIX

The term–document matrix (TDM) is often large and sparse, so reducing its
dimensionality is necessary for efficient processing.

Dimensionality can be reduced by: (1) manually removing irrelevant terms with the help
of a domain expert,

(2) eliminating terms that appear very infrequently, or

(3) transforming the matrix using Singular Value Decomposition (SVD).

SVD reduces the matrix to a lower-dimensional space, similar to principal components
analysis, where each dimension captures the greatest variability between terms and
documents. By identifying the most salient dimensions, the latent semantic space is
revealed, effectively extracting the underlying meaning from the documents.

Task 3: Extract the Knowledge

Task 3 involves extracting novel patterns from the structured TDM, often combined with
other structured data, using methods like classification, clustering, association, and trend
analysis.

Classification:

Classification (or text categorization) assigns documents to predefined topics or concepts

using labeled training data. Approaches include knowledge engineering (expert rules) and
machine learning (learning from examples). Applications include spam filtering,
automatic indexing, web page categorization, metadata generation, and genre detection.
Machine learning is increasingly preferred due to document growth and limited expert
availability.

Clustering:

Clustering groups unlabeled documents into meaningful clusters based on content

similarity. Benefits include improved search recall (returning entire clusters) and search
precision (grouping documents by relevance). Popular methods are scatter/gather
clustering (dynamic table of contents) and query-specific clustering (hierarchical clusters
with relevance levels).

RLJIT CSE [DATA SCIENCE] 10

BUSINESS ANALYTICS Module 6

Association:

Association identifies relationships between terms or concepts in text using support

(percentage of documents containing all concepts) and confidence (percentage of
documents containing concept C within those that contain concept A). Example:
“Software Implementation Failure” frequently appears with “ERP” and “CRM.”
Applications include tracking trends like disease outbreaks.
Trend Analysis:

Trend analysis compares concept distributions across different document collections or

over time to detect evolving patterns. Example: Delen and Crossland (2008) analyzed
academic articles to track the evolution of key concepts in information systems.
Conclusion:
Text mining uses classification, clustering, association, and trend analysis to extract
meaningful patterns. Techniques like clustering and topic modeling are applied to analyze
large collections of academic literature.

6.5 Sentiment Analysis and Topic Modeling

Text analytics has many applications wherever large volumes of text exist. Among them,
sentiment analysis and topic modeling are the two most prominent application areas.

Sentiment Analysis

Humans rely heavily on others’ opinions for decision-making, especially in areas where they lack
expertise. With the rapid growth of social media, online reviews, blogs, and forums, opinion-rich
textual data has increased exponentially, making automated sentiment analysis essential.
Sentiment refers to a settled opinion reflecting feelings and is related to beliefs, views, and
opinions. Unlike topic classification, which deals with many topics, sentiment analysis typically
focuses on polarity—positive vs. negative, graded ratings, or strength of opinion. Despite fewer
classes, sentiment analysis is complex.

Sentiment analysis is closely related to NLP, computational linguistics, and text mining, and is
also known as opinion mining, subjectivity analysis, and appraisal extraction. It aims to
automatically extract opinions, emotions, and attitudes from text. The growing use of sentiment
analysis presents both opportunities and risks, as online opinions are permanently traceable and
analyzable.
RLJIT CSE [DATA SCIENCE] 11
BUSINESS ANALYTICS Module 6

In business contexts, especially marketing and CRM, sentiment analysis identifies favorable and
unfavorable opinions toward products and services using sources such as tweets, blogs, and
customer feedback.

Sentiment in text appears in two forms:

• Explicit sentiment: Directly stated opinions (e.g., “This product is excellent”)

• Implicit sentiment: Opinions implied through facts (e.g., “The battery dies quickly”)

Earlier research focused on explicit sentiment; modern approaches address both types. Sentiment
polarity may be positive, negative, or mixed, which differs from purely objective text.

Real-time collection and analysis of textual data from multiple sources is crucial for proactive,
customer-focused organizations and is often presented through dashboards.

Sentiment Analysis Applications

Modern sentiment analysis overcomes limitations of traditional surveys by enabling large-scale,

automated analysis using NLP and data mining. It processes data from social media, reviews,
forums, call center logs, search logs, and more.

Major Applications:

• Voice of the Customer (VOC)

Uses sentiment analysis to understand customer complaints and praises across all
touchpoints (emails, surveys, calls, social media). It supports CRM and customer
experience management by linking opinions to transactions and customer profiles.

• Voice of the Market (VOM)

Focuses on aggregate opinions and trends about a company and its competitors,
supporting competitive intelligence and product positioning.
• Voice of the Employee (VOE)
Uses textual employee feedback to assess employee sentiment, extending beyond
traditional surveys. Positive employee sentiment enhances customer satisfaction.

• Brand Management
Monitors social media opinions to protect or enhance brand reputation. It is product- and
company-focused and aims to shape public perception.

RLJIT CSE [DATA SCIENCE] 12

BUSINESS ANALYTICS Module 6

• Financial Markets
Applies sentiment analysis to news and social media to predict short-term stock
movements, as markets are often sentiment driven.

• Politics
Analyzes public sentiment to predict election outcomes, understand voter concerns, and
assess candidate positioning. It was successfully used in the 2008 and 2012 U.S.
elections.

• Government Intelligence
Used by agencies to monitor hostile or negative communications, analyze public opinion
on policies, and detect spikes in negative sentiment for security purposes.

• Other Areas
Includes e-commerce personalization, targeted advertising, opinion-based search engines,
email filtering, and citation analysis to determine supportive or dismissive references.

Sentiment Analysis Process

Sentiment analysis is complex due to language ambiguity, context, and expression styles. There
is no universally accepted standard process. It can be performed using supervised or
unsupervised (lexicon-based) learning.

• Supervised learning uses manually labeled data and machine-learning models.

• Unsupervised learning uses sentiment lexicons without manual labeling and is more
flexible but complex.

A multistep iterative process is commonly used for unsupervised lexicon-based sentiment

analysis.

RLJIT CSE [DATA SCIENCE] 13

BUSINESS ANALYTICS Module 6

Fig 6.4 : A Multistep Process to Lexicon-based Sentiment Analysis.

Step 1: Sentiment (Objectivity) Detection

The goal is to distinguish facts from opinions by classifying text as objective or subjective.
This is represented using O–S polarity (0 to 1).

• Value close to 1 → objective (fact) → skip

• Value close to 0 → subjective (opinion) → analyze

Detection is often based on adjectives.

Step 2: N–P Polarity Classification

Opinionated text is classified as positive or negative or placed on a sentiment continuum.

• Binary classification: positive / negative

• Also identifies sentiment strength (mild to very strong)

Challenges include:

RLJIT CSE [DATA SCIENCE] 14

BUSINESS ANALYTICS Module 6

• Mixed sentiments

• Implicit opinions

• Lengthy documents
Polarity identification may occur at word, phrase, sentence, and document levels, using
outputs of lower levels as inputs to higher levels.

Step 3: Target Identification

Identifies the entity (person, product, event) being discussed.

• Easy in product/movie reviews

• Difficult in news articles and blogs
Comparative sentences may contain multiple targets, detected using comparative and
superlative words (better, more, best, prefer).

Step 4: Collection and Aggregation

All detected sentiments are aggregated into a single sentiment score for the document.
Aggregation can be:
• Simple (summing polarities)

• Complex (semantic NLP-based aggregation)

Methods for Polarity Identification

Polarity can be identified at word, sentence, or document level using two main approaches:

1. Lexicon-Based Approach (Unsupervised)

Uses predefined sentiment dictionaries.

• WordNet: A general-purpose lexical database of English.

• Extended WordNet: Adds Positive–Negative and Subjective–Objective labels.

• SentiWordNet: Assigns positivity, negativity, and objectivity scores (0–1) to synsets.

• WordNet-Affect: Labels synsets with emotional categories.

RLJIT CSE [DATA SCIENCE] 15

BUSINESS ANALYTICS Module 6

• Seed-based methods: Expand sentiment terms using synonymy and antonymy.

Fig 6.5: A graphical Representation of the P-N polarity ans S-O polarity Relationship.

2. Training Document-Based Approach (Supervised)

Uses labeled datasets such as product and movie reviews.

• Star/point ratings act as gold standards

• Common data sources: Amazon, IMDb, Rotten Tomatoes

• Algorithms used:

o Neural Networks

o SVM

o Naive Bayes

o k-NN

o Decision Trees
o EM-based clustering

Semantic Orientation Identification

Word → Phrase / Sentence Level

Polarity of words is aggregated (often averaged) to determine phrase or sentence sentiment.

Document Level

Used for summarization and information retrieval.

Aggregation is usually averaging and is suitable for small to medium documents.

RLJIT CSE [DATA SCIENCE] 16

BUSINESS ANALYTICS Module 6

Topic Modeling

Topic modeling is a probabilistic machine learning approach used to discover hidden (latent)
themes in large document collections.

Core Assumptions:

1. Documents contain multiple topics

2. Topics consist of groups of words

Early Topic Modeling Approaches

Clustering

Documents are grouped using word frequency matrices and labeled manually.

Latent Semantic Analysis (LSA / LSI)

• Uses document–term matrix

• Employs tf-idf instead of raw counts

• Applies Singular Value Decomposition (SVD):

o A= U × S ×V

• Documents are grouped using cosine similarity

Limitation of Early Methods

They assume:

• One document → one topic

This limitation is overcome by Latent Dirichlet Allocation (LDA), where:

• Documents can belong to multiple topics

• Words can contribute to multiple topic

Latent Dirichlet Allocation (LDA)

• Latent Dirichlet Allocation (LDA) is one of the most widely used and effective topic
modeling techniques. Although newer approaches using deep learning and word
embeddings exist, LDA remains dominant in practice. LDA uses Dirichlet distributions
RLJIT CSE [DATA SCIENCE] 17
BUSINESS ANALYTICS Module 6

as priors to model document–topic and topic–word associations. The Dirichlet

distribution is a multivariate generalization of the beta distribution and is parameterized
by a vector α.

• LDA is an unsupervised, generative probabilistic model that discovers latent topic

structures by assuming that each document is a mixture of topics and each topic is a
mixture of words. It follows a three-level hierarchical structure involving documents,
topics, and terms. The foundational LDA algorithm was introduced by Blei et al. (2003).
Further illustrations and large-scale applications of LDA were presented by David M.
Blei (2012), including experiments on thousands of scientific articles demonstrating its
effectiveness.

Fig 6.6 : A Graphical Depiction of the LDA Components—Documents, Words, and Latent
Topics

RLJIT CSE [DATA SCIENCE] 18

BUSINESS ANALYTICS Module 6

6.6 Web Mining Overview

The Internet has permanently transformed business, creating vast opportunities to reach global
customers while also increasing competition. Today, having an active online presence is a
necessity, not a choice. Customers not only purchase online but also share opinions, experiences,
and feedback, influencing business reputation and performance.

Advancements in Internet and social media technologies have made data creation, sharing, and
collection easier and more transparent. Service delays, delivery issues, and customer complaints
are now public. Successful organizations embrace web technologies to improve customer
communication, understand needs, and enhance satisfaction. Customer-centricity has become
crucial in the digital age.

The World Wide Web as a Data Repository

The World Wide Web (WWW) is the world’s largest repository of data and text. It contains:

• Web page content (HTML/XML)

• Hyperlink information (connections between pages)

• Usage data (clicks, searches, transactions, logs)

Every user interaction generates valuable data. Analyzing this information helps organizations
optimize websites, improve customer relationships, and increase business value.

Challenges in Web Mining

Mining the Web is difficult due to several challenges:

1. Web is too large

Its massive and rapidly growing size makes full data collection and integration
impractical.

2. Web is too complex

Web pages lack uniform structure and show high variation in content and design.

3. Web is too dynamic

Content such as news, blogs, prices, advertisements, and reports is frequently updated.
4. Web is domain-independent
Users have diverse interests and limited understanding of Web structure, leading to
inefficient searches.

RLJIT CSE [DATA SCIENCE] 19

BUSINESS ANALYTICS Module 6

5. Web has excessive information

Only a small fraction of Web content is relevant to a given user or task, making relevance
identification difficult.

Limitations of Keyword-Based Search

Search engines index Web pages using keywords, but keyword-based search has limitations:

• Returns too many marginally relevant documents

• Misses relevant documents lacking exact keywords

• Fails to resolve ambiguity and authority

These limitations led to the development of Web mining, which enhances search accuracy by
identifying patterns, authority, and relevance.

Web Mining Definition and Scope

Web mining is the process of discovering useful and meaningful patterns from Web data,
including:

• Textual data

• Linkage data

• Usage data

First introduced by Etzioni (1996), Web mining is an extension of data mining applied to
Internet-generated data. Its goal is to convert Web transactions, interactions, and usage data into
actionable knowledge for better decision-making.
Although often confused, Web mining ≠ Web analytics:

• Web analytics: Descriptive, metrics-driven, focuses on website usage

• Web mining: Predictive/prescriptive, discovers unknown patterns across all Web data

Web analytics is a subset of Web mining.

Taxonomy of Web Mining

Web mining is divided into three main areas:
1. Web Content Mining
RLJIT CSE [DATA SCIENCE] 20
BUSINESS ANALYTICS Module 6

2. Web Structure Mining

3. Web Usage Mining

These areas are often used together and rely heavily on data mining and text mining techniques.

Fig 6. 7 A simple taxonomy of Web Mining

Web Content Mining

Web content mining extracts useful information from Web page content using automated tools
like Web crawlers (spiders). Extracted information includes:

• Text features

• Document structure
• Hierarchical relationships

Applications include:

• Competitive intelligence
• News and opinion collection
RLJIT CSE [DATA SCIENCE] 21
BUSINESS ANALYTICS Module 6

• Sentiment analysis

• Automated data gathering for predictive models

Example: Automated collection and validation of movie-related data from multiple websites to
predict box-office success, ensuring data quality and saving time.

Web Structure Mining

Web pages contain hyperlinks, which represent hidden human judgments. A link often acts as an
endorsement, helping identify:

• Authoritative pages (important and trusted)

• Hubs (pages linking to multiple authoritative sources)

Authority and Hub Concept

• Authority: A highly referenced, trusted page

• Hub: A page that links to many authoritative pages

Authorities and hubs have a mutually reinforcing relationship.

HITS Algorithm
The Hyperlink-Induced Topic Search (HITS) algorithm:

• Analyzes Web links to compute hub and authority scores

• Builds a document set based on a query

• Iteratively refines scores until convergence

• Ranks pages based on importance and relevance

HITS is a foundational link-analysis method in Web search.

Importance of Web Structure Mining

Web structure mining:

• Identifies authorities and hubs

RLJIT CSE [DATA SCIENCE] 22

BUSINESS ANALYTICS Module 6

• Supports page-ranking algorithms (e.g., Google, Yahoo!)

• Reveals relationships among Web pages

• Helps understand Web communities, cliques, and topic coverage depth

6.7 Search Engines

Search engines play a central role in Internet usage by helping users locate relevant information
on the rapidly growing and complex World Wide Web. They are widely used for product
research, price comparison, decision-making, travel planning, and general information search.
Search engines have become the backbone of Internet-based activities, with Google being the
most prominent example.

A search engine is a software system that retrieves documents or Web pages based on user-
provided keywords or queries. Technically, search engines are information retrieval systems and
are used not only on the Web but also for desktop and document search. The main goal of a
search engine is to return the most relevant documents, usually in a ranked order. Search engines
are evaluated using two metrics: effectiveness (quality of results) and efficiency (speed of
response). Achieving both simultaneously is challenging.

Anatomy of a Search Engine

A search engine consists of two main cycles:

1. Development Cycle

2. Response Cycle
The development cycle interacts with the Web, while the response cycle interacts with the user.

RLJIT CSE [DATA SCIENCE] 23

BUSINESS ANALYTICS Module 6

Fig 6.8 :Structure of a typical Internet Search Engine

Development Cycle
The purpose of the development cycle is to collect, organize, and index Web pages into a
searchable database.
Web Crawler

A Web crawler (spider) automatically browses the Web to find and fetch Web pages. It starts with
a list of seed URLs, extracts hyperlinks from visited pages, and recursively adds new URLs to a
scheduler. Due to resource limitations, crawlers prioritize which pages to download.

Document Indexer
The document indexer processes fetched pages and stores them in the document database.

Step 1: Preprocessing
Documents in different formats are converted into a standard representation, and content types
such as text and links are separated.

Step 2: Parsing
Text mining and NLP techniques are applied. This includes tokenization, stop-word removal,
stemming, spelling correction, and identification of synonyms and homonyms.

Step 3: Term-by-Document Matrix Creation

Relationships between terms and documents are established using weights. Advanced weighting
methods such as TF-IDF are used to improve document discrimination.

RLJIT CSE [DATA SCIENCE] 24

BUSINESS ANALYTICS Module 6

Response Cycle

The response cycle handles user queries and retrieves the best-matched documents.

Query Analyzer

The query analyzer converts the user’s search request into a standardized format using the same
processes as document indexing, such as tokenization, stemming, stop-word removal, and
disambiguation.

Document Matcher/Ranker

The matcher compares the processed query with indexed documents and ranks results by
relevance and importance. Ranking algorithms are a key differentiator among search engines.
Early search engines relied on simple keyword matching, which produced poor results. Google
introduced PageRank, which ranks pages based on importance and link structure in addition to
keyword relevance. Modern search engines also analyze user click behavior to continuously
refine ranking accuracy.

Search Engine Optimization (SEO)

Search Engine Optimization (SEO) is the process of improving a website’s visibility and ranking
in a search engine’s organic (unpaid) search results. Higher ranking and frequent appearance in
search results increase website traffic. SEO considers how search engines work, user search
behavior, keywords used, and preferred search engines of the target audience.

SEO activities include editing website content, HTML, and coding to improve keyword
relevance and remove indexing barriers. Another important tactic is increasing backlinks
(inbound links) to improve visibility and authority.

Evolution of SEO and Indexing

Earlier, Webmasters had to submit URLs to search engines, which used spiders to crawl pages,
extract links, and index content. The spider downloaded pages, while an indexer analyzed words,
weights, and links, storing them for future crawling.
Today, search engines proactively and continuously crawl the Web without relying on URL
submissions.

RLJIT CSE [DATA SCIENCE] 25

BUSINESS ANALYTICS Module 6

Importance of Ranking

Being indexed is not sufficient; businesses must rank higher than competitors on popular search
engines. Ranking improvement methods include:

• Cross-linking pages within a website

• Using frequently searched keywords in content

• Regularly updating content

• Optimizing metadata (title tags, meta descriptions)

• URL normalization, canonical links, and redirects to consolidate link popularity

SEO Techniques Classification

SEO techniques are broadly divided into:

1. White-hat SEO

2. Black-hat SEO

White-Hat SEO
White-hat SEO follows search engine guidelines and avoids deception. It focuses on creating
quality content for users, ensuring that search engines index the same content users see. It
emphasizes accessibility and long-term results.

Black-Hat SEO

Black-hat SEO uses deceptive practices to manipulate rankings. Techniques include hidden text,
off-screen content, and cloaking (showing different content to users and search engines). Search
engines may penalize such sites by lowering rankings or removing them entirely. Penalties may
be automatic or manual.

Risks and Business Considerations

SEO can offer a high return on investment, but it carries risks:

• Search engines do not charge for organic traffic

• Algorithms change frequently

• No guarantee of consistent traffic

Google alone made over 500 algorithm changes in one year, making SEO unpredictable. To
manage this risk, businesses may:

RLJIT CSE [DATA SCIENCE] 26

BUSINESS ANALYTICS Module 6

1. Hire SEO specialist companies

2. Use paid search advertising

3. Reduce dependence on search engine traffic

Final Business Perspective

Regardless of traffic source—organic, paid, or referral—the primary goal of an e-commerce

website is to convert visitors into customers. High traffic without transactions does not fulfill
business objectives.

6.8 Web Usage Mining (Web Analytics)

Web usage mining, also known as Web analytics, is the process of extracting useful information
from data generated by Web page visits and online transactions. It helps organizations understand
user behavior by analyzing Web server data, commonly called clickstream analysis. Using data
and text mining techniques, companies can discover patterns in user navigation behavior, such as
search sequences and access timings, which can support advertising placement, customer support
planning, and bandwidth management.

Clickstream analysis provides insights into what users do, when they visit, and how they
navigate, helping organizations improve websites, business processes, and customer value. The
extracted knowledge is used to enhance the Web site, optimize operations, and increase overall
business effectiveness.

Fig 6.9 Extraction of knowledge from Web Usage Data

RLJIT CSE [DATA SCIENCE] 27
BUSINESS ANALYTICS Module 6

Web Analytics Technologies

Web analytics tools are widely used to measure, collect, and analyze Internet data to understand
and optimize Web usage. Web analytics is not limited to traffic measurement; it also supports e-
business analysis, market research, advertising effectiveness measurement, and ROI assessment.
It helps analyze visitor counts, page views, traffic patterns, and popularity trends.

There are two main categories of Web analytics:

Off-Site Web Analytics

Off-site Web analytics analyzes data outside the organization’s Web site, including:
• Potential audience size

• Share of voice (visibility)

• Online buzz (opinions, comments, word-of-mouth)

On-Site Web Analytics

On-site Web analytics measures visitor behavior within the Web site, including:

• Traffic drivers

• Landing pages

• Conversions and online purchases

It evaluates Web site performance using key performance indicators (KPIs) and helps improve
Web site design and marketing campaigns. Popular tools include Google Analytics, along with
services from Yahoo! and Microsoft.

Web Analytics Data Collection Methods

On-site Web analytics data is collected using two main technical approaches:
1. Server log file analysis, where Web servers record browser requests

2. Page tagging, which uses JavaScript to send data to third-party analytics servers

Additional data sources such as email campaigns, sales records, direct mail data, and social
media data may be integrated to enrich analysis.

RLJIT CSE [DATA SCIENCE] 28

BUSINESS ANALYTICS Module 6

Web Analytics Metrics

Web analytics provides near real-time marketing data that supports decision-making, campaign
evaluation, and ROI measurement. Among many metrics, four actionable metric categories
directly impact business objectives:

1. Web Site Usability – How users interact with the site

2. Traffic Sources – Where visitors come from

3. Visitor Profiles – Characteristics of visitors

4. Conversion Statistics – Business impact of visits

Web Site Usability Metrics

Key usability metrics include:

• Page views: Average number of pages viewed per visitor; low values may indicate design
or content mismatch.

• Time on site: Measures engagement duration; longer times usually indicate better
interaction but must be evaluated alongside page views.

• Downloads: Tracks access to PDFs, videos, and resources; helps assess content
effectiveness and conversion influence.

• Click maps: Show percentage of clicks on page elements to assess whether important
items receive attention.

• Click paths: Analyze navigation sequences to identify where visitors drop out and how
effectively predefined paths guide users through education, motivation, and purchase
processes.

Traffic Sources

Traffic sources identify where Web site visitors originate. Web analytics automatically
categorizes traffic into search engines, referral sites, and direct visits, and can also track traffic
from offline and online advertising campaigns using coded or dedicated URLs.

• Referral Web sites are external sites that send visitors via links. Analytics helps determine
which referrals generate the most traffic, conversions, and new visitors.

RLJIT CSE [DATA SCIENCE] 29

BUSINESS ANALYTICS Module 6

• Search engines traffic is divided into paid search and organic search. Analytics identifies
keywords used by visitors, helping assess keyword relevance, customer intent, and search
behavior variations.

• Direct traffic comes from bookmarked pages or users typing the URL directly, often
influenced by business cards, brochures, print ads, or radio ads, making coded URLs
useful.
• Offline campaigns use dedicated URLs in non-Web advertisements to measure response
and landing-page visits.

• Online campaigns such as banner ads, search ads, and email campaigns are measured
using unique URLs to track individual campaign performance.

Visitor Profiles

Visitor profiling uses segmentation to combine analytics data and identify distinct user groups.

• Keywords reveal visitor intent and product awareness, distinguishing informed users
from those seeking general solutions.

• Content groupings allow analysis of site sections related to specific products, services, or
campaigns.

• Geography shows visitor locations (country, state, city), useful for geo-targeted marketing
and regional visibility analysis.

• Time of day analysis identifies browsing and buying patterns and helps plan customer
support hours.
• Landing page profiles measure how targeted campaigns attract specific demographic
groups through dedicated landing pages.

Conversion Statistics

Conversion statistics measure how well a Web site meets its marketing objectives, often referred
to as goals.

• New visitors indicate growth in visibility.

• Returning visitors measure loyalty and long purchase-cycle engagement.

RLJIT CSE [DATA SCIENCE] 30

BUSINESS ANALYTICS Module 6

• Leads are generated when forms are completed; analytics calculates completion and
abandonment rates.

• Sales/conversions track purchases, registrations, or submissions to monitor success and

detect performance changes.

• Abandonment and exit rates identify where users leave processes or pages, helping detect
usability or expectation mismatches.

Performance Monitoring

Organizations can define custom metrics and create dashboards (e.g., using Google Analytics) to
monitor performance regularly. Consistent evaluation of these metrics, combined with other
marketing data, enables a quantified and effective marketing strategy.

6.9 Social Analytics

Social analytics has different meanings depending on the field. Philosophically, it originated
from Lars-Henrik Schmidt, focusing on socius—a shared social commonness distinct from
philosophy and sociology.
In business and analytics, social analytics emphasizes the analysis of digital interactions and
relationships rather than philosophical theory.

According to Gartner, social analytics involves monitoring, analyzing, measuring, and

interpreting digital interactions related to people, ideas, topics, and content. It includes analyzing
social media text (sentiment analysis, NLP) and social networks to understand customer
behavior, preferences, and future trends.

Social analytics is broadly divided into two branches:

1. Social Network Analysis (SNA)

2. Social Media Analytics

Social Network Analysis (SNA)

A social network consists of individuals or organizations (nodes) connected by relationships
(ties).
SNA provides a holistic view of social structures, helping identify patterns, influential entities,
and network dynamics.

RLJIT CSE [DATA SCIENCE] 31

BUSINESS ANALYTICS Module 6

SNA is interdisciplinary, drawing from sociology, psychology, statistics, and graph theory. Its
mathematical foundations began in the 1950s, with major theoretical development in the 1980s.
Today, SNA is widely used in business analytics, consumer intelligence, and social sciences.

Social networks are self-organizing, emergent, and complex, where global patterns arise from
local interactions.

Types of Social Networks Relevant to Business

1. Communication Networks
Represent information flow between individuals or entities. Used by telecom companies
to optimize operations and customer relationships.

2. Community Networks
Include traditional geographic communities and modern online communities. Data from
online communities helps organizations gain actionable insights.
3. Criminal Networks
Analyze relationships among criminal actors to understand and prevent crime. Modern
surveillance uses internet-based tools while still relying on traditional social theories.

4. Innovation Networks
Focus on how ideas and innovations spread. Used to understand why certain networks
adopt innovations earlier and are more innovative.

Social Network Analysis Metrics

SNA metrics are grouped into three categories:

1. Connections
• Homophily: Tendency to connect with similar individuals

• Multiplexity: Multiple relationship types between two actors

• Mutuality/Reciprocity: Degree of mutual interaction

• Network Closure / Transitivity: Likelihood that friends are also connected

• Propinquity: Influence of geographic closeness on relationships

RLJIT CSE [DATA SCIENCE] 32

BUSINESS ANALYTICS Module 6

2. Distributions

• Bridge: A node connecting otherwise separate groups

• Centrality: Importance or influence of a node (degree, betweenness, closeness,

eigenvector, etc.)

• Density: Ratio of actual connections to possible connections

• Distance: Shortest path between two nodes

• Structural Holes: Gaps in the network that can provide competitive advantage

• Tie Strength: Strength of relationships based on time, emotion, intimacy, and reciprocity

3. Segmentation

• Cliques & Social Circles: Fully or partially connected groups

• Clustering Coefficient: Degree of cliquishness

• Cohesion: Strength of connectivity within a group; structural cohesion measures

resilience

Social Media Analytics

Social media refers to Web 2.0–based platforms enabling users to create, share, and exchange
content in virtual communities. It significantly changes how individuals and organizations
communicate.

Social media platforms include:

• Blogs and microblogs
• Social networks

• Wikis

• Online forums

• Podcasts

• Images and videos

• Product and service reviews

RLJIT CSE [DATA SCIENCE] 33

BUSINESS ANALYTICS Module 6

These platforms generate vast amounts of data useful for customer insights, sentiment analysis,
and behavior prediction.

Kaplan and Haenlein (2010) classified social media using media richness and social presence
theories, identifying six types (e.g., collaborative projects like Wikipedia, blogs, microblogs like
Twitter).

Web-Based Social Media

Web-based social media include content communities (YouTube), social networking sites
(Facebook), virtual game worlds (World of Warcraft), and virtual social worlds (Second Life).
Social media differ from traditional industrial media (newspapers, TV, films) because they are
inexpensive, accessible, decentralized, and interactive, allowing anyone to create and consume
content.

Key Differences Between Social and Industrial Media

• Quality: Social media content has high quality variance, ranging from excellent to low-
quality or abusive, unlike industrial media with controlled quality.

• Reach: Both can reach global audiences, but industrial media are centralized, whereas
social media are decentralized with multiple content creators.

• Frequency: Social media content is updated more frequently due to lower cost and ease of
publishing.

• Accessibility: Industrial media production is costly and controlled; social media tools are
low-cost or free.

• Usability: Industrial media require specialized skills; social media require minimal skills.

• Immediacy: Social media enable near-instant communication, unlike delayed industrial

publishing.

• Updatability: Social media content can be edited or commented on instantly, unlike fixed
industrial content.

How People Use Social Media

Social media usage and engagement have increased significantly. Research identifies six levels
of user engagement, with users steadily moving toward higher participation. A large portion of
previously inactive users now actively use social media, indicating mass adoption.

RLJIT CSE [DATA SCIENCE] 34

BUSINESS ANALYTICS Module 6

Social Media Analytics

Social media analytics refers to systematic methods for analyzing large volumes of social media
content to improve organizational competitiveness. It supports integrated marketing, customer
understanding, and communication strategies. Organizations use social media analytics to engage
with global audiences and measure campaign effectiveness.

Despite widespread adoption, many organizations still treat social media as experimental while
trying to integrate it into their strategies.

Measuring Social Media Impact

Social media analytics tools help extract insights from vast user-generated content and measure
impact. These tools fall into three main categories:

1. Descriptive analytics – Identifies activity levels and trends (followers, posts, reviews).

2. Social network analysis – Identifies influence patterns and key influencers.

3. Advanced analytics – Uses predictive and text analytics to detect sentiment, themes, and
hidden relationships.

Advanced social media analytics typically use all three analytics types progressively.

Best Practices in Social Media Analytics

• Use measurement as guidance, not evaluation, to identify what works and what does not.

• Track sentiment accurately by using text analytics to classify opinions as positive,

negative, or neutral, avoiding mixed-polarity misclassification.
• Continuously improve text analytics accuracy by refining rules, taxonomies, and NLP
models.
• Analyze ripple effects to understand which content goes viral and why.

• Look beyond the brand to understand broader conversations around customer interests.

• Identify key influencers who shape discussions, not just brand advocates.

• Evaluate analytics accuracy, noting that accuracy varies by platform and content type.

• Integrate social media insights into planning, linking activity spikes to business actions or
events.
RLJIT CSE [DATA SCIENCE] 35
BUSINESS ANALYTICS Module 6

RLJIT CSE [DATA SCIENCE] 36